Difference between revisions of "Interop Testing Plan"

Revision as of 19:39, 3 February 2014

The test material is based upon the current OASIS consultation draft of the specification.

The test material described below can be found in the Paho testing git repository.

Conformance Statements

The specification contains a list of identified conformance statements, which are labelled with sequence numbers. Although the statements are to a certain extent arbitrary, due to the use of language in the specification (wherever the words MUST or SHOULD are used), nevertheless they are a good place to start in determining conformance. I anticipate adding more statements to this list in due course, but for now we will go with the list as identified in the current version of the specification.

I have extracted the list into a spreadsheet where tests for both MQTT clients and servers are briefly described. I am using this as a coverage checklist to view progress of the current test material.

Model Based Testing

I have taken the approach of what can be described as model based testing. A fully automated test at its core is comprised of two sets of data:

• input sequences
• expected results, or outputs.

MQTT is a client/server protocol, meaning that in general there are two components, clients and servers to test.

Test Material Components

I'll start by describing the components of the test material.

Conformance Statements Spreadsheet

Conformance statements extracted into a spreadsheet, associated with client and server tests, and coverage statements checked off.

Test Broker

A test or "model" broker is in the package mqtt/broker. You can run the broker with the command:

python3 startbroker.py

and if running successfully, you will see this:

INFO 20140203 220904 MQTT 3.1.1 Paho Test Broker
INFO 20140203 220904 Optional behaviour, publish on pubrel: True
INFO 20140203 220904 Optional behaviour, single publish on overlapping topics: True
INFO 20140203 220904 Optional behaviour, drop QoS 0 publications to disconnected clients: True
INFO 20140203 220904 Starting the MQTT server on port 1883

The first question I will be asked (thanks Ian Skerrett) is why write another broker, when Mosquitto, RSMB or others already exist. The reasons are:

• I already had one written before (but that is not really important)
• a model broker need not have all the implementation details of a production server
• the model is seeded with conformance statements - it knows when they have been executed
• the model is written in Python and is intended to be simple and serve as an illustration and clarification for the specification.

I hope the source can be scrutinized and discussed by the community, both for understanding the specification, and discussion of MQTT server implementation. The source consists of the following files:

• start.py - Python socket server wrapper for the MQTT code
• MQTTV311.py - MQTT packet formatting
• MQTTBrokers.py - MQTT packet interactions of the broker
• Brokers.py - core broker behaviour, without MQTT packet details
• SubscriptionEngines.py - subscription and retained message processing
• Subscriptions.py - handling of individual subscriptions
• Topics.py - validation and matching of topics

I've put these into a literate programming system for a pretty printed display. I intend to add descriptions to this source, so that the reasons for the implementation are explained, and also to allow comments on the source pages.

As the broker is executed, it notes when it encounters any embedded conformance statements, and will display at the end of its run coverage figures, like this:

INFO 20140203 131616 exceptions 2 out of 17 11%
INFO 20140203 131616 coverages 35 out of 55 63%

so you can run a test suite against it and determine the level of coverage it achieves, and whether more tests are needed.

Test Client

The mqtt.client package contains a test MQTT 3.1.1 client which is used for sanity checking the function of the test broker and also for various clean-up operations when running tests.

An example of its use is shown in client_test.py. This program can be used as a basic test of an MQTT 3.1.1 broker implementation. It exercises:

• message queueing for offline clients
• retained messages
• clearing retained messages
• setting and receiving will messages
• $ topics
• overlapping subscriptions
• keepalive timeout

and when added:

• username & password (subscribe failure)
• redelivery on reconnect.

The coverage of the test broker conformance statements as of today (Feb 3rd) by running this program is:

INFO 20140203 220602 exceptions 2 out of 17 11%
INFO 20140203 220602 coverages 35 out of 55 63%

Run this program against your own server as a first test.

Test Proxy

The mqtt.proxy package contains a test "proxy" which can be used to redirect traffic from localhost:1883 (typically) to another host and port.

This makes it useful when you want to run a test against a remote system, but your test is coded for localhost:1883 for instance. It will also trace the packets being sent and received, for information. To run the proxy:

python3 run_proxy.py remote_hostname remote_port

This currently runs in text mode. The module wmqttsas.py is graphical wxPython version, but I need to get that running.

Model-Based Testing Package

The mbt package contains a test suite generation tool which exercises paths through an input model. It takes steps to maximize coverage, such as avoiding paths already trodden. I want to make it more coverage driven, so that it will actively seek out coverage statements of the kind embedded in the test broker.

An example of its use is shown in MQTTV311_spec.py, which is used to generate the broker test suites. It is desribed below.

Broker Test Input Model

Test Procedures

Testing an MQTT Client Library

Start the test broker, as described above.

Run your test suite against this broker. Note the coverage achieved when you stop the broker. Try and get more coverage!

The client_test.py program, as described above, is a good basis for the sort of coverage that ought to be achieved.

With client libraries that ensure the data that is sent to the server consists of well-formed MQTT packets, the tests are likely hit the good paths in the broker rather than the exceptions. So you don't need to worry if your exception coverage is low or non-existent.

When I've generated more test suites, they could be used as templates for client test suites. I'm not sure about that yet.

Testing an MQTT Server

Run:

python3 client_test.py [hostname:port]

as a first test. If hostname:port are not specified, localhost:1883 is assumed.

The full goal of this approach is to run a generated broker test suite. Assuming that test suite is in the tests subdirectory, you do this with the command:

python3 run_test.py

which will run the test suite against localhost:1883. To run the test suite against a remote server, or one on a different port, use the test proxy.

Generating a Broker Test Suite

Set the parameters on the broker for the MQTT 3.1.1 optional behaviour you want. Currently these options are:

• publish_on_pubrel - whether, for QoS 2, outgoing publishes are propagated on receipt of an incoming pubrel rather than publish, default True
• overlapping_single - whether, for multiple subscriptions matching one publication, one message is sent out rather than one for each subscription, default True
• dropQoS0 - whether, when a client is disconnected, QoS 0 messages are thrown away, default True

These are set in mqtt/broker/start.py as default parameters to the run function. I'll implement that more elegantly when I get some time...

Then call:

python3 suite_generate.py

which will stop at 10 tests right now. I'm working on it stopping when desired coverage is achieved.

To Do List