Interop Testing Plan
The test material is based upon the current OASIS consultation draft of the specification.
The test material described below can be found in the Paho testing git repository.
- 1 Principles
- 2 Conformance Statements
- 3 Model Based Testing
- 4 Test Material Components
- 5 Test Procedures
- 6 To Do List
1) We need tests for both MQTT client libraries and servers. Servers may also act as clients connecting to other servers (called a bridge in Mosquitto/RSMB).
2) The MQTT OASIS specification now has a public consultation draft. It contains normative statements which are listed, and the various degrees of compliance required categorized as in RFC 2119.
"The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119."
3) I propose we take each of the listed normative statements and turn them into one or two test cases, with a link back to the specification. Each statement will apply to either server or client library, some to both. Each test case will test either client library or server and will consist of input sequences along with assertions about the results that should/must be observed.
4) Server tests can be described at the MQTT packet level. Each of the server tests will be a sequence of MQTT flows into a server, interleaved with the expected MQTT packets out of the server.
5) Because the interfaces used to drive client libraries are all different, the client tests will need to be more abstract. They could also be termed in MQTT packet flows, in this sort of way "make calls to the client library to cause MQTT packet flows foo", "following packet flow bar, make a message available to the application", and so forth. Using these test scenarios, any client library can be tested against any server.
6) Server bridge tests will be similar to client tests - a subset as bridges have more limited scope for behaviour in general.
7) On the day, servers would be expected to be configured in a standard way to allow the client tests to work against them. We'll publish that configuration as part of the client tests.
8) We will have at least one Mosquitto instance available for anyone to test against on the day. We expect to have Mosquitto updated to allow OASIS conforming clients to connect.
9) Other aspects which may not be in the specification directly, might have additional tests, but the top priority is testing of the new specification. Such aspects as SSL/TLS connections, High Availability client library support.
The specification contains a list of identified conformance statements, which are labelled with sequence numbers. Although the statements are to a certain extent arbitrary, due to the use of language in the specification (wherever the words MUST or SHOULD are used), nevertheless they are a good place to start in determining conformance. I anticipate adding more statements to this list in due course, but for now we will go with the list as identified in the current version of the specification.
I have extracted the list into a spreadsheet where tests for both MQTT clients and servers are briefly described. I am using this as a coverage checklist to view progress of the current test material.
Model Based Testing
I have taken the approach of what can be described as model based testing. A fully automated test at its core is comprised of two sets of data:
- input sequences
- expected results, or outputs.
MQTT is a client/server protocol, meaning that in general there are two components, clients and servers to test.
Test Material Components
I'll start by describing the components of the test material.
Conformance Statements Spreadsheet
Conformance statements extracted into a spreadsheet, associated with client and server tests, and coverage statements checked off.
A test or "model" broker is in the package mqtt/broker. You can run the broker with the command:
and if running successfully, you will see this:
INFO 20140203 220904 MQTT 3.1.1 Paho Test Broker
INFO 20140203 220904 Optional behaviour, publish on pubrel: True
INFO 20140203 220904 Optional behaviour, single publish on overlapping topics: True
INFO 20140203 220904 Optional behaviour, drop QoS 0 publications to disconnected clients: True
INFO 20140203 220904 Starting the MQTT server on port 1883
The first question I will be asked (thanks Ian Skerrett) is why write another broker, when Mosquitto, RSMB or others already exist. The reasons are:
- I already had one written before (but that is not really important)
- a model broker need not have all the implementation details of a production server
- the model is seeded with conformance statements - it knows when they have been executed
- the model is written in Python and is intended to be simple and serve as an illustration and clarification for the specification.
I hope the source can be scrutinized and discussed by the community, both for understanding the specification, and discussion of MQTT server implementation. The source consists of the following files:
- start.py - Python socket server wrapper for the MQTT code
- MQTTV311.py - MQTT packet formatting
- MQTTBrokers.py - MQTT packet interactions of the broker
- Brokers.py - core broker behaviour, without MQTT packet details
- SubscriptionEngines.py - subscription and retained message processing
- Subscriptions.py - handling of individual subscriptions
- Topics.py - validation and matching of topics
I've put these into a literate programming system for a pretty printed display. I intend to add descriptions to this source, so that the reasons for the implementation are explained, and also to allow comments on the source pages.
As the broker is executed, it notes when it encounters any embedded conformance statements, and will display at the end of its run coverage figures, like this:
INFO 20140203 131616 exceptions 2 out of 17 11%
INFO 20140203 131616 coverages 35 out of 55 63%
so you can run a test suite against it and determine the level of coverage it achieves, and whether more tests are needed.
The mqtt.client package contains a test MQTT 3.1.1 client which is used for sanity checking the function of the test broker and also for various clean-up operations when running tests.
An example of its use is shown in client_test.py. This program can be used as a basic test of an MQTT 3.1.1 broker implementation. It exercises:
- message queueing for offline clients
- retained messages
- clearing retained messages
- setting and receiving will messages
- $ topics
- overlapping subscriptions
- keepalive timeout
and when added:
- username & password (subscribe failure)
- redelivery on reconnect.
The coverage of the test broker conformance statements as of today (Feb 3rd) by running this program is:
INFO 20140203 220602 exceptions 2 out of 17 11%
INFO 20140203 220602 coverages 35 out of 55 63%
Run this program against your own server as a first test.
The mqtt.proxy package contains a test "proxy" which can be used to redirect traffic from localhost:1883 (typically) to another host and port.
This makes it useful when you want to run a test against a remote system, but your test is coded for localhost:1883 for instance. It will also trace the packets being sent and received, for information. To run the proxy:
python3 run_proxy.py remote_hostname remote_port
This currently runs in text mode. The module wmqttsas.py is graphical wxPython version, but I need to get that running.
Model-Based Testing Package
The mbt package contains a test suite generation tool which exercises paths through an input model. It takes steps to maximize coverage, such as avoiding paths already trodden. I want to make it more coverage driven, so that it will actively seek out coverage statements of the kind embedded in the test broker.
An example of its use is shown in MQTTV311_spec.py, which is used to generate the broker test suites. It is desribed below.
Broker Test Input Model
The broker test input model describes the input data to be used when generating broker tests. There are a set of actions, which are functions annotated with the "@mbt.action" label. In this program, the actions are:
which are chosen partly randomly to generate input sequences. The data for parameters are listed in the mbt.choices statements. For instance, the set of clientids used is defined in the statement
mbt.choices("clientids", ("", "normal", "23 characters4567890123", "A clientid that is too long - should fail"))
The input sequences generated are combined with the results from the model broker to construct complete self-checking tests.
Testing an MQTT Client Library
Start the test broker, as described above.
Run your test suite against this broker. Note the coverage achieved when you stop the broker. Try and get more coverage!
The client_test.py program, as described above, is a good basis for the sort of coverage that ought to be achieved.
With client libraries that ensure the data that is sent to the server consists of well-formed MQTT packets, the tests are likely hit the good paths in the broker rather than the exceptions. So you don't need to worry if your exception coverage is low or non-existent.
When I've generated more test suites, they could be used as templates for client test suites. I'm not sure about that yet.
Testing an MQTT Server
python3 client_test.py [hostname:port]
as a first test. If hostname:port are not specified, localhost:1883 is assumed.
The full goal of this approach is to run a generated broker test suite. There is a basic generated test suite in testsuites/basic. The syntax of the command is:
python3 run_test.py --testname test_file --testdir test_dirname --hostname hostname --port port
It will run the test suite against localhost:1883 by default. To run the test suite against a remote server, or one on a different port, use the hostname/port options. For example, to run the test suite in testsuites/basic against a remote server:
python3 run_test.py --testdir testsuites/basic --hostname m2m.eclipse.org --port 1883
You can run an individual test with the --testname option: the value should be the name of a test log file, like testsuites/basic/test.log.1.
Generating an MQTT Server Test Suite
Set the parameters on the broker for the MQTT 3.1.1 optional behaviour you want. Currently these options are:
- publish_on_pubrel - whether, for QoS 2, outgoing publishes are propagated on receipt of an incoming pubrel rather than publish, default True
- overlapping_single - whether, for multiple subscriptions matching one publication, one message is sent out rather than one for each subscription, default True
- dropQoS0 - whether, when a client is disconnected, QoS 0 messages are thrown away, default True
These are set in mqtt/broker/start.py as default parameters to the run function. I'll implement that more elegantly when I get some time...
which will stop at 10 tests right now. I'm working on it stopping when desired coverage is achieved.
Testing an MQTT Bridge
A bridge is an MQTT client embedded inside an MQTT server, which allows multiple servers to be linked together. A pair of test programs has been created which send messages to and from each other. First link two or more servers together, making sure they will send messages between each other using the bridge/toA and bridge/toB topics.
python3 bridge_testB.py --hostname hostname --port port
with hostname and port parameters set to connect to the server which is receving messages on the bridge/toB topic, and sending messages on the bridge/toA topic.
Within 30 seconds, run
python3 bridge_testA.py --hostname hostname --port port
with hostname and port parameters set to connect to the server which is receiving messages on the bridge/toA topic, and sending messages on the bridge/toB topic.
No assertions should be violated.
To Do List
In some sort of priority order.
- make the running of generated tests cope with arbitrary packet ids sent from the broker
- add zero-length clientid behaviour to the model broker
- make the generation of broker test suites directed by the model broker coverage
- support for websockets tests somehow (added to test broker)
- add pre-generated broker test suites to the repository for immediate use
- complete seeding the model broker with the conformance statements from the spec
- comment and explain the model broker code thoroughly
- add more conformance statements to the model broker for "proper" completeness (although the complete set from the spec is a good start, and I'm thinking good enough for the interops testing day)
- add some bridge testing material
SSL/TLS testing will not be part of the interops testing day. The point is to test implementations of the MQTT protocol, whereas SSL/TLS is a layer above.