Jump to: navigation, search

Difference between revisions of "Paho/MQTT Interop Testing Day"

(Testing Plans)
(Test Approach)
 
(20 intermediate revisions by 2 users not shown)
Line 28: Line 28:
 
* Jan. 15 - Test cases made available
 
* Jan. 15 - Test cases made available
 
* Feb. 1 - First phone call meeting to discuss the testing requirements
 
* Feb. 1 - First phone call meeting to discuss the testing requirements
 +
* Feb 20 - Deadline to register
 +
* Feb 26 - Second planning phone call
 
* March 17 - Testing day
 
* March 17 - Testing day
  
== Testing Plans ==
+
== Test Approach ==
  
This is a starting point for a plan for testing. More details [[Interop Testing Plan|here]].
+
Details of tests and and test material are held [[Interop Testing Plan|here]].
  
1) We need tests for both MQTT client libraries and servers.  Servers may also act as clients connecting to other servers (called a bridge in Mosquitto/RSMB).
+
There are two major types of MQTT components which we will be testing:
  
2) The MQTT OASIS specification is nearing a public consultation draft.  It will contain normative statements which will be listed, and the various degrees of compliance required categorized as in RFC 2119.  
+
# MQTT servers
 +
# MQTT client libraries.
  
"The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
+
Where an MQTT server has a built-in MQTT client capability (sometimes called an MQTT bridge), we aim to test that too, by linking several servers together.  An MQTT bridge is not a requirement for an MQTT server, but some implementations do have one, it seems like a fun idea to link them together and see what happens!
"OPTIONAL" in this document are to be interpreted as described in RFC 2119."
+
  
The drafts are available here: https://www.oasis-open.org/committees/documents.php?wg_abbrev=mqtt.  Working draft 15 contains the normative statement list.
+
The test material linked to above is made available so that developers of MQTT servers and clients can make an attempt at independent validation of their implmentations before the day, and maximize the possibility of successful interoperation.
  
3) I propose we take each of the listed normative statements and turn them into one or two test cases, with a link back to the specification.  Each statement will apply to either server or client library, some to both.  Each test case will test either client library or server and will consist of input sequences along with assertions about the results that should/must be observed.
+
Each MQTT server can use the client_test.py test program or generated test suites in the test material to ensure correct behaviour of the server.
  
4) Server tests can be described at the MQTT packet levelI already have some tests which do this - so does Roger. Each of the server tests will be a sequence of MQTT flows into a server, interleaved with the expected MQTT packets out of the server.
+
Each MQTT client library should build a test suite which can be used along with the client library, that matches the client_test.py test program at a minimumThe features which the test suite should include are:
  
5) Because the interfaces used to drive client libraries are all different, the client tests will need to be more abstract. They could also be termed in MQTT packet flows, in this sort of way "make calls to the client library to cause MQTT packet flows foo", "following packet flow bar, make a message available to the application", and so forth.  Using these test scenarios, any client library can be tested against any server.
+
* connecting to an MQTT server using the MQTT 3.1.1 standard (connect packet has protocol name MQTT and version 4)
 +
* 0-length clientid
 +
* sending and receiving messages with various QoS levels (0, 1 and 2)
 +
* receiving messages queued up for a non-cleansession client on reconnect
 +
* the setting and receiving of retained messages
 +
* unsetting retained messages
 +
* setting and receiving will messages
 +
* more than one subscription matching a publication
 +
* correct keepalive processing (when no other messages are being sent/received, PINGs should be)
  
6) Server bridge tests will be similar to client tests - probably a subset as bridges have more limited scope for behaviour in general.
+
== Participating Products ==
  
7) On the day, servers would be expected to be configured in a standard way to allow the client tests to work against them.  We'll publish that configuration as part of the client tests.
+
This is a current list of products that plan to participate in the testing day.
  
8) We will have at least one Mosquitto instance available for anyone to test against on the day. We expect to have Mosquitto updated to allow OASIS conforming clients to connect.  
+
{| cellpadding="2" border="1" class="wikitable"
 
+
|-
9) Other aspects, which may not be in the specification directly, that we might like tests for: SSL/TLS connections, High Availability client library support.
+
! Product
 +
! | MQTT Client or Server
 +
! | People Attending
 +
|-
 +
| IBM MessageSight
 +
| Server
 +
| Dave Locke
 +
|-
 +
| RabbitMQ
 +
| Server
 +
| Andy Piper
 +
|-
 +
| WSO2 ESB and Message Broker
 +
| Client and Server
 +
| Paul Fremantle
 +
|-
 +
| ClearBlade
 +
| Client and Server
 +
| Aaron Allsbrook
 +
|-
 +
| Mosquitto
 +
| Server
 +
| Ian Craggs
 +
|-
 +
| 2lemetry
 +
| Server
 +
| Kyle Roche, Tim Kellogg, Chris Chiappone
 +
|-
 +
| HiveMQ
 +
| Server
 +
| Klemens Edler and Florian Pirchner
 +
|-
 +
| AirVantage
 +
| Server
 +
| Manuel Sangoi
 +
|-
 +
| Active MQ, JBoss A-MQ
 +
| Client and Server
 +
| Dhiraj Bokde
 +
|-
 +
| Gabriel
 +
| Server
 +
| Tim Kellog
 +
|-
 +
| WunderBar
 +
| Client
 +
| Paul Hopton
 +
|-
 +
| Skynet
 +
| Client and Server
 +
| Chris Matthieu
 +
|-
 +
| Skynet
 +
| Client and Server
 +
| Chris Matthieu
 +
|-
 +
| Loop
 +
| Server
 +
| Shah Vatsal
 +
|-
 +
| Smart Sensors, IP Devices
 +
| Client and Server
 +
| Paul Hancock
 +
|-
 +
| Xively
 +
| Client and Server
 +
| Philip DesAutels
 +
|-
 +
| Node RED
 +
| Client
 +
| Dave Conway-Jones
 +
|-
 +
| openpicus
 +
| Client
 +
| Andy Piper
 +
|-
 +
| Mosca
 +
| Server
 +
| Andy Piper
 +
|-
 +
| Software AG Universal Messaging
 +
| Server
 +
| Alex Kritikos
 +
|-
 +
| Kura
 +
| Client
 +
| Wes Johnson
 +
|}
  
 
== Participation ==
 
== Participation ==
  
Organizations and individuals that have implemented a 'product' using MQTT client or server are welcome to participate. Each 'product' can send 2 representatives to participate in the testing. Due to space and logistics there will be a limit of 20 products that can participate in the testing day. 'Prduct' = a commercial product, open source project or cloud service.
+
Organizations and individuals that have implemented a 'product' using MQTT client or server are welcome to participate. Each 'product' can send 2 representatives to participate in the testing. Due to space and logistics there will be a limit of 20 products that can participate in the testing day. 'Product' = a commercial product, open source project or cloud service.
  
 
Participation in the Interop Testing Day will be free to all EclipseCon attendees. If you don't want to attend EclipseCon you might have to pay a nominal fee to pay for the food. We still need to work out these details but we promise it won't have more than $100/participant. Of course everyone should really attend EclipseCon since there will be lots of M2M and IoT sessions.
 
Participation in the Interop Testing Day will be free to all EclipseCon attendees. If you don't want to attend EclipseCon you might have to pay a nominal fee to pay for the food. We still need to work out these details but we promise it won't have more than $100/participant. Of course everyone should really attend EclipseCon since there will be lots of M2M and IoT sessions.
  
To register please [http://mqtttestingday.eventbrite.com sign-up using our eventbrite site.]   
+
Deadline to register is February 20, 2014. To register please [http://mqtttestingday.eventbrite.com sign-up using our eventbrite site.]   
  
 
NOTE: We are not making provisions for remote participation. Part of the success for these types of events is the face to face contact.
 
NOTE: We are not making provisions for remote participation. Part of the success for these types of events is the face to face contact.

Latest revision as of 10:55, 24 February 2014

We will be hosting an MQTT Interop Testing Day on Monday, March 17, 2014 in Burlingame, CA. The goal is to have as many different MQTT client and server implementations participate in interoperability testing to validate the implementation of the upcoming OASIS MQTT standard.

The outcome of this day will be a report that shows the products that successfully participated in the interop testing. Hoepfully we will demonstrate the true power of an open standard for the IoT and M2M industry.

Date and Location

Monday, March 17
9:00am-5:00pm
Hyatt Regency San Francisco Airport (same place at EclipseCon 2014)

Agenda (DRAFT)

1. Introduction and update from OASIS TC

2. First round of testing - client product match with a server product

3. Second round - multiple different clients with one server

4. Grand finale - multiple servers bridged together

5. Creating the testing report

6. Conversation and Discovering what cool things other people are doing with MQTT

Schedule

  • Dec. 1 - Sign-up open
  • Jan. 15 - Test cases made available
  • Feb. 1 - First phone call meeting to discuss the testing requirements
  • Feb 20 - Deadline to register
  • Feb 26 - Second planning phone call
  • March 17 - Testing day

Test Approach

Details of tests and and test material are held here.

There are two major types of MQTT components which we will be testing:

  1. MQTT servers
  2. MQTT client libraries.

Where an MQTT server has a built-in MQTT client capability (sometimes called an MQTT bridge), we aim to test that too, by linking several servers together. An MQTT bridge is not a requirement for an MQTT server, but some implementations do have one, it seems like a fun idea to link them together and see what happens!

The test material linked to above is made available so that developers of MQTT servers and clients can make an attempt at independent validation of their implmentations before the day, and maximize the possibility of successful interoperation.

Each MQTT server can use the client_test.py test program or generated test suites in the test material to ensure correct behaviour of the server.

Each MQTT client library should build a test suite which can be used along with the client library, that matches the client_test.py test program at a minimum. The features which the test suite should include are:

  • connecting to an MQTT server using the MQTT 3.1.1 standard (connect packet has protocol name MQTT and version 4)
  • 0-length clientid
  • sending and receiving messages with various QoS levels (0, 1 and 2)
  • receiving messages queued up for a non-cleansession client on reconnect
  • the setting and receiving of retained messages
  • unsetting retained messages
  • setting and receiving will messages
  • more than one subscription matching a publication
  • correct keepalive processing (when no other messages are being sent/received, PINGs should be)

Participating Products

This is a current list of products that plan to participate in the testing day.

Product MQTT Client or Server People Attending
IBM MessageSight Server Dave Locke
RabbitMQ Server Andy Piper
WSO2 ESB and Message Broker Client and Server Paul Fremantle
ClearBlade Client and Server Aaron Allsbrook
Mosquitto Server Ian Craggs
2lemetry Server Kyle Roche, Tim Kellogg, Chris Chiappone
HiveMQ Server Klemens Edler and Florian Pirchner
AirVantage Server Manuel Sangoi
Active MQ, JBoss A-MQ Client and Server Dhiraj Bokde
Gabriel Server Tim Kellog
WunderBar Client Paul Hopton
Skynet Client and Server Chris Matthieu
Skynet Client and Server Chris Matthieu
Loop Server Shah Vatsal
Smart Sensors, IP Devices Client and Server Paul Hancock
Xively Client and Server Philip DesAutels
Node RED Client Dave Conway-Jones
openpicus Client Andy Piper
Mosca Server Andy Piper
Software AG Universal Messaging Server Alex Kritikos
Kura Client Wes Johnson

Participation

Organizations and individuals that have implemented a 'product' using MQTT client or server are welcome to participate. Each 'product' can send 2 representatives to participate in the testing. Due to space and logistics there will be a limit of 20 products that can participate in the testing day. 'Product' = a commercial product, open source project or cloud service.

Participation in the Interop Testing Day will be free to all EclipseCon attendees. If you don't want to attend EclipseCon you might have to pay a nominal fee to pay for the food. We still need to work out these details but we promise it won't have more than $100/participant. Of course everyone should really attend EclipseCon since there will be lots of M2M and IoT sessions.

Deadline to register is February 20, 2014. To register please sign-up using our eventbrite site.

NOTE: We are not making provisions for remote participation. Part of the success for these types of events is the face to face contact.

Test Code Collection

Organizers

  • Ian Craggs, IBM - Setting up the testing plans
  • Ian Skerrett, Eclipse Foundation - Helping with the logistics and report writing

Planning Meeting Minutes