Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Paho/MQTT Interop Testing Day 2015"

(Agenda (DRAFT))
 
(11 intermediate revisions by 2 users not shown)
Line 1: Line 1:
We will be hosting an MQTT Interop Testing Day on Monday, March 9, 2014 in Burlingame, CA. The goal is to have as many different MQTT client and server implementations participate in interoperability testing to validate the implementation of the OASIS MQTT 3.1.1 specification.
+
 
 +
'''NOTE: This event has been cancelled. Please follow the Eclipse Paho mailing list for more information about conformance test suites.'''
 +
 
 +
We will be hosting an MQTT Interop Testing Day on Monday, March 9, 2015 in Burlingame, CA. The goal is to have as many different MQTT client and server implementations participate in interoperability testing to validate the implementation of the OASIS MQTT 3.1.1 specification.
  
 
The outcome of this day will be a report that shows the products that successfully participated in the interop testing. Hoepfully we will demonstrate the true power of an open standard for the IoT and M2M industry.
 
The outcome of this day will be a report that shows the products that successfully participated in the interop testing. Hoepfully we will demonstrate the true power of an open standard for the IoT and M2M industry.
Line 7: Line 10:
 
Monday, March 9<br>
 
Monday, March 9<br>
 
9:00am-5:00pm<br>
 
9:00am-5:00pm<br>
Hyatt Regency San Francisco Airport  (same place at EclipseCon 2014)
+
Hyatt Regency San Francisco Airport  (same place at EclipseCon 2015)
  
 
== Agenda (DRAFT) ==
 
== Agenda (DRAFT) ==
Line 15: Line 18:
 
== Schedule ==
 
== Schedule ==
  
* Dec. 1 - Sign-up open
+
* Oct 23 - Sign-up open
 
* Jan. 15 - Test cases made available
 
* Jan. 15 - Test cases made available
* Feb. 1 - First phone call meeting to discuss the testing requirements
+
* aprpox. Feb. 1 - First phone call meeting to discuss the testing requirements
* Feb 20 - Deadline to register
+
* Jan 20 - Deadline to register
* Feb 26 - Second planning phone call
+
* approx Feb 26 - Second planning phone call
* March 17 - Testing day
+
* March 9 - Testing day
  
 
== Test Approach ==
 
== Test Approach ==
  
Details of tests and and test material are held [[Interop Testing Plan|here]].
+
More details will be available close to the time but here is a starting point from [http://modelbasedtesting.co.uk/?p=144 Ian Craggs ]
  
There are two major types of MQTT components which we will be testing:
+
This year will will focus on validation that each product conforms to the 3.1.1 specification and interoperability testing.
 
+
# MQTT servers
+
# MQTT client libraries.
+
 
+
Where an MQTT server has a built-in MQTT client capability (sometimes called an MQTT bridge), we aim to test that too, by linking several servers together.  An MQTT bridge is not a requirement for an MQTT server, but some implementations do have one, it seems like a fun idea to link them together and see what happens!
+
 
+
The test material linked to above is made available so that developers of MQTT servers and clients can make an attempt at independent validation of their implmentations before the day, and maximize the possibility of successful interoperation.
+
 
+
Each MQTT server can use the client_test.py test program or generated test suites in the test material to ensure correct behaviour of the server.
+
 
+
Each MQTT client library should build a test suite which can be used along with the client library, that matches the client_test.py test program at a minimum.  The features which the test suite should include are:
+
 
+
* connecting to an MQTT server using the MQTT 3.1.1 standard (connect packet has protocol name MQTT and version 4)
+
* 0-length clientid
+
* sending and receiving messages with various QoS levels (0, 1 and 2)
+
* receiving messages queued up for a non-cleansession client on reconnect
+
* the setting and receiving of retained messages
+
* unsetting retained messages
+
* setting and receiving will messages
+
* more than one subscription matching a publication
+
* correct keepalive processing (when no other messages are being sent/received, PINGs should be)
+
  
 
== Participating Products ==
 
== Participating Products ==
  
This is a current list of products that plan to participate in the testing day.
+
To be updated closer to the time.
 
+
{| cellpadding="2" border="1" class="wikitable"
+
|-
+
! Product
+
!  | MQTT Client or Server
+
!  | People Attending
+
|-
+
| IBM MessageSight
+
| Server
+
| Dave Locke
+
|-
+
| RabbitMQ
+
| Server
+
| Andy Piper
+
|-
+
| WSO2 ESB and Message Broker
+
| Client and Server
+
| Paul Fremantle
+
|-
+
| ClearBlade
+
| Client and Server
+
| Aaron Allsbrook
+
|-
+
| Mosquitto
+
| Server
+
| Ian Craggs
+
|-
+
| 2lemetry
+
| Server
+
| Kyle Roche, Tim Kellogg, Chris Chiappone
+
|-
+
| HiveMQ
+
| Server
+
| Klemens Edler and Florian Pirchner
+
|-
+
| AirVantage
+
| Server
+
| Manuel Sangoi
+
|-
+
| Active MQ, JBoss A-MQ
+
| Client and Server
+
| Dhiraj Bokde
+
|-
+
| Gabriel
+
| Server
+
| Tim Kellog
+
|-
+
| WunderBar
+
| Client
+
| Paul Hopton
+
|-
+
| Skynet
+
| Client and Server
+
| Chris Matthieu
+
|-
+
| Skynet
+
| Client and Server
+
| Chris Matthieu
+
|-
+
| Loop
+
| Server
+
| Shah Vatsal
+
|-
+
| Smart Sensors, IP Devices
+
| Client and Server
+
| Paul Hancock
+
|-
+
| Xively
+
| Client and Server
+
| Philip DesAutels
+
|-
+
| Node RED
+
| Client
+
| Dave Conway-Jones
+
|-
+
| openpicus
+
| Client
+
| Andy Piper
+
|-
+
| Mosca
+
| Server
+
| Andy Piper
+
|-
+
| Software AG Universal Messaging
+
| Server
+
| Alex Kritikos
+
|-
+
| Kura
+
| Client
+
| Wes Johnson
+
|}
+
  
 
== Participation ==
 
== Participation ==
 +
 +
'''NOTE: This event has been cancelled. Please follow the Eclipse Paho mailing list for more information about conformance test suites.'''
  
 
Organizations and individuals that have implemented a 'product' using MQTT client or server are welcome to participate. Each 'product' can send 2 representatives to participate in the testing. Due to space and logistics there will be a limit of 20 products that can participate in the testing day. 'Product' = a commercial product, open source project or cloud service.
 
Organizations and individuals that have implemented a 'product' using MQTT client or server are welcome to participate. Each 'product' can send 2 representatives to participate in the testing. Due to space and logistics there will be a limit of 20 products that can participate in the testing day. 'Product' = a commercial product, open source project or cloud service.
  
Participation in the Interop Testing Day will be free to all EclipseCon attendees. If you don't want to attend EclipseCon you might have to pay a nominal fee to pay for the food. We still need to work out these details but we promise it won't have more than $100/participant. Of course everyone should really attend EclipseCon since there will be lots of M2M and IoT sessions.
+
Participation in the Interop Testing Day will be free to all EclipseCon attendees with All Access passes. If you don't want to attend EclipseCon the cost will be $150/participant if you are a member of the Eclipse IoT Working Group or $300/participant if you are not. You should really attend EclipseCon since there will be lots of IoT sessions.
  
Deadline to register is February 20, 2014. To register please [http://mqtttestingday.eventbrite.com sign-up using our eventbrite site.] 
 
  
 
NOTE: We are not making provisions for remote participation. Part of the success for these types of events is the face to face contact.
 
NOTE: We are not making provisions for remote participation. Part of the success for these types of events is the face to face contact.
 
==Test Code Collection==
 
* [[File:Mqtt-test.zip]]
 
  
 
== Organizers ==
 
== Organizers ==
Line 161: Line 50:
 
* Ian Craggs, IBM  - Setting up the testing plans
 
* Ian Craggs, IBM  - Setting up the testing plans
 
* Ian Skerrett, Eclipse Foundation - Helping with the logistics and report writing
 
* Ian Skerrett, Eclipse Foundation - Helping with the logistics and report writing
 
Planning Meeting Minutes
 
 
* [[Paho/MQTT_Interop_Testing_Day/Meeting Minutes Nov 4]]
 

Latest revision as of 17:20, 26 January 2015

NOTE: This event has been cancelled. Please follow the Eclipse Paho mailing list for more information about conformance test suites.

We will be hosting an MQTT Interop Testing Day on Monday, March 9, 2015 in Burlingame, CA. The goal is to have as many different MQTT client and server implementations participate in interoperability testing to validate the implementation of the OASIS MQTT 3.1.1 specification.

The outcome of this day will be a report that shows the products that successfully participated in the interop testing. Hoepfully we will demonstrate the true power of an open standard for the IoT and M2M industry.

Date and Location

Monday, March 9
9:00am-5:00pm
Hyatt Regency San Francisco Airport (same place at EclipseCon 2015)

Agenda (DRAFT)

TBD

Schedule

  • Oct 23 - Sign-up open
  • Jan. 15 - Test cases made available
  • aprpox. Feb. 1 - First phone call meeting to discuss the testing requirements
  • Jan 20 - Deadline to register
  • approx Feb 26 - Second planning phone call
  • March 9 - Testing day

Test Approach

More details will be available close to the time but here is a starting point from Ian Craggs

This year will will focus on validation that each product conforms to the 3.1.1 specification and interoperability testing.

Participating Products

To be updated closer to the time.

Participation

NOTE: This event has been cancelled. Please follow the Eclipse Paho mailing list for more information about conformance test suites.

Organizations and individuals that have implemented a 'product' using MQTT client or server are welcome to participate. Each 'product' can send 2 representatives to participate in the testing. Due to space and logistics there will be a limit of 20 products that can participate in the testing day. 'Product' = a commercial product, open source project or cloud service.

Participation in the Interop Testing Day will be free to all EclipseCon attendees with All Access passes. If you don't want to attend EclipseCon the cost will be $150/participant if you are a member of the Eclipse IoT Working Group or $300/participant if you are not. You should really attend EclipseCon since there will be lots of IoT sessions.


NOTE: We are not making provisions for remote participation. Part of the success for these types of events is the face to face contact.

Organizers

  • Ian Craggs, IBM - Setting up the testing plans
  • Ian Skerrett, Eclipse Foundation - Helping with the logistics and report writing

Back to the top