Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Mihini/QA Plan/0.9

< Mihini‎ | QA Plan
Revision as of 08:46, 9 August 2013 by Lbarthelemy.sierrawireless.com (Talk | contribs) (Mihini Updates)

(coche) : test done, success


(erreur) : test done, failure


(avertissement) : test not done yet, or in progress


(ampoule grise) : test not consistent for this release


Results and Status

Device(s) used for tests RaspberryPi for GPIO tests, regular PC, 
Status ( ok tests / total tests) 10 / 15 suites
Comments
Final status with date




Miscellaneous [done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_1


Check optional dependency of agent.srvcon on agent.netman


Disable agent.netman (agent.config.network.activate = false).
Reboot the device and verify the correct initialization of the Mihini




(coche)

Config Store (agent.config )  [done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_2.1


Data Reading


Test reading whole config, command arg = agent.config.get(""). Verify execution.




(coche)


Mihini_2.2


Data Reading


Test reading sub part of config, command arg = agent.config.server. Verify execution.




(coche)


Mihini_2.3


Data Reading


1- Modify some config entry, read it using ReadNode command,
2- Call agent.config.default(), read entry again to check default value is read.



(coche)


Mihini_2.5


Test default()


Call default (agent.config.default()). Verify that default values are restored.




(coche)


Mihini_2.6


Test diff()


Change some values in agent.config. Verify that the differences are correctly reported in the Lua shell.




(coche)


Server Connection (agent.srvcon) [done]



Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_3.0.1


Connection policy: Cron


agent.config.server.autoconnect = { cron="mm hh dd MM YY"}. Reboot and verify connection according to cron entry.




Check.png

Mihini_3.0.2


Connection policy: On reboot


agent.config.server.autoconnect = { onboot = 300}. Reboot and verify connection ~300 seconds after reboot.




Check.png

Mihini_3.0.3


Connection policy: Periodic


agent.config.server.autoconnect = { period = 2 }. Reboot and verify a periodic connection every 2 minutes.




Check.png

Commands (agent.devman.cmds) [done]

This chapter must be done using platform jobs: retrieve data, apply settings, send command.


Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_3.1.1


Connect


Send job to device using platform server. Verify execution.




Check.png

Mihini_3.1.2


ReadNode


Send read job on Mihini asset to device using platform server. Use empty path (""). Verify execution.




Check.png

Mihini_3.1.3


ReadNode


Send job to device using platform server. Use existing path. Verify execution.




Check.png

Mihini_3.1.4


ReadNode


Send job to device using platform server. Use non-existing path. Verify execution.





Check.png


Mihini_3.1.5


Reboot


Send command to device using platform server. Verify execution.




Check.png

Mihini_3.1.6


UnknownCommand


Send command to device. Verify failure as command does not exist.




Check.png




Data Writing [done]

Test ID


Test Purpose


Test process


Automatic test


Result

Mihini_3.2.1


Data writing on agent.config


Executes from platform server a data writing on @sys.config. Verify execution.


1- Using datamodel from platform, modify Mihini > Configuration > Shell > HistorySize.
2- On device :":agent.srvcon.connect()"
3- On device: ":devicetree.get("config.shell.historysize")"


Check.png

Mihini_3.2.2


Data writing on asset datatree


Create a new asset (donkey) Executes from platform server a data writing on donkey.test. Verify execution.


testPolicies.test_SeveralJobs()


Check.png

Network Manager (agent.netman) [done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_4.1


Check that agent.netman is optional


De-activate agent.netman (agent.config.network.activate = false)
activate Fake Netman init signal (agent.config.network.initsignal="SOME_BEARER", with "SOME_BEARER" present in agent.config.mediation.pollingperiod)
Obviously, the target must provide somehow a pre-existent network connection




Check.png

Mihini_4.2


Connectivity with ETH or GPRS


Set agent.config.network.activate = true, agent.config.network.bearerpriority = {"ETH","GPRS"}, agent.config.network.maxconnectiontime = 30
Set agent.config.network.bearer.GPRS.apn="sim_apn" and agent.config.network.bearer.ETH.mode = "dhcp"
Reboot the device with an ethernet cable plugged.
Verify that the device is connected on the net with the ethernet bearer




Check.png

Mihini_4.3




Unplugged the ethernet cable, and execute a agent.srvcon.connecttoserver() command.
Verify that the device is connected with the GPRS bearer. Verify that after 30s a new selection of bearer is triggered.




Check.png

Mihini_4.4




Set agent.config.network.bearer.retry = 2, agent.config.network.bearer.retryperiod = 10, verify that agent.config.network.bearerpriority is {"ETH", "GPRS"}
Reboot the device without any ethernet cable plugged.
Verify that agent.netman tries to connect three times to the ethernet bearer (first attempt + 2 retries). Verify that agent.netman waits 10 seconds between each attempt.




Check.png

Mihini_4.5




Re-execute previous test with agent.config.network.bearer.ETH.retry = 3 and agent.config.network.bearer.ETH.retryperiod = 50.
Verify that the Mihini uses the specific bearer configuration instead of the "general" network configuration.




Check.png

Mihini_4.6




Set agent.config.network.activate = true, agent.config.network.bearerpriority = {"ETH","GPRS"}, agent.config.network.maxconnectiontime = 30
Set agent.config.network.bearer.GPRS.apn = "sim_apn" and agent.config.network.bearer.ETH.mode = "dhcp".
Reboot the device with an ethernet cable plugged.
Verify that the device is connected with the ethernet bearer.




Check.png

Mihini_4.7


Connectivity with selected bearer


Execute agent.netman.connect('GPRS').
Verify that the device is connected with the GPRS bearer.




Check.png

Mihini_4.8




Execute agent.netman.connect('ETH').
Verify that the device is connected with the ETH bearer.




Check.png

Software Update [done]

Test ID


Test Purpose


Test process


Automatic test


Result

Mihini_5.1


Update job success by AWTDA


add the SoftwareUpdate Command, configure the device so that initial conditions are met, connect to Server




Check.png




Mihini_5.2


Local update


put update package in drop folder, check update is done, package removed from drop folder, no update done on next boot


put file in update/drop/updatepackage.tar
 :agent.update.localupdate()
ls update/drop/



Check.png






Mihini_5.5


Update job errors


test each possible error cause:
1- job already in progress,
2- in manifest file, create a sw_list incompatibility (e.g. Mihini="<2.0"),
3- too many retries for a component





Check.png




Mihini_5.6


Update job retries


network errors, Mihini reboot during update (from platform)


testMihiniBigSize.zip
 :agent.appcon.list()
ls apps/




Check.png




Mihini_5.7


Data Reading


Send ReadNode Command on some Update table, like {cmd = ReadNode, path=@sys, args=update.swlist}




Check.png


Mihini_5.8


Data Writing


check data writing on update data is forbidden:
1- from the platform
2- from the asset/treemgr API.


1- on platform Mihini > Update > SWlist > Components > 3 > Version
2- :devicetree.set("update.swlist.components.3.version", 33)



Check.png


Mihini_5.9


Update Script


Update job with update package containing software component which Package field = @sys.update.**


create a updatepackage.tar and drop it in /runtime/update/drop, then use require"agent.update".localupdate() in shell agent to execute the local update



Check.png


Mihini_5.10


Software Update Lua API


Update job with update package targeting an asset registered using "airvantage.asset"


test_Update()
put file in update/drop/updatepackage.tar
 :agent.update.localupdate()
 :agent.update.getstatus()



Check.png



Application Container   [done]

(Linux Only)


Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_6.1


Application monitor daemon on Linux: setup


Run appmon_daemon before starting RA.
Be sure appmon_daemon has sufficient rights to do setuid/setgid actions:


  • either run it as root
  • or force uid/gid param to be current user




(coche)

Mihini_6.2


Application monitor app restart delay


test application with run function that reports error "immediately", check restart delay





(coche)

Mihini_6.3


Application Container and Application monitor communication


launch appmon, RA, add std apps in RA.AppCon, kill RA, check apps are still running, restart RA.
Check apps haven't been started a second time, add some new app to check communication is still ok





(coche)

Time[done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_7.1


Time unit test suite


Activate time but with no periodic synchronization. Require "tests.time", unittest.run().
Note: time unitest #3 need 64 bits system to pass.




Check.png

Mihini_7.2


On boot only synchronization


Set agent.config.time.ntppolling = 0. Verify that a NTP sync is done on boot.




Check.png

Mihini_7.3


On boot and periodic synchronization


Set agent.config.time.ntppolling = <cron entry>. Verify that a NTP sync is done on boot at <cron entry>.
example: "*/3 * * * *" for synchronization every 3 minutes .




Check.png

Mihini_7.4


On boot and periodic synchronization


Set agent.config.time.ntppooling = <number>. Verify that a NTP sync is done on boot and every <number> minutes.




Check.png

Serial Framework (modbus) [done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_8.1


All API functions in serial (RTU) mode


Run modbus_test.lua file calling all the API functions in an infinite loop. Verify execution.


automatique test with the modbus_test.lua file



Check.png


Mihini_8.2


All API functions in TCP mode


Run test.lua file calling all the API functions in an infinite loop. Verify execution.






 not tested Important.png


Messaging  [done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_10.1


Send SMS


Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). From platform, send SMS using messaging SendSMS command.




(coche) SMS reception is working properly. SMS sending was OK when SMS was received by another device

(avertissement) SMS sending was not fully tested because reception of 8bits SMS is not widely supported, on cell phones, the result varies from no SMS received at all to SMs received without the correct content.


Mihini_10.2


SMS fallback


Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). Activate SMS fallback (agent.config.network.smsfallback = <phonenumber>. Verify SMS fallback if network bearers (ETH and/or GPRS) are down.




NA

Mihini_10.3


SMS notification


Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). From platform, send AWTDA notification (ie Connect command through SMS acting like as a WakeUp). Verify execution.




NA

Migration Helper [done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_11.1 Test migration helper default script

Compile a brand new runtime, start it,

  • Check the message "GENERAL-INFO: Migration executed successfully" appears 
  • Verify that no migration are attempted on the following boots
(coche)

Mihini_11.2


Test migration helper without migration code


  • Remove migration script (if any):
    can runtime/lua/agent/migration.lua or runtime/lua/agent/migration.so
  • Executes "persist.save("MihiniVersion", Mihini_11.2)" to trigger a migration attempt on next boot.
  • Verify that the next boot reports no migration script found.
  • Verify that no migration are attempted on the following boots.





(coche)

Mihini_11.3


Test migration helper with migration code written in Lua


  • persist.save("MihiniVersion", Mihini_11.3)
  • put file runtime/lua/agent/migration.lua
  • reboot agent
  • check logs



(coche)

Mihini_11.4


Test migration helper with migration code written in C


  • persist.save("MihiniVersion", Mihini_11.4)
  • compile C MigrationScript using Migration doc examples, see section "Migration script templates"
  • Stop agent
  • Execute in shell `export SWI_LOG_VERBOSITY="MIGRATIONSCRIPT:info"`
    (in order to logs from migration script)
  • Start agent
  • check logs



(coche)

M2M Server[done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_13.0.1


Racon Lua sample on Linux


Build and run agent. Run Lua sample in standalone Lua VM. Verify execution.




(coche)

M2M Server APIs [done]

Test ID


Test Purpose


Test process


Automatic test


Result


Mihini_13.1.1


Racon unit test suite


Require "tests.racon", unittest.run().


require "tests.airvantageTest"
unittest.run()


Check.png


Mihini_13.1.2


Racon data sending


Create an asset (av=require "racon"; myasset = av.newasset("myasset"), av.init() myasset:start()). Use the newly created asset to push data. Verify data sending.


test_PolicyAll()


Check.png

Mihini_13.1.3


Racon Get-Set-Notify API


Create an asset. Use the newly created asset to set data in the Mihini data tree. Verify execution.


testJobs.test_SeveralJobs()


Check.png

Mihini_13.1.4


Racon Get-Set-Notify API


Create an asset. Use the newly created asset to get data from the Mihini data tree. Verify execution.


testJobs.test_SeveralJobs() Check.png

Mihini_13.1.5


Racon Get-Set-Notify API


Create an asset. Use the newly created asset to register a callback on the Mihini data tree modifications. Verify execution.


testJobs.test_asset_devicetree()




Check.png

Mihini_13.1.8


Racon Update API


Create an asset. Use the newly created to register a callback on update. Initialize an update of the asset. Verify execution.


testJobs.test_SeveralJobs()


Check.png


Device Status Access (N/A) [done]

Test ID


Test Purpose


Test Process


Automatic test


Result


Mihini_14.4.1


Poll & Notify


From an application in the Mihini, Poll the device status.
From an application in the Mihini, Register on the RSSI.(:devicetree.get('system.net.rssi')




NA

Mihini_14.4.2


Radio Information


Display all data defined in the M3DA Dev Tree and check they are relevant.

Check the availability of Radio Module data:
- Hardware Version: system.cellular.hw_info.hw_ver
- Firmware Version: system.cellular.sw_info.fw_ver






NA

Mihini_14.4.3


Router Information


Display all data defined in the M3DA Dev Tree and check they are relevant.

Check the availability of Router data:
- Hardware Version: system.hw_info.hw_ver
- Firmware Version: system.sw_info.fw_ver




NA

Mihini_14.4.4


Device Versioning


Display all data defined in the M3DA Dev Tree and check they are relevant.

From an application in the Mihini, Read the the Mihini version.


(:require "agent.update.common".data.swlist)




NA

Mihini Configuration Access

Test ID


Test Purpose


Test Process


Automatic test


Result


Mihini_14.5.1


Set, Poll & Notify


Change some RA data using the TreeMgr
Then read this value and ensure that the modification have been taken into account
Place a notification hook on a variable and modify it. Check that your hook handle has been triggered


test_devicetree_RSSI()




Factory Reset APIs [done]

Test ID


Test Purpose


Test Process


Automatic test


Result


Mihini_14.6.1


Factory Reset Integration


Install an one or more applications in the user space. Change some data in the treemgr.
Perform a Reset to factory defaults using the UI and check that all applications and data have been removed. The server sends this command, must be the one defined by default in defaultconfig.lua, because it need to receive the acknowlege from the device after



Check.png




Mihini Updates [done]


Test ID


Test Purpose


Test Process


Automatic test


Result

Mihini_14.7.1


Mihini updated as part of system update


Install one or more applications in the user space. Then perform an update of the Mihini component. Checks the installed applications are still here.




N/A Mihini agent update is dependent on each integration.

Mihini_14.7.2


Packaged apps & libraries


Prerequisite:
Install an application (here app1.tar) in its version 1.0

- Test: Install an application requiring a newest version and check that the installation is rejected.

- Test : Install testMihini application (SW_update_1_depends.tar by example) , then uninstall it by updating with the uninstall_testMihini archive. Check that Mihini is still running (no crash) and testMihini application has been removed.


SW_update_1_depends.tar
SW_update_1_depends_fail.tar
agent.update.localupdate(path,true) with path=/tmp/RA/


(coche)


Mihini_14.7.3


Can update existing app w/o uninstall


- Install an application in version 1.0
- Check the version of the application(:require"agent.update.common".data.swlist)
- Perform an update to a new version
- Check the new version of the application






(coche)



Mihini_14.7.4


Fresh-install option


Install an application containing several files with the parameter "purge=true"
Make an update to a new version with new files and new versions of old files.
Check that the new files have been taken into account.


SW_update_2_old.tar
SW_update_2_new.tar
agent.update.localupdate()


(coche)



Mihini_14.7.5


FIO - Base OS version checking


Try to install an application requiring a specific version of the Agent 
- with a wrong one: Rejected
- with a good one: Accepted





(coche)




Mihini_14.7.6


FIO - API Version checking


199 : Populate the Software Update component/feature list by installing some applications to provide Mihini version, applications versions etc. as features with corresponding versions.
This is the cas if you ran the previous tests, and your swlist is not empty :).

Check in the DeviceTree that those versions are available and up-to-date

(example of commands to run dt.get("update.swlist.components.1.version"))




(coche)




Mihini_14.7.7


FIO - Component/Library/App dependency version checking


Try to install an application requiring a library not present in the update package nor in the installed list apps. The installation shall be rejected.
Try to install an application requiring a library present in the update package. The installation shall be accepted.


SW_update_4.tar



(coche)



Mihini_14.7.8


Default exec app per package


Check that the 'autostart=true' parameter works


SW_update_5.tar


(coche)



Mihini_14.7.9


Global file resource sandboxing


Check that the Mihini agent runs with specific user rights
Check that an application runs with application specific user rights (different from Mihini's user)


1)
start appmon daemon using a command like

sudo ./bin/appmon_daemon -a /home/pi/mihini/start.sh -w /home/pi/mihini -u SOMEUSER -g SOMEUSERGROUP -n 5
check mihini agent is run as root, applications are run as SOMEUSER / SOMEUSERGROUP

start appmon daemon using a command like
sudo ./bin/appmon_daemon -a /home/pi/mihini/start.sh -w /home/pi/mihini -v ANOTHERUSER -h ANOTHERUSERGROUP -u SOMEUSER -g SOMEUSERGROUP -n 5
check mihini agent is run as ANOTHERUSER / ANOTHERUSERGROUP, applications are run as SOMEUSER / SOMEUSERGROUP


(coche)



Mihini_14.7.10


Check application rights
use an app that can:


  • create file and folder in working dir
  • rm files and folders created in working dir
  • use persist module to create Lua persisted files.
  • etc

Use app in attachement, try those test cycles:


  • install it, check install status, check app traces, update the app, check install status, check app traces, uninstall it, check uinstall status
  • install it, check install status, check app traces, reboot the device, check app traces, update the app, check install status, check app traces, uninstall it, check uinstall status
  • install it, check install status, check app traces, reboot the device, check app traces, uninstall it, check uninstall status

test_uasuser_rights_persist_installation.tar
test_uasuser_rights_persist_update.tar
test_uasuser_rights_persist_uninstallation.tar

agent.update.localupdate(path,true)
agent.update.getstatus()
 :agent.appcon.list()
 :devicetree.get("update.swlist.components.3.version")



(coche)




Mihini_14.7.11


Check many Mihini apps can be installed without rebooting the device


choose an app, install it and update it at least 10 times, see if everything is ok
at least 10 cycle of install/uninstall for same app
install 10 different apps (can be the same application code but 10 different app ids)




(coche)



Security: M3DA authentication and encryption[done]



Test ID Test purpose Test Process Automatic test Result
Mihini_14.9.1 none/none Test a simple connection from LUA shell and check the status. No errors should be raised

agent.config.server.authentication =nil

agent.config.server.encryption = nil

require 'agent.provisioning'.registration_password 'xy' (where x is first 4 numbers of S/N and y first 4 numbers of ENS)

(coche)
Mihini_14.9.2 MD5/AES128 idem

agent.config.server.authentication ='hmac-md5'

agent.config.server.encryption ='aes-cbc-128

require 'agent.provisioning'.registration_password 'xy

(coche)
Mihini_14.9.3 SHA1/AES128 idem

default config for authentication and encryption

<code>require 'agent.provisioning'.registration_password 'xy'

N/A (only HMAC-MD5 and AES-CBC-128 are suppotred)
Mihini_14.9.4 SHA1/AESCTR256 idem

agent.config.server.authentication ='hmac-md5'

agent.config.server.encryption ='aes-ctr-256'<code>require 'agent.provisioning'.registration_password 'xy'</code>

N/A (only HMAC-MD5 and AES-CBC-128 are suppotred)
Mihini_14.9.8 change password change password on Server side and Mihini. Check that connection is still OK




0.8 features [done]

Test ID


Test Purpose


Test Process


Result



AppCon - Provide a watchdog library for user apps


TBD: time before killing app
TBC: diff between kick and kill
Make an application with functions doing some very strong jobs (no wait) that will cause the app to not respond for several seconds
Test the following cases:
- check watchdog is enabled and start the job => app shall be killed/restarted
- use API the disable temporarly the watchdog before doing the job => app shall not be killed/restarted
- use API to kick the watchdog and start the job => app shall not be killed/restarted.
- use API the disable temporarly the watchdog before doing the job and job last more than the watchdog delay => app shall be killed/restarted


NA - Removed from 0.8 and 0.9 scope.



RA C API Implementation - Airvantage Asset Update API


Use an update package with an extra parameter in the manifest file and check that extra param is correctly sent to app updater


Check.png






Airvantage Lua API - make asset update result to be sent asynchronously


Create an asset in which we attached a callback function. The latter have to return "async" and then that is a part of this asset which calls sendUpdateResult() to send a asynchronous response to RA



Check.png




Update - Package Download


Agent receives a notification on DWL from the server, it download the package and send the knowlegement to the server. Check the agent log messages : :

  • UPDATE-INFO: Update Step: [4][dispatch_update] finished successfully
  • UPDATE-INFO: Acknowledging update command to M3DA server



Check.png





Update - Package Download Resume 1


Agent receives a notification on DWL from the server, Agent is starting download the package. During this time, kill the agent and then restart it. The agent shall restart the DL from the last received byte and not from the beginning. Check the agent log messages :

  • UPDATE-INFO: Update Step: [4][dispatch_update] finished successfully
  • UPDATE-INFO: Acknowledging update command to M3DA server

To know if the server supports or not the resumed download, launch the command in the terminal linux:

  • curl -H Range:bytes=16- -I http://your.url.server  : a 206 code must be displayed to support the resume download feature



Check.png




Update - Package Download Resume 2


Agent receives a notification on DWL from the server, Agent is starting download the package. During this time, unplug the network cable (about 60s) so that the timout event will be trigged. The next download retry will be done by the resume DL feature ( don't forget to replug the network cable) which shall restart the DL from the last received byte and not from the beginning. Check the agent log messages :

  • UPDATE-INFO: Update Step: [4][dispatch_update] finished successfully
  • UPDATE-INFO: Acknowledging update command to M3DA server



Check.png





Update - Package Download Resume 3


A C application receiving notification on DWL, shall pause and resume DL. If the server supports the partial content, the resume DL feature shall restart the DL from the last received byte and not from the beginning. Otherwise, it will restart the download from the beginning. Check the agent log messages :

  • UPDATE-INFO: Update Step: [4][dispatch_update] finished successfully
  • UPDATE-INFO: Acknowledging update command to M3DA server



Check.png




Add timeout management in EMP libs


make an asset command blocking for more than 60 seconds
install and start it
use the RA to call synchronously the command and check that after 60 seconds the RA returns an error.



Check.png




Update - Download: support user notification and request during download


  • Start downloading an update file
  • "Pause" the download
  • Reboot Mihini
  • "Resume" the download of update file


Check.png




Update - check space left before archive extraction


Create a tar.gz file which unzipped file is greater than free space disk and give it to updater. The updater shall return an error: file too long



Check.png




Handle the case Mihini reboot while apps were running


Use the Airvantage sample in both lua and C to:
- start RA with appmon deamon
- install application
- make the RA crash (force stop)



Check.png




Update - Add elastic retries in update package download mecanism


TBC: Disconnect the device from the network and start a remote application installation. The download will fail and the first retry shall occur after 1minute, the second shall occur only after 30 minutes and the last one after one hour.



Check.png





Message dispatch: no ack is sent when msg from srv targets unregistred asset - M3DA


Send a command to an unknown asset. Check that the message has been acknowledged.



Check.png





Security [done]



Test ID


Test Purpose


Test Process


Result




use defaultconfig.lua to configure auth and cipher parameters

MD5 / AES-CBC-128


Check that with this configuration of authentication method and encryption :


  • device is able to be authenticated by the server and authenticate the server (keys have been generated and stored in the keystore)
  • device is able to execute commands sent from server
  • device is able to receive data from the server
  • server is able to receive data from the device
Check.png

MD5 / NONE

Check that with this configuration of authentication method and encryption :


  • device is able to be authenticated by the server and authenticate the server (keys have been generated and stored in the keystore)
  • device is able to execute commands sent from server
  • device is able to receive data from the server
  • server is able to receive data from the device
Check.png

NONE / NONE

Check that with this configuration of authentication method and encryption :


  • device is able to connect to the server
  • device is able to execute commands sent from server
  • device is able to receive data from the server
  • server is able to receive data from the device
Check.png

Reset To Factory
  • Configure the device into one of the security configuration above and ensure that device/server are communicating correctly
  • Perform a reset to factory
  • Check that device and server are still communicating correctly (keys have been conserved)
Check.png

REST API [done]


Device Tree  [done]

Test ID


Test Purpose


Test Process


Result



GET Node


  • Perform a GET on "devicetree/config"
  • Check the returned configuration table match with the one returned by devicetree.get(config)

(coche)




  • Perform a GET on "devicetree/config/server/url"
  • Check the returned configuration table match with the one returned by devicetree.get(config.server.url)
(coche)


  • Perform a GET on "devicetree/config.server.url"
  • Check the returned configuration table match with the one returned by devicetree.get(config.server.url)
(coche)


  • Perform a GET on "devicetree/config.server"
  • Check the returned configuration table match with the one returned by devicetree.get(config.server)
(coche)


  • Create a node in config table :
config["1234"] = { my-leaf = "valid leaf";}
  • Perform a GET on "devicetree/config/1234/my-leaf"
  • Check "valid leaf" is returned
Check.png


  • Create a node in config table : config["@test"] = { my-leaf = "valid leaf";}
  • Perform a GET on "devicetree/config/@test/my-leaf"
  • Check "valid leaf" is returned
Check.png


  • Perform a GET on "update/config"
  • Check an error is returned
Check.png

PUT Node
  • Perform a PUT on "devicetree/config/server" with JSON payload {url : "somewhere.com", port : 8088}
  • Check success and persistence over a reboot
Check.png


  • Perform a PUT on "devicetree/config/server/url" with JSON payload "somewhere.com"
  • Check success and persistence over a reboot
Check.png


  • Perform a PUT on "devicetree/ALEOS" with JSON payload { avms : { url : "toto"; interval : 20000} }
  • Check success and persistence over a reboot
Check.png


  • Perform a PUT on "devicetree/config/notexistingnode/leaf" with JSON payload "valid leaf"
  • Check success and persistence over a reboot
(coche)


  • Perform a PUT on "devicetree/config/1234/my-leaf" with JSON payload

"valid leaf" 

  • Check success and persistence over reboot
(coche)


  • Perform a PUT on "devicetree/config/server/url" without JSON payload
  • Check an error is returned
(coche)

POST Node
  • Perform a POST on "devicetree/config/server/url" with JSON payload
{url : "somewhere.com", port : 8088}
  • Check an error is returned
(coche)

UPDATE Node
  • Perform an UPDATE on "devicetree/config/server/url" with JSON payload
{url : "somewhere.com", port : 8088}
  • Check an error is returned
(coche)

Application Container [done]

Test ID


Test Purpose


Test Process


Result



List applications

Prerequisite:


  • Install some application in application container

Tests steps:


  • Perform a GET on application
  • Check that returned list match with installed application list

(coche)

[LBA]
The test was: "Perform a GET on application/ "
The Mihini REST spec was also mentionning "application/"

However when I do a GET on "application/", it failed with "status: 404 No handler found for application/"

On the contrary, "GET on application" works correctly.

I read appcon code, feature export to REST confirm it is done to support "application" without trailing /.

Reading some forum topics and speaking about it here and there, I think supporting only "GET on application" is OK for now.

We may want to support both for next release, but the question can be extended to any REST URL . TBD.

For now, I've edited this test and Mihini REST spec to conform to current behavior, and kept the test result as succesful.








Prerequisite:


  • Install some application in application container

Tests steps:


  • Perform a POST on application
  • Check error is returned

(coche)


Status

Prerequisite:


  • Install an application named "testapp" in application container

Tests steps:


  • Perform a GET on application/testapp
  • Check that status is returned correctly

(coche)

[RPE]

The output is correct but I am not completly sure about the variable "appname" (with or without Rest), it contains "2" here... There is a bunch of code in agent/agent/appcon.lua which converts a string coming from the appmon daemon to a table. Please check if this is a bug or not

[LBA]

the field "appname" comes from appmon_daemon, and it would be better if it was named "appid" indeed. So I don't think this is a bug, likely a badly named variable. As this variable is already used in ApplicationContainer and that the issue is only seen when displaying the application status to user, I don't think it's worth renaming it for now.
We can open an issue for 0.10 I guess, so I set this test as successful.



Prerequisite:


  • noapp is a not an installed application

Tests steps:


  • Perform a GET on application/noapp
  • Check that an error is returned

(coche)


Start

Prerequisite:


  • Install an application named "testapp" in application container
  • Ensure application is stopped

Tests steps:


  • Perform a PUT on application/testapp/start
  • Check that application is now running


(coche)

[LBA] The Body must be set to "{}" (empty JSON payload), but the body can't be empty like stated in the Rest doc.



Prerequisite:


  • noapp is a not an installed application

Tests steps:


  • Perform a PUT on application/noapp/start
  • Check that an error is returned

(coche)


Stop

Prerequisite:


  • Install an application named "testapp" in application container
  • Ensure application is running

Tests steps:


  • Perform a PUT on application/testapp/stop
  • Check that application is now stopped

(coche)

[LBA]

The Body must be set to "{}" (empty JSON payload), but the body can't be empty like stated in the Rest doc.




Prerequisite:


  • noapp is a not an installed application

Tests steps:


  • Perform a PUT on application/noapp/stop
  • Check that an error is returned
(coche)

Configure

Prerequisite:


  • Install an application named "testapp" in application container
  • Ensure application is running

Tests steps:


  • Perform a PUT on application/testapp/configure with JSON payload "true"
  • Check that autostart parameter has been modified

(coche) LBA

I used ` curl -X PUT  -d "true" http://localhost:8357/application/toto/configure `



Prerequisite:


  • Install an application named "testapp" in application container
  • Ensure application is running

Tests steps:


  • Perform a PUT on application/testapp/configure without JSON payload
  • Check that an error is returned
(coche)

Update [done]

the specification of REST API is here http://wiki.eclipse.org/Mihini/Rest

Test ID


Test Purpose


Test Process


Result



Local update
  • Perform a POST on update/ with HTTP payload containing the binary application (tar)
  • Check that application has successfully been installed
=> ex : curl -i -X POST --data-binary @/your/path/to/file.tar localhost:8357/update/


(coche)





  • Perform a POST on update/ without payload
  • Check that an error is returned
=> ex : curl -i -X POST --data-binary @ localhost:8357/update/


(coche)






  • Perform a PUT on update/ with HTTP payload containing the binary application (tar)
  • Check that an error is returned
=> ex : curl -i -X PUT --data-binary @/your/path/to/file.tar localhost:8357/update/


(coche)





  • Perform a UPDATE on update/ with HTTP payload containing the binary application (tar)
  • Check that an error is returned
=> ex : curl -i -X UPDATE --data-binary @/your/path/to/file.tar localhost:8357/update/


(coche)





Status
  • Perform a GET on update/ without payload
  • Check that returned status matches with expected status
=> ex : curl -i -X GET localhost:8357/update/


(coche)



GPIO [done]

For all the GPIO tests, here are the prerequisites:

  • Use a device with system conforming to GPIO lib requirements (see download.eclipse.org/mihini/api/lua/gpio.html )
    RaspberryPI with Raspbian system is a good candidate.
  • Build GPIO targets:
    GPIO targets are not built by default, do it by typing:
    make gpio agent_treemgr_gpio
  • Start mihini agent / application with root user rights to be able to access to the GPIO.

    Here is a hardware setup to test the GPIO:
    TBD

GPIO Lib [done]

Test ID


Test Purpose


Test Process


Result



GPIO read
  • Connect push button to some device GPIO
  • Call gpio.configure on the GPIO with config:
    {direction="in", edge="none", activelow="0"}
  • Call gpio.read and check returned values are consistent with button state:
    button pushed -> 1
    button released -> 0


(coche)



GPIO write
  • Connect LED to somedevice GPIO
  • Call gpio.configure on the GPIO with config:
    {direction="out", edge="none", activelow="0"}
  • Call gpio.write(id, 0), then gpio.write(id, 1)


(coche)




GPIO register / notify
  • Connect LED to some device GPIO and connect push button to another device GPIO
  • Call gpio.configure on the GPIO connected to the button with config:
    {direction="in", edge="both", activelow="0"}
  • Call gpio.register on the GPIO connected to the button, for the hook function, you can use:

    function myhook(id, value)
       print(string.format("GPIO %d has changed! New value =%s", id, value))
       gpio.write(GPIO_led, value)
    end
  • Check LED mimics button actions


(coche)



GPIO automatic configuration
  • Clear GPIO configuration (e.g. by rebooting the device!)
  • Redo 3 first tests without calling configure explicitely
  • Check each gpio automatic config after doing test action (read/write/register) by calling gpio.getconfig(id)


(coche)



GPIO listing APIs:
  • call gpio.enabledlist() after doing at least one of the previous test without rebooting the device
  • check GPIO used in tests are listed



  • call gpio.availablelist()
  • check all device GPIO are listed


(coche)



GPIO devicetree handler [done]


Call devicetree.init() before doing any devicetree test.


Test ID

Test Purpose


Test Process


Result



GPIO device tree read
  • Connect push button to some device GPIO
  • Configure GPIO :
    devicetree.set("system.gpio.settings.id", {direction="in", edge="none", activelow=0})
  • Call devicetree.get("sytem.gpio.id")


(coche)



GPIO device tree write
  • Connect LED to some device GPIO
  • Configure GPIO :
    devicetree.set("system.gpio.settings.id", {direction="out", edge="none", activelow=0})
  • Call devicetree.set("system.gpio.id", 0) then devicetree.set("system.gpio.id", 1)


(coche)



GPIO device tree register / notify
  •  Connect LED to some device GPIO and connect push button to another device GPIO
  • Call devicetree.set("system.gpio.settings.id", {direction="in", edge="both", activelow=0}) on GPIO connected to the button
  • Call devicetree.register on the GPIO connected to the button (path: system.gpio.$ID_BUTTON) , for the hook function, you can use:
    local function hook(notif)
    devicetree.set("system.gpio.$ID_LED", notif["system.gpio.$ID_BUTTON"])
    end
  • Check LED mimics button actions


(coche)



GPIO device tree listing
  • Do a get on "system.gpio" and "system.gpio.settings", check only enabled GPIO are returned
  • Do a get on system.gpio.available, checks all system GPIO are returned as a single leaf value.


(coche)



  Unittests (Auto)   [done]

All the following tests shall have been successfully run on the release tag.


Test ID


Environement


Target x86 Target x86_64
bysant lua framework

(coche)


(coche)

luatobin lua framework

(coche)


(coche)

posixsignal lua framework

(coche)


(coche)

rpc lua framework

(coche)


(coche)

sched lua framework

(coche)


(coche)

persist lua framework

(coche)


(coche)

timer lua framework

(coche)


(coche)

bysant agent

(coche)


(coche)

luatobin agent

(coche)


(coche)

posixsignal agent

(coche)


(coche)

rpc agent

(coche)


(coche)

sched agent

(coche)


(coche)

persist agent

(coche)


(coche)

crypto agent

(coche)


(coche)

treemgr agent

(coche)


(coche)

config agent

(coche)


(coche)

update agent

(coche)


(coche)

stagedb agent

(coche)


(coche)

monitoring agent

(coche)


(coche)

asset_tree.lua racon

(coche)


(coche)

airvantage.lua racon

(coche)


(coche)

devicetree.lua racon

(coche)


(coche)

sms.lua racon

(coche)


(coche)

system.lua racon

(coche)


(coche)

emp.lua racon

(coche)


(coche)

dset_test (natif) racon

(coche)


(coche)

emp_test (natif) racon

(coche)


(coche)

av_test (natif) racon

(coche)


(coche)

sms_test (natif) racon

(coche)


(coche)

sys_test (natif) racon

(coche)


(coche)

updatetest (natif) racon

(coche)


(coche)

dt_test (natif) racon

(coche)


(coche)


Back to the top