Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Mihini/QA Plan/0.9
|
Contents
Miscellaneous
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_1
|
Check optional dependency of agent.srvcon on agent.netman
|
Disable agent.netman (agent.config.network.activate = false).
|
|
|
Config Store (agent.config)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_2.1
|
Data Reading
|
Test reading whole config, command arg = agent.config.get(""). Verify execution.
|
|
|
Mihini_2.2
|
Data Reading
|
Test reading sub part of config, command arg = agent.config.server. Verify execution.
|
|
|
Mihini_2.3
|
Data Reading
|
1- Modify some config entry, read it using ReadNode command,
|
|
|
Mihini_2.4
|
Data Writing
|
Data writing from platform: path @sys.config.mediation, variable = timeout, check that current value is replaced by written one and persisted
|
|
|
Mihini_2.5
|
Test default()
|
Call default (agent.config.default()). Verify that default values are restored.
|
|
|
Mihini_2.6
|
Test diff()
|
Change some values in agent.config. Verify that the differences are correctly reported in the Lua shell.
|
|
|
Server Connection (agent.srvcon)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_3.0.2
|
Connection policy: On Demand
|
agent.config.server.autoconnect.ondemand = <number>, agent.srvcon.connecttoserver(). Send data via agent.devman.hl. ange
|
|
|
Mihini_3.0.3
|
Connection policy: Cron
|
agent.config.server.autoconnect = { cron="mm hh dd MM YY"}. Reboot and verify connection according to cron entry.
|
agent.config.server.autoconnect={cron="* * * * *"} (no jitter)
|
|
Mihini_3.0.4
|
Connection policy: On reboot
|
agent.config.server.autoconnect = { onboot = 300}. Reboot and verify connection ~300 seconds after reboot.
|
|
|
Mihini_3.0.5
|
Connection policy: Periodic
|
agent.config.server.autoconnect = { period = <number>}. Reboot and verify a periodic connection every <number> minutes.
|
|
|
Commands (agent.devman.cmds)
This chapter must be done using platform jobs: retrieve data, apply settings, send command.
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result |
---|---|---|---|---|
Mihini_3.1.1
|
Connect
|
Send job to device using platform server. Verify execution.
|
test_SeveralJobs()
|
|
Mihini_3.1.2
|
ReadNode
|
Send read job on Mihini asset to device using platform server. Use empty path (""). Verify execution.
|
|
|
Mihini_3.1.3
|
ReadNode
|
Send job to device using platform server. Use existing path. Verify execution.
|
test_SeveralJobs()
|
|
Mihini_3.1.4
|
ReadNode
|
Send job to device using platform server. Use non-existing path. Verify execution.
|
test_SeveralJobs()
|
|
Mihini_3.1.5
|
Reboot
|
Send command to device using platform server. Verify execution.
|
test_SeveralJobs()
|
|
Mihini_3.1.8
|
UnknownCommand
|
Send command to device. Verify failure as command does not exist.
|
test_SeveralJobs()
|
|
Data Writing
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result |
---|---|---|---|---|
Mihini_3.2.1
|
Data writing on agent.config
|
Executes from platform server a data writing on @sys.config. Verify execution.
|
1- Using datamodel from platform, modify Mihini > Configuration > Shell > HistorySize.
|
|
Mihini_3.2.2
|
Data writing on asset datatree
|
Create a new asset (donkey) Executes from platform server a data writing on donkey.test. Verify execution.
|
testPolicies.test_SeveralJobs()
|
|
Network Manager (agent.netman)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_4.1
|
Check that agent.netman is optional
|
De-activate agent.netman (agent.config.network.activate = false)
|
|
|
Mihini_4.2
|
Connectivity with ETH or GPRS
|
Set agent.config.network.activate = true, agent.config.network.bearerpriority = {"ETH","GPRS"}, agent.config.network.maxconnectiontime = 30
|
|
|
Mihini_4.3
|
|
Unplugged the ethernet cable, and execute a agent.srvcon.connecttoserver() command.
|
|
|
Mihini_4.4
|
|
Set agent.config.network.bearer.retry = 2, agent.config.network.bearer.retryperiod = 10, verify that agent.config.network.bearerpriority is {"ETH", "GPRS"}
|
|
|
Mihini_4.5
|
|
Re-execute previous test with agent.config.network.bearer.ETH.retry = 3 and agent.config.network.bearer.ETH.retryperiod = 50.
|
|
|
Mihini_4.6
|
|
Set agent.config.network.activate = true, agent.config.network.bearerpriority = {"ETH","GPRS"}, agent.config.network.maxconnectiontime = 30
|
|
|
Mihini_4.7
|
Connectivity with selected bearer
|
Execute agent.netman.connect('GPRS').
|
|
|
Mihini_4.8
|
|
Execute agent.netman.connect('ETH').
|
|
|
Software Update
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result |
---|---|---|---|---|
Mihini_5.1
|
Update job success by AWTDA
|
add the SoftwareUpdate Command, configure the device so that initial conditions are met, connect to Server
|
|
|
Mihini_5.2
|
Local update
|
put update package in drop folder, check update is done, package removed from drop folder, no update done on next boot
|
put file in update/drop/updatepackage.tar
|
|
Mihini_5.5
|
Update job errors
|
test each possible error cause:
|
|
|
Mihini_5.6
|
Update job retries
|
network errors, Mihini reboot during update (from platform)
|
testMihiniBigSize.zip
|
|
Mihini_5.7
|
Data Reading
|
Send ReadNode Command on some Update table, like {cmd = ReadNode, path=@sys, args=update.swlist}
|
|
|
Mihini_5.8
|
Data Writing
|
check data writing on update data is forbidden:
|
1- on platform Mihini > Update > SWlist > Components > 3 > Version
|
|
Mihini_5.9
|
Update Script
|
Update job with update package containing software component which Package field = @sys.update.**
|
SW_update_4_atsys_update.tar
|
|
Mihini_5.10
|
Software Update Lua API
|
Update job with update package targeting an asset registered using "airvantage.asset"
|
test_Update()
|
|
Application Container
(Linux Only)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_6.1
|
Application monitor daemon on Linux: setup
|
Run appmon_daemon before starting RA.
|
|
|
Mihini_6.2
|
Application monitor app restart delay
|
test application with run function that reports error "immediately", check restart delay
|
|
|
Mihini_6.3
|
Application Container and Application monitor communication
|
launch appmon, RA, add std apps in RA.AppCon, kill RA, check apps are still running, restart RA.
|
|
|
Time
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_7.1
|
Time unit test suite
|
Activate time but with no periodic synchronization. Require "tests.time", unittest.run().
|
|
|
Mihini_7.2
|
On boot only synchronization
|
Set agent.config.time.ntppolling = 0. Verify that a NTP sync is done on boot.
|
|
|
Mihini_7.3
|
On boot and periodic synchronization
|
Set agent.config.time.ntppolling = <cron entry>. Verify that a NTP sync is done on boot at <cron entry>.
|
|
|
Mihini_7.4
|
On boot and periodic synchronization
|
Set agent.config.time.ntppooling = <number>. Verify that a NTP sync is done on boot and every <number> minutes.
|
|
|
Serial Framework (modbus)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_8.1
|
All API functions in serial (RTU) mode
|
Run test.lua file calling all the API functions in an infinite loop. Verify execution.
|
project serial_modbus_test file main.lua
|
|
Mihini_8.2
|
All API functions in TCP mode
|
Run test.lua file calling all the API functions in an infinite loop. Verify execution.
|
|
|
Messaging
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_10.1
|
Send SMS
|
Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). From platform, send SMS using messaging SendSMS command.
|
|
|
Mihini_10.2
|
SMS fallback
|
Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). Activate SMS fallback (agent.config.network.smsfallback = <phonenumber>. Verify SMS fallback if network bearers (ETH and/or GPRS) are down.
|
|
|
Mihini_10.3
|
SMS notification
|
Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). From platform, send AWTDA notification (ie Connect command through SMS acting like as a WakeUp). Verify execution.
|
|
|
Migration Helper
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_11.1
|
Test migration helper without migration code
|
Executes "persist.save("MihiniVersion", 1)" to trigger a migration attempt on next boot.
|
|
|
Mihini_11.2
|
Test migration helper with migration code written in Lua
|
|
persist.save("MihiniVersion",1)
|
|
Mihini_11.3
|
Test migration helper with migration code written in C
|
|
|
|
M2M Server
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_13.0.1
|
Racon Lua sample on Linux
|
Build and run agent. Run Lua sample in standalone Lua VM. Verify execution.
|
|
|
M2M Server APIs
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_13.1.1
|
Racon unit test suite
|
Require "tests.airvantageTest", unittest.run().
|
require "tests.airvantageTest"
|
|
Mihini_13.1.2
|
Racon data sending
|
Create an asset (av=require "airvantage"; myasset = av.newasset("myasset"), av.init() myasset:start()). Use the newly created asset to push data. Verify data sending.
|
test_PolicyAll()
|
|
Mihini_13.1.3
|
Racon Get-Set-Notify API
|
Create an asset. Use the newly created asset to set data in the Mihini data tree. Verify execution.
|
testJobs.test_SeveralJobs()
|
|
Mihini_13.1.4
|
Racon Get-Set-Notify API
|
Create an asset. Use the newly created asset to get data from the Mihini data tree. Verify execution.
|
testJobs.test_SeveralJobs() | |
Mihini_13.1.5
|
Racon Get-Set-Notify API
|
Create an asset. Use the newly created asset to register a callback on the Mihini data tree modifications. Verify execution.
|
testJobs.test_asset_devicetree()
|
|
Mihini_13.1.8
|
Racon Update API
|
Create an asset. Use the newly created to register a callback on update. Initialize an update of the asset. Verify execution.
|
testJobs.test_SeveralJobs()
|
|
Device Status Access
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_14.4.1
|
Poll & Notify
|
From an application in the Mihini, Poll the device status.
|
|
|
Mihini_14.4.2
|
Radio Information
|
Display all data defined in the AWTDA Dev Tree and check they are relevant.
|
|
|
Mihini_14.4.3
|
Router Information
|
Display all data defined in the AWTDA Dev Tree and check they are relevant.
|
|
|
Mihini_14.4.4
|
Device Versioning
|
Display all data defined in the AWTDA Dev Tree and check they are relevant.
|
|
|
Mihini Configuration Access
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_14.5.1
|
Set, Poll & Notify
|
Change some RA data using the TreeMgr
|
test_devicetree_RSSI()
|
|
Factory Reset
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_14.6.1
|
Factory Reset Integration
|
Install an one or more applications in the user space. Change some data in the treemgr. |
|
|
Mihini Updates
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result |
---|---|---|---|---|
Mihini_14.7.1
|
Mihini updated as part of system update
|
Install one or more applications in the user space. Then perform an update of the Mihini component. Checks the installed applications are still here.
|
|
|
Mihini_14.7.2
|
Packaged apps & libraries
|
Prerequisite:
|
SW_update_1_depends.tar
|
|
Mihini_14.7.3
|
Can update existing app w/o uninstall
|
- Install an application in version 1.0
|
|
|
Mihini_14.7.4
|
Fresh-install option
|
Install an application containing several files with the parameter "purge=true"
|
SW_update_2_old.tar
|
|
Mihini_14.7.5
|
FIO - Base OS version checking
|
Try to install an application requiring a specific version of the OS
|
|
|
Mihini_14.7.6
|
FIO - API Version checking
|
199 : [Waikiki] Populate the Software Update component/feature list to provide ALEOS, Mihini, etc. as features with corresponding versions.
|
|
|
Mihini_14.7.7
|
FIO - Component/Library/App dependency version checking
|
Try to install an application requiring a library not present in the update package nor the installed list apps. The installation shall be rejected.
|
SW_update_4.tar
|
|
Mihini_14.7.8
|
Default exec app per package
|
Check that the 'autostart=true' parameter works
|
SW_update_5.tar
|
|
Mihini_14.7.9
|
Global file resource sandboxing
|
Check that the RA runs with specific user rights
|
|
|
Mihini_14.7.10
|
Check application rights
|
Use app in attachement, try those test cycles:
|
test_uasuser_rights_persist_installation.tar
|
|
Mihini_14.7.11
|
Check many Mihini apps can be installed without rebooting the device
|
choose an app, install it and update it at least 10 times, see if everything is ok
|
|
|
Security: M3DA authentication and encryption
Test ID | Test purpose | Test Process | Automatic test | Result |
---|---|---|---|---|
Mihini_14.9.1 | none/none | Test a simple connection from LUA shell and check the status. No errors should be raised |
|
|
Mihini_14.9.2 | MD5/AES128 | idem |
|
|
Mihini_14.9.3 | SHA1/AES128 | idem |
default config for authentication and encryption
|
|
Mihini_14.9.4 | SHA1/AESCTR256 | idem |
|
|
Mihini_14.9.8 | change password | change password on Server side and Mihini. Check that connection is still OK | |
15. 0.8 features
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
AppCon - Provide a watchdog library for user apps
|
TBD: time before killing app
|
|
|
RA C API Implementation - Airvantage Asset Update API
|
Use an update package with an extra parameter in the manifest file and check that extra param is correctly sent to app updater
|
|
|
Airvantage Lua API - make asset update result to be sent asynchronously
|
Create an asset in which we attached a callback function. The latter have to return "async" and then that is a part of this asset which calls sendUpdateResult() to send a asynchronous response to RA
|
|
|
Update - Package Download Resume
|
A C application receiving notification on DWL shall interrupt and resume DL. The resume DL feature shall restart the DL from the last received byte and not from the beginning.
|
|
|
Add timeout management in EMP libs
|
make an asset command blocking for more than 60 seconds
|
|
|
Update - Download: support user notification and request during download
|
|
|
|
Update - check space left before archive extraction
|
Create a tar.gz file which unzipped file is greater than free space disk and give it to updater. The updater shall return an error: file too long
|
|
|
Handle the case Mihini reboot while apps were running
|
Use the Airvantage sample in both lua and C to:
|
|
|
Update - Add elastic retries in update package download mecanism
|
TBC: Disconnect the device from the network and start a remote application installation. The download will fail and the first retry shall occur after 1minute, the second shall occur only after 30 minutes and the last one after one hour.
|
|
|
Message dispatch: no ack is sent when msg from srv targets unregistred asset - AWTDA3
|
Send a command to an unknown asset. Check that the message has been acknowledged.
|
|
|
Airvantage C API: EMP SendData cmd handler is still using hessian to deserialize data
|
Test that a C program is able to receive data
|
|
16. Security
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
|
use defaultconfig.lua to configure auth and cipher parameters | |
|
MD5 / AES-CBC-128
|
Check that with this configuration of authentication method and encryption :
|
|
|
MD5 / AES-CTR-128 |
Check that with this configuration of authentication method and encryption :
|
|
|
MD5 / AES-CBC-256 |
Check that with this configuration of authentication method and encryption :
|
|
|
MD5 / AES-CTR-256 |
Check that with this configuration of authentication method and encryption :
|
|
|
MD5 / NONE |
Check that with this configuration of authentication method and encryption :
|
|
|
SHA1 / AES-CBC-128 |
Check that with this configuration of authentication method and encryption :
|
|
|
SHA1 / AES-CTR-128 |
Check that with this configuration of authentication method and encryption :
|
|
|
SHA1 / AES-CBC-256 |
Check that with this configuration of authentication method and encryption :
|
|
|
SHA1 / AES-CTR-256 |
Check that with this configuration of authentication method and encryption :
|
|
|
SHA1 / NONE |
Check that with this configuration of authentication method and encryption :
|
|
|
Reset To Factory |
|
|
17. REST API
WARNING: Fix the question about URL path separator: "." or "/"
17.1. Device Tree
17.2. Application Container
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
List applications |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
|
Status |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
|
Start |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
|
Stop |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
|
Configure |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
17.3. Update
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
Local update |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Status |
|
|
18. Automated tests
All the following tests shall have been successfully run on the release tag.
Test ID
|
Environement
|
Target x86 | Target x86_64 |
---|---|---|---|
bysant | lua framework | |
|
luatobin | lua framework | |
|
posixsignal | lua framework | |
|
rpc | lua framework | |
|
sched | lua framework | |
|
persist | lua framework | |
|
bysant | agent | |
|
luatobin | agent | |
|
posixsignal | agent | |
|
rpc | agent | |
|
sched | agent | |
|
persist | agent | |
|
crypto | agent | |
|
treemgr | agent | |
|
config | agent | |
|
update | agent | |
|
stagedb | agent | |
|
monitoring | agent | |
|
asset_tree.lua | racon | |
|
airvantage.lua | racon | |
|
devicetree.lua | racon | |
|
sms.lua | racon | |
|
system.lua | racon | |
|
emp.lua | racon | |
|
dset_test (natif) | racon | |
|
emp_test (natif) | racon | |
|
av_test (natif) | racon | |
|
sms_test (natif) | racon | |
|
sys_test (natif) | racon | |
|
updatetest (natif) | racon | |
|
dt_test (natif) | racon | |
|