Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Mihini/QA Plan/0.9
|
Contents
- 1 Results and Status
- 2 Miscellaneous [done]
- 3 Config Store (agent.config ) [done]
- 4 Server Connection (agent.srvcon)
- 5 Software Update
- 6 Application Container [done]
- 7 Time
- 8 Serial Framework (modbus)
- 9 Messaging (not supported in 0.9) [done]
- 10 Migration Helper
- 11 M2M Server
- 12 0.8 features
- 13 Security [done]
- 14 REST API
- 15 GPIO
- 16 Automated tests [done]
Results and Status
Device(s) used for tests | RaspberryPi for GPIO tests, regular PC, |
Status ( ok tests / total tests) | |
Comments | |
Final status with date | |
Miscellaneous [done]
Config Store (agent.config ) [done]
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_2.1
|
Data Reading
|
Test reading whole config, command arg = agent.config.get(""). Verify execution.
|
|
|
Mihini_2.2
|
Data Reading
|
Test reading sub part of config, command arg = agent.config.server. Verify execution.
|
|
|
Mihini_2.3
|
Data Reading
|
1- Modify some config entry, read it using ReadNode command,
|
|
|
Mihini_2.5
|
Test default()
|
Call default (agent.config.default()). Verify that default values are restored.
|
|
|
Mihini_2.6
|
Test diff()
|
Change some values in agent.config. Verify that the differences are correctly reported in the Lua shell.
|
|
|
Server Connection (agent.srvcon)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_3.0.2
|
Connection policy: On Demand
|
agent.config.server.autoconnect.ondemand = <number>, agent.srvcon.connecttoserver(). Send data via agent.devman.hl. ange
|
|
[in progress] SCH |
Mihini_3.0.3
|
Connection policy: Cron
|
agent.config.server.autoconnect = { cron="mm hh dd MM YY"}. Reboot and verify connection according to cron entry.
|
|
|
Mihini_3.0.4
|
Connection policy: On reboot
|
agent.config.server.autoconnect = { onboot = 300}. Reboot and verify connection ~300 seconds after reboot.
|
|
|
Mihini_3.0.5
|
Connection policy: Periodic
|
agent.config.server.autoconnect = { period = 2 }. Reboot and verify a periodic connection every 2 minutes.
|
|
Commands (agent.devman.cmds)
This chapter must be done using platform jobs: retrieve data, apply settings, send command.
Data Writing
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result |
---|---|---|---|---|
Mihini_3.2.1
|
Data writing on agent.config
|
Executes from platform server a data writing on @sys.config. Verify execution.
|
1- Using datamodel from platform, modify Mihini > Configuration > Shell > HistorySize.
|
|
Mihini_3.2.2
|
Data writing on asset datatree
|
Create a new asset (donkey) Executes from platform server a data writing on donkey.test. Verify execution.
|
testPolicies.test_SeveralJobs()
|
|
Network Manager (agent.netman)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_4.1
|
Check that agent.netman is optional
|
De-activate agent.netman (agent.config.network.activate = false)
|
|
|
Mihini_4.2
|
Connectivity with ETH or GPRS
|
Set agent.config.network.activate = true, agent.config.network.bearerpriority = {"ETH","GPRS"}, agent.config.network.maxconnectiontime = 30
|
|
|
Mihini_4.3
|
|
Unplugged the ethernet cable, and execute a agent.srvcon.connecttoserver() command.
|
|
|
Mihini_4.4
|
|
Set agent.config.network.bearer.retry = 2, agent.config.network.bearer.retryperiod = 10, verify that agent.config.network.bearerpriority is {"ETH", "GPRS"}
|
|
|
Mihini_4.5
|
|
Re-execute previous test with agent.config.network.bearer.ETH.retry = 3 and agent.config.network.bearer.ETH.retryperiod = 50.
|
|
|
Mihini_4.6
|
|
Set agent.config.network.activate = true, agent.config.network.bearerpriority = {"ETH","GPRS"}, agent.config.network.maxconnectiontime = 30
|
|
|
Mihini_4.7
|
Connectivity with selected bearer
|
Execute agent.netman.connect('GPRS').
|
|
|
Mihini_4.8
|
|
Execute agent.netman.connect('ETH').
|
|
|
Software Update
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result |
---|---|---|---|---|
Mihini_5.1
|
Update job success by AWTDA
|
add the SoftwareUpdate Command, configure the device so that initial conditions are met, connect to Server
|
|
|
Mihini_5.2
|
Local update
|
put update package in drop folder, check update is done, package removed from drop folder, no update done on next boot
|
put file in update/drop/updatepackage.tar
|
|
Mihini_5.5
|
Update job errors
|
test each possible error cause:
|
|
|
Mihini_5.6
|
Update job retries
|
network errors, Mihini reboot during update (from platform)
|
testMihiniBigSize.zip
|
|
Mihini_5.7
|
Data Reading
|
Send ReadNode Command on some Update table, like {cmd = ReadNode, path=@sys, args=update.swlist}
|
|
|
Mihini_5.8
|
Data Writing
|
check data writing on update data is forbidden:
|
1- on platform Mihini > Update > SWlist > Components > 3 > Version
|
|
Mihini_5.9
|
Update Script
|
Update job with update package containing software component which Package field = @sys.update.**
|
SW_update_4_atsys_update.tar
|
|
Mihini_5.10
|
Software Update Lua API
|
Update job with update package targeting an asset registered using "airvantage.asset"
|
test_Update()
|
|
Application Container [done]
(Linux Only)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_6.1
|
Application monitor daemon on Linux: setup
|
Run appmon_daemon before starting RA.
|
|
|
Mihini_6.2
|
Application monitor app restart delay
|
test application with run function that reports error "immediately", check restart delay
|
|
|
Mihini_6.3
|
Application Container and Application monitor communication
|
launch appmon, RA, add std apps in RA.AppCon, kill RA, check apps are still running, restart RA.
|
|
|
Time
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_7.1
|
Time unit test suite
|
Activate time but with no periodic synchronization. Require "tests.time", unittest.run().
|
|
|
Mihini_7.2
|
On boot only synchronization
|
Set agent.config.time.ntppolling = 0. Verify that a NTP sync is done on boot.
|
|
|
Mihini_7.3
|
On boot and periodic synchronization
|
Set agent.config.time.ntppolling = <cron entry>. Verify that a NTP sync is done on boot at <cron entry>.
|
|
|
Mihini_7.4
|
On boot and periodic synchronization
|
Set agent.config.time.ntppooling = <number>. Verify that a NTP sync is done on boot and every <number> minutes.
|
|
|
Serial Framework (modbus)
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_8.1
|
All API functions in serial (RTU) mode
|
Run test.lua file calling all the API functions in an infinite loop. Verify execution.
|
project serial_modbus_test file main.lua
|
|
Mihini_8.2
|
All API functions in TCP mode
|
Run test.lua file calling all the API functions in an infinite loop. Verify execution.
|
|
|
Messaging (not supported in 0.9) [done]
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_10.1
|
Send SMS
|
Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). From platform, send SMS using messaging SendSMS command.
|
|
NA
|
Mihini_10.2
|
SMS fallback
|
Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). Activate SMS fallback (agent.config.network.smsfallback = <phonenumber>. Verify SMS fallback if network bearers (ETH and/or GPRS) are down.
|
|
NA |
Mihini_10.3
|
SMS notification
|
Activate SMS management (agent.config.modem = true, agent.config.modem.sms = true). From platform, send AWTDA notification (ie Connect command through SMS acting like as a WakeUp). Verify execution.
|
|
NA |
Migration Helper
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_11.1
|
Test migration helper without migration code
|
Executes "persist.save("MihiniVersion", 1)" to trigger a migration attempt on next boot.
|
|
(avertissement) LBA |
Mihini_11.2
|
Test migration helper with migration code written in Lua
|
|
persist.save("MihiniVersion",1)
|
(avertissement) LBA |
Mihini_11.3
|
Test migration helper with migration code written in C
|
|
|
(avertissement) LBA |
M2M Server
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_13.0.1
|
Racon Lua sample on Linux
|
Build and run agent. Run Lua sample in standalone Lua VM. Verify execution.
|
|
|
M2M Server APIs
Test ID
|
Test Purpose
|
Test process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_13.1.1
|
Racon unit test suite
|
Require "tests.airvantageTest", unittest.run().
|
require "tests.airvantageTest"
|
|
Mihini_13.1.2
|
Racon data sending
|
Create an asset (av=require "airvantage"; myasset = av.newasset("myasset"), av.init() myasset:start()). Use the newly created asset to push data. Verify data sending.
|
test_PolicyAll()
|
|
Mihini_13.1.3
|
Racon Get-Set-Notify API
|
Create an asset. Use the newly created asset to set data in the Mihini data tree. Verify execution.
|
testJobs.test_SeveralJobs()
|
|
Mihini_13.1.4
|
Racon Get-Set-Notify API
|
Create an asset. Use the newly created asset to get data from the Mihini data tree. Verify execution.
|
testJobs.test_SeveralJobs() | |
Mihini_13.1.5
|
Racon Get-Set-Notify API
|
Create an asset. Use the newly created asset to register a callback on the Mihini data tree modifications. Verify execution.
|
testJobs.test_asset_devicetree()
|
|
Mihini_13.1.8
|
Racon Update API
|
Create an asset. Use the newly created to register a callback on update. Initialize an update of the asset. Verify execution.
|
testJobs.test_SeveralJobs()
|
|
Device Status Access
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_14.4.1
|
Poll & Notify
|
From an application in the Mihini, Poll the device status.
|
|
|
Mihini_14.4.2
|
Radio Information
|
Display all data defined in the AWTDA Dev Tree and check they are relevant.
|
|
|
Mihini_14.4.3
|
Router Information
|
Display all data defined in the AWTDA Dev Tree and check they are relevant.
|
|
|
Mihini_14.4.4
|
Device Versioning
|
Display all data defined in the AWTDA Dev Tree and check they are relevant.
|
|
|
Mihini Configuration Access
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_14.5.1
|
Set, Poll & Notify
|
Change some RA data using the TreeMgr
|
test_devicetree_RSSI()
|
|
Factory Reset
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result
|
---|---|---|---|---|
Mihini_14.6.1
|
Factory Reset Integration
|
Install an one or more applications in the user space. Change some data in the treemgr. |
|
|
Mihini Updates
Test ID
|
Test Purpose
|
Test Process
|
Automatic test
|
Result |
---|---|---|---|---|
Mihini_14.7.1
|
Mihini updated as part of system update
|
Install one or more applications in the user space. Then perform an update of the Mihini component. Checks the installed applications are still here.
|
|
|
Mihini_14.7.2
|
Packaged apps & libraries
|
Prerequisite:
|
SW_update_1_depends.tar
|
|
Mihini_14.7.3
|
Can update existing app w/o uninstall
|
- Install an application in version 1.0
|
|
|
Mihini_14.7.4
|
Fresh-install option
|
Install an application containing several files with the parameter "purge=true"
|
SW_update_2_old.tar
|
|
Mihini_14.7.5
|
FIO - Base OS version checking
|
Try to install an application requiring a specific version of the OS
|
|
|
Mihini_14.7.6
|
FIO - API Version checking
|
199 : [Waikiki] Populate the Software Update component/feature list to provide ALEOS, Mihini, etc. as features with corresponding versions.
|
|
|
Mihini_14.7.7
|
FIO - Component/Library/App dependency version checking
|
Try to install an application requiring a library not present in the update package nor the installed list apps. The installation shall be rejected.
|
SW_update_4.tar
|
|
Mihini_14.7.8
|
Default exec app per package
|
Check that the 'autostart=true' parameter works
|
SW_update_5.tar
|
|
Mihini_14.7.9
|
Global file resource sandboxing
|
Check that the RA runs with specific user rights
|
|
|
Mihini_14.7.10
|
Check application rights
|
Use app in attachement, try those test cycles:
|
test_uasuser_rights_persist_installation.tar
|
|
Mihini_14.7.11
|
Check many Mihini apps can be installed without rebooting the device
|
choose an app, install it and update it at least 10 times, see if everything is ok
|
|
|
Security: M3DA authentication and encryption
Test ID | Test purpose | Test Process | Automatic test | Result |
---|---|---|---|---|
Mihini_14.9.1 | none/none | Test a simple connection from LUA shell and check the status. No errors should be raised |
|
|
Mihini_14.9.2 | MD5/AES128 | idem |
|
|
Mihini_14.9.3 | SHA1/AES128 | idem |
default config for authentication and encryption
|
N/A (only HMAC-MD5 and AES-CBC-128 are suppotred) |
Mihini_14.9.4 | SHA1/AESCTR256 | idem |
|
N/A (only HMAC-MD5 and AES-CBC-128 are suppotred) |
Mihini_14.9.8 | change password | change password on Server side and Mihini. Check that connection is still OK | |
0.8 features
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
AppCon - Provide a watchdog library for user apps
|
TBD: time before killing app
|
|
|
RA C API Implementation - Airvantage Asset Update API
|
Use an update package with an extra parameter in the manifest file and check that extra param is correctly sent to app updater
|
|
|
Airvantage Lua API - make asset update result to be sent asynchronously
|
Create an asset in which we attached a callback function. The latter have to return "async" and then that is a part of this asset which calls sendUpdateResult() to send a asynchronous response to RA
|
|
|
Update - Package Download Resume
|
A C application receiving notification on DWL shall interrupt and resume DL. The resume DL feature shall restart the DL from the last received byte and not from the beginning.
|
|
|
Add timeout management in EMP libs
|
make an asset command blocking for more than 60 seconds
|
|
|
Update - Download: support user notification and request during download
|
|
|
|
Update - check space left before archive extraction
|
Create a tar.gz file which unzipped file is greater than free space disk and give it to updater. The updater shall return an error: file too long
|
|
|
Handle the case Mihini reboot while apps were running
|
Use the Airvantage sample in both lua and C to:
|
|
|
Update - Add elastic retries in update package download mecanism
|
TBC: Disconnect the device from the network and start a remote application installation. The download will fail and the first retry shall occur after 1minute, the second shall occur only after 30 minutes and the last one after one hour.
|
|
|
Message dispatch: no ack is sent when msg from srv targets unregistred asset - AWTDA3
|
Send a command to an unknown asset. Check that the message has been acknowledged.
|
|
|
Airvantage C API: EMP SendData cmd handler is still using hessian to deserialize data
|
Test that a C program is able to receive data
|
|
Security [done]
REST API
Device Tree [done]
Application Container
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
List applications |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
|
Status |
Prerequisite:
Tests steps:
|
The output is correct but I am not completly sure about the variable "appname" (with or without Rest), it contains "2" here... There is a bunch of code in agent/agent/appcon.lua which converts a string coming from the appmon daemon to a table. Please check if this is a bug or not |
|
|
Prerequisite:
Tests steps:
|
|
|
Start |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
|
Stop |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
|
Configure |
Prerequisite:
Tests steps:
|
|
|
|
Prerequisite:
Tests steps:
|
|
Update
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
Local update |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Status |
|
|
GPIO
For all the GPIO tests, here are the prerequisites:
- Use a device with system conforming to GPIO lib requirements (see download.eclipse.org/mihini/api/lua/gpio.html )
RaspberryPI with Raspbian system is a good candidate.
- Build GPIO targets:
GPIO targets are not built by default, do it by typing:
make gpio agent_treemgr_gpio
- Start mihini agent / application with root user rights to be able to access to the GPIO.
Here is a hardware setup to test the GPIO:
TBD
GPIO Lib
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
GPIO read |
|
|
|
GPIO write |
|
|
|
GPIO register / notify |
|
|
|
GPIO automatic configuration |
|
|
|
GPIO listing APIs: |
|
|
GPIO devicetree handler
Test ID
|
Test Purpose
|
Test Process
|
Result
|
---|---|---|---|
|
| ||
|
| ||
|
| ||
|
| ||
|
| ||
|
|
Automated tests [done]
All the following tests shall have been successfully run on the release tag.
Test ID
|
Environement
|
Target x86 | Target x86_64 |
---|---|---|---|
bysant | lua framework | |
|
luatobin | lua framework | |
|
posixsignal | lua framework | |
|
rpc | lua framework | |
|
sched | lua framework | |
|
persist | lua framework | |
|
timer | lua framework | |
|
bysant | agent | |
|
luatobin | agent | |
|
posixsignal | agent | |
|
rpc | agent | |
|
sched | agent | |
|
persist | agent | |
|
crypto | agent | |
|
treemgr | agent | |
|
config | agent | |
|
update | agent | |
|
stagedb | agent | |
|
monitoring | agent | |
|
asset_tree.lua | racon | |
|
airvantage.lua | racon | |
|
devicetree.lua | racon | |
|
sms.lua | racon | |
|
system.lua | racon | |
|
emp.lua | racon | |
|
dset_test (natif) | racon | |
|
emp_test (natif) | racon | |
|
av_test (natif) | racon | |
|
sms_test (natif) | racon | |
|
sys_test (natif) | racon | |
|
updatetest (natif) | racon | |
|
dt_test (natif) | racon | |
|