Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "COSMOS i10 QA Phase Execution"
(→i10 ERs - JUnits/Manual Tests Inspection) |
(→Basic Operational efficiency tests) |
||
Line 886: | Line 886: | ||
==Basic Operational efficiency tests == | ==Basic Operational efficiency tests == | ||
+ | |||
+ | '''Configuration Tests''' | ||
{|{{BMTableStyle}} | {|{{BMTableStyle}} | ||
|-{{BMTHStyle}} | |-{{BMTHStyle}} | ||
− | ! Test Description | + | ! Test Description |
− | ! | + | ! Test Status |
− | + | ! Remarks | |
− | ! Remarks | + | |
|- | |- | ||
+ | |Testing the scenario where COSMOSUI running on windows and all COSMOS services running on Linux | ||
+ | #Install COSMOS service components under linux box | ||
+ | #Install COSMOSUI,COSMOSReportViewer webapps application under windows | ||
+ | #From windows, open http://localhost:8080/COSMOSUI | ||
+ | |||
+ | Expected Result: | ||
+ | #COSMOS UI should display all 6 Data Managers and all it's operations should work normally | ||
+ | |Passed | ||
+ | |None | ||
+ | |||
+ | |- | ||
+ | |Testing the scenario where COSMOSUI running on linux and all COSMOS services running on windows | ||
+ | #Install COSMOS service components under windows | ||
+ | #Install COSMOSUI,COSMOSReportViewer webapps application under linux | ||
+ | #From linux, open http://localhost:8080/COSMOSUI | ||
+ | |||
+ | Expected Result: | ||
+ | #COSMOS UI should display all 6 Data Managers and all it's operations should work normally | ||
+ | |Passed | ||
+ | |None | ||
+ | |} | ||
+ | |||
+ | '''Concurrency Tests''' | ||
+ | |||
+ | {|{{BMTableStyle}} | ||
+ | |-{{BMTHStyle}} | ||
+ | ! Test Description | ||
+ | ! Test Status | ||
+ | ! Remarks | ||
+ | |||
+ | |- | ||
+ | |Concurrency tests, where multiple COSMOSUI's running on one machine | ||
+ | #Install all COSMOS components on windows PC | ||
+ | #open 2 COSMOSUI's under IE and 2 COSMOSUI's under Firebox browsers | ||
+ | #Perform various test operations - Generating Stat/Log reports on all COSMOSUI's simultaneously,Executing CMDBf queries on all COSMOSUI's simultaneously | ||
+ | |||
+ | Expected Result: | ||
+ | #All COSMOS UI's should work normally and should get appropriate results without any errors | ||
+ | |Passed | ||
+ | |None | ||
+ | |||
+ | |- | ||
+ | |Concurrency tests, where multiple COSMOSUI's and multiple COSMOS Command Line Clients running on one machine | ||
+ | #Install all COSMOS components on windows PC | ||
+ | #open 1 COSMOSUI under IE and 1 COSMOSUI under Firebox browser | ||
+ | #open 2 COSMOS Command Line Shells | ||
+ | #Perform various test operations - Execute CMDBf queries on all COSMOS Clients simultaneously | ||
+ | |||
+ | Expected Result: | ||
+ | #All COSMOS clients should work normally and should get appropriate results without any errors | ||
+ | |Passed | ||
+ | |None | ||
+ | |||
+ | |||
+ | |- | ||
+ | |Testing the scenario where COSMOSUI running on linux and all COSMOS services running on windows | ||
+ | #Install COSMOS service components under windows | ||
+ | #Install COSMOSUI,COSMOSReportViewer webapps application under linux | ||
+ | #From linux, open http://localhost:8080/COSMOSUI | ||
+ | |||
+ | Expected Result: | ||
+ | #COSMOS UI should display all 6 Data Managers and all it's operations should work normally | ||
+ | |Passed | ||
+ | |None | ||
+ | |} | ||
+ | |||
+ | '''Volume/Performance Tests''' | ||
+ | |||
+ | {|{{BMTableStyle}} | ||
+ | |-{{BMTHStyle}} | ||
+ | ! Test Description | ||
+ | ! Test Status | ||
+ | ! Remarks | ||
+ | |- | ||
+ | | | ||
+ | Expected Result: | ||
+ | # | ||
+ | |Passed | ||
+ | |None | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | Expected Result: | ||
+ | # | ||
+ | |Failed | ||
+ | |None | ||
+ | |} | ||
+ | |||
+ | '''Scalability/Stability Tests''' | ||
+ | |||
+ | {|{{BMTableStyle}} | ||
+ | |-{{BMTHStyle}} | ||
+ | ! Test Description | ||
+ | ! Test Status | ||
+ | ! Remarks | ||
+ | |- | ||
+ | | | ||
+ | Expected Result: | ||
+ | # | ||
+ | |Passed | ||
+ | |None | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | Expected Result: | ||
+ | # | ||
+ | |Failed | ||
+ | |None | ||
|} | |} |
Revision as of 01:47, 1 May 2008
This page is created by QA to track all i10 test execution activities from 21 Apr'08 to 02 May'08. This page will be filled in by the QA team as i10 testing phase proceeds.
Contents
QA Test Execution Activities for i10
Primary Tasks:
- E2E testing on Windows XP, 2003 and Red Hat Enterprise Linux Server 5.0 (http://wiki.eclipse.org/COSMOS_DEMO_i10 )
- i10 ERs – JUnits/Manual Tests inspection (http://wiki.eclipse.org/Cosmos1.0Features#Iteration_10_Enhancements)
- Data Visualization TPTP Manual Tests Execution (DataVisualization.UI.Component.Widget.testsuite / UI.Reports.testsuite)
Secondary Tasks:
- Execution of i10 supplement e2e tests (negative) from QA.
- Basic operational efficiency tests (limited performance tests)
For complete i10 QA Activities, refer http://wiki.eclipse.org/COSMOS_QA_i10_Activities
QA Status Summary
E2E testing status (WinXP SP2/Win2K3 SP1/RHES 5.0)
Build Installation and Configuration
Test Description | Test Status
Windows XP SP2 |
Test Status
Windows 2003 SP1 |
Test Status
RHES 5.0 |
Remarks |
---|---|---|---|---|
<cosmos-demo>/bin/configDemo.bat <tomcatDir> <cosmosDir> <birt-path> <dojo-path> <axis2war-path> | Passed | Passed | Passed | None |
Starting COSMOS components : Management Console
Test Description | Test Status
Windows XP SP2 |
Test Status
Windows 2003 SP1 |
Test Status
RHES 5.0 |
Remarks |
---|---|---|---|---|
<cosmos-demo>\cosmos-client\cosmosClient.bat http://localhost:8080/axis2/services/Broker | Passed | Passed | Passed | None |
COSMOS> broker registerAllDataManagers | Passed | Passed | Passed | Found warnings - "log4j:WARN No appenders could be found for logger (org.apache.axis2.util.Loader).
log4j:WARN Please initialize the log4j system properly." |
COSMOS> broker getDataManagers | Passed | Passed | Passed | None |
COSMOS> finddatamanagers http://localhost:8080/axis2/services/CosmosServiceFinder | Passed | Passed | Passed | None |
COSMOS> broker deregister domsr01-xp StatDataManager | Passed | Passed | Passed | None |
COSMOS> broker deregister domsr01-xp LoggingDataManager | Passed | Passed | Passed | None |
COSMOS> broker deregister domsr01-xp TestFederatingCMDB | Passed | Passed | Passed | None |
COSMOS> broker deregister domsr01-xp ExampleMdr | Passed | Passed | Passed | None |
COSMOS> broker deregister domsr01-xp SMLMDR | Passed | Passed | Passed | None |
COSMOS> broker deregister domsr01-xp AperiMdr | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp SMLMDR D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\SMLMDR\testInstanceIdQuery.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp SMLMDR D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\SMLMDR\testPropertyQuery.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp ExampleMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\ExampleMDR\all-students.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp ExampleMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\ExampleMDR\all-teachers.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp ExampleMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\ExampleMDR\relationship-all-classes.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp ExampleMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\ExampleMDR\teaches-relationship.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp AperiMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\AperiMdr\StorageSystem_all_items.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp AperiMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\AperiMdr\StorageSystem_localid.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp AperiMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\AperiMdr\StorageSystem_display_name.txt | Passed | Passed | Passed | None |
COSMOS> graphQuery domsr01-xp AperiMdr D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\AperiMdr\StorageSystem_relationship.txt | Passed | Passed | Passed | None |
COSMOS> COSMOS> graphQuery domsr01-xp TestFederatingCMDB D:\cosmosi10\cosmos-demo\cosmos-client\cmdbfQuery\TestFederatingCMDB\all_items.txt | Passed | Passed | Passed | Empty Response, no data found. |
Using the COSMOS Web User Interface
Test Description | Test Status
Windows XP SP2 |
Test Status
Windows 2003 SP1 |
Test Status
RHES 5.0 |
Remarks |
---|---|---|---|---|
http://localhost:8080/COSMOSUI/?page=cosmos | Passed | Passed | Passed | Found All 6 DataManagers under COSMOS Web UI |
StatDataManager
server 33 tomcat1181829221515 CBE |
Passed | Passed | Passed | Statistical Report generated successfully |
StatDataManager
server 33 tomcat1181829221515 TomcatSet |
Passed | Passed | Passed | Statistical Report generated successfully |
LoggingDataManager
access_log.cbe.xml |
Passed | Passed | Passed | Top 10 Log Report generated successfully |
LoggingDataManager
windows_app_log.cbe.xml |
Passed | Passed | Passed | Top 10 Log Report generated successfully |
Sample Repository with Registration
Submit Query |
Passed | Passed | Passed | Query for all-items resulted Empty Reponse, no data found. |
Sample Repository with Registration
Register Configuration Items... |
Passed | Passed | Passed | Registration with Example MDR is successful |
Sample Repository with Registration
Deregister Configuration Items... |
Passed | Passed | Passed | Deregistration with Example MDR is successful |
Sample Repository with Registration
Submit Query |
Passed | Passed | Passed | Query for all-items resulted Empty Reponse, no data found. |
Sample Repository with Registration
View MetaData |
Passed | Passed | Passed | No Metadata available |
Resource Repository
Submit Query |
Passed | Passed | Passed | Queries executed successfully with proper results |
Resource Repository
View MetaData |
Passed | Passed | Passed | No Metadata available |
AperiMDR
Submit Query |
Passed | Passed | Passed | Using Query builder able to execute and validate queries properly |
AperiMDR
View MetaData |
Passed | Passed | Passed | Displayed MetaData properly |
Sample Repository with Query
Get All Students |
Passed | Passed | Passed | Displayed Query Result for Students data correctly |
Sample Repository with Query
Get All Teachers |
Passed | Passed | Passed | Displayed Query Result for Teachers data correctly |
Sample Repository with Query
Get All Classes |
Passed | Passed | Passed | Displayed Query Result for Classes data correctly |
Sample Repository with Query
Staff01 |
Passed | Passed | Passed | Displayed data for 'students taught by teacher-staff01' correctly |
Sample Repository with Query
Staff02 |
Passed | Passed | Passed | Displayed data for 'students taught by teacher-staff02' correctly |
Sample Repository with Query
Submit Query |
Passed | Passed | Passed | Using Query Builder, able to create and execute queries correctly |
Sample Repository with Query
View MetaData |
Passed | Passed | Passed | Displayed MetaData information correctly |
Data Visualization TPTP Manual Tests
/org.eclipse.cosmos.examples.e2e.tests/manual/DataVisualization.UI.Component.Widget.testsuite
Test Description | Test Status
Windows XP SP2 |
Test Status
RHES 5.0 |
Remarks |
---|---|---|---|
navigate | Passed | Passed | 6 nodes should be shown under "Data Managers"
No "Monitoring Data Repository" node, it should be "StatDataManager" node |
navigate2 | Passed | Passed | No "Create Query" found, it should be "Submit Query" |
properties | Passed | Passed | Given EPR is wrong, it should be http://localhost:8080/axis2/services/ExampleMdrDataManager |
visualize log report | Passed | Passed | No "Monitoring Data Repository" available, it should be "StatDataManager |
visualize stat report | Passed | Passed | None |
properties 2 | Passed | Passed | None |
example cmdbf query | Passed | Passed | XML output contents seems to be wrong, actual result shows only one student detail - "firstName="Mike" id="03 lastname="Lee"" |
SML CMDBF Graph Response Viewer | Passed | Passed | None |
Example MDR CMDBF Graph Response Viewer | Passed | Passed | No cancel button available under details view window, it should be OK button. |
Multiple Graph Response Viewers | Passed | Passed | No "Cancel" button available under detail area, it should be Ok button |
Registration Example | Passed | Passed | None |
DeRegistration Example | Passed | Passed | None |
Partial Registration of MDR with CMDBf | Passed | Passed | None |
Partial DeRegistration of MDR with CMDBf | Passed | Passed | None |
/org.eclipse.cosmos.examples.e2e.tests/manual/DataVisualization.UI.Reports.testsuite
Test Description | Test Status
Windows XP SP2 |
Test Status
RHES 5.0 |
Remarks |
---|---|---|---|
LogReport | Passed | Passed | None |
StatReport | Passed | Passed | None |
i10 End2End Supplement Tests (Negative)
COSMOS Command Line Client : Management Console
Test Description | Test Status
Windows XP SP2 |
Test Status
RHES 5.0 |
Remarks |
---|---|---|---|
Configure tomcat port to 9080 (other than default port : 8080) and test the following:
Expected Result:
|
Passed | Passed | None |
Test Broker registration other than defaul port 8080 (say 9080):
Expected Result:
|
Failed | Failed | DataManagers are getting registered on default port 8080 only, ignored actual port 9080 |
Test Broker registration multiple times:
Expected Result:
ed." |
Passed | Passed | Found expected message under tomcat console window |
Without broker registration, test the following:
Expected Result:
|
Passed | Passed | None |
After succesful broker registration, test the following:
Expected Result:
|
Passed | Passed | None |
Without any broker registration, test the following
Expected Result:
|
Passed | Passed | None |
Test deregistration of StatDataManager with valid hostname and valid datamanager name :
Expected Result:
|
Passed | Passed | None |
Test deregistration of StatDataManager with invalid hostname and valid datamanager name :
Expected Result:
|
Failed | Failed | No Failure message, it still shows "DataManager deregistered" |
Test deregistration of StatDataManager with valid hostname and invalid datamanager name :
Expected Result:
|
Failed | Failed | No Failure message, it still shows "DataManager deregistered" |
Test graphQuery for SMLMDR, with valid hostname,valid DataManager,valid query path:
Expected Result:
|
Passed | Passed | This is depended on Broker registration issue, fixed manually to test this. |
Test graphQuery for SMLMDR, with invalid hostname,valid DataManager,valid query path:
Expected Result:
|
Passed | Passed | Displaying error message as "SMLMDR is not registered at the broker." |
Test graphQuery for SMLMDR, with valid hostname,invalid DataManager,valid query path:
Expected Result:
|
Passed | Passed | Displayed error message as "StatDataManager is not registered at the broker." |
Test graphQuery for SMLMDR, with valid hostname,valid DataManager,invalid query path:
Expected Result:
|
Passed | Passed | Displayed "java.io.FileNotFoundException:" error message tree |
Test graphQuery for SMLMDR, with valid hostname,valid DataManager,valid query path with spaces:
Expected Result:
|
Failed | Failed | Displayed "java.io.FileNotFoundException:" error message tree |
Test graphQuery for AperMdr with valid data, with valid Aperi database settings:
Expected Result:
|
Passed | Passed | Query executed successfully |
Test graphQuery for AperMdr with invalid query data, with valid Aperi database settings:
Expected Result:
|
Passed | Passed | Query execution failed with appropriate error message |
Test graphQuery for AperMdr with valid data, with invalid Aperi database settings:
Expected Result:
|
Passed | Passed | Displayed "aperiDB.getStorageSystems error" |
Using the COSMOS Web User Interface
Test Description | Test Status
Windows XP SP2 |
Test Status
RHES 5.0 |
Remarks |
---|---|---|---|
Open COSMOSUI with 8090 port (Make sure all COSMOS Components are up and running at this port)
Expected Result:
|
Passed | Passed | Found All 6 DataManagers under COSMOS Web UI |
Open COSMOSUI, Verify Properties of all DataManagers
Expected Result:
|
Passed | Passed | All EPR's under properties window displayed correctly |
Test StatDataManager, "Statistical Report" generation:
Expected Result:
|
Passed | Passed | Statistical Report generated successfully |
Test "LoggingDataManager", Top 10 Log Report generation:
Expected Result:
|
Passed | Passed | Top 10 Log Report generated successfully |
Test "Sample Repository with Registration",self registration
Expected Result
|
Failed | Failed | Registration successful, but no result data |
Test "Sample Repository with Registration",Resource Repository registration
|
Failed | Failed | Registration successful, but no result data |
Test "Sample Repository with Registration",AperiMdr registration
Expected Result
|
Failed | Failed | Error "org.apache.axis2.AxisFault: AperiMDR error: relationshipTemplate 'all-relationships' is missing a recordConstraint" |
Test Resource Repository, with valid query
Expected Result:
|
Passed | Passed | Queries executed successfully with proper results |
Test Resource Repository, with invalid query
Expected Result:
|
Passed | Passed | Query execution failed with appropriate error message |
Test AperiMDR with proper db configuration and with valid query
Expected Result:
|
Passed | Passed | Using Query builder able to execute and validate queries properly |
Test AperiMDR with proper db configuration and with invalid query
Expected Result:
|
Passed | Passed | Query execution failed and displayed error message |
Test AperiMDR with invalid db configuration and with valid query
Expected Result:
|
Passed | Passed | Query failed with db connection error |
Test "Sample Repository with Query", with valid query
Expected Result:
|
Passed | Passed | Query executed successfully |
Test "Sample Repository with Query", with invalid query
Expected Result:
|
Passed | Passed | Empty Query response displayed under GraphResponseViewer |
i10 ERs - JUnits/Manual Tests Inspection
Bugzilla ID | ID Summary | JUnits Tests | Manual Tests | Remarks |
---|---|---|---|---|
214153 | Data Visualization Customization - Customized CMDBf Query Builder | |||
214143 | Predefined CMDBf Queries | |||
220334 | Additional Record Types for the Graph Reponse Viewer | |||
221554 | Provide a simple view to show the service meta data from an MDR. | |||
217303 | Support partial registering and deregister configuration Items | |||
221901 | Service side responses should be cached to reduce network calls | |||
205825 | Update SML validator implementation based on changes to the SML latest draft | |||
222982 | Widget factory should generate unique ids when instantiating widgets | |||
220949 | MDRs need to accept vanilla web service requests | |||
220952 | Broker to manage MDRs that do not support WSA and WSRF | |||
220955 | deregister data managers from broker | |||
221324 | Additional Record Types for the Graph Reponse Viewer - Widget | |||
221322 | Additional Record Types for the Graph Reponse Viewer |
Basic Operational efficiency tests
Configuration Tests
Test Description | Test Status | Remarks |
---|---|---|
Testing the scenario where COSMOSUI running on windows and all COSMOS services running on Linux
Expected Result:
|
Passed | None |
Testing the scenario where COSMOSUI running on linux and all COSMOS services running on windows
Expected Result:
|
Passed | None |
Concurrency Tests
Test Description | Test Status | Remarks |
---|---|---|
Concurrency tests, where multiple COSMOSUI's running on one machine
Expected Result:
|
Passed | None |
Concurrency tests, where multiple COSMOSUI's and multiple COSMOS Command Line Clients running on one machine
Expected Result:
|
Passed | None
|
Testing the scenario where COSMOSUI running on linux and all COSMOS services running on windows
Expected Result:
|
Passed | None |
Volume/Performance Tests
Test Description | Test Status | Remarks |
---|---|---|
Expected Result: |
Passed | None |
Expected Result: |
Failed | None |
Scalability/Stability Tests
Test Description | Test Status | Remarks |
---|---|---|
Expected Result: |
Passed | None |
Expected Result: |
Failed | None |