COSMOS QA i9 Activities
- 1 COSMOS QA Activities for i9
- 2 Terminologies/Acronyms
- 3 Scope Definition
- 4 Iteration QA Entrance and Exit Criteria
- 5 i9 Test Cases
- 6 Resources & timeframe
- 7 TBD
- 8 Task Breakdown
- 9 Open Issues/Questions
COSMOS QA Activities for i9
This has been put together to address Bugzilla ER 216529.
The terminologies/acronyms below are commonly used throughout this document. The list below defines each term regarding how it is used in this document.
|Quality Expectations||Is a statement of some behaviour, characteristic or operational facility that a product must exhibit for it to be deemed ‘fit for purpose’. Quality expectations are normally grouped into four main categories: functional/behavioural, operational efficiency, inter operability factors; and admin/management factors (to control TCO).|
|Acceptance Criteria||This is a quantification of how a quality expectation is to be validated. For functional/behavioural quality expectations this is a simple Boolean test – it either works or it doesn’t. Hence, for most scope docs there is no need to specifically define functional acceptance criteria. However, other types of quality expectations – especially performance related areas – do require specific acceptance criteria because the quantification is normally some form of numeric threshold (with optional margin/tolerance) that states minimum levels of acceptable operational efficiency.|
The COSMOS quality expectations and the matching acceptance criteria, that would serve as a preamble to the COSMOS QA team while executing their work, were completed via ER 214576.
Since i9 is the first iteration to utilize the QA Expectations, we need to define the i9 QA activities upfront. This will enable us to translate the QA Expectations into an actionable series of steps that ensure QA coverage for i9. This will also serve as the QA plan for i9. Depending on how we execute the QA cycle this time around, we may append to the COSMOS Development Process.
In scope i9 ERs
|Priority||Enhancement||Description||Estimate (PW)||Subproject||Assigned owner(s) to drive designs||Assigned for implementation|
|P1 - blocks 215521||215123||Complete CMDBf 1.0 Service Metadata implementation (design)||Bill Muldoon||Bill Muldoon, Joel Hawkins|
|P1 - blocks 214672||214903||Provide a mechanism for testing the registration service and client (design)||2||DC||Ali Mehregani||Ali Mehregani|
|P1 - blocks ERs||215267||Provide support for adding a federating CMDB to COSMOS framework (design)||4.6||DC||Ali Mehregani||Ali Mehregani|
|P1 - blocks 205826||205825||Update SML validator implementation based on changes to the SML latest draft (design)||5||RM||Valentina Popescu||Ali Mehregani, Valentina Popescu|
|P2||205826||Update DataCenter model based on latest SML changes (design)||0.2||RM||Valentina Popescu||Valentina Popescu|
|P2||208274||Create a data manager toolkit that will allow adopters to easily register and use a data provider inside COSMOS framework (design)||6||ME||David Whiteman||David Whiteman, Hubert Leung|
|P2||214145||Generic CMDBf Graph Response View (design)||3.4||DV||Sheldon Lee-Loy||Martin Simmonds, John Todd|
|P2||214672||Registration of MDR configuration items with a federating CMDB (design)||4.4||DV||Sheldon Lee-Loy||Sheldon Lee-Loy|
|P2||214794||Generic CMDBf Query Builder (design)||3.4||DV||Sheldon Lee-Loy||Bill Muldoon|
|P2||209980||MDR (and Data Manager) deployment outside of OSGi environment (design)||3||DC||Hubert Leung||Hubert Leung|
|P2||216809||Line up JAX-WS annotations and management annotations in ME (design)||2||DC||Hubert Leung||Hubert Leung, Joel Hawkins|
|P2||215135||Establish a process for running JUnits against a COSMOS build||Project/Process||Balan Subramanian|
|P2||216210||Define COSMOS 1.0 hardware & software operational guidelines, recommendations, and best practices||RE||Balan Subramanian|
|P2||216211||We need an ongoing Build process to facilitate a continuous test loop||RE||Balan Subramanian|
|P2||215534||Need Component value cosmos.doc in Bugzilla||Project/Process||Mark Weitzel|
|216591||build "fire alarm" notification email||RE||Balan Subramanian|
|216332||Complete design for COSMOS Security - phase 2||Jimmy Mohsin|
|216529||Define and detail the i9 QA activities||Shivy||Shivy|
In scope platforms, OS's & configurations
Because i9 is a compressed iteration, Windows will remain the primary OS, with a "smoke test" on Linux. i9 QA will cover the following configurations:
- One Management Domain and Broker running on one machine
- TBD MDRs running on TBD machines
- TBD Data Manager(s) running on TBD machines
In scope and out of scope QA activities at the ER and iteration level
For i9, QA will complete (and ***not*** complete) the following activities:
- QA will not be doing build-related tasks. They will commence the i9 test phase on Februaury 25 with the following pre-requisites
- Since ER 215135 "JUnits Establish a process for running JUnits against a COSMOS build" will ***not*** be completed in i9, Development will run the JUnits against the weekly integration builds.
- The sub-team leads need to ensure that this stop gap activity happens in i9 until 215135 is completed sometime in i10.
- To reiterate, QA will commence the i9 testing phase with the KEY assumption that multiple weekly build have occurred, and ALL JUnits have been run against these weekly builds
- QA will verify that the JUnits or alternate testing mechanisms are in place for each of the i9 ERs listed above
- QA will ***not*** run the JUnits; they will simply verify that the JUnits have been run and this is documented in Bugzilla
- If any ER is missing the JUnits, QA will immediately punt the ER back to Development
- QA will run the end to end test as specified on http://wiki.eclipse.org/COSMOS_i9_e2e_testing
- QA will identify the platforms and configurations via Test Cases they will document below.
- The weekly integration build will NOT be run on ALL platforms. They will run only on Windows. How do we address the lack of ongoing testing on additional platforms, i.e. Linux in i9?
Iteration QA Entrance and Exit Criteria
The following items need to be in place before the 2 week i9 QA phase can commence:
- A stable i9 build that has undergone multiple integration build during the course of the Development phase which ends on February 22, 2008
- http://wiki.eclipse.org/COSMOS_i9_e2e_testing is complete
The following items need to be in place before the 2 week i9 QA phase can be declared complete:
- All ERs should have JUnits in place. If JUnits are not applicable to a given ER, alternative verification means need to be specified. To reiterate, QA are not expected to run the JUnits. In i9, this activity will be completed as part of the weekly iteration builds. In i10, this will be fully automated once ER 215135 is completed.
- The e2e tests for i9 (http://wiki.eclipse.org/COSMOS_i9_e2e_testing) are run against the final i9 build
- QA needs to publish a date by when they need the final i9 build. I believe they can do so once http://wiki.eclipse.org/COSMOS_i9_e2e_testing is complete.
i9 Test Cases
Should these be documented here? If not, where should they live?
Resources & timeframe
i9 QA will be completed by two dedicated CA resources. The i9 QA phase will run from February 25 thru March 7.
This section includes the tasks required to complete this enhancement.
- Jimmy Mohsin has generated this page to address bugzilla 216529
- The COSMOS team needs to identify the relevant section for this page.
- Shivvy, representing the QA team, is supposed to complete this activity by Februrary 22, 2008. This is prior to the commencement of the QA phase for i9.
All reviewer feedback should go in the Talk page for 216529.