Skip to main content
Jump to: navigation, search

Difference between revisions of "COSMOS QA i9 Activities"

(New page: == '''COSMOS QA Activities for i9''' == This has been put together to address [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216529 Bugzilla ER 216529]. == '''Terminologies/Acronyms''' =...)
 
('''i9 Test Cases''')
 
(62 intermediate revisions by 5 users not shown)
Line 19: Line 19:
 
|}
 
|}
  
== '''Purpose''' ==
+
== '''Scope Definition''' ==
We intent to set the quality expectations of COSMOS (the software) and matching acceptance criteria that would serve as a preamble to the COSMOS QA team while executing their work.
+
The COSMOS quality expectations and the matching acceptance criteria, that would serve as a preamble to the COSMOS QA team while executing their work, were completed via ER 214576.
  
 
<p>
 
<p>
The QA team will use these criteria as a specification to define and plan all their testing efforts.
+
Since i9 is the first iteration to utilize the QA Expectations, we need to define the i9 QA activities upfront.  This will enable us to translate the QA Expectations into an actionable series of steps that ensure QA coverage for i9.  This will also serve as the QA plan for i9.  Depending on how we execute the QA cycle this time around, we may append to the [http://wiki.eclipse.org/COSMOS_dev_process COSMOS Development Process].  
 
</p>
 
</p>
  
== '''Is COSMOS 1.0 well-formed software?''' ==
+
'''QA Scope for i9 testing:'''
  
COSMOS QA perceives COSMOS 1.0 as being a well formed software if it is built upon valid use cases, on bug free technology, with accompanying API documentation and providing easy installations, on some widely used platforms.
+
# Writing e2e regression test cases for i8 and i9 (with traceability to use cases)
 +
# Updating COSMOS QA i9 Activities on Wiki
 +
# Reviewing and understanding i9 ERs (http://wiki.eclipse.org/Cosmos1.0Features#Iteration_9_Enhancements)
 +
# i9 weekly integration build smoke test on Linux
 +
# i9 Test execution (Feb 25 to Mar 07, 2008):
 +
## e2e testing on Windows xp, 2003 and Linux ES 5.0 (http://wiki.eclipse.org/COSMOS_i9_e2e_testing)
 +
## i9 ERs specific JUnits inspection & Closure
 +
## Execution of regression test cases mentioned in item #1. '''This will be done after Mar 07, 2008 '''
 +
## Attempt to perform basic operational efficiency tests (Basic scenario testing). '''This will be done after Mar 07,2008'''
  
 +
=== '''In scope i9 ERs''' ===
  
{|{{BMTableStyle}}
+
Please refer to http://wiki.eclipse.org/Cosmos1.0Features#Iteration_9_Enhancements for the list of ERs that QA will need to test as part of the i9 testing phase that runs from February 25 thru March 7.
|+{{BMTableCaptionStyle}}|Quality perspective 1: Is COSMOS well formed ?
+
|-{{BMTHStyle}}
+
! Quality Expectation
+
! Acceptance Criteria
+
! QA Role
+
|-
+
| align="left" | Valid use cases
+
| COSMOS team to define use cases [[COSMOS_Use_Cases]]
+
| Validate ERs against the use cases - manual
+
|-
+
| align="left" | Bug free implementation
+
| COSMOS team must provide JUnits covering 100% of code; code and JUnits walk-through to QA must be provided [https://bugs.eclipse.org/bugs/show_bug.cgi?id=215135 215135 (i9)][https://bugs.eclipse.org/bugs/show_bug.cgi?id=208604 208604 (i9)]
+
| Run the JUnits and validate their code / ER coverage – TPTP; Black box functional testing – manual / SOAPUI recorded as TPTP manual tests
+
|-
+
| align="left" | API documentation
+
| COSMOS team must provide API documentation [https://bugs.eclipse.org/bugs/show_bug.cgi?id=215534 215534 (i9)]
+
| Manual verification of API documentation
+
|-
+
| align="left" | Easy to use deployment package
+
| Release Engineering team to provide an easy install [[Build_Packaging_for_COSMOS]]
+
| Validate the ease of use of the package and accompanying install instructions – manual; RE process will not be scrutinized
+
|-
+
| align="left" | Base platforms support
+
| COSMOS team to specify the supported platforms
+
| QA certifies release on specified platforms
+
|-
+
| align="left" | Wiki documentation
+
| Owners take responsibility of the quality of content [https://bugs.eclipse.org/bugs/show_bug.cgi?id=197652 197652 (no target)]
+
| QA will not validate wiki content
+
|}
+
  
== '''Is COSMOS 1.0 a consumable entity?''' ==
+
=== '''In scope platforms, OS's & configurations''' ===
  
COSMOS QA perceives COSMOS 1.0 as a consumable (adoptable) software if we can demonstrate its capability to successfully integrate with participating data managers and MDRs through integration and performance testing reports, code samples and other adopter aids.
+
Because i9 is a compressed iteration, Windows will remain the primary OS, with a "smoke test" on Linux.
<p>
+
Clear documentation, process definition for fixing COSMOS bugs or supporting future COSMOS enhancements and a plan for scaling COSMOS as dependent software upgrade themselves can also contribute.
+
</p>
+
  
{|{{BMTableStyle}}
+
For i9, QA will certify the following platforms:
|+{{BMTableCaptionStyle}}|Quality perspective 2: Is COSMOS consumable entity?
+
# Microsoft Windows XP SP2
|-{{BMTHStyle}}
+
# Micorsoft Windows 2003 Server SP1
! Quality Expectation
+
# RedHat Linux Enterprise Server 5.0 (Smoke test/Limited End2End testing only)
! Acceptance Criteria
+
! QA Role
+
|-
+
| align="left" | Successful integration of COSMOS components
+
| COSMOS team must provide helper applications for integration testing with scenarios [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208274 208274 (i9)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=209990 209990 (future)]
+
| Perform integration testing and execute scenarios – manual.  Recorded as manual TPTP tests
+
|-
+
| align="left" | COSMOS stability during production deployments
+
| COSMOS team must state the minimum system requirements for production.  Also recommend parameters (number of data Managers / MDRs that may be added, volume of data that can be queried, etc.) that should be considered for these tests.
+
| Execute performance / scalability /volume/stress/ availability testing with minimum resources recommended
+
|-
+
| align="left" | COSMOS support across products / data sources
+
| COSMOS team must state the kinds of MDRs that can be integrated and provide samples [https://bugs.eclipse.org/bugs/show_bug.cgi?id=211093 211093 (future)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=214766 214766 (i8)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=209987 209987 (i9)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=201302 201302 (i8)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=201317 201317 (i8)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=212187 212187 (i8)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=212189 212189 (i8)]
+
| Execute integration tests with these samples – manual.  Recorded as TPTP manual tests
+
|-
+
| align="left" | User documentation
+
| COSMOS team will write manuals [https://bugs.eclipse.org/bugs/show_bug.cgi?id=214805 214805 (i9)]
+
| QA validates the information - manual
+
|-
+
| align="left" | Samples / skeleton MDR implementation / any collateral
+
| COSMOS team must provide samples [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208274 208274 (i9)] [https://bugs.eclipse.org/bugs/show_bug.cgi?id=202332 202332 (no target)]  [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200567 200567 (i8)]
+
| QA validates the existence and may require assistance from COSMOS Team while using the skeleton implementations – manual.  Recorded as TPTP manual tests
+
|-
+
| align="left" | Additional platforms
+
| COSMOS team must specify
+
| QA will certify the product on these platforms - manual
+
|-
+
| align="left" | Dependencies on other open source software
+
| COSMOS team to define a process to integrate with the newer versions of these dependent software [https://bugs.eclipse.org/bugs/show_bug.cgi?id=215609 215609 (no target)]
+
| QA will validate the process - manual
+
|-
+
| align="left" | Future enhancements / bug reporting mechanism
+
| COSMOS team to set a process [https://bugs.eclipse.org/bugs/ Bugzilla]
+
| QA validates the process - manual
+
|}
+
  
== '''Operational Efficiency considerations''' ==
+
For i9, QA will cover the following configurations:
 +
# One Management Domain and One Broker running on one machine
 +
# 2 MDRs on Windows and 2 Data Managers on linux machine
 +
# 2 MDRs on Linux and 2 Data Managers on windows machine
  
{|{{BMTableStyle}}
+
For i9, QA will use following softwares:
|+{{BMTableCaptionStyle}}|Quality Perspective 3: COSMOS Operational Efficiency
+
# Tomcat 5.5.25
|-{{BMTHStyle}}
+
# SUN JDK 1.5.0_13
! Quality Characteristic
+
# Firefox version (2.0.0.11)
! Acceptance Criteria
+
# Internet Explorer (6.0.2900.2180.XP SP2)
|-
+
# Eclipse 3.3.1
| align="left" | Availability
+
 
| There are no Availability quality expectations
+
=== '''In scope and out of scope QA activities at the ER and iteration level''' ===
|-
+
For i9, QA will complete (and ***not*** complete) the following activities:
| align="left" | Capacity
+
 
| There are no Capacity quality expectations
+
# QA will not be doing build-related tasks.  They will commence the i9 test phase on Februaury 25 with the following pre-requisites
|-
+
## Since ER 215135 "JUnits Establish a process for running JUnits against a COSMOS build" will ***not*** be completed in i9,  Development will run the JUnits against the weekly integration builds.
| align="left" | Concurrency
+
## The sub-team leads need to ensure that this stop gap activity happens in i9 until 215135 is completed sometime in i10.
| How many queries / clients may run simultaneously?  Should there be any other concurrency considerations? [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]
+
## To reiterate, QA will commence the i9 testing phase with the KEY assumption that multiple weekly build have occurred, and ALL JUnits have been run against these weekly builds
|-
+
# QA will verify that the JUnits or alternate testing mechanisms are in place for each of the i9 ERs listed above
| align="left" | Data Volumes
+
## QA will ***not*** run the JUnits; they will simply verify that the JUnits have been run and this is documented in Bugzilla
| Are there any restrictions on the amount of data that may be returned by a query? How many queries / clients may run at a time? [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]
+
## If any ER is missing the JUnits, QA will immediately punt the ER back to Development
|-
+
# QA will run the end to end test as specified on http://wiki.eclipse.org/COSMOS_i9_e2e_testing
| align="left" | Performance
+
# QA will identify the platforms and configurations via Test Cases they will document below.
| The Data Managers and MDRs should not degrade the performance of Data Adapters by more than 15%. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]
+
# QA does not have any responsibility for testing weekly integration builds, however QA will do minimal of a "smoke test" on Linux Platform.
|-
+
# QA will not run the full JUnit suite.  As stated above, this will be done by Development.
| align="left" | Scalability
+
 
| COSMOS 1.0 will support a single Data Manager and Management DomainHow many MDRs and Data Managers may be added?  Should there be any other scalilbity considerations? [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]
+
'''Scope of Smoke Test for Linux Platform'''
|-
+
# Limited End2End testing : Build setup,Basic query execution tests,Statistical and Log report generation tests
| align="left" | Stability
+
 
| There are no Stability quality expectations
+
ISSUES:
|-
+
# The weekly integration build testing will NOT be run on ALL platforms. They will run only on Windows.  How do we address the lack of ongoing testing on additional platforms, i.e. Linux in i9?  Suggestion from DW: Perhaps this is one place we might want to engage QA, to smoke test an integration build on Linux, since it doesn't appear that any developers are setup on that platform?  We could define as minimal of a smoke test as makes sense given the timeframes?
| align="left" | Stress
+
 
| There are no Stress quality expectations
+
== '''Iteration QA Entrance and Exit Criteria''' ==
|-
+
 
| align="left" | Manageability
+
The following items need to be in place before the 2 week i9 QA phase can commence:
| Monitor Administrator should have best practices, guidelines, and tooling to administer a COSMOS environment. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]
+
# A stable i9 build that has undergone multiple integration build during the course of the Development phase which ends on February 22, 2008
|}
+
# The wiki page http://wiki.eclipse.org/COSMOS_i9_e2e_testing is completed (by Development)
 +
 
 +
The following items need to be in place before the 2 week i9 QA phase can be declared complete:
 +
# All ERs should have JUnits or a manual TPTP test in place.  If JUnits are not applicable to a given ER, alternative verification means need to be specified.  To reiterate, QA are not expected to run the JUnits. In i9, this activity will be completed as part of the weekly iteration builds. In i10, this will be fully automated once ER 215135 is completed.
 +
# The e2e tests for i9 (http://wiki.eclipse.org/COSMOS_i9_e2e_testing) are run against the final i9 build
 +
## QA needs to publish a date by when they need the final i9 buildI believe they can do so once http://wiki.eclipse.org/COSMOS_i9_e2e_testing is complete.
 +
# Validate that the use cases for the iteration are functional
 +
 
 +
== '''i9 Test Cases''' ==
 +
QA is preparing word document with End2End i9 Test cases and will provide the link shortly over here with an attachment to ER-216529.
 +
 
 +
https://bugs.eclipse.org/bugs/attachment.cgi?id=90146
 +
 
 +
== '''Resources & timeframe''' ==
 +
i9 QA will be completed by two dedicated CA resources.  The i9 QA phase will run from February 25 thru March 7.
 +
 
 +
== '''TBD''' ==
  
 
== '''Task Breakdown''' ==
 
== '''Task Breakdown''' ==
Line 150: Line 110:
 
This section includes the tasks required to complete this enhancement.
 
This section includes the tasks required to complete this enhancement.
  
# Jimmy Mohsin has generated this page to address [https://bugs.eclipse.org/bugs/show_bug.cgi?id=214576 bugzilla 214576]
+
# Jimmy Mohsin has generated this page to address [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216529 bugzilla 216529]
# The COSMOS team needs to provide input to this page
+
# The COSMOS team needs to identify the relevant section for this page.
# Shivvy, representing the QA team, is supposed to review and sign-off on these criteria
+
# Kishore Adivishnu, representing the QA team, is supposed to complete this activity by Februrary 22, 2008.  This is prior to the commencement of the QA phase for i9.
  
 
== '''Open Issues/Questions '''==
 
== '''Open Issues/Questions '''==
  
All reviewer feedback should go in the [[Talk:COSMOS_Design_214576|Talk page for 214576]].
+
All reviewer feedback should go in the [http://wiki.eclipse.org/Talk:COSMOS_QA_i9_Activities Talk page for 216529 "Define scope and detail the i9 QA activities"].
 
+
A formal review is required to agree the content and detail of the quality expectations laid out in the three ‘quality perspective’ tables. This is a three step process…
+
1. Review each quality expectation to agree whether it should be included.
+
2. If it is to be included then determine whether the acceptance criteria can be met with an r1.0 timeframe, or be postponed to a subsequent release.
+
3. All acceptance criteria must be linked to an ER that delivers the required quality. This can be illustrated by linking the acceptance criteria table cell content to either other relevant wiki pages or an ER. If the link is to an ER then this should also state the ER’s status (e.g. not started / WIP /completed)
+
  
 
----
 
----
 
[[Category:COSMOS_Bugzilla_Designs]]
 
[[Category:COSMOS_Bugzilla_Designs]]

Latest revision as of 02:23, 20 February 2008

COSMOS QA Activities for i9

This has been put together to address Bugzilla ER 216529.

Terminologies/Acronyms

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term regarding how it is used in this document.

Term Definition
Quality Expectations Is a statement of some behaviour, characteristic or operational facility that a product must exhibit for it to be deemed ‘fit for purpose’. Quality expectations are normally grouped into four main categories: functional/behavioural, operational efficiency, inter operability factors; and admin/management factors (to control TCO).
Acceptance Criteria This is a quantification of how a quality expectation is to be validated. For functional/behavioural quality expectations this is a simple Boolean test – it either works or it doesn’t. Hence, for most scope docs there is no need to specifically define functional acceptance criteria. However, other types of quality expectations – especially performance related areas – do require specific acceptance criteria because the quantification is normally some form of numeric threshold (with optional margin/tolerance) that states minimum levels of acceptable operational efficiency.

Scope Definition

The COSMOS quality expectations and the matching acceptance criteria, that would serve as a preamble to the COSMOS QA team while executing their work, were completed via ER 214576.

Since i9 is the first iteration to utilize the QA Expectations, we need to define the i9 QA activities upfront. This will enable us to translate the QA Expectations into an actionable series of steps that ensure QA coverage for i9. This will also serve as the QA plan for i9. Depending on how we execute the QA cycle this time around, we may append to the COSMOS Development Process.

QA Scope for i9 testing:

  1. Writing e2e regression test cases for i8 and i9 (with traceability to use cases)
  2. Updating COSMOS QA i9 Activities on Wiki
  3. Reviewing and understanding i9 ERs (http://wiki.eclipse.org/Cosmos1.0Features#Iteration_9_Enhancements)
  4. i9 weekly integration build smoke test on Linux
  5. i9 Test execution (Feb 25 to Mar 07, 2008):
    1. e2e testing on Windows xp, 2003 and Linux ES 5.0 (http://wiki.eclipse.org/COSMOS_i9_e2e_testing)
    2. i9 ERs specific JUnits inspection & Closure
    3. Execution of regression test cases mentioned in item #1. This will be done after Mar 07, 2008
    4. Attempt to perform basic operational efficiency tests (Basic scenario testing). This will be done after Mar 07,2008

In scope i9 ERs

Please refer to http://wiki.eclipse.org/Cosmos1.0Features#Iteration_9_Enhancements for the list of ERs that QA will need to test as part of the i9 testing phase that runs from February 25 thru March 7.

In scope platforms, OS's & configurations

Because i9 is a compressed iteration, Windows will remain the primary OS, with a "smoke test" on Linux.

For i9, QA will certify the following platforms:

  1. Microsoft Windows XP SP2
  2. Micorsoft Windows 2003 Server SP1
  3. RedHat Linux Enterprise Server 5.0 (Smoke test/Limited End2End testing only)

For i9, QA will cover the following configurations:

  1. One Management Domain and One Broker running on one machine
  2. 2 MDRs on Windows and 2 Data Managers on linux machine
  3. 2 MDRs on Linux and 2 Data Managers on windows machine

For i9, QA will use following softwares:

  1. Tomcat 5.5.25
  2. SUN JDK 1.5.0_13
  3. Firefox version (2.0.0.11)
  4. Internet Explorer (6.0.2900.2180.XP SP2)
  5. Eclipse 3.3.1

In scope and out of scope QA activities at the ER and iteration level

For i9, QA will complete (and ***not*** complete) the following activities:

  1. QA will not be doing build-related tasks. They will commence the i9 test phase on Februaury 25 with the following pre-requisites
    1. Since ER 215135 "JUnits Establish a process for running JUnits against a COSMOS build" will ***not*** be completed in i9, Development will run the JUnits against the weekly integration builds.
    2. The sub-team leads need to ensure that this stop gap activity happens in i9 until 215135 is completed sometime in i10.
    3. To reiterate, QA will commence the i9 testing phase with the KEY assumption that multiple weekly build have occurred, and ALL JUnits have been run against these weekly builds
  2. QA will verify that the JUnits or alternate testing mechanisms are in place for each of the i9 ERs listed above
    1. QA will ***not*** run the JUnits; they will simply verify that the JUnits have been run and this is documented in Bugzilla
    2. If any ER is missing the JUnits, QA will immediately punt the ER back to Development
  3. QA will run the end to end test as specified on http://wiki.eclipse.org/COSMOS_i9_e2e_testing
  4. QA will identify the platforms and configurations via Test Cases they will document below.
  5. QA does not have any responsibility for testing weekly integration builds, however QA will do minimal of a "smoke test" on Linux Platform.
  6. QA will not run the full JUnit suite. As stated above, this will be done by Development.

Scope of Smoke Test for Linux Platform

  1. Limited End2End testing : Build setup,Basic query execution tests,Statistical and Log report generation tests

ISSUES:

  1. The weekly integration build testing will NOT be run on ALL platforms. They will run only on Windows. How do we address the lack of ongoing testing on additional platforms, i.e. Linux in i9? Suggestion from DW: Perhaps this is one place we might want to engage QA, to smoke test an integration build on Linux, since it doesn't appear that any developers are setup on that platform? We could define as minimal of a smoke test as makes sense given the timeframes?

Iteration QA Entrance and Exit Criteria

The following items need to be in place before the 2 week i9 QA phase can commence:

  1. A stable i9 build that has undergone multiple integration build during the course of the Development phase which ends on February 22, 2008
  2. The wiki page http://wiki.eclipse.org/COSMOS_i9_e2e_testing is completed (by Development)

The following items need to be in place before the 2 week i9 QA phase can be declared complete:

  1. All ERs should have JUnits or a manual TPTP test in place. If JUnits are not applicable to a given ER, alternative verification means need to be specified. To reiterate, QA are not expected to run the JUnits. In i9, this activity will be completed as part of the weekly iteration builds. In i10, this will be fully automated once ER 215135 is completed.
  2. The e2e tests for i9 (http://wiki.eclipse.org/COSMOS_i9_e2e_testing) are run against the final i9 build
    1. QA needs to publish a date by when they need the final i9 build. I believe they can do so once http://wiki.eclipse.org/COSMOS_i9_e2e_testing is complete.
  3. Validate that the use cases for the iteration are functional

i9 Test Cases

QA is preparing word document with End2End i9 Test cases and will provide the link shortly over here with an attachment to ER-216529.

https://bugs.eclipse.org/bugs/attachment.cgi?id=90146

Resources & timeframe

i9 QA will be completed by two dedicated CA resources. The i9 QA phase will run from February 25 thru March 7.

TBD

Task Breakdown

This section includes the tasks required to complete this enhancement.

  1. Jimmy Mohsin has generated this page to address bugzilla 216529
  2. The COSMOS team needs to identify the relevant section for this page.
  3. Kishore Adivishnu, representing the QA team, is supposed to complete this activity by Februrary 22, 2008. This is prior to the commencement of the QA phase for i9.

Open Issues/Questions

All reviewer feedback should go in the Talk page for 216529 "Define scope and detail the i9 QA activities".


Copyright © Eclipse Foundation, Inc. All Rights Reserved.