Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "4.5 Test Automation Initiative"
(→Meeting Minutes) |
(→Testing Process) |
||
(45 intermediate revisions by 5 users not shown) | |||
Line 14: | Line 14: | ||
* Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.). | * Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.). | ||
* Automatic test execution and report generation with every build (BVT). | * Automatic test execution and report generation with every build (BVT). | ||
+ | * Integrate EMMA with the build and test process for generating code coverage reports for all manual and automated testing. | ||
== Benefits == | == Benefits == | ||
Line 26: | Line 27: | ||
* November 14: | * November 14: | ||
− | ** | + | ** First draft of the first testing process document for review. |
** Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds. | ** Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds. | ||
+ | ** Status update for the PMC. | ||
** Measure cost savings. | ** Measure cost savings. | ||
* November 21: | * November 21: | ||
+ | ** Review of the first testing process document. | ||
+ | ** First draft of the remaining testing process documents for review | ||
** Identify and resolve bugs. | ** Identify and resolve bugs. | ||
* November 28: | * November 28: | ||
− | ** | + | ** Final version of the the first testing process document. |
+ | ** Review of the remaining testing process documents for review | ||
+ | ** Status update including cost savings for the PMC. | ||
** Presentation to TPTP committers. | ** Presentation to TPTP committers. | ||
− | ** | + | ** Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure. |
+ | |||
+ | == Testing Process == | ||
+ | |||
+ | * '''Summary''': | ||
+ | ** A very lightweight testing process for TPTP. | ||
+ | ** Extends the existing TPTP Testing Strategy. The existing technical content will remain unchanged, aside from review changes. | ||
+ | ** Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario. | ||
+ | ** Sections include | ||
+ | *** Getting tests. | ||
+ | *** Creating tests. | ||
+ | *** Running tests. | ||
+ | *** Save test results. | ||
+ | *** Reporting results. | ||
+ | *** BVTs. | ||
+ | ** Two (or more) documents: | ||
+ | # High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s). | ||
+ | # Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders. | ||
+ | # Additional documents detailing the specifics of a testing topics. | ||
+ | |||
+ | * '''Draft Documents''': | ||
+ | ** Part 1: | ||
+ | *** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1_Draft_1.doc Draft 1]: [[Testing Process Part 1 Draft 1 Comments| Comments]] | ||
+ | *** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1_Draft_2.html Draft 2]: [[Testing Process Part 1 Draft 2 Comments| Comments]] | ||
+ | ** Part 2: | ||
+ | *** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_2_Draft_1.html Draft 1]: [[Testing Process Part 2 Draft 1 Comments| Comments]] | ||
+ | |||
+ | |||
+ | * '''Final Documents''': | ||
+ | ** Part 1: | ||
+ | *** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1.html Final] | ||
+ | ** Part 2: | ||
+ | *** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_2.html Final] | ||
+ | |||
+ | == Common Test Infrastructure == | ||
+ | |||
+ | * '''Summary''': | ||
+ | ** Reference platform: | ||
+ | *** Developers are expected to unit test on reference platform before checking-in code to CVS. | ||
+ | *** Target: IBM Java 1.5 (latest SR) and Windows XP/x86. | ||
+ | *** Goal to expand to a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86). | ||
+ | *** IBM JREs for Windows and Linux (see [http://www.ibm.com/developerworks/java/jdk/eclipse/index.html IBM Development Package for Eclipse]) are publicly available from [http://www.ibm.com/developerworks/java/jdk/ developerWorks]. | ||
+ | *** Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure. | ||
+ | ** ASF integrated with the TPTP builds by invoking the automated tests on reference platform via ANT: | ||
+ | *** Invoke tests. | ||
+ | *** Interrogate results. | ||
+ | *** Generate test reports. | ||
+ | ** TPTP test types: | ||
+ | *** AGR for functional UI testing. | ||
+ | *** TPTP JUnit and TPTP JUnit Plug-in test for unit testing. | ||
+ | ** TPTP build infrastructure: | ||
+ | *** Extract test suites from CVS. | ||
+ | *** Provision reference platform (Agent Controller and Eclipse/TPTP). | ||
+ | *** Install and configure Agent Controller and Eclipse/TPTP on the reference platform. | ||
+ | *** Clean-up reference platform. | ||
+ | *** Post test reports (test and code coverage) on TPTP web site. | ||
+ | *** Email notifications to component leads for failing test suites. | ||
+ | ** Limitations/Issues: | ||
+ | *** Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform. | ||
+ | *** Intel needs to inform IBM that their portion of the build is complete (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200351 defect 200351]). | ||
+ | |||
+ | * '''Architecture''': [[Common Test Infrastructure Comments| Comments]] | ||
+ | |||
+ | [[Image:AsfBVT.JPG]] | ||
+ | |||
+ | * '''Demonstration''': | ||
+ | ** [http://www.eclipse.org/tptp/home/documents/process/test/automation/infastructure/viewlet/TestAutomationInitiative_viewlet_swf.html Viewlet]. | ||
+ | |||
+ | == Metrics == | ||
+ | |||
+ | In order to evaluate this initiative, are measuring the labor costs (as accurately as possible) in Person Weeks (PW) to run: | ||
+ | |||
+ | # full test pass | ||
+ | # smoke test pass | ||
+ | |||
+ | |||
+ | * Before Initiative: | ||
+ | |||
+ | <table width="100%" border="1" cellpadding="5"> | ||
+ | <tr><th>Project</th><th>Full Test Pass Cost (PW)</th><th>Smoke Test Pass Cost (PW)</th></tr> | ||
+ | <tr><td>Platform</td><td>5</td><td>2.5</td></tr> | ||
+ | <tr><td>Test</td><td>5</td><td>2.5</td></tr> | ||
+ | <tr><td>Trace</td><td>3</td><td>1.5</td></tr> | ||
+ | <tr><td>Monitoring</td><td>3</td><td>1.5</td></tr> | ||
+ | <tr><td><b>Total</b></td><td><b>16</b></td><td><b>8</b></td></tr> | ||
+ | </table> | ||
+ | |||
+ | |||
+ | * With Test Automation: | ||
+ | |||
+ | <table width="100%" border="1" cellpadding="5"> | ||
+ | <tr><th>Project</th><th>Full Test Pass Cost (PW)</th><th>Smoke Test Pass Cost (PW)</th></tr> | ||
+ | <tr><td>Platform</td><td>4</td><td>2</td></tr> | ||
+ | <tr><td>Test</td><td>4</td><td>1.5</td></tr> | ||
+ | <tr><td>Trace</td><td>3</td><td>1.5</td></tr> | ||
+ | <tr><td>Monitoring</td><td>2.5</td><td>1</td></tr> | ||
+ | <tr><td><b>Total</b></td><td><b>13.5</b></td><td><b>6</b></td></tr> | ||
+ | </table> | ||
+ | |||
+ | == References == | ||
+ | |||
+ | * [http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/tlaunching-tests-from-scripts-and-applications.htm ASF] | ||
+ | * [http://www.eclipse.org/tptp/test/documents/tutorials/eclipseWorld2006/Achieving_Continuous_Integration.html ASF Tutorial] | ||
+ | * [http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html TPTP Testing Strategy] | ||
+ | * [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928]) | ||
+ | * [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in] | ||
== Meeting Minutes == | == Meeting Minutes == | ||
Line 43: | Line 154: | ||
* [[TPTP-Test-Automation-20071022| October 22, 2007]] | * [[TPTP-Test-Automation-20071022| October 22, 2007]] | ||
* [[TPTP-Test-Automation-20071106| November 6, 2007]] | * [[TPTP-Test-Automation-20071106| November 6, 2007]] | ||
+ | * [[TPTP-Test-Automation-20071112| November 12, 2007]] | ||
+ | * [[TPTP-Test-Automation-20071119| November 19, 2007]] | ||
+ | * [[TPTP-Test-Automation-20071126| November 26, 2007]] | ||
+ | * [[TPTP-Test-Automation-20071129| November 29, 2007]] | ||
+ | * [[TPTP-Test-Automation-20071203| December 3, 2007]] |
Latest revision as of 10:28, 21 June 2008
Contents
Members
- Paul Slauenwhite (IBM)
- Alan Haggarty (IBM)
- Joel Cayne (IBM)
- Jonathan West (IBM)
- Joanna Kubasta (IBM)
- Kiryl Kazakevich (Intel)
Goals
- Consolidation of our testing process and infrastructure.
- Consolidation of custom test frameworks (Test Dashboard, AC/Profiler test server, variants).
- Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.).
- Automatic test execution and report generation with every build (BVT).
- Integrate EMMA with the build and test process for generating code coverage reports for all manual and automated testing.
Benefits
- Specialization of testing skills (e.g. builds).
- Provide an infrastructure that makes it easy for committers to contribute automated test cases/suites.
- Manual labor savings by using automated tests, which are automatically executed with every build.
- Decrease time/cost to resolve defects since uncovered earlier in the code-test-build cycle.
- Decrease testing overlap by a) localizing test executing and reporting to one group and b) using code coverage statistics.
Delivery dates
- November 14:
- First draft of the first testing process document for review.
- Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds.
- Status update for the PMC.
- Measure cost savings.
- November 21:
- Review of the first testing process document.
- First draft of the remaining testing process documents for review
- Identify and resolve bugs.
- November 28:
- Final version of the the first testing process document.
- Review of the remaining testing process documents for review
- Status update including cost savings for the PMC.
- Presentation to TPTP committers.
- Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.
Testing Process
- Summary:
- A very lightweight testing process for TPTP.
- Extends the existing TPTP Testing Strategy. The existing technical content will remain unchanged, aside from review changes.
- Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
- Sections include
- Getting tests.
- Creating tests.
- Running tests.
- Save test results.
- Reporting results.
- BVTs.
- Two (or more) documents:
- High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s).
- Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
- Additional documents detailing the specifics of a testing topics.
Common Test Infrastructure
- Summary:
- Reference platform:
- Developers are expected to unit test on reference platform before checking-in code to CVS.
- Target: IBM Java 1.5 (latest SR) and Windows XP/x86.
- Goal to expand to a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
- IBM JREs for Windows and Linux (see IBM Development Package for Eclipse) are publicly available from developerWorks.
- Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.
- ASF integrated with the TPTP builds by invoking the automated tests on reference platform via ANT:
- Invoke tests.
- Interrogate results.
- Generate test reports.
- TPTP test types:
- AGR for functional UI testing.
- TPTP JUnit and TPTP JUnit Plug-in test for unit testing.
- TPTP build infrastructure:
- Extract test suites from CVS.
- Provision reference platform (Agent Controller and Eclipse/TPTP).
- Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
- Clean-up reference platform.
- Post test reports (test and code coverage) on TPTP web site.
- Email notifications to component leads for failing test suites.
- Limitations/Issues:
- Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
- Intel needs to inform IBM that their portion of the build is complete (see defect 200351).
- Reference platform:
- Architecture: Comments
- Demonstration:
Metrics
In order to evaluate this initiative, are measuring the labor costs (as accurately as possible) in Person Weeks (PW) to run:
- full test pass
- smoke test pass
- Before Initiative:
Project | Full Test Pass Cost (PW) | Smoke Test Pass Cost (PW) |
---|---|---|
Platform | 5 | 2.5 |
Test | 5 | 2.5 |
Trace | 3 | 1.5 |
Monitoring | 3 | 1.5 |
Total | 16 | 8 |
- With Test Automation:
Project | Full Test Pass Cost (PW) | Smoke Test Pass Cost (PW) |
---|---|---|
Platform | 4 | 2 |
Test | 4 | 1.5 |
Trace | 3 | 1.5 |
Monitoring | 2.5 | 1 |
Total | 13.5 | 6 |
References
- ASF
- ASF Tutorial
- TPTP Testing Strategy
- Test Dashboard (see enhancement 112928)
- Test Dashboard Plug-in