Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "4.5 Test Automation Initiative"
(→Testing Process) |
(→Testing Process) |
||
Line 46: | Line 46: | ||
== Testing Process == | == Testing Process == | ||
− | * A very lightweight testing process for TPTP. | + | * '''Summary''': |
− | * Extends the existing TPTP Testing Strategy. | + | ** A very lightweight testing process for TPTP. |
− | * Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario. | + | ** Extends the existing TPTP Testing Strategy. The existing technical content will remain unchanged, aside from review changes. |
− | * | + | ** Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario. |
− | * | + | ** Sections include |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
*** Getting tests. | *** Getting tests. | ||
*** Creating tests. | *** Creating tests. | ||
Line 66: | Line 57: | ||
*** Reporting results. | *** Reporting results. | ||
*** BVTs. | *** BVTs. | ||
− | ** | + | ** Two (or more) documents: |
− | + | # High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s). | |
− | + | # Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders. | |
+ | # Additional documents detailing the specifics of a testing topics. | ||
− | * '''Draft Documents''' | + | * '''Draft Documents''': |
** Part 1: | ** Part 1: | ||
− | ** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1_Draft_1.doc Draft 1]: [[Testing Process Part 1 Draft 1 Comments| Comments]] | + | *** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1_Draft_1.doc Draft 1]: [[Testing Process Part 1 Draft 1 Comments| Comments]] |
== Common Test Infrastructure == | == Common Test Infrastructure == |
Revision as of 13:08, 19 November 2007
Contents
Members
- Paul Slauenwhite (IBM)
- Alan Haggarty (IBM)
- Joel Cayne (IBM)
- Jonathan West (IBM)
- Joanna Kubasta (IBM)
- Kiryl Kazakevich (Intel)
Goals
- Consolidation of our testing process and infrastructure.
- Consolidation of custom test frameworks (Test Dashboard, AC/Profiler test server, variants).
- Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.).
- Automatic test execution and report generation with every build (BVT).
- Integrate EMMA with the build and test process for generating code coverage reports for all manual and automated testing.
Benefits
- Specialization of testing skills (e.g. builds).
- Provide an infrastructure that makes it easy for committers to contribute automated test cases/suites.
- Manual labor savings by using automated tests, which are automatically executed with every build.
- Decrease time/cost to resolve defects since uncovered earlier in the code-test-build cycle.
- Decrease testing overlap by a) localizing test executing and reporting to one group and b) using code coverage statistics.
Delivery dates
- November 14:
- First draft of the first testing process document for review.
- Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds.
- Status update for the PMC.
- Measure cost savings.
- November 21:
- Review of the first testing process document.
- First draft of the remaining testing process documents for review
- Identify and resolve bugs.
- November 28:
- Final version of the the first testing process document.
- Review of the remaining testing process documents for review
- Status update including cost savings for the PMC.
- Presentation to TPTP committers.
- Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.
Testing Process
- Summary:
- A very lightweight testing process for TPTP.
- Extends the existing TPTP Testing Strategy. The existing technical content will remain unchanged, aside from review changes.
- Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
- Sections include
- Getting tests.
- Creating tests.
- Running tests.
- Save test results.
- Reporting results.
- BVTs.
- Two (or more) documents:
- High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s).
- Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
- Additional documents detailing the specifics of a testing topics.
Common Test Infrastructure
- Demonstration viewlet.
- Diagram detailing the architecture: