4.5 Test Automation Initiative

Members

• Paul Slauenwhite (IBM)
• Alan Haggarty (IBM)
• Joel Cayne (IBM)
• Jonathan West (IBM)
• Joanna Kubasta (IBM)
• Kiryl Kazakevich (Intel)

Goals

• Consolidation of our testing process and infrastructure.
• Consolidation of custom test frameworks (Test Dashboard, AC/Profiler test server, variants).
• Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.).
• Automatic test execution and report generation with every build (BVT).
• Integrate EMMA with the build and test process for generating code coverage reports for all manual and automated testing.

Benefits

• Specialization of testing skills (e.g. builds).
• Provide an infrastructure that makes it easy for committers to contribute automated test cases/suites.
• Manual labor savings by using automated tests, which are automatically executed with every build.
• Decrease time/cost to resolve defects since uncovered earlier in the code-test-build cycle.
• Decrease testing overlap by a) localizing test executing and reporting to one group and b) using code coverage statistics.

Delivery dates

• November 14:
  • First draft of the first testing process document for review.
  • Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds.
  • Status update for the PMC.
  • Measure cost savings.

• November 21:
  • Review of the first testing process document.
  • First draft of the remaining testing process documents for review
  • Identify and resolve bugs.

• November 28:
  • Final version of the the first testing process document.
  • Review of the remaining testing process documents for review
  • Status update including cost savings for the PMC.
  • Presentation to TPTP committers.
  • Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.

Testing Process

• A very lightweight testing process for TPTP.
• Extends the existing TPTP Testing Strategy.
• Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
• Two (or more) documents:
  • 1) High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s).
  • 2) Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
  • 3) Additional documents detailing the specifics of a testing topics.

• Discussion:
  • Naming conventions for root level test suites and execution results should include test plug-in ID.
  • Execution results need to be saved in a directory with a meaningful name (e.g. OS, JRE, type (full, smoke, BVT)).
  • Test suites for specific JREs and OSes (e.g. JVMPI versus JVMTI) will not always run on the reference platform(s).
  • The existing technical content will remain unchanged, aside from review changes.
  • Need sections for:
    • Getting tests.
    • Creating tests.
    • Running tests.
    • Save test results.
    • Reporting results.
    • BVTs.
  • The Agent Controller and profiler test suites need to migrated to use the TPTP Test Tools instead of requiring a custom configuration.
  • The average tester/reader does not need to know about the technical details, but the steps to perform a certain testing task.

• Draft Documents
  • Part 1:
  • Draft 1: Comments

Common Test Infrastructure

• Demonstration viewlet.

• Diagram detailing the architecture:

Meeting Minutes

• September 24, 2007
• October 22, 2007
• November 6, 2007
• November 12, 2007
• November 19, 2007