TPTP-Test-Automation-20071126

Attendees

• Present:
  • Paul
  • Kiryl
  • Joanna
  • Jonathan
  • Joel

• Absent:
  • Alan - ill

Deliverables

• Testing Process:
  • Only Paul and Jonathan reviewed/commented on the first draft of the Testing Process (part 1) document.
  • Alan will integrate review comments and post a second draft of the Testing Process (part 1) document for review by Wednesday, November 28.
  • Alan created a preliminary first draft of the Testing Process (part 2) containing the remaining sections of the existing TPTP Testing Strategy document.
  • Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
  • Please review the Testing Process (part 2) draft document by Thursday, November 29.

• Common Test Infrastructure:
  • Completed:
    • CVS check-out of all of the test plug-ins in the /test-results/<project>/* modules.
  • Outstanding:
    • Check for Intel build to complete (every x hours) before launching the BVTs.
    • Configure and run the Agent Controller on the reference platform.
      • Joel is working with Jonathan to integrate the existing code (target: today/tomorrow).
    • Create the root-level test suite for BVTs. The test infrastructure will execute each test plug-in's AllBVTTest.testsuite, as referenced by the org.eclipse.hyades.tests/All<test type>BVTTests.testsuite. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure.
    • Automatically generating the Test Report.
      • Until the blocking issue with remote test execution is resolved, we can start generating a 0% complete test report for the PMC to evaluate the TPTP test bucket for 4.5 iterations.
    • Automatically posting the Test Report to /tptp/test/reports/<release>.
      • Will require a new report generator for generating detailed test reports from a root-level execution result.
    • Cleanup of the reference platform.
    • Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
    • Email notifications to component leads for failing test suites.
  • Blocking Issues:

1. When generating reports, the report has no entries: Paul and Sheldon are investigating.
2. Defect 208616: Jerome are working on a fix. Paul/Kendric/Joel are verifying the fix.
3. Defect 200194: Symptoms have not been reproduced. Retargeting to i5.
4. Defect 190453: Joel has a work-around. Retargeting to i6.
5. Defect 210289: Joel has a work-around. Retargeting to i6.
6. Defect 210292: Duwayne triaged last week and provided a work-around for Joel. Retargeting to i6.
7. Defect 210955: Paul is investigating.

• Metrics:
  • No project leads have replied to a request to estimate the current testing costs.
    • Paul and Oliver Cole has reminded the TPTP Project Leads to provide a sizing.
    • The PMC has determined that this must be done by Wednesday, November 28.

Action Items

• All:
  • Review the Testing Process (part 2) draft document by Thursday, November 29.

• Alan:
  • Complete the second draft of the Testing Process (part 1) document.
    • Integrate review comments by Wednesday, November 28.
  • Add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.

• Joel:
  • Outstanding issues for the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).

• Paul:
  • Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
  • Encourage Project Leads to automate existing manual test suites.

Upcoming Meeting

• We will meet next on Thursday, November 29 on the Agent Controller Status Call (9 AM ET).

Upcoming Discussion

• Test framework reuse:
  • See defect 171827.
  • Existing test frameworks include:
    • Test Server for the Agent Controller/Profiler
    • Test Server for the JVMTI Profiler.
  • Proposed test frameworks include:
    • Test Dashboard (see enhancement 112928 and Test Dashboard Plug-in).
    • Test automation service (see defect 87285)