Difference between revisions of "TPTP-Test-Automation-20071119"

Revision as of 13:20, 19 November 2007

Attendees

• Present:
  • Paul
  • Joel
  • Alan
  • Kiryl
  • Jonathan

• Absent:
  • Joanna

Reference Platform

• Target: IBM Java 1.5 (latest SR) and Windows XP/x86.

• Developers are expected to unit test on reference platform before check-in.

• Nightly builds invoke automated test execution on reference platform.

• Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).

• Move to Java 6 once a complete test pass is completed to expose initial defects.

• The dedicate test machine for the reference platform has been clean and does not need to be re-imaged.

• IBM JREs for Windows and Linux (see IBM Development Package for Eclipse) are publicly available from developerWorks.

• Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.

Deliverables

• All:
  • Review draft of part 1.

• Alan:
  • Delivered draft of part 1.
  • Working on draft 2 of part 1.
  • Will work on second document with testing infrastructure and removed content from the TPTP Test Strategy.
  • Integrate comments form comment page.

• Joel:
  • Eclipse/TPTP and the Agent Controller is now deployed and expanded on the target platform.
  • Jonathan will assist Joel with configuring and running the Agent Controller.
    • Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform.
      • Joel to work with Jonathan to use Cygwin. (does not include environment variables set in Cygwin, defaults to the Windows envirooemnt variables).
      • Legal approval for freeSSH.
      • Jonathan has the code written.
  • A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting.
  • Since there is defect when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests.
  • Executes the Common Base Event JUnit tests.
  • Joel will work on automatically posting the Test Report to /tptp/test/reports/<release>.
  • Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
  • Joel will check out all test plugins and run root level test suite (e.g. AllJUnitBVTTests.testsuite).
    • may need a new reprot format for running root level tests.
  • Joel will work on automate check-in of test report.
  • We will not check in test reuslts due to disk space limimtations and reduces, developer can reproduce.
  • Email notiifcxaiton for failures.
  • Outstanding/Issues:

1. defect: Paul and Jerome.
2. Need to import test plug-ins into workbench before running the tests.

• ASF provides this service (importExistingProjects task).
• Exception.
• Working with Duwayne

1. When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne

• File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file.

1. Checking out a project using the CVS command, you need to know the test plug-in's name to extract to the correct direcotry. OTherwize, it is extracted as /test-resutls/....: Joel

• Paul:
  • Answeer comments to draft document.
  • Working on the defect when deploying TPTP JUnit tests remotely.
  • No project leads have replied to a request to estimate the current testing costs.
    • Paul will remind the TPTP Project Leads to provide a sizing.

Action Items

• Alan: Complete the first draft of the first testing process document for review by November 14, 2007.
• Alan: Complete the new test process.

• Jonathan: Assist Joel with configuring and running the Agent Controller.

• Joel: Prepare a small discussion on the architecture of the test infrastructure for next week's call.
• Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
• Joel: Create a small slide deck for demoing the PoC to the PMC by November 14, 2007.
• Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard.
• Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.

• Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
• Paul: Encourage Project Leads to automate existing manual test suites.
• Paul: Book another meeting for next week (November 19, 2007).
• Paul: Fix defect when deploying TPTP JUnit tests remotely.

Upcoming Discussion

• Test framework reuse:
  • See defect 171827.
  • Existing test frameworks include:
    • Test Server for the Agent Controller/Profiler
    • Test Server for the JVMTI Profiler.
  • Proposed test frameworks include:
    • Test Dashboard (see enhancement 112928 and Test Dashboard Plug-in).
    • Test automation service (see defect 87285)

References

• ASF
• ASF Tutorial
• TPTP Testing Strategy
• Test Dashboard (see enhancement 112928)
• Test Dashboard Plug-in