View source for TPTP-Test-Automation-20071129

== Attendees ==

* Present:
** Paul
** Kiryl
** Joanna
** Jonathan
** Joel
** Alan

== Deliverables ==

* Testing Process:
** Alan integrated the [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] and Paul posted the second draft of the Testing Process (part 1) document for review.
*** Allan will integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_2_Comments review comments] for the second draft of the Testing Process (part 1) document by Monday, December 3. 
** Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
** Please review the latest drafts for both [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process documents] for Monday, December 3.

* Common Test Infrastructure: 
** Completed:
*** CVS check-out of all of the test plug-ins in the <code>/test-results/&lt;project&gt;/*</code> modules.
*** Configure and run the Agent Controller on the reference platform.
*** Paul created the [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/BVT/?root=TPTP_Project root-level test suites for BVTs].  The test infrastructure will execute the <code>org.eclipse.hyades.tests/AllBVT&lt;test type&gt;Tests.testsuite</code> test suites.  We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results.  Each developer can rerun the automated tests to reproduce a failure.
*** Paul created the [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/deployment/?root=TPTP_Project BVT deployment] to the reference platform.
*** Automatically generating the Test Report based on the existing test execution results in CVS.
*** Paul modified the BIRT tabular test report to walk an execution result to generate a detailed test report from a root-level execution result.
** Outstanding:
*** Check for Intel build to complete (every x hours) before launching the BVTs.
*** Automatically posting the Test Report to <code>/tptp/test/reports/&lt;release&gt;</code>.
*** Cleanup of the reference platform.
*** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
*** Email notifications to component leads for failing test suites.
** Issues:
# When generating reports, the report has no entries: Paul and Sheldon are investigating.  
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Fixed and verified by Kendric/Joel.
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200194 Defect 200194]: Targeted to i5.
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=190453 Defect 190453]: Targeted to i6.
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210289 Defect 210289]: Targeted to i6.
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210292 Defect 210292]: Targeted to i6.
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210955 Defect 210955]: DTP has resolved the issue, available in the December 1st build.  Paul has request a patch.
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=211476 Defect 211476 ]: Paul will fix in i4.

* Metrics: 
** The monitoring project lead has replied to a request to estimate the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Metrics current testing costs].
*** Joanna and Guru are gathering their estimates.

== Action Items ==

* All:
** Review the latest drafts for both [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process documents] for Monday, December 3.

* Alan: 
** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_2_Comments review comments] for the second draft of the Testing Process (part 1) document by Monday, December 3. 
** Add more detailed content to the first draft of the Testing Process (part 2) by Monday, December 3.
 
* Joel: 
** Outstanding issues for the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).  

* Paul: 
** Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
** Encourage Project Leads to automate existing manual test suites.

== Upcoming Meeting ==

* We will meet next on Monday, December 3 (9 AM ET).

== Upcoming Discussion ==

* Test framework reuse:
** See [https://bugs.eclipse.org/bugs/show_bug.cgi?id=171827 defect 171827].
** Existing test frameworks include:
*** Test Server for the Agent Controller/Profiler
*** Test Server for the JVMTI Profiler.
** Proposed test frameworks include:
*** [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928] and [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in]).
*** Test automation service (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=87285 defect 87285])