Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "TPTP-Test-Automation-20071129"

(New page: == Attendees == * Present: ** Paul ** Kiryl ** Joanna ** Jonathan ** Joel ** Alan == Deliverables == * Testing Process: ** Only Paul and Jonathan reviewed/commented on the first draft o...)
 
(→‎Deliverables)
 
(One intermediate revision by one other user not shown)
Line 12: Line 12:
  
 
* Testing Process:
 
* Testing Process:
−
** Only Paul and Jonathan reviewed/commented on the first draft of the Testing Process (part 1) document.
+
** Alan integrated the [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] and Paul posted the second draft of the Testing Process (part 1) document for review.
−
** Alan will integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] and post a second draft of the Testing Process (part 1) document for review by Wednesday, November 28.
+
*** Allan will integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_2_Comments review comments] for the second draft of the Testing Process (part 1) document by Monday, December 3.  
−
** Alan created a preliminary first draft of the Testing Process (part 2) containing the remaining sections of the existing TPTP Testing Strategy document.
+
 
** Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
 
** Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
−
** Please review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process (part 2)] draft document by Thursday, November 29.
+
** Please review the latest drafts for both [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process documents] for Monday, December 3.
  
 
* Common Test Infrastructure:  
 
* Common Test Infrastructure:  
 
** Completed:
 
** Completed:
 
*** CVS check-out of all of the test plug-ins in the <code>/test-results/&lt;project&gt;/*</code> modules.
 
*** CVS check-out of all of the test plug-ins in the <code>/test-results/&lt;project&gt;/*</code> modules.
 +
*** Configure and run the Agent Controller on the reference platform.
 +
*** Paul created the [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/BVT/?root=TPTP_Project root-level test suites for BVTs].  The test infrastructure will execute the <code>org.eclipse.hyades.tests/AllBVT&lt;test type&gt;Tests.testsuite</code> test suites.  We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results.  Each developer can rerun the automated tests to reproduce a failure.
 +
*** Paul created the [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/deployment/?root=TPTP_Project BVT deployment] to the reference platform.
 +
*** Automatically generating the Test Report based on the existing test execution results in CVS.
 +
*** Paul modified the BIRT tabular test report to walk an execution result to generate a detailed test report from a root-level execution result.
 
** Outstanding:
 
** Outstanding:
 
*** Check for Intel build to complete (every x hours) before launching the BVTs.
 
*** Check for Intel build to complete (every x hours) before launching the BVTs.
−
*** Configure and run the Agent Controller on the reference platform.
 
−
****Joel is working with Jonathan to integrate the existing code (target: today/tomorrow).
 
−
*** Create the root-level test suite for BVTs.  The test infrastructure will execute each test plug-in's <code>AllBVTTest.testsuite</code>, as referenced by the <code>org.eclipse.hyades.tests/All&lt;test type&gt;BVTTests.testsuite</code>.  We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results.  Each developer can rerun the automated tests to reproduce a failure.
 
−
*** Automatically generating the Test Report.
 
−
**** Until the blocking issue with remote test execution is resolved, we can start generating a 0% complete test report for the PMC to evaluate the TPTP test bucket for 4.5 iterations.
 
 
*** Automatically posting the Test Report to <code>/tptp/test/reports/&lt;release&gt;</code>.
 
*** Automatically posting the Test Report to <code>/tptp/test/reports/&lt;release&gt;</code>.
−
**** Will require a new report generator for generating detailed test reports from a root-level execution result.
 
 
*** Cleanup of the reference platform.
 
*** Cleanup of the reference platform.
 
*** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
 
*** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
 
*** Email notifications to component leads for failing test suites.
 
*** Email notifications to component leads for failing test suites.
−
** Blocking Issues:
+
** Issues:
 
# When generating reports, the report has no entries: Paul and Sheldon are investigating.   
 
# When generating reports, the report has no entries: Paul and Sheldon are investigating.   
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Jerome are working on a fix.  Paul/Kendric/Joel are verifying the fix.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Fixed and verified by Kendric/Joel.
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200194 Defect 200194]: Symptoms have not been reproduced.  Retargeting to i5.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200194 Defect 200194]: Targeted to i5.
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=190453 Defect 190453]: Joel has a work-around.  Retargeting to i6.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=190453 Defect 190453]: Targeted to i6.
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210289 Defect 210289]: Joel has a work-around.  Retargeting to i6.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210289 Defect 210289]: Targeted to i6.
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210292 Defect 210292]: Duwayne triaged last week and provided a work-around for Joel. Retargeting to i6.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210292 Defect 210292]: Targeted to i6.
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210955 Defect 210955]: Paul is investigating.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210955 Defect 210955]: DTP has resolved the issue, available in the December 1st build.  Paul has request a patch.
 +
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=211476 Defect 211476 ]: Paul will fix in i4.
  
 
* Metrics:  
 
* Metrics:  
−
** No project leads have replied to a request to estimate the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Metrics current testing costs].
+
** The monitoring project lead has replied to a request to estimate the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Metrics current testing costs].
−
*** Paul and Oliver Colee has reminded the TPTP Project Leads to provide a sizing.
+
*** Joanna and Guru are gathering their estimates.
−
*** The PMC has determined that this must be done by Wednesday, November 28.
+
  
 
== Action Items ==
 
== Action Items ==
  
 
* All:
 
* All:
−
** Review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process (part 2)] draft document by Thursday, November 29.
+
** Review the latest drafts for both [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process documents] for Monday, December 3.
  
 
* Alan:  
 
* Alan:  
−
** Complete the second draft of the Testing Process (part 1) document.
+
** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_2_Comments review comments] for the second draft of the Testing Process (part 1) document by Monday, December 3.  
−
*** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] by Wednesday, November 28.  
+
** Add more detailed content to the first draft of the Testing Process (part 2) by Monday, December 3.
−
** Add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
+
 
   
 
   
 
* Joel:  
 
* Joel:  
Line 66: Line 63:
 
== Upcoming Meeting ==
 
== Upcoming Meeting ==
  
−
* We will meet next on Thursday, November 29 on the Agent Controller Status Call (9 AM ET).
+
* We will meet next on Monday, December 3 (9 AM ET).
  
 
== Upcoming Discussion ==
 
== Upcoming Discussion ==

Latest revision as of 16:12, 29 November 2007

Attendees

  • Present:
    • Paul
    • Kiryl
    • Joanna
    • Jonathan
    • Joel
    • Alan

Deliverables

  • Testing Process:
    • Alan integrated the review comments and Paul posted the second draft of the Testing Process (part 1) document for review.
      • Allan will integrate review comments for the second draft of the Testing Process (part 1) document by Monday, December 3.
    • Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
    • Please review the latest drafts for both Testing Process documents for Monday, December 3.
  • Common Test Infrastructure:
    • Completed:
      • CVS check-out of all of the test plug-ins in the /test-results/<project>/* modules.
      • Configure and run the Agent Controller on the reference platform.
      • Paul created the root-level test suites for BVTs. The test infrastructure will execute the org.eclipse.hyades.tests/AllBVT<test type>Tests.testsuite test suites. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure.
      • Paul created the BVT deployment to the reference platform.
      • Automatically generating the Test Report based on the existing test execution results in CVS.
      • Paul modified the BIRT tabular test report to walk an execution result to generate a detailed test report from a root-level execution result.
    • Outstanding:
      • Check for Intel build to complete (every x hours) before launching the BVTs.
      • Automatically posting the Test Report to /tptp/test/reports/<release>.
      • Cleanup of the reference platform.
      • Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
      • Email notifications to component leads for failing test suites.
    • Issues:
  1. When generating reports, the report has no entries: Paul and Sheldon are investigating.
  2. Defect 208616: Fixed and verified by Kendric/Joel.
  3. Defect 200194: Targeted to i5.
  4. Defect 190453: Targeted to i6.
  5. Defect 210289: Targeted to i6.
  6. Defect 210292: Targeted to i6.
  7. Defect 210955: DTP has resolved the issue, available in the December 1st build. Paul has request a patch.
  8. Defect 211476 : Paul will fix in i4.
  • Metrics:
    • The monitoring project lead has replied to a request to estimate the current testing costs.
      • Joanna and Guru are gathering their estimates.

Action Items

  • Alan:
    • Integrate review comments for the second draft of the Testing Process (part 1) document by Monday, December 3.
    • Add more detailed content to the first draft of the Testing Process (part 2) by Monday, December 3.
  • Joel:
    • Outstanding issues for the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).
  • Paul:
    • Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
    • Encourage Project Leads to automate existing manual test suites.

Upcoming Meeting

  • We will meet next on Monday, December 3 (9 AM ET).

Upcoming Discussion

Retrieved from "https://wiki.eclipse.org/index.php?title=TPTP-Test-Automation-20071129&oldid=65017"

This page was last modified 16:12, 29 November 2007 by Paul Slauenwhite.

Back to the top