Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "TPTP-Test-Automation-20071126"
(New page: == Attendees == * Present: ** Paul ** Joel ** Kiryl ** Jonathan * Absent: ** Alan - ill ** Joanna == Deliverables == * Alan: ** Delivered the first draft of the Testing Process (part ...) |
(→Deliverables) |
||
(3 intermediate revisions by one other user not shown) | |||
Line 3: | Line 3: | ||
* Present: | * Present: | ||
** Paul | ** Paul | ||
− | |||
** Kiryl | ** Kiryl | ||
+ | ** Joanna | ||
** Jonathan | ** Jonathan | ||
+ | ** Joel | ||
* Absent: | * Absent: | ||
** Alan - ill | ** Alan - ill | ||
− | |||
== Deliverables == | == Deliverables == | ||
− | * | + | * Testing Process: |
− | ** | + | ** Only Paul and Jonathan reviewed/commented on the first draft of the Testing Process (part 1) document. |
+ | ** Alan will integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] and post a second draft of the Testing Process (part 1) document for review by Wednesday, November 28. | ||
+ | ** Alan created a preliminary first draft of the Testing Process (part 2) containing the remaining sections of the existing TPTP Testing Strategy document. | ||
+ | ** Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28. | ||
+ | ** Please review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process (part 2)] draft document by Thursday, November 29. | ||
− | * | + | * Common Test Infrastructure: |
− | ** | + | ** Completed: |
− | *** | + | *** CVS check-out of all of the test plug-ins in the <code>/test-results/<project>/*</code> modules. |
− | + | ||
** Outstanding: | ** Outstanding: | ||
− | *** | + | *** Check for Intel build to complete (every x hours) before launching the BVTs. |
− | *** | + | *** Configure and run the Agent Controller on the reference platform. |
+ | ****Joel is working with Jonathan to integrate the existing code (target: today/tomorrow). | ||
+ | *** Create the root-level test suite for BVTs. The test infrastructure will execute each test plug-in's <code>AllBVTTest.testsuite</code>, as referenced by the <code>org.eclipse.hyades.tests/All<test type>BVTTests.testsuite</code>. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure. | ||
+ | *** Automatically generating the Test Report. | ||
+ | **** Until the blocking issue with remote test execution is resolved, we can start generating a 0% complete test report for the PMC to evaluate the TPTP test bucket for 4.5 iterations. | ||
*** Automatically posting the Test Report to <code>/tptp/test/reports/<release></code>. | *** Automatically posting the Test Report to <code>/tptp/test/reports/<release></code>. | ||
− | **** | + | **** Will require a new report generator for generating detailed test reports from a root-level execution result. |
+ | *** Cleanup of the reference platform. | ||
*** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders. | *** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders. | ||
*** Email notifications to component leads for failing test suites. | *** Email notifications to component leads for failing test suites. | ||
− | ** Issues: | + | ** Blocking Issues: |
− | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: | + | # When generating reports, the report has no entries: Paul and Sheldon are investigating. |
− | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id= | + | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Jerome are working on a fix. Paul/Kendric/Joel are verifying the fix. |
− | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id= | + | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200194 Defect 200194]: Symptoms have not been reproduced. Retargeting to i5. |
− | # | + | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=190453 Defect 190453]: Joel has a work-around. Retargeting to i6. |
+ | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210289 Defect 210289]: Joel has a work-around. Retargeting to i6. | ||
+ | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210292 Defect 210292]: Duwayne triaged last week and provided a work-around for Joel. Retargeting to i6. | ||
+ | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210955 Defect 210955]: Paul is investigating. | ||
− | * | + | * Metrics: |
− | + | ** No project leads have replied to a request to estimate the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Metrics current testing costs]. | |
− | ** No project leads have replied to a request to estimate the current testing costs. | + | *** Paul and Oliver Cole has reminded the TPTP Project Leads to provide a sizing. |
− | *** Paul | + | *** The PMC has determined that this must be done by Wednesday, November 28. |
== Action Items == | == Action Items == | ||
* All: | * All: | ||
− | ** Review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process] draft document by | + | ** Review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process (part 2)] draft document by Thursday, November 29. |
* Alan: | * Alan: | ||
** Complete the second draft of the Testing Process (part 1) document. | ** Complete the second draft of the Testing Process (part 1) document. | ||
− | *** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments]. | + | *** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] by Wednesday, November 28. |
− | ** | + | ** Add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28. |
− | + | ||
* Joel: | * Joel: | ||
− | ** | + | ** Outstanding issues for the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code). |
− | + | ||
− | + | ||
* Paul: | * Paul: | ||
− | |||
** Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads. | ** Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads. | ||
** Encourage Project Leads to automate existing manual test suites. | ** Encourage Project Leads to automate existing manual test suites. | ||
− | * | + | |
− | + | == Upcoming Meeting == | |
+ | |||
+ | * We will meet next on Thursday, November 29 on the Agent Controller Status Call (9 AM ET). | ||
== Upcoming Discussion == | == Upcoming Discussion == |
Latest revision as of 16:10, 29 November 2007
Attendees
- Present:
- Paul
- Kiryl
- Joanna
- Jonathan
- Joel
- Absent:
- Alan - ill
Deliverables
- Testing Process:
- Only Paul and Jonathan reviewed/commented on the first draft of the Testing Process (part 1) document.
- Alan will integrate review comments and post a second draft of the Testing Process (part 1) document for review by Wednesday, November 28.
- Alan created a preliminary first draft of the Testing Process (part 2) containing the remaining sections of the existing TPTP Testing Strategy document.
- Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
- Please review the Testing Process (part 2) draft document by Thursday, November 29.
- Common Test Infrastructure:
- Completed:
- CVS check-out of all of the test plug-ins in the
/test-results/<project>/*
modules.
- CVS check-out of all of the test plug-ins in the
- Outstanding:
- Check for Intel build to complete (every x hours) before launching the BVTs.
- Configure and run the Agent Controller on the reference platform.
- Joel is working with Jonathan to integrate the existing code (target: today/tomorrow).
- Create the root-level test suite for BVTs. The test infrastructure will execute each test plug-in's
AllBVTTest.testsuite
, as referenced by theorg.eclipse.hyades.tests/All<test type>BVTTests.testsuite
. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure. - Automatically generating the Test Report.
- Until the blocking issue with remote test execution is resolved, we can start generating a 0% complete test report for the PMC to evaluate the TPTP test bucket for 4.5 iterations.
- Automatically posting the Test Report to
/tptp/test/reports/<release>
.- Will require a new report generator for generating detailed test reports from a root-level execution result.
- Cleanup of the reference platform.
- Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
- Email notifications to component leads for failing test suites.
- Blocking Issues:
- Completed:
- When generating reports, the report has no entries: Paul and Sheldon are investigating.
- Defect 208616: Jerome are working on a fix. Paul/Kendric/Joel are verifying the fix.
- Defect 200194: Symptoms have not been reproduced. Retargeting to i5.
- Defect 190453: Joel has a work-around. Retargeting to i6.
- Defect 210289: Joel has a work-around. Retargeting to i6.
- Defect 210292: Duwayne triaged last week and provided a work-around for Joel. Retargeting to i6.
- Defect 210955: Paul is investigating.
- Metrics:
- No project leads have replied to a request to estimate the current testing costs.
- Paul and Oliver Cole has reminded the TPTP Project Leads to provide a sizing.
- The PMC has determined that this must be done by Wednesday, November 28.
- No project leads have replied to a request to estimate the current testing costs.
Action Items
- All:
- Review the Testing Process (part 2) draft document by Thursday, November 29.
- Alan:
- Complete the second draft of the Testing Process (part 1) document.
- Integrate review comments by Wednesday, November 28.
- Add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
- Complete the second draft of the Testing Process (part 1) document.
- Joel:
- Outstanding issues for the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).
- Paul:
- Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
- Encourage Project Leads to automate existing manual test suites.
Upcoming Meeting
- We will meet next on Thursday, November 29 on the Agent Controller Status Call (9 AM ET).
Upcoming Discussion
- Test framework reuse:
- See defect 171827.
- Existing test frameworks include:
- Test Server for the Agent Controller/Profiler
- Test Server for the JVMTI Profiler.
- Proposed test frameworks include:
- Test Dashboard (see enhancement 112928 and Test Dashboard Plug-in).
- Test automation service (see defect 87285)