Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "TPTP-Test-Automation-20071119"
(→Reference Platform) |
(→Deliverables) |
||
(4 intermediate revisions by the same user not shown) | |||
Line 10: | Line 10: | ||
* Absent: | * Absent: | ||
** Joanna | ** Joanna | ||
− | |||
− | |||
− | |||
− | |||
== Deliverables == | == Deliverables == | ||
− | |||
− | |||
− | |||
* Alan: | * Alan: | ||
− | ** Delivered draft of part 1 | + | ** Delivered the first draft of the Testing Process (part 1) document. |
− | + | ||
− | + | ||
− | + | ||
* Joel: | * Joel: | ||
− | ** | + | ** Working with Jonathan to configure and run the Agent Controller. |
− | ** | + | *** Code is written. |
− | *** Jonathan | + | *** Jonathan just received legal approval for using freeSSH on Windows. Note, Cygwin cannot be used since it does not support custom environment variables on Windows. |
− | + | ** Outstanding: | |
− | ** | + | *** CVS check-out of all of the test plug-ins in the <code>/test-results/<project>/*</code> modules. |
− | *** | + | *** Running the correct test suite(s). For example, defining a root-level test suite for BVTs (possibly replacing the smoke tests). The test infrastructure will execute each test plug-in's <code>AllBVTTest.testsuite</code>, as referenced by the <code>org.eclipse.hyades.tests/All<test type>BVTTests.testsuite</code>. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure. |
− | + | *** Automatically posting the Test Report to <code>/tptp/test/reports/<release></code>. | |
− | + | **** May require a new report generator for generating detailed test reports from a root-level execution result. | |
− | ** | + | *** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders. |
− | ** | + | *** Email notifications to component leads for failing test suites. |
− | ** | + | ** Issues: |
− | ** | + | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Paul and Jerome are working on a fix. |
− | * | + | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210289 Defect 210289]: Duwayne is investigating. |
− | + | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210292 Defect 210292]: Duwayne is investigating. | |
− | ** | + | # When checking out a project using the CVS command, the name of the test plug-in is required to extract it to the correct directory. Otherwise, the test plug-in is extracted to the <code>/test-results/<project>/<test plug-in></code> directory. |
− | + | ||
− | + | ||
− | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 | + | |
− | # | + | |
− | + | ||
− | + | ||
− | + | ||
− | # | + | |
− | + | ||
− | # | + | |
* Paul: | * Paul: | ||
− | ** | + | ** Worked with Jerome on reproducing [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect 208616] when deploying TPTP JUnit plug-in tests remotely. |
− | + | ||
** No project leads have replied to a request to estimate the current testing costs. | ** No project leads have replied to a request to estimate the current testing costs. | ||
*** Paul will remind the TPTP Project Leads to provide a sizing. | *** Paul will remind the TPTP Project Leads to provide a sizing. | ||
Line 61: | Line 40: | ||
== Action Items == | == Action Items == | ||
− | * | + | * All: |
− | + | ** Review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process] draft document by Wednesday, November 21. | |
− | * | + | * Alan: |
+ | ** Complete the second draft of the Testing Process (part 1) document. | ||
+ | *** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments]. | ||
+ | ** Complete the first draft of the Testing Process (part 2) document by Wednesday, November 21. | ||
+ | *** Includes the remaining sections of the existing TPTP Testing Strategy document, and [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Common_Test_Infrastructure Common Test Infrastructure]. | ||
+ | |||
+ | * Joel: | ||
+ | ** Prepare a small discussion on the architecture of the test infrastructure for next week's call. | ||
+ | ** Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders. | ||
+ | ** Finish the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code). | ||
− | * | + | * Paul: |
− | * | + | ** Answer [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] by Tuesday, November 20. |
− | + | ** Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads. | |
− | * | + | ** Encourage Project Leads to automate existing manual test suites. |
− | * | + | ** Book another meeting for next week (November 26, 2007). |
− | + | ** Work on [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect 208616] when deploying TPTP JUnit plug-in tests remotely. | |
− | + | ||
− | * | + | |
− | * | + | |
− | * | + | |
== Upcoming Discussion == | == Upcoming Discussion == | ||
Line 87: | Line 71: | ||
*** [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928] and [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in]). | *** [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928] and [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in]). | ||
*** Test automation service (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=87285 defect 87285]) | *** Test automation service (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=87285 defect 87285]) | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Latest revision as of 15:39, 19 November 2007
Attendees
- Present:
- Paul
- Joel
- Alan
- Kiryl
- Jonathan
- Absent:
- Joanna
Deliverables
- Alan:
- Delivered the first draft of the Testing Process (part 1) document.
- Joel:
- Working with Jonathan to configure and run the Agent Controller.
- Code is written.
- Jonathan just received legal approval for using freeSSH on Windows. Note, Cygwin cannot be used since it does not support custom environment variables on Windows.
- Outstanding:
- CVS check-out of all of the test plug-ins in the
/test-results/<project>/*
modules. - Running the correct test suite(s). For example, defining a root-level test suite for BVTs (possibly replacing the smoke tests). The test infrastructure will execute each test plug-in's
AllBVTTest.testsuite
, as referenced by theorg.eclipse.hyades.tests/All<test type>BVTTests.testsuite
. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure. - Automatically posting the Test Report to
/tptp/test/reports/<release>
.- May require a new report generator for generating detailed test reports from a root-level execution result.
- Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
- Email notifications to component leads for failing test suites.
- CVS check-out of all of the test plug-ins in the
- Issues:
- Working with Jonathan to configure and run the Agent Controller.
- Defect 208616: Paul and Jerome are working on a fix.
- Defect 210289: Duwayne is investigating.
- Defect 210292: Duwayne is investigating.
- When checking out a project using the CVS command, the name of the test plug-in is required to extract it to the correct directory. Otherwise, the test plug-in is extracted to the
/test-results/<project>/<test plug-in>
directory.
- Paul:
- Worked with Jerome on reproducing defect 208616 when deploying TPTP JUnit plug-in tests remotely.
- No project leads have replied to a request to estimate the current testing costs.
- Paul will remind the TPTP Project Leads to provide a sizing.
Action Items
- All:
- Review the Testing Process draft document by Wednesday, November 21.
- Alan:
- Complete the second draft of the Testing Process (part 1) document.
- Integrate review comments.
- Complete the first draft of the Testing Process (part 2) document by Wednesday, November 21.
- Includes the remaining sections of the existing TPTP Testing Strategy document, and Common Test Infrastructure.
- Complete the second draft of the Testing Process (part 1) document.
- Joel:
- Prepare a small discussion on the architecture of the test infrastructure for next week's call.
- Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
- Finish the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).
- Paul:
- Answer review comments by Tuesday, November 20.
- Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
- Encourage Project Leads to automate existing manual test suites.
- Book another meeting for next week (November 26, 2007).
- Work on defect 208616 when deploying TPTP JUnit plug-in tests remotely.
Upcoming Discussion
- Test framework reuse:
- See defect 171827.
- Existing test frameworks include:
- Test Server for the Agent Controller/Profiler
- Test Server for the JVMTI Profiler.
- Proposed test frameworks include:
- Test Dashboard (see enhancement 112928 and Test Dashboard Plug-in).
- Test automation service (see defect 87285)