Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "TPTP-Test-Automation-20071119"
(→Deliverables) |
(→Deliverables) |
||
Line 97: | Line 97: | ||
** Jonathan will assist Joel with configuring and running the Agent Controller. | ** Jonathan will assist Joel with configuring and running the Agent Controller. | ||
*** Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform. | *** Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform. | ||
− | **** Joel to work with Jonathan to use Cygwin. | + | **** Joel to work with Jonathan to use Cygwin. (does not include environment variables set in Cygwin, defaults to the Windows envirooemnt variables). |
+ | **** Legal approval for freeSSH. | ||
+ | **** Jonathan has the code written. | ||
** A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting. | ** A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting. | ||
** Since there is [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests. | ** Since there is [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests. | ||
Line 103: | Line 105: | ||
** Joel will work on automatically posting the Test Report to <code>/tptp/test/reports/<release></code>. | ** Joel will work on automatically posting the Test Report to <code>/tptp/test/reports/<release></code>. | ||
** Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders. | ** Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders. | ||
+ | ** Joel will check out all test plugins and run root level test suite (e.g. AllJUnitBVTTests.testsuite). | ||
+ | *** may need a new reprot format for running root level tests. | ||
+ | ** Joel will work on automate check-in of test report. | ||
+ | ** We will not check in test reuslts due to disk space limimtations and reduces, developer can reproduce. | ||
+ | ** Email notiifcxaiton for failures. | ||
** Outstanding/Issues: | ** Outstanding/Issues: | ||
− | + | # [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect]: Paul and Jerome. | |
− | + | # Need to import test plug-ins into workbench before running the tests. | |
+ | * ASF provides this service (importExistingProjects task). | ||
+ | * Exception. | ||
+ | * Working with Duwayne | ||
+ | # When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne | ||
+ | * File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file. | ||
+ | # Checking out a project using the CVS command, you need to know the test plug-in's name to extract to the correct direcotry. OTherwize, it is extracted as /test-resutls/....: Joel | ||
* Paul: | * Paul: | ||
− | ** Answeer comments. | + | ** Answeer comments to draft document. |
** Working on the [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely. | ** Working on the [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely. | ||
** No project leads have replied to a request to estimate the current testing costs. | ** No project leads have replied to a request to estimate the current testing costs. |
Revision as of 10:42, 19 November 2007
Contents
Attendees
- Present:
- Paul
- Joel
- Alan
- Kiryl
- Jonathan
- Absent:
- Joanna
Testing Process
- A very lightweight testing process for TPTP.
- Extends the existing TPTP Testing Strategy.
- Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
- Two (or more) documents:
- 1) High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s).
- 2) Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
- 3) Additional documents detailing the specifics of a testing topis.
- Discussion:
- Naming conventions for root level test suites and execution results should include test plug-in ID.
- Execution results need to be saved in a directory with a meaningful name (e.g. OS, JRE, type (full, smoke, BVT)).
- Test suites for specific JREs and OSes (e.g. JVMPI versus JVMTI) will not always run on the reference platform(s).
- The existing technical content will remain unchanged, aside from review changes.
- Need sections for:
- Getting tests.
- Creating tests.
- Running tests.
- Save test results.
- Reporting results.
- BVTs.
- The Agent Controller and profiler test suites need to migrated to use the TPTP Test Tools instead of requiring a custom configuration.
- The average tester/reader does not need to know about the technical details, but the steps to perform a certain testing task.
Reference Platform
- Target: IBM Java 1.5 (latest SR) and Windows XP/x86.
- Developers are expected to unit test on reference platform before check-in.
- Nightly builds invoke automated test execution on reference platform.
- Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
- Move to Java 6 once a complete test pass is completed to expose initial defects.
- The dedicate test machine for the reference platform has been clean and does not need to be re-imaged.
- IBM JREs for Windows and Linux (see IBM Development Package for Eclipse) are publicly available from developerWorks.
- Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.
Common Test Infrastructure
- ASF integrated with the TPTP builds (invoked via ANT):
- Invoke tests.
- Interrogate results.
- Generate test reports.
- TPTP test types:
- AGR for functional UI testing.
- TPTP JUnit and TPTP JUnit Plug-in test for unit testing.
- TPTP build infrastructure:
- Extract test suites from CVS.
- Provision reference platform (Agent Controller and Eclipse/TPTP).
- Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
- Clean-up reference platform.
- Post test reports (test and code coverage) on TPTP web site.
- See the diagram of this infrastructure.
- Limitations/Issues:
- Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
- Intel needs to inform IBM that their portion of the build is complete (see defect 200351).
- Discussion:
- The test infrastructure will execute each test plug-in's
AllTest.testsuite
, as referenced by theorg.eclipse.hyades.tests/All<test type>Tests.testsuite
. - Need to check out the test plug-ins in the
/test-results/<sub-project>/
directory.
- The test infrastructure will execute each test plug-in's
Deliverables
- All:
- Review draft of part 1.
- Alan:
- Delivered draft of part 1.
- Working on draft 2 of part 1.
- Will work on second document with testing infrastructure and removed content from the TPTP Test Strategy.
- Integrate comments form comment page.
- Joel:
- Eclipse/TPTP and the Agent Controller is now deployed and expanded on the target platform.
- Jonathan will assist Joel with configuring and running the Agent Controller.
- Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform.
- Joel to work with Jonathan to use Cygwin. (does not include environment variables set in Cygwin, defaults to the Windows envirooemnt variables).
- Legal approval for freeSSH.
- Jonathan has the code written.
- Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform.
- A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting.
- Since there is defect when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests.
- Executes the Common Base Event JUnit tests.
- Joel will work on automatically posting the Test Report to
/tptp/test/reports/<release>
. - Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
- Joel will check out all test plugins and run root level test suite (e.g. AllJUnitBVTTests.testsuite).
- may need a new reprot format for running root level tests.
- Joel will work on automate check-in of test report.
- We will not check in test reuslts due to disk space limimtations and reduces, developer can reproduce.
- Email notiifcxaiton for failures.
- Outstanding/Issues:
- defect: Paul and Jerome.
- Need to import test plug-ins into workbench before running the tests.
- ASF provides this service (importExistingProjects task).
- Exception.
- Working with Duwayne
- When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne
- File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file.
- Checking out a project using the CVS command, you need to know the test plug-in's name to extract to the correct direcotry. OTherwize, it is extracted as /test-resutls/....: Joel
- Paul:
- Answeer comments to draft document.
- Working on the defect when deploying TPTP JUnit tests remotely.
- No project leads have replied to a request to estimate the current testing costs.
- Paul will remind the TPTP Project Leads to provide a sizing.
Action Items
- Alan: Complete the first draft of the first testing process document for review by November 14, 2007.
- Alan: Complete the new test process.
- Jonathan: Assist Joel with configuring and running the Agent Controller.
- Joel: Prepare a small discussion on the architecture of the test infrastructure for next week's call.
- Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
- Joel: Create a small slide deck for demoing the PoC to the PMC by November 14, 2007.
- Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard.
- Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
- Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
- Paul: Encourage Project Leads to automate existing manual test suites.
- Paul: Book another meeting for next week (November 19, 2007).
- Paul: Fix defect when deploying TPTP JUnit tests remotely.
Upcoming Discussion
- Test framework reuse:
- See defect 171827.
- Existing test frameworks include:
- Test Server for the Agent Controller/Profiler
- Test Server for the JVMTI Profiler.
- Proposed test frameworks include:
- Test Dashboard (see enhancement 112928 and Test Dashboard Plug-in).
- Test automation service (see defect 87285)
References
- ASF
- ASF Tutorial
- TPTP Testing Strategy
- Test Dashboard (see enhancement 112928)
- Test Dashboard Plug-in