Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "TPTP-Test-Automation-20071119"

(→‎Deliverables)
(→‎Deliverables)
 
(7 intermediate revisions by the same user not shown)
Line 10: Line 10:
 
* Absent:
 
* Absent:
 
** Joanna
 
** Joanna
−
 
−
== Testing Process ==
 
−
 
−
* A very lightweight testing process for TPTP.
 
−
* Extends the existing TPTP Testing Strategy.
 
−
* Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
 
−
* Two (or more) documents:
 
−
** 1) High steps to use the testing infrastructure intended for first time users.  This document will reference the other document(s).
 
−
** 2) Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
 
−
** 3) Additional documents detailing the specifics of a testing topis. 
 
−
 
−
* Discussion:
 
−
** Naming conventions for root level test suites and execution results should include test plug-in ID.
 
−
** Execution results need to be saved in a directory with a meaningful name (e.g. OS, JRE, type (full, smoke, BVT)).
 
−
** Test suites for specific JREs and OSes (e.g. JVMPI versus JVMTI) will not always run on the reference platform(s).
 
−
** The existing technical content will remain unchanged, aside from review changes.
 
−
** Need sections for:
 
−
*** Getting tests.
 
−
*** Creating tests.
 
−
*** Running tests.
 
−
*** Save test results.
 
−
*** Reporting results.
 
−
*** BVTs.
 
−
** The Agent Controller and profiler test suites need to migrated to use the TPTP Test Tools instead of requiring a custom configuration.
 
−
** The average tester/reader does not need to know about the technical details, but the steps to perform a certain testing task.
 
−
 
−
== Reference Platform ==
 
−
 
−
* Target: IBM Java 1.5 (latest SR) and Windows XP/x86.
 
−
 
−
* Developers are expected to unit test on reference platform before check-in.
 
−
 
−
* Nightly builds invoke automated test execution on reference platform.
 
−
 
−
* Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
 
−
 
−
* Move to Java 6 once a complete test pass is completed to expose initial defects.
 
−
 
−
* The dedicate test machine for the reference platform has been clean and does not need to be re-imaged.
 
−
 
−
* IBM JREs for Windows and Linux (see [http://www.ibm.com/developerworks/java/jdk/eclipse/index.html IBM Development Package for Eclipse]) are publicly available from [http://www.ibm.com/developerworks/java/jdk/ developerWorks].
 
−
 
−
* Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.
 
−
 
−
== Common Test Infrastructure ==
 
−
 
−
* ASF integrated with the TPTP builds (invoked via ANT):
 
−
** Invoke tests.
 
−
** Interrogate results.
 
−
** Generate test reports.
 
−
 
−
* TPTP test types:
 
−
** AGR for functional UI testing.
 
−
** TPTP JUnit and TPTP JUnit Plug-in test for unit testing.
 
−
 
−
* TPTP build infrastructure:
 
−
** Extract test suites from CVS.
 
−
** Provision reference platform (Agent Controller and Eclipse/TPTP).
 
−
** Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
 
−
** Clean-up reference platform.
 
−
** Post test reports (test and code coverage) on TPTP web site.
 
−
 
−
* See the [http://wiki.eclipse.org/images/b/b5/AsfBVT.JPG diagram] of this infrastructure.
 
−
 
−
* Limitations/Issues:
 
−
** Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
 
−
** Intel needs to inform IBM that their portion of the build is complete (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200351 defect 200351]).
 
−
 
−
* Discussion:
 
−
** The test infrastructure will execute each test plug-in's <code>AllTest.testsuite</code>, as referenced by the <code>org.eclipse.hyades.tests/All&lt;test type&gt;Tests.testsuite</code>.
 
−
** Need to check out the test plug-ins in the <code>/test-results/&lt;sub-project&gt;/</code> directory.
 
  
 
== Deliverables ==
 
== Deliverables ==
−
 
−
* All:
 
−
** Review draft of part 1.
 
  
 
* Alan:  
 
* Alan:  
−
** Delivered draft of part 1.
+
** Delivered the first draft of the Testing Process (part 1) document.
−
** Working on draft 2 of part 1.
+
−
** Will work on second document with testing infrastructure and removed content from the TPTP Test Strategy.
+
−
** Integrate comments form comment page.
+
  
 
* Joel:  
 
* Joel:  
−
** Eclipse/TPTP and the Agent Controller is now deployed and expanded on the target platform.
+
** Working with Jonathan to configure and run the Agent Controller.
−
** Jonathan will assist Joel with configuring and running the Agent Controller.
+
*** Code is written.
−
*** Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform.
+
*** Jonathan just received legal approval for using freeSSH on Windows. Note, Cygwin cannot be used since it does not support custom environment variables on Windows.
−
**** Joel to work with Jonathan to use Cygwin. (does not include environment variables set in Cygwin, defaults to the Windows envirooemnt variables).
+
** Outstanding:
−
**** Legal approval for freeSSH.
+
*** CVS check-out of all of the test plug-ins in the <code>/test-results/&lt;project&gt;/*</code> modules.
−
**** Jonathan has the code written.
+
*** Running the correct test suite(s). For example, defining a root-level test suite for BVTs (possibly replacing the smoke tests). The test infrastructure will execute each test plug-in's <code>AllBVTTest.testsuite</code>, as referenced by the <code>org.eclipse.hyades.tests/All&lt;test type&gt;BVTTests.testsuite</code>.  We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results.  Each developer can rerun the automated tests to reproduce a failure.
−
** A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting.
+
*** Automatically posting the Test Report to <code>/tptp/test/reports/&lt;release&gt;</code>.
−
** Since there is [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests.
+
**** May require a new report generator for generating detailed test reports from a root-level execution result.
−
** Executes the Common Base Event [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.logging.core.tests/junit/?root=TPTP_Project JUnit tests].  
+
*** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
−
** Joel will work on automatically posting the Test Report to <code>/tptp/test/reports/&lt;release&gt;</code>.
+
*** Email notifications to component leads for failing test suites.
−
** Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
+
** Issues:
−
** Joel will check out all test plugins and run root level test suite (e.g. AllJUnitBVTTests.testsuite).
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Paul and Jerome are working on a fix.
−
*** may need a new reprot format for running root level tests.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210289 Defect 210289]: Duwayne is investigating.
−
** Joel will work on automate check-in of test report.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=210292 Defect 210292]: Duwayne is investigating.
−
** We will not check in test reuslts due to disk space limimtations and reduces, developer can reproduce.
+
# When checking out a project using the CVS command, the name of the test plug-in is required to extract it to the correct directory.  Otherwise, the test plug-in is extracted to the <code>/test-results/&lt;project&gt;/&lt;test plug-in&gt;</code> directory.
−
** Email notiifcxaiton for failures.
+
−
** Outstanding/Issues:
+
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect]: Paul and Jerome.
+
−
# Need to import test plug-ins into workbench before running the tests.
+
−
* ASF provides this service (importExistingProjects task).
+
−
* Exception.
+
−
* Working with Duwayne
+
−
# When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne
+
−
* File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file.
+
−
# Checking out a project using the CVS command, you need to know the test plug-in's name to extract to the correct direcotry.  OTherwize, it is extracted as /test-resutls/....: Joel
+
  
 
* Paul:  
 
* Paul:  
−
** Answeer comments to draft document.
+
** Worked with Jerome on reproducing [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect 208616] when deploying TPTP JUnit plug-in tests remotely.
−
** Working on the [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely.
+
 
** No project leads have replied to a request to estimate the current testing costs.
 
** No project leads have replied to a request to estimate the current testing costs.
 
*** Paul will remind the TPTP Project Leads to provide a sizing.
 
*** Paul will remind the TPTP Project Leads to provide a sizing.
Line 128: Line 40:
 
== Action Items ==
 
== Action Items ==
  
−
* Alan: Complete the first draft of the first testing process document for review by November 14, 2007.
+
* All:
−
* Alan: Complete the new test process.
+
** Review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process] draft document by Wednesday, November 21.
  
−
* Jonathan: Assist Joel with configuring and running the Agent Controller.
+
* Alan:  
 +
** Complete the second draft of the Testing Process (part 1) document.
 +
*** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments].
 +
** Complete the first draft of the Testing Process (part 2) document by Wednesday, November 21.
 +
*** Includes the remaining sections of the existing TPTP Testing Strategy document, and [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Common_Test_Infrastructure Common Test Infrastructure].
 +
 +
* Joel:
 +
** Prepare a small discussion on the architecture of the test infrastructure for next week's call.
 +
** Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
 +
** Finish the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).
  
−
* Joel: Prepare a small discussion on the architecture of the test infrastructure for next week's call.
+
* Paul:  
−
* Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
+
** Answer [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] by Tuesday, November 20.  
−
* Joel: Create a small slide deck for demoing the PoC to the PMC by November 14, 2007.
+
** Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
−
* Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard. 
+
** Encourage Project Leads to automate existing manual test suites.
−
* Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
+
** Book another meeting for next week (November 26, 2007).
−
 
+
** Work on [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect 208616] when deploying TPTP JUnit plug-in tests remotely.
−
* Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
+
−
* Paul: Encourage Project Leads to automate existing manual test suites.
+
−
* Paul: Book another meeting for next week (November 19, 2007).
+
−
* Paul: Fix defect when deploying TPTP JUnit tests remotely.
+
  
 
== Upcoming Discussion ==
 
== Upcoming Discussion ==
Line 154: Line 71:
 
*** [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928] and [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in]).
 
*** [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928] and [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in]).
 
*** Test automation service (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=87285 defect 87285])
 
*** Test automation service (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=87285 defect 87285])
−
 
−
== References ==
 
−
 
−
* [http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/tlaunching-tests-from-scripts-and-applications.htm ASF]
 
−
* [http://www.eclipse.org/tptp/test/documents/tutorials/eclipseWorld2006/Achieving_Continuous_Integration.html ASF Tutorial]
 
−
* [http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html TPTP Testing Strategy]
 
−
* [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928])
 
−
* [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in]
 

Latest revision as of 15:39, 19 November 2007

Attendees

  • Present:
    • Paul
    • Joel
    • Alan
    • Kiryl
    • Jonathan
  • Absent:
    • Joanna

Deliverables

  • Alan:
    • Delivered the first draft of the Testing Process (part 1) document.
  • Joel:
    • Working with Jonathan to configure and run the Agent Controller.
      • Code is written.
      • Jonathan just received legal approval for using freeSSH on Windows. Note, Cygwin cannot be used since it does not support custom environment variables on Windows.
    • Outstanding:
      • CVS check-out of all of the test plug-ins in the /test-results/<project>/* modules.
      • Running the correct test suite(s). For example, defining a root-level test suite for BVTs (possibly replacing the smoke tests). The test infrastructure will execute each test plug-in's AllBVTTest.testsuite, as referenced by the org.eclipse.hyades.tests/All<test type>BVTTests.testsuite. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure.
      • Automatically posting the Test Report to /tptp/test/reports/<release>.
        • May require a new report generator for generating detailed test reports from a root-level execution result.
      • Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
      • Email notifications to component leads for failing test suites.
    • Issues:
  1. Defect 208616: Paul and Jerome are working on a fix.
  2. Defect 210289: Duwayne is investigating.
  3. Defect 210292: Duwayne is investigating.
  4. When checking out a project using the CVS command, the name of the test plug-in is required to extract it to the correct directory. Otherwise, the test plug-in is extracted to the /test-results/<project>/<test plug-in> directory.
  • Paul:
    • Worked with Jerome on reproducing defect 208616 when deploying TPTP JUnit plug-in tests remotely.
    • No project leads have replied to a request to estimate the current testing costs.
      • Paul will remind the TPTP Project Leads to provide a sizing.

Action Items

  • Alan:
    • Complete the second draft of the Testing Process (part 1) document.
    • Complete the first draft of the Testing Process (part 2) document by Wednesday, November 21.
  • Joel:
    • Prepare a small discussion on the architecture of the test infrastructure for next week's call.
    • Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
    • Finish the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).
  • Paul:
    • Answer review comments by Tuesday, November 20.
    • Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
    • Encourage Project Leads to automate existing manual test suites.
    • Book another meeting for next week (November 26, 2007).
    • Work on defect 208616 when deploying TPTP JUnit plug-in tests remotely.

Upcoming Discussion

Retrieved from "https://wiki.eclipse.org/index.php?title=TPTP-Test-Automation-20071119&oldid=62264"

This page was last modified 15:39, 19 November 2007 by Eclipsepedia anonymous user Unnamed Poltroon.

Back to the top