Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "TPTP-Test-Automation-20071119"

(→‎References)
Line 12: Line 12:
  
 
== Deliverables ==
 
== Deliverables ==
−
 
−
* All:
 
−
** Review draft of part 1.
 
  
 
* Alan:  
 
* Alan:  
−
** Delivered draft of part 1.
+
** Delivered the first draft of the Testing Process (part 1) document.
−
** Working on draft 2 of part 1.
+
−
** Will work on second document with testing infrastructure and removed content from the TPTP Test Strategy.
+
−
** Integrate comments form comment page.
+
  
 
* Joel:  
 
* Joel:  
−
** Eclipse/TPTP and the Agent Controller is now deployed and expanded on the target platform.
+
** Working with Jonathan to configure and run the Agent Controller.
−
** Jonathan will assist Joel with configuring and running the Agent Controller.
+
*** Code is written.
−
*** Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform.
+
*** Jonathan just received legal approval for using freeSSH on Windows. Note, Cygwin cannot be used since it does not support custom environment variables on Windows.
−
**** Joel to work with Jonathan to use Cygwin. (does not include environment variables set in Cygwin, defaults to the Windows envirooemnt variables).
+
** Outstanding:
−
**** Legal approval for freeSSH.
+
*** CVS check-out of all of the test plug-ins in the <code>/test-results/&lt;project&gt;/*</code> modules.
−
**** Jonathan has the code written.
+
*** Running the correct test suite(s). For example, defining a root-level test suite for BVTs (possibly replacing the smoke tests). The test infrastructure will execute each test plug-in's <code>AllBVTTest.testsuite</code>, as referenced by the <code>org.eclipse.hyades.tests/All&lt;test type&gt;BVTTests.testsuite</code>.  We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results.  Each developer can rerun the automated tests to reproduce a failure.
−
** A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting.
+
*** Automatically posting the Test Report to <code>/tptp/test/reports/&lt;release&gt;</code>.
−
** Since there is [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests.
+
**** May require a new report generator for generating detailed test reports from a root-level execution result.
−
** Executes the Common Base Event [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.logging.core.tests/junit/?root=TPTP_Project JUnit tests].  
+
*** Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
−
** Joel will work on automatically posting the Test Report to <code>/tptp/test/reports/&lt;release&gt;</code>.
+
*** Email notifications to component leads for failing test suites.
−
** Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
+
** Issues:
−
** Joel will check out all test plugins and run root level test suite (e.g. AllJUnitBVTTests.testsuite).
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Paul and Jerome are working on a fix.
−
*** may need a new reprot format for running root level tests.  
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Duwayne is investigating.
−
** Joel will work on automate check-in of test report.
+
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 Defect 208616]: Duwayne is investigating.
−
** We will not check in test reuslts due to disk space limimtations and reduces, developer can reproduce.
+
−
** Email notiifcxaiton for failures.
+
−
** Outstanding/Issues:
+
−
# [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect]: Paul and Jerome.
+
 
# Need to import test plug-ins into workbench before running the tests.
 
# Need to import test plug-ins into workbench before running the tests.
 
* ASF provides this service (importExistingProjects task).
 
* ASF provides this service (importExistingProjects task).
Line 47: Line 37:
 
# When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne
 
# When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne
 
* File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file.
 
* File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file.
−
# Checking out a project using the CVS command, you need to know the test plug-in's name to extract to the correct direcotry.  OTherwize, it is extracted as /test-resutls/....: Joel
+
# When checking out a project using the CVS command, the name of the test plug-in is required to extract it to the correct directory.  Otherwise, the test plug-in is extracted to the <code>/test-results/&lt;project&gt;/&lt;test plug-in&gt;</code> directory.
  
 
* Paul:  
 
* Paul:  
−
** Answeer comments to draft document.
+
** Worked with Jerome on reproducing [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect 208616] when deploying TPTP JUnit plug-in tests remotely.
−
** Working on the [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect] when deploying TPTP JUnit tests remotely.
+
 
** No project leads have replied to a request to estimate the current testing costs.
 
** No project leads have replied to a request to estimate the current testing costs.
 
*** Paul will remind the TPTP Project Leads to provide a sizing.
 
*** Paul will remind the TPTP Project Leads to provide a sizing.
Line 57: Line 46:
 
== Action Items ==
 
== Action Items ==
  
−
* Alan: Complete the first draft of the first testing process document for review by November 14, 2007.
+
* All:
−
* Alan: Complete the new test process.
+
** Review the [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process Testing Process] draft document by Wednesday, November 21.
  
−
* Jonathan: Assist Joel with configuring and running the Agent Controller.
+
* Alan:  
 +
** Complete the second draft of the Testing Process (part 1) document.
 +
*** Integrate [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments].
 +
** Complete the first draft of the Testing Process (part 2) document by Wednesday, November 21.
 +
*** Includes the remaining sections of the existing TPTP Testing Strategy document, and [http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Common_Test_Infrastructure Common Test Infrastructure].
 +
 +
* Joel:
 +
** Prepare a small discussion on the architecture of the test infrastructure for next week's call.
 +
** Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
 +
** Finish the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).
  
−
* Joel: Prepare a small discussion on the architecture of the test infrastructure for next week's call.
+
* Paul:  
−
* Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
+
** Answer [http://wiki.eclipse.org/Testing_Process_Part_1_Draft_1_Comments review comments] by Tuesday, November 20.  
−
* Joel: Create a small slide deck for demoing the PoC to the PMC by November 14, 2007.
+
** Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
−
* Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard. 
+
** Encourage Project Leads to automate existing manual test suites.
−
* Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
+
** Book another meeting for next week (November 26, 2007).
−
 
+
** Work on [https://bugs.eclipse.org/bugs/show_bug.cgi?id=208616 defect 208616] when deploying TPTP JUnit plug-in tests remotely.
−
* Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
+
−
* Paul: Encourage Project Leads to automate existing manual test suites.
+
−
* Paul: Book another meeting for next week (November 19, 2007).
+
−
* Paul: Fix defect when deploying TPTP JUnit tests remotely.
+
  
 
== Upcoming Discussion ==
 
== Upcoming Discussion ==

Revision as of 13:57, 19 November 2007

Attendees

  • Present:
    • Paul
    • Joel
    • Alan
    • Kiryl
    • Jonathan
  • Absent:
    • Joanna

Deliverables

  • Alan:
    • Delivered the first draft of the Testing Process (part 1) document.
  • Joel:
    • Working with Jonathan to configure and run the Agent Controller.
      • Code is written.
      • Jonathan just received legal approval for using freeSSH on Windows. Note, Cygwin cannot be used since it does not support custom environment variables on Windows.
    • Outstanding:
      • CVS check-out of all of the test plug-ins in the /test-results/<project>/* modules.
      • Running the correct test suite(s). For example, defining a root-level test suite for BVTs (possibly replacing the smoke tests). The test infrastructure will execute each test plug-in's AllBVTTest.testsuite, as referenced by the org.eclipse.hyades.tests/All<test type>BVTTests.testsuite. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure.
      • Automatically posting the Test Report to /tptp/test/reports/<release>.
        • May require a new report generator for generating detailed test reports from a root-level execution result.
      • Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
      • Email notifications to component leads for failing test suites.
    • Issues:
  1. Defect 208616: Paul and Jerome are working on a fix.
  2. Defect 208616: Duwayne is investigating.
  3. Defect 208616: Duwayne is investigating.
  4. Need to import test plug-ins into workbench before running the tests.
  • ASF provides this service (importExistingProjects task).
  • Exception.
  • Working with Duwayne
  1. When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne
  • File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file.
  1. When checking out a project using the CVS command, the name of the test plug-in is required to extract it to the correct directory. Otherwise, the test plug-in is extracted to the /test-results/<project>/<test plug-in> directory.
  • Paul:
    • Worked with Jerome on reproducing defect 208616 when deploying TPTP JUnit plug-in tests remotely.
    • No project leads have replied to a request to estimate the current testing costs.
      • Paul will remind the TPTP Project Leads to provide a sizing.

Action Items

  • Alan:
    • Complete the second draft of the Testing Process (part 1) document.
    • Complete the first draft of the Testing Process (part 2) document by Wednesday, November 21.
  • Joel:
    • Prepare a small discussion on the architecture of the test infrastructure for next week's call.
    • Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
    • Finish the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).
  • Paul:
    • Answer review comments by Tuesday, November 20.
    • Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
    • Encourage Project Leads to automate existing manual test suites.
    • Book another meeting for next week (November 26, 2007).
    • Work on defect 208616 when deploying TPTP JUnit plug-in tests remotely.

Upcoming Discussion

Retrieved from "https://wiki.eclipse.org/index.php?title=TPTP-Test-Automation-20071119&oldid=62225"

Back to the top