Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "TPTP-Test-Automation-20071119"

(→‎Deliverables)
(→‎Testing Process)
Line 11: Line 11:
 
** Joanna
 
** Joanna
  
−
== Testing Process ==
 
  
−
* A very lightweight testing process for TPTP.
 
−
* Extends the existing TPTP Testing Strategy.
 
−
* Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
 
−
* Two (or more) documents:
 
−
** 1) High steps to use the testing infrastructure intended for first time users.  This document will reference the other document(s).
 
−
** 2) Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
 
−
** 3) Additional documents detailing the specifics of a testing topis. 
 
−
 
−
* Discussion:
 
−
** Naming conventions for root level test suites and execution results should include test plug-in ID.
 
−
** Execution results need to be saved in a directory with a meaningful name (e.g. OS, JRE, type (full, smoke, BVT)).
 
−
** Test suites for specific JREs and OSes (e.g. JVMPI versus JVMTI) will not always run on the reference platform(s).
 
−
** The existing technical content will remain unchanged, aside from review changes.
 
−
** Need sections for:
 
−
*** Getting tests.
 
−
*** Creating tests.
 
−
*** Running tests.
 
−
*** Save test results.
 
−
*** Reporting results.
 
−
*** BVTs.
 
−
** The Agent Controller and profiler test suites need to migrated to use the TPTP Test Tools instead of requiring a custom configuration.
 
−
** The average tester/reader does not need to know about the technical details, but the steps to perform a certain testing task.
 
  
 
== Reference Platform ==
 
== Reference Platform ==

Revision as of 13:15, 19 November 2007

Attendees

  • Present:
    • Paul
    • Joel
    • Alan
    • Kiryl
    • Jonathan
  • Absent:
    • Joanna


Reference Platform

  • Target: IBM Java 1.5 (latest SR) and Windows XP/x86.
  • Developers are expected to unit test on reference platform before check-in.
  • Nightly builds invoke automated test execution on reference platform.
  • Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
  • Move to Java 6 once a complete test pass is completed to expose initial defects.
  • The dedicate test machine for the reference platform has been clean and does not need to be re-imaged.
  • Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.

Common Test Infrastructure

  • ASF integrated with the TPTP builds (invoked via ANT):
    • Invoke tests.
    • Interrogate results.
    • Generate test reports.
  • TPTP test types:
    • AGR for functional UI testing.
    • TPTP JUnit and TPTP JUnit Plug-in test for unit testing.
  • TPTP build infrastructure:
    • Extract test suites from CVS.
    • Provision reference platform (Agent Controller and Eclipse/TPTP).
    • Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
    • Clean-up reference platform.
    • Post test reports (test and code coverage) on TPTP web site.
  • See the diagram of this infrastructure.
  • Limitations/Issues:
    • Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
    • Intel needs to inform IBM that their portion of the build is complete (see defect 200351).
  • Discussion:
    • The test infrastructure will execute each test plug-in's AllTest.testsuite, as referenced by the org.eclipse.hyades.tests/All<test type>Tests.testsuite.
    • Need to check out the test plug-ins in the /test-results/<sub-project>/ directory.

Deliverables

  • All:
    • Review draft of part 1.
  • Alan:
    • Delivered draft of part 1.
    • Working on draft 2 of part 1.
    • Will work on second document with testing infrastructure and removed content from the TPTP Test Strategy.
    • Integrate comments form comment page.
  • Joel:
    • Eclipse/TPTP and the Agent Controller is now deployed and expanded on the target platform.
    • Jonathan will assist Joel with configuring and running the Agent Controller.
      • Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform.
        • Joel to work with Jonathan to use Cygwin. (does not include environment variables set in Cygwin, defaults to the Windows envirooemnt variables).
        • Legal approval for freeSSH.
        • Jonathan has the code written.
    • A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting.
    • Since there is defect when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests.
    • Executes the Common Base Event JUnit tests.
    • Joel will work on automatically posting the Test Report to /tptp/test/reports/<release>.
    • Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
    • Joel will check out all test plugins and run root level test suite (e.g. AllJUnitBVTTests.testsuite).
      • may need a new reprot format for running root level tests.
    • Joel will work on automate check-in of test report.
    • We will not check in test reuslts due to disk space limimtations and reduces, developer can reproduce.
    • Email notiifcxaiton for failures.
    • Outstanding/Issues:
  1. defect: Paul and Jerome.
  2. Need to import test plug-ins into workbench before running the tests.
  • ASF provides this service (importExistingProjects task).
  • Exception.
  • Working with Duwayne
  1. When generating execution results when the test suite path does not have a tailing slash (/folder/filename/).: Duwayne
  • File will be created in the wrong location and results interrogation will throw an exception because ASF cannot find the file.
  1. Checking out a project using the CVS command, you need to know the test plug-in's name to extract to the correct direcotry. OTherwize, it is extracted as /test-resutls/....: Joel
  • Paul:
    • Answeer comments to draft document.
    • Working on the defect when deploying TPTP JUnit tests remotely.
    • No project leads have replied to a request to estimate the current testing costs.
      • Paul will remind the TPTP Project Leads to provide a sizing.

Action Items

  • Alan: Complete the first draft of the first testing process document for review by November 14, 2007.
  • Alan: Complete the new test process.
  • Jonathan: Assist Joel with configuring and running the Agent Controller.
  • Joel: Prepare a small discussion on the architecture of the test infrastructure for next week's call.
  • Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
  • Joel: Create a small slide deck for demoing the PoC to the PMC by November 14, 2007.
  • Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard.
  • Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
  • Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
  • Paul: Encourage Project Leads to automate existing manual test suites.
  • Paul: Book another meeting for next week (November 19, 2007).
  • Paul: Fix defect when deploying TPTP JUnit tests remotely.

Upcoming Discussion

References

Retrieved from "https://wiki.eclipse.org/index.php?title=TPTP-Test-Automation-20071119&oldid=62200"

Back to the top