View source for 4.5 Test Automation Initiative

== Members ==

* Paul Slauenwhite (IBM)
* Alan Haggarty (IBM)
* Joel Cayne (IBM)
* Jonathan West (IBM)
* Joanna Kubasta (IBM)
* Kiryl Kazakevich (Intel)

== Goals ==
 
* Consolidation of our testing process and infrastructure.
* Consolidation of custom test frameworks (Test Dashboard, AC/Profiler test server, variants).
* Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.).
* Automatic test execution and report generation with every build (BVT).
* Integrate EMMA with the build and test process for generating code coverage reports for all manual and automated testing.

== Benefits == 

* Specialization of testing skills (e.g. builds). 
* Provide an infrastructure that makes it easy for committers to contribute automated test cases/suites.
* Manual labor savings by using automated tests, which are automatically executed with every build.
* Decrease time/cost to resolve defects since uncovered earlier in the code-test-build cycle.
* Decrease testing overlap by a) localizing test executing and reporting to one group and b) using code coverage statistics.

== Delivery dates ==

* November 14:
** First draft of the first testing process document for review.
** Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds.
** Status update for the PMC.
** Measure cost savings.

* November 21:
** Review of the first testing process document.
** First draft of the remaining testing process documents for review
** Identify and resolve bugs.

* November 28:
** Final version of the the first testing process document.
** Review of the remaining testing process documents for review
** Status update including cost savings for the PMC.
** Presentation to TPTP committers.
** Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.

== Testing Process ==

* '''Summary''':
** A very lightweight testing process for TPTP.
** Extends the existing TPTP Testing Strategy.  The existing technical content will remain unchanged, aside from review changes.
** Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario. 
** Sections include
*** Getting tests.
*** Creating tests.
*** Running tests.
*** Save test results.
*** Reporting results.
*** BVTs.
** Two (or more) documents:
# High steps to use the testing infrastructure intended for first time users.  This document will reference the other document(s).
# Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
# Additional documents detailing the specifics of a testing topics.  

* '''Draft Documents''':
** Part 1:
***  [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1_Draft_1.doc Draft 1]:  [[Testing Process Part 1 Draft 1 Comments| Comments]]
*** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1_Draft_2.html Draft 2]:  [[Testing Process Part 1 Draft 2 Comments| Comments]]
** Part 2:
*** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_2_Draft_1.html Draft 1]:  [[Testing Process Part 2 Draft 1 Comments| Comments]]


* '''Final Documents''':
** Part 1:
*** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_1.html Final]
** Part 2:
*** [http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_2.html Final]

== Common Test Infrastructure ==

* '''Summary''':
** Reference platform:
*** Developers are expected to unit test on reference platform before checking-in code to CVS.
*** Target: IBM Java 1.5 (latest SR) and Windows XP/x86.
*** Goal to expand to a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
*** IBM JREs for Windows and Linux (see [http://www.ibm.com/developerworks/java/jdk/eclipse/index.html IBM Development Package for Eclipse]) are publicly available from [http://www.ibm.com/developerworks/java/jdk/ developerWorks].
*** Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.
** ASF integrated with the TPTP builds by invoking the automated tests on reference platform via ANT:
*** Invoke tests.
*** Interrogate results.
*** Generate test reports.
** TPTP test types:
*** AGR for functional UI testing.
*** TPTP JUnit and TPTP JUnit Plug-in test for unit testing.
** TPTP build infrastructure:
*** Extract test suites from CVS.
*** Provision reference platform (Agent Controller and Eclipse/TPTP).
*** Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
*** Clean-up reference platform.
*** Post test reports (test and code coverage) on TPTP web site.
*** Email notifications to component leads for failing test suites.
** Limitations/Issues:
*** Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
*** Intel needs to inform IBM that their portion of the build is complete (see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=200351 defect 200351]).

* '''Architecture''': [[Common Test Infrastructure Comments| Comments]]

[[Image:AsfBVT.JPG]]

* '''Demonstration''':
** [http://www.eclipse.org/tptp/home/documents/process/test/automation/infastructure/viewlet/TestAutomationInitiative_viewlet_swf.html Viewlet].

== Metrics ==

In order to evaluate this initiative, are measuring the labor costs (as accurately as possible) in Person Weeks (PW) to run:

# full test pass
# smoke test pass


* Before Initiative:

<table width="100%" border="1" cellpadding="5">
 <tr><th>Project</th><th>Full Test Pass Cost (PW)</th><th>Smoke Test Pass Cost (PW)</th></tr>
 <tr><td>Platform</td><td>5</td><td>2.5</td></tr>
 <tr><td>Test</td><td>5</td><td>2.5</td></tr>
 <tr><td>Trace</td><td>3</td><td>1.5</td></tr>
 <tr><td>Monitoring</td><td>3</td><td>1.5</td></tr>
 <tr><td><b>Total</b></td><td><b>16</b></td><td><b>8</b></td></tr>
</table>


* With Test Automation:

<table width="100%" border="1" cellpadding="5">
 <tr><th>Project</th><th>Full Test Pass Cost (PW)</th><th>Smoke Test Pass Cost (PW)</th></tr>
 <tr><td>Platform</td><td>4</td><td>2</td></tr>
 <tr><td>Test</td><td>4</td><td>1.5</td></tr>
 <tr><td>Trace</td><td>3</td><td>1.5</td></tr>
 <tr><td>Monitoring</td><td>2.5</td><td>1</td></tr>
 <tr><td><b>Total</b></td><td><b>13.5</b></td><td><b>6</b></td></tr>
</table>

== References ==

* [http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/tlaunching-tests-from-scripts-and-applications.htm ASF]
* [http://www.eclipse.org/tptp/test/documents/tutorials/eclipseWorld2006/Achieving_Continuous_Integration.html ASF Tutorial]
* [http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html TPTP Testing Strategy]
* [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928])
* [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in]

== Meeting Minutes ==

* [[TPTP-Test-Automation-20070924| September 24, 2007]]
* [[TPTP-Test-Automation-20071022| October 22, 2007]]
* [[TPTP-Test-Automation-20071106| November 6, 2007]]
* [[TPTP-Test-Automation-20071112| November 12, 2007]]
* [[TPTP-Test-Automation-20071119| November 19, 2007]]
* [[TPTP-Test-Automation-20071126| November 26, 2007]]
* [[TPTP-Test-Automation-20071129| November 29, 2007]]
* [[TPTP-Test-Automation-20071203| December 3, 2007]]