Difference between revisions of "How to test TPTP with line level code coverage enabled"

Revision as of 14:44, 29 October 2007

Background

EMMA is an open-source toolkit written in Java for measuring and reporting line level code coverage for Java applications.

TPTP is currently conducting a Proof of Concept (PoC) to evaluate EMMA for generating and reporting line level code coverage statistics when testing TPTP.

Download

The PoC requires EMMA v2.0.5312.

Instrumentation

To generate line level code coverage statistics, the TPTP Software Development Kit (SDK) is statically instrumented using EMMA.

The Release Engineering Team has integrated the instrumentation with the TPTP build process to generate statically instrumented TPTP SDK ZIP for testing. The instrumented TPTP SDK ZIP are available for all candidate builds from the TPTP download site (see the 'EMMA Instrumented' link).

To manually generate statically instrumented TPTP plug-ins:

• JARed plug-ins:

   java.exe -cp emma.jar emma instr -verbose -instrpath org.eclipse.tptp.*.jar -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*

• JARs in plug-ins:

   java.exe -cp %EMMA_JAR% emma instr -verbose -instrpath org.eclipse.tptp.*\*.jar  -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*

where

• org.eclipse.tptp.*.jar: TPTP JARed plug-in.
• org.eclipse.tptp.*\*.jar: TPTP JARs in plug-ins.
• EMMA_Coverage.metadata: EMMA metadata file.

Note: Directory paths are omitted.

When manually generating a statically instrumented TPTP SDK ZIP, an EMMA metadata file is generated. This file describes the instruments classes, used when generating EMMA coverage reports. The EMMA metadata file for instrumented TPTP SDK ZIP is available from the TPTP download site (see the 'EMMA Metadata' link).

For more information, see the EMMA instructions.

Running

To run the statically instrumented TPTP SDK, Eclipse must be started with several environment variables:

• emma.coverage.out.file=EMMA_Coverage.data
• emma.rt.control=false

where

• EMMA_Coverage.data: EMMA coverage file.

Note: Directory paths are omitted.

These environment variables can be set globally or passed to the Eclipse executable as VM argument (-vmargs), using the -D command line option. For example:

eclipse.exe -vmargs -Demma.coverage.out.file=%ECLIPSE_HOME%\emma\EMMA_Coverage.data -Dcoverage.out.merge=false -Demma.rt.control=false

In addition, the EMMA JAR (emma.jar) must be accessible on runtime classpath of each plug-in. The EMMA JAR (emma.jar) can be added to the runtime classpath of each plug-in by starting Eclipse in development mode (-vmargs). For example:

   eclipse.exe -dev emma.jar

Finally, the EMMA JAR (emma.jar) must be accessible on runtime classpath of the Agent Controller. The EMMA JAR (emma.jar) can be added to the runtime classpath of the Agent Controller by . For example:

   eclipse.exe -dev emma.jar

When manually generating a statically instrumented TPTP SDK ZIP, an EMMA coverage file is generated. This file contains the raw line level code coverage statistics used when generating EMMA coverage reports. Each time Eclipse is started, the line level code coverage statistics for that instance are appended to the EMMA coverage file.

At the end of each test pass, the EMMA coverage file(s) are emailed to Paul Slauenwhite for generating the EMMA coverage report for that test pass.

Reporting

To view the line level code coverage statistics, the line level code coverage metadata and data files created when instrumenting and running TPTP are merged and reported using EMMA.

The Test Tools Project/Release Engineering Team generates EMMA coverage reports during each test pass. The EMMA coverage reports are available from the TPTP test report site for the specific release. For example:

TPTP 4.4.0.3 EMMA Coverage Report

Note: The line level code coverage statistics in this report are generated from executing only a fraction of the test cases for the TPTP 4.4.0.3 smoke test pass.

To manually generate a EMMA coverage report:

java.exe -cp emma.jar emma report -verbose -input EMMA_Coverage.metadata,EMMA_Coverage.data -report html -Dreport.out.file=EMMA_Coverage_Report.html

where

• EMMA_Coverage.metadata: One or more EMMA metadata file(s).
• EMMA_Coverage.data: One or more EMMA coverage file(s).
• EMMA_Coverage_Report.html: HTML EMMA coverage report.

Note: Directory paths are omitted.

For more information, see the EMMA instructions.

Demo

To demo this PoC, we created a Windows batch script that instruments the TPTP plug-ins (org.eclipse.tptp.* and org.eclipse.hyades.* plug-ins), starts Eclipse with the necessary configuration, and generates a coverage report.

Instructions:

• Save the Windows batch script in your <Eclipse installation directory>/eclipse directory.
• Save the EMMA JAR (emma.jar) in your <Eclipse installation directory>/eclipse directory.
• Run the Windows batch script.

After execution, the <Eclipse installation directory>/eclipse/emma directory contains:

   EMMA metadata file (EMMA_Coverage.metadata)
   EMMA coverage file (EMMA_Coverage.data)
   EMMA coverage report (EMMA_Coverage_Report.html)
   Instrumentation (EMMA_Instrumentation.log) and report (EMMA_Report.log) logs