Difference between revisions of "How to test TPTP with line level code coverage enabled"

Revision as of 07:51, 30 October 2007

Background

EMMA is an open-source toolkit written in Java for measuring and reporting line level code coverage for Java applications.

TPTP is currently conducting a Proof of Concept (PoC) to evaluate EMMA for generating and reporting line level code coverage statistics when testing TPTP.

Download

The PoC requires EMMA v2.0.5312.

Instrumentation

To generate line level code coverage statistics, the TPTP Software Development Kit (SDK) is statically instrumented using EMMA.

The Release Engineering Team has integrated the instrumentation with the TPTP build process to generate statically instrumented TPTP SDK ZIP for testing. The instrumented TPTP SDK ZIP are available for all candidate builds from the TPTP download site (see the 'EMMA Instrumented' link).

To manually generate statically instrumented TPTP plug-ins:

• JARed plug-ins:

   java -cp emma.jar emma instr -instrpath org.eclipse.tptp.*.jar -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*

• JARs in plug-ins:

   java -cp %EMMA_JAR% emma instr -instrpath org.eclipse.tptp.*\*.jar  -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*

where

• org.eclipse.tptp.*.jar: TPTP JARed plug-in.
• org.eclipse.tptp.*\*.jar: TPTP JARs in plug-ins.
• EMMA_Coverage.metadata: EMMA metadata file.

Note: Directory paths are omitted.

When manually generating a statically instrumented TPTP SDK ZIP, an EMMA metadata file is generated. This file describes the instruments classes, used when generating EMMA coverage reports. The EMMA metadata file for instrumented TPTP SDK ZIP is available from the TPTP download site (see the 'EMMA Metadata' link).

For more information, see the EMMA instructions.

Running

Before running the statically instrumented TPTP SDK, the EMMA JAR (emma.jar) must be added to the default environment of both the Agent Controller and Integrated Agent Controller. This is easily done by adding a <Variable> to the <AgentControllerEnvironment> in <Agent Controller installation directory>\config\serviceconfig.xml (Agent Controller) and <Eclipse installation directory>\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_<build ID>\agent_controller\config\serviceconfig.xml (Integrated Agent Controller). For example:

   <Variable name="CLASSPATH" position="append" value="emma.jar"/>

where

• emma.jar: EMMA JAR file path.

When running the statically instrumented TPTP SDK, the EMMA JAR (emma.jar) must be added to the runtime classpath of each plug-in. This is easily done by starting Eclipse in development mode (-dev). For example:

   eclipse -dev emma.jar

where

• emma.jar: EMMA JAR file path.

After running the statically instrumented TPTP SDK and/or Integrated Agent Controller, one or more EMMA coverage files (coverage.ec) are generated. These files contain the raw line level code coverage statistics used when generating EMMA coverage reports. Each time Eclipse and/or the Integrated Agent Controller is started, the line level code coverage statistics for that instance are appended to the associated EMMA coverage file. The default location of these EMMA coverage files is:

• Eclipse: Eclipse home directory (ECLIPSE_HOME).
• Integrated Agent Controller: User's temporary directory (user.dir JVM system property).

Note: Currently, the Agent Controller is not statically instrumented so it does not generate a EMMA coverage file.

To override the default location of the EMMA coverage file generated by Eclipse, Eclipse can be started with the following environment variable:

• emma.coverage.out.file=EMMA_Coverage.data

where

• EMMA_Coverage.data: EMMA coverage file path.

This environment variable can be passed to the Eclipse executable as VM argument (-vmargs), using the -D command line option. For example:

   eclipse -vmargs -Demma.coverage.out.file=EMMA_Coverage.data

At the end of each test pass, the EMMA coverage file(s) are emailed to Paul Slauenwhite to be merged using the EMMA aggregator and reported for that test pass.

Reporting

To view the line level code coverage statistics, the line level code coverage metadata and data files created when instrumenting and running TPTP are merged and reported using EMMA.

The Test Tools Project/Release Engineering Team generates EMMA coverage reports during each test pass. The EMMA coverage reports are available from the TPTP test report site for the specific release. For example:

TPTP 4.4.0.3 EMMA Coverage Report

Note: The line level code coverage statistics in this report are generated from executing only a fraction of the test cases for the TPTP 4.4.0.3 smoke test pass.

To manually generate a EMMA coverage report:

   java -cp emma.jar emma report -input EMMA_Coverage.metadata,EMMA_Coverage.data -report html -Dreport.out.file=EMMA_Coverage_Report.html

where

• EMMA_Coverage.metadata: One or more EMMA metadata file(s).
• EMMA_Coverage.data: One or more EMMA coverage file(s).
• EMMA_Coverage_Report.html: HTML EMMA coverage report.

Note: Directory paths are omitted.

For more information, see the EMMA instructions.

Demo

To demo this PoC, we created a Windows batch script that instruments the TPTP plug-ins (org.eclipse.tptp.* and org.eclipse.hyades.* plug-ins), starts Eclipse with the necessary configuration, and generates a coverage report.

Instructions:

• Save the Windows batch script in your <Eclipse installation directory>/eclipse directory.
• Save the EMMA JAR (emma.jar) in your <Eclipse installation directory>/eclipse directory.
• Run the Windows batch script.

After execution, the <Eclipse installation directory>/eclipse/emma directory contains:

• EMMA metadata file (EMMA_Coverage.metadata).
• EMMA coverage file (EMMA_Coverage.data).
• EMMA coverage report (EMMA_Coverage_Report.html).
• Instrumentation (EMMA_Instrumentation.log) and report (EMMA_Report.log) logs.