Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "How to test TPTP with line level code coverage enabled"
(→Running) |
(→Running) |
||
| Line 48: | Line 48: | ||
eclipse.exe -vmargs -Demma.coverage.out.file=EMMA_Coverage.data -Demma.rt.control=false | eclipse.exe -vmargs -Demma.coverage.out.file=EMMA_Coverage.data -Demma.rt.control=false | ||
| + | |||
| + | In addition, these environment variables must be added to the Agent Controller/Integrated Agent Controller's default environment. These environment variables can be added to the Agent Controller/Integrated Agent Controller's default environment by adding <code><Variable></code>s to the <code><AgentControllerEnvironment></code> in <code><Agent Controller installation directory>\config\serviceconfig.xml</code> (Agent Controller) and <code><Eclipse installation directory>\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_<build ID>\agent_controller\config\serviceconfig.xml</code> (Integrated Agent Controller). For example: | ||
| + | |||
| + | <Variable name="emma.coverage.out.file" position="append" value="EMMA_Coverage.data"/> | ||
| + | <Variable name="emma.rt.control" position="append" value="false"/> | ||
The EMMA JAR (emma.jar) must be accessible on runtime classpath of each plug-in. The EMMA JAR (emma.jar) can be added to the runtime classpath of each plug-in by starting Eclipse in development mode (<code>-vmargs</code>). For example: | The EMMA JAR (emma.jar) must be accessible on runtime classpath of each plug-in. The EMMA JAR (emma.jar) can be added to the runtime classpath of each plug-in by starting Eclipse in development mode (<code>-vmargs</code>). For example: | ||
Revision as of 06:30, 30 October 2007
Background
EMMA is an open-source toolkit written in Java for measuring and reporting line level code coverage for Java applications.
TPTP is currently conducting a Proof of Concept (PoC) to evaluate EMMA for generating and reporting line level code coverage statistics when testing TPTP.
Download
The PoC requires EMMA v2.0.5312.
Instrumentation
To generate line level code coverage statistics, the TPTP Software Development Kit (SDK) is statically instrumented using EMMA.
The Release Engineering Team has integrated the instrumentation with the TPTP build process to generate statically instrumented TPTP SDK ZIP for testing. The instrumented TPTP SDK ZIP are available for all candidate builds from the TPTP download site (see the 'EMMA Instrumented' link).
To manually generate statically instrumented TPTP plug-ins:
- JARed plug-ins:
java.exe -cp emma.jar emma instr -instrpath org.eclipse.tptp.*.jar -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*
- JARs in plug-ins:
java.exe -cp %EMMA_JAR% emma instr -instrpath org.eclipse.tptp.*\*.jar -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*
where
- org.eclipse.tptp.*.jar: TPTP JARed plug-in.
- org.eclipse.tptp.*\*.jar: TPTP JARs in plug-ins.
- EMMA_Coverage.metadata: EMMA metadata file.
Note: Directory paths are omitted.
When manually generating a statically instrumented TPTP SDK ZIP, an EMMA metadata file is generated. This file describes the instruments classes, used when generating EMMA coverage reports. The EMMA metadata file for instrumented TPTP SDK ZIP is available from the TPTP download site (see the 'EMMA Metadata' link).
For more information, see the EMMA instructions.
Running
To run the statically instrumented TPTP SDK, Eclipse must be started with several environment variables:
- emma.coverage.out.file=EMMA_Coverage.data
- emma.rt.control=false
where
- EMMA_Coverage.data: EMMA coverage file.
Note: Directory paths are omitted.
These environment variables can be passed to the Eclipse executable as VM argument (-vmargs), using the -D command line option. For example:
eclipse.exe -vmargs -Demma.coverage.out.file=EMMA_Coverage.data -Demma.rt.control=false
In addition, these environment variables must be added to the Agent Controller/Integrated Agent Controller's default environment. These environment variables can be added to the Agent Controller/Integrated Agent Controller's default environment by adding <Variable>s to the <AgentControllerEnvironment> in <Agent Controller installation directory>\config\serviceconfig.xml (Agent Controller) and <Eclipse installation directory>\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_<build ID>\agent_controller\config\serviceconfig.xml (Integrated Agent Controller). For example:
<Variable name="emma.coverage.out.file" position="append" value="EMMA_Coverage.data"/> <Variable name="emma.rt.control" position="append" value="false"/>
The EMMA JAR (emma.jar) must be accessible on runtime classpath of each plug-in. The EMMA JAR (emma.jar) can be added to the runtime classpath of each plug-in by starting Eclipse in development mode (-vmargs). For example:
eclipse.exe -dev emma.jar -vmargs -Demma.coverage.out.file=EMMA_Coverage.data -Demma.rt.control=false
In addition, the EMMA JAR (emma.jar) must be added to the Agent Controller/Integrated Agent Controller's default environment. The EMMA JAR (emma.jar) can be added to the Agent Controller/Integrated Agent Controller's default environment by adding a <Variable> to the <AgentControllerEnvironment> in <Agent Controller installation directory>\config\serviceconfig.xml (Agent Controller) and <Eclipse installation directory>\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_<build ID>\agent_controller\config\serviceconfig.xml (Integrated Agent Controller). For example:
<Variable name="CLASSPATH" position="append" value="emma.jar"/>
When manually generating a statically instrumented TPTP SDK ZIP, an EMMA coverage file is generated. This file contains the raw line level code coverage statistics used when generating EMMA coverage reports. Each time Eclipse is started, the line level code coverage statistics for that instance are appended to the EMMA coverage file.
At the end of each test pass, the EMMA coverage file(s) are emailed to Paul Slauenwhite to be merged using the EMMA aggregator and reported for that test pass.
Reporting
To view the line level code coverage statistics, the line level code coverage metadata and data files created when instrumenting and running TPTP are merged and reported using EMMA.
The Test Tools Project/Release Engineering Team generates EMMA coverage reports during each test pass. The EMMA coverage reports are available from the TPTP test report site for the specific release. For example:
TPTP 4.4.0.3 EMMA Coverage Report
Note: The line level code coverage statistics in this report are generated from executing only a fraction of the test cases for the TPTP 4.4.0.3 smoke test pass.
To manually generate a EMMA coverage report:
java.exe -cp emma.jar emma report -input EMMA_Coverage.metadata,EMMA_Coverage.data -report html -Dreport.out.file=EMMA_Coverage_Report.html
where
- EMMA_Coverage.metadata: One or more EMMA metadata file(s).
- EMMA_Coverage.data: One or more EMMA coverage file(s).
- EMMA_Coverage_Report.html: HTML EMMA coverage report.
Note: Directory paths are omitted.
For more information, see the EMMA instructions.
Demo
To demo this PoC, we created a Windows batch script that instruments the TPTP plug-ins (org.eclipse.tptp.* and org.eclipse.hyades.* plug-ins), starts Eclipse with the necessary configuration, and generates a coverage report.
Instructions:
- Save the Windows batch script in your <Eclipse installation directory>/eclipse directory.
- Save the EMMA JAR (emma.jar) in your <Eclipse installation directory>/eclipse directory.
- Run the Windows batch script.
After execution, the <Eclipse installation directory>/eclipse/emma directory contains:
- EMMA metadata file (EMMA_Coverage.metadata).
- EMMA coverage file (EMMA_Coverage.data).
- EMMA coverage report (EMMA_Coverage_Report.html).
- Instrumentation (EMMA_Instrumentation.log) and report (EMMA_Report.log) logs.