Jump to: navigation, search

Difference between revisions of "Virgo/Test"

m (Virgo Test Framework)
(Integration Tests)
 
(32 intermediate revisions by 3 users not shown)
Line 33: Line 33:
  
 
Virgo tests are implemented using JUnit and EasyMock. In addition, integration tests which require an OSGi environment use the [[#Virgo_Test_Framework|Virgo test framework]], described below.
 
Virgo tests are implemented using JUnit and EasyMock. In addition, integration tests which require an OSGi environment use the [[#Virgo_Test_Framework|Virgo test framework]], described below.
 +
 +
Note that integration tests can be run under Eclipse provided the containing git repository has been freshly built using Ant. If Eclipse rebuilds some projects, integration tests may fail with an "invalid plugin" error. The solution is to do a full build under Ant and then re-run the test under Eclipse without refreshing or cleaning projects.
  
 
== Automated Testing  ==
 
== Automated Testing  ==
Line 45: Line 47:
 
</pre>  
 
</pre>  
 
and, since this will always report "build successful", remember to open <tt>target/test-results/index.html</tt> at the end to see which tests passed and which failed.  
 
and, since this will always report "build successful", remember to open <tt>target/test-results/index.html</tt> at the end to see which tests passed and which failed.  
 +
 +
=== Running a Subset of the Tests ===
 +
 +
To run the tests in a particular project, change directory to the project directory and then issue:
 +
<pre>ant test
 +
</pre>
 +
 +
To run a single test case, change directory to the project containing the testcase (not strictly necessary, but saves time skipping all the other tests in other projects) and issue:
 +
<pre>ant -Dtestcase=fully.qualified.ClassName test
 +
</pre>
 +
 +
=== Known Issues ===
 +
Sometimes on '''Windows''' machines a number of '''kernel repository''' tests can fail with the following error:
 +
<pre> org.eclipse.virgo.kernel.deployer.core.DeploymentException: listFiles() failed for file ... </pre>
 +
 +
This is due to a limitation of the Windows FS - it is not supporting filepaths longer than 260 characters.
 +
In order to solve that issue it is recommended to either always execute the tests with ''-Dci.build=true'' option or move your kernel repository closer to the root and shorten its folder name.
 +
Here's an example. Instead of: <pre> C:\myDir\org.eclipse.virgo.kernel\<repo_content> </pre> use <pre> C:\o.e.v.k\<repo_content> </pre>
 +
That way the tests will pass and you won't have to include ''-Dci.build=true''.
  
 
== System Verification Tests  ==
 
== System Verification Tests  ==
  
SVTs are present in the Virgo Kernel System Verification Tests and the Virgo System Verification Tests repositories. Building these repositories runs the test. The latter project requires a package web server zip file to be placed in <tt>build-svt/target/artifacts/</tt> before the following ant target is run:  
+
SVTs are present in the Virgo Kernel System Verification Tests and the Virgo System Verification Tests repositories. Building these repositories runs the tests.
 +
 
 +
=== Manual Steps for Virgo System Verification Tests ===
 +
The Virgo System Verification Tests repository requires a certain amount of manual hand-holding.
 +
 
 +
You need to copy a packaged Tomcat server zip file into <tt>build-svt/target/artifacts/</tt> and then run the following ant target:  
 
<pre>ant test-svt
 
<pre>ant test-svt
</pre>  
+
</pre>
 +
 
 +
At the (successful or unsuccessful) end, some processes are typically left running: the Virgo web server ("Launcher") and/or a database ("Server"). This is convenient if you want to run some specific SVT test in Eclipse but not if you want to do other work, such as run the tests again, in which case you must kill these processes, e.g.:
 +
<pre>$ jps
 +
29925 Launcher
 +
29908 Server
 +
29955 Jps
 +
$ kill -9 29925 29908
 +
</pre>
 +
 
 
== Performance Tests  ==
 
== Performance Tests  ==
  
Line 96: Line 131:
 
== Annotations  ==
 
== Annotations  ==
  
{| width="400" border="1" cellpadding="1" cellspacing="1"
+
{| width="600" border="1" cellpadding="1" cellspacing="1"
 
|+ Test Framework Annotations  
 
|+ Test Framework Annotations  
 
|-
 
|-
! '''Annotation'''  
+
! width="10%" | '''Annotation'''  
! '''Type'''  
+
! width="5%" | '''Type'''  
! '''Purpose'''  
+
! width="50%" | '''Purpose'''  
! '''Default Value'''
+
! width="35%" | '''Default Value'''
 
|-
 
|-
 
| ConfigLocation  
 
| ConfigLocation  
Line 123: Line 158:
 
|+ Test Kernel Launch Properties  
 
|+ Test Kernel Launch Properties  
 
|-
 
|-
! '''Property name'''  
+
! width="30%"| '''Property name'''  
 
! '''Definition'''  
 
! '''Definition'''  
! '''Default value'''
+
! width="10%"| '''Default value'''
 
|-
 
|-
 
|  
 
|  
Line 139: Line 174:
 
<pre>protected final BundleContext testBundleContext = FrameworkUtil.getBundle(getClass()).getBundleContext();
 
<pre>protected final BundleContext testBundleContext = FrameworkUtil.getBundle(getClass()).getBundleContext();
 
</pre>
 
</pre>
 +
 +
= build.properties =
 +
 +
Certain test properties may be set for all the projects of a given git repository in its build.properties file. For an example, see the kernel's [http://git.eclipse.org/c/virgo/org.eclipse.virgo.kernel.git/tree/build.properties build.properties].
 +
 +
<br>
 +
 +
{| width="70%" border="1" cellpadding="1" cellspacing="1"
 +
|+ Test-related build.properties
 +
|-
 +
! width="15%"| '''Property name'''
 +
! '''Definition'''
 +
! width="25%"| '''Default value'''
 +
|-
 +
| test.forkmode
 +
| "perTest" if a new JVM is to be launched for each test class. "perBatch" if a new JVM is to be launched for each project. See "forkMode" in the [http://ant.apache.org/manual/Tasks/junit.html Ant JUnit task].
 +
| perBatch
 +
|-
 +
| test.java.dir
 +
| The directory containing the source code for a project's tests.
 +
| <project>/src/test/java
 +
|-
 +
| test.resources.dir
 +
| The directory containing resources for a project's tests.
 +
| <project>/src/test/resources
 +
|-
 +
| test.output.dir
 +
| The directory in which a project's compiled tests are stored.
 +
| <project>/target/test-classes
 +
|-
 +
| test-results.output.dir
 +
| The directory in which a project's test results are stored.
 +
| <project>/target/test-results
 +
|-
 +
| test.vm.args
 +
| JVM parameters for running the test classes. See the "jvmarg" nested element of the [http://ant.apache.org/manual/Tasks/junit.html Ant JUnit task].
 +
<br>
 +
This property is typically used to set the JVM's heap and permgen sizes and to trigger a heapdump when the heap is exhausted:
 +
<pre>
 +
test.vm.args= -Xmx1024M -XX:MaxPermSize=512M -XX:+HeapDumpOnOutOfMemoryError
 +
</pre>
 +
The property is useful for debugging tests which either cannot be run in Eclipse or which do not fail in Eclipse. The following values are then added to suspend the JVM ready to attach a debugger:
 +
<pre>
 +
test.vm.args= <as above> -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y
 +
</pre>
 +
| <the empty string>
 +
|-
 +
| ci.build
 +
| Set to "true" if the build should continue after errors are encountered. When needed, this property is passed on the Ant invocation by specifying "-Dci.build=true" and is otherwise left to default. This property is used to specify the "haltOnError" and "haltOnFailure" parameters to the [http://ant.apache.org/manual/Tasks/junit.html Ant JUnit task].
 +
| false
 +
|}
 +
 +
Virgo build drives JUnit for unit and integration tests in the "test-run" macro in [http://git.eclipse.org/c/virgo/org.eclipse.virgo.virgo-build.git/tree/common/quality.xml common/quality.xml]. The above properties parameterise the macro.
 +
 +
The properties are defaulted in [http://git.eclipse.org/c/virgo/org.eclipse.virgo.virgo-build.git/tree/common/common.properties common/common.properties] and are typically overridden for all the projects of a given git repository in the build.properties file.

Latest revision as of 08:17, 25 September 2012



Testing

Virgo has a strong emphasis on testing. Unit tests check the behaviour of individual classes or small groups of classes. Integration tests check the behaviour of groups of Virgo bundles running together in Equinox. System verification tests (SVTs), including some simple performance tests, check the behaviour of packaging builds of the kernel and web server from a user's perspective.

JUnit is used for all these types of tests. Each testcase is structured as a class with class name ending in Tests. Tests typically reside in the src/test/java folder of a project.

If you need a thorough introduction to Test Driven Development, we recommend "Growing Object-Oriented Software Guided by Tests". This book makes the important point that unless test code is as clean as the code under test, the tests will inevitably end up slowing down the whole project. Good test coverage enables changes to be made with confidence, so test code should be refactored and kept clean just like the code under test.

We find that an emphasis on testing has a strong beneficial effect on code design. Writing tests is much simpler if the code is well-structured and modular. Designing for testability encourages good structure and small modifiable pieces.

Unit Tests

Each unit test tests a small unit, typically a single class. Dependencies are usually stubbed or mocked out.

Stubs are hand-written, dummy implementations of interfaces just sufficient to simulate the behaviour of a full implementation of the interface and in some cases to check that the interface has been used correctly. Stubs for several standard OSGi interfaces are provided in the OSGi Test Stubs git repository. Other projects may supply stubs for commonly used interfaces. For example, the Quasi Framework interfaces have stubs [add link once kernel checked in] provided.

Mocks are generated on the fly from interfaces. EasyMock is typically used. Expectations of how an interface is used can be set and checked. It is important not to code expectations that are too specific to an implementation otherwise the resultant test will be fragile and is likely to break if the implementation being tested is refactored.

Some tests of complex classes use a combination of stubs and mocks.

In general, unit tests aim to provide at least 80% coverage of Virgo code, but some components fall short of this and more tests need to be written. The precommit ant target used to check coverage using Clover, but this check is not currently supported as the Eclipse Hudson server does not support Clover.

Some complex classes are simply too messy to unit test and refactoring is required to enable maintainable unit tests to be created. So if significant refactoring of existing code is planned, that is often a good time to add a suite of unit tests.

New code should be written with unit tests from the start. This is the only way to make sure that code can be thoroughly unit tested.

Integration Tests

Integration tests install a collection of bundles into Equinox and then run the testcase as part of a bundle. Some integration tests install sufficient bundles to start a Virgo kernel or web server, but others install a relatively small subset.

Virgo tests are implemented using JUnit and EasyMock. In addition, integration tests which require an OSGi environment use the Virgo test framework, described below.

Note that integration tests can be run under Eclipse provided the containing git repository has been freshly built using Ant. If Eclipse rebuilds some projects, integration tests may fail with an "invalid plugin" error. The solution is to do a full build under Ant and then re-run the test under Eclipse without refreshing or cleaning projects.

Automated Testing

Test dependencies are either checked in to src/test/resources or are downloaded into an Ivy repository when the tests are built.

The unit and integration tests of a Virgo git repository can be run by changing into the build-xxx directory and issuing:

ant clean clean-integration test

This will halt when the first test fails. To run all the tests and not halt on failure, issue:

ant -Dci.build=true clean clean-integration test

and, since this will always report "build successful", remember to open target/test-results/index.html at the end to see which tests passed and which failed.

Running a Subset of the Tests

To run the tests in a particular project, change directory to the project directory and then issue:

ant test

To run a single test case, change directory to the project containing the testcase (not strictly necessary, but saves time skipping all the other tests in other projects) and issue:

ant -Dtestcase=fully.qualified.ClassName test

Known Issues

Sometimes on Windows machines a number of kernel repository tests can fail with the following error:

 org.eclipse.virgo.kernel.deployer.core.DeploymentException: listFiles() failed for file ... 

This is due to a limitation of the Windows FS - it is not supporting filepaths longer than 260 characters. In order to solve that issue it is recommended to either always execute the tests with -Dci.build=true option or move your kernel repository closer to the root and shorten its folder name.

Here's an example. Instead of:
 C:\myDir\org.eclipse.virgo.kernel\<repo_content> 
use
 C:\o.e.v.k\<repo_content> 

That way the tests will pass and you won't have to include -Dci.build=true.

System Verification Tests

SVTs are present in the Virgo Kernel System Verification Tests and the Virgo System Verification Tests repositories. Building these repositories runs the tests.

Manual Steps for Virgo System Verification Tests

The Virgo System Verification Tests repository requires a certain amount of manual hand-holding.

You need to copy a packaged Tomcat server zip file into build-svt/target/artifacts/ and then run the following ant target:

ant test-svt

At the (successful or unsuccessful) end, some processes are typically left running: the Virgo web server ("Launcher") and/or a database ("Server"). This is convenient if you want to run some specific SVT test in Eclipse but not if you want to do other work, such as run the tests again, in which case you must kill these processes, e.g.:

$ jps
29925 Launcher
29908 Server
29955 Jps
$ kill -9 29925 29908

Performance Tests

A small number of SVTs in the Virgo Performance Tests git repository check that the performance of Virgo does not worsen. These tests are very approximate since it is impossible to enforce precise performance goals when tests are run in a general CI server which may be subject to fluctuation in its load and performance.

Ignoring Tests

Ignoring a failing test is always a last resort. If this is required, then a bug must be raised and the failing test method or, in rare situations, the whole test class should be annotated using the org.junit.Ignore annotation:

@Ignore("bug <bugzilla number>: <reason for ignoring the test>")

The bug should also detail which test or tests are ignored.

Virgo Test Framework

The Virgo Test Framework is designed to ease the construction of integration tests that require an OSGi framework. It is maintained in its own git repository. It is fairly primitive, providing just two test runners and an annotation.

Test Runners

The test framework provides the following test runners.

OsgiTestRunner

The OsgiTestRunner is for testing components that need a simple OSGi environment. It is invoked via the JUnit @RunWith class annotation, for example:

import org.junit.runner.RunWith;
import org.eclipse.virgo.test.framework.OsgiTestRunner;

@RunWith(OsgiTestRunner.class)
public class EventLogIntegrationTests { ...

OsgiTestRunner launches Equinox, installs the project containing the testcase class as a bundle, starts the bundle, and then runs the testcase class from inside the bundle using a standard JUnit runner.

DmKernelTestRunner

The DmKernelTestRunner, which should probably be renamed to VirgoKernelTestRunner, is for testing components that need a Virgo kernel environment complete with kernel and user regions. It must be used in combination with suitable configuration to launch the kernel bundles and the bundle on which the kernel depends.

It is invoked via the JUnit @RunWith class annotation, for example:

import org.junit.runner.RunWith;
import org.eclipse.virgo.test.framework.dmkernel.DmKernelTestRunner;
 
@RunWith(DmKernelTestRunner.class)
public abstract class AbstractKernelIntegrationTest { ...

The DmKernelTestRunner extends the function of the OsgiTestRunner to wait for the user region to start before running the testcase.

Annotations

Test Framework Annotations
Annotation Type Purpose Default Value
ConfigLocation class Specifies the location from which the test framework should load its configuration META-INF/test.config.properties

Test Configuration

The test configuration file specifies a list of bundles to be installed and optionally started as well as kernel and other properties. It is an extended form of the kernel launch properties configuration file.

The Virgo web layer has a fairly typical test configuration file.

The following properties may be specified which would not normally appear in a non-test kernel launch properties configuration file.


Test Kernel Launch Properties
Property name Definition Default value


org.eclipse.virgo.test.properties.include

A comma separated list of "file:" URLs relative to the current working directory each referring to a properties file. Properties files may provide property placeholders in other configuration files. Empty list

Utilities

The standard OSGi utility class FrameworkUtil is used to obtain the test bundle context as in the following example:

protected final BundleContext testBundleContext = FrameworkUtil.getBundle(getClass()).getBundleContext();

build.properties

Certain test properties may be set for all the projects of a given git repository in its build.properties file. For an example, see the kernel's build.properties.


Test-related build.properties
Property name Definition Default value
test.forkmode "perTest" if a new JVM is to be launched for each test class. "perBatch" if a new JVM is to be launched for each project. See "forkMode" in the Ant JUnit task. perBatch
test.java.dir The directory containing the source code for a project's tests. <project>/src/test/java
test.resources.dir The directory containing resources for a project's tests. <project>/src/test/resources
test.output.dir The directory in which a project's compiled tests are stored. <project>/target/test-classes
test-results.output.dir The directory in which a project's test results are stored. <project>/target/test-results
test.vm.args JVM parameters for running the test classes. See the "jvmarg" nested element of the Ant JUnit task.


This property is typically used to set the JVM's heap and permgen sizes and to trigger a heapdump when the heap is exhausted:

test.vm.args= -Xmx1024M -XX:MaxPermSize=512M -XX:+HeapDumpOnOutOfMemoryError

The property is useful for debugging tests which either cannot be run in Eclipse or which do not fail in Eclipse. The following values are then added to suspend the JVM ready to attach a debugger:

test.vm.args= <as above> -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y
<the empty string>
ci.build Set to "true" if the build should continue after errors are encountered. When needed, this property is passed on the Ant invocation by specifying "-Dci.build=true" and is otherwise left to default. This property is used to specify the "haltOnError" and "haltOnFailure" parameters to the Ant JUnit task. false

Virgo build drives JUnit for unit and integration tests in the "test-run" macro in common/quality.xml. The above properties parameterise the macro.

The properties are defaulted in common/common.properties and are typically overridden for all the projects of a given git repository in the build.properties file.