Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Eclipse/Testing/JUnit4 Changes"

(Moving to JUnit4: JUnit-4-style test suites)
 
(20 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
=Introduction=
 
=Introduction=
  
This page describes steps taken to provide support for running JUnit4-style tests within the [[Automated Testing|Eclipse Test Framework]] (ETF).  Included is a brief description of why each of the main changes was made.  More information about the development process can be found in the [https://bugs.eclipse.org/bugs/show_bug.cgi?id=153429 bug report].  A few simple [[ETF_JUnit4_Changes#Steps to Run JUnit4 Tests|steps]] are required on the part of the user to run tests which require JUnit 4.x.  As for JUnit 3.x tests, the Framework is fully backwards compatible.
+
JUnit version 4.0 was released in March 2006, and further maintenance on JUnit 3.x stopped around the same time with the 3.8.2 release. Due to its dependency on Java SE 5, JUnit 4 was not adopted by the platform test framework, but there has been strong community interest in running the Eclipse test framework with JUnit4. This page describes the enhancement of the [[Eclipse/Testing|Eclipse Test Framework]] (ETF) to support JUnit version 4.0This change is being made in the '''[[Helios]] M4''' milestone build.
  
For the duration of this page, tests that use any of the new features provided by JUnit 4.x (annotations, static imports, etc.) will be referred to as "JUnit4 tests". Tests that do not use JUnit 4's features will be referred to as "JUnit3 tests".
+
This will be a transparent change for many users of the test framework, but some tests may be affected by this and will need to react accordingly. This document outlines the steps required to either transition to JUnit4 or to continue using JUnit3 with the Eclipse Test Framework.
 +
 
 +
The org.apache.ant bundle contains the ant.jar as well as jars for optional tasks, like JUnit. Unfortunately, the manifest file for this bundle doesn't have JUnit and other in its dependency list. To enable the Ant JUnit tasks to work, the Eclipse Test Framework adds a fragment to org.apache.ant which adds the JUnit bundle to the classpath of the Ant bundle. However, since the fragment will only be bound to a '''single''' version of JUnit by OSGi, the test framework '''can only run one version of JUnit for any given invocation of the test framework'''. As a result of this limitation, clients must select either JUnit3 or JUnit4 to run their tests against.
 +
 
 +
Clients of the ETF that wish to transition to using JUnit4 may require some migration in their tests due to the change in version number, and some minor incompatibilities between versions of JUnit. Clients that wish to remain on JUnit3 will need to either ''remove JUnit4'' from their test environment, or run with JDK 1.4 which will cause JUnit4 to be ignored by the OSGi runtime.
  
 
=Test framework changes =
 
=Test framework changes =
Line 11: Line 15:
 
org.junit4 is now "empty", and simply re-exports the packages exported by the org.junit version 4.x bundle.
 
org.junit4 is now "empty", and simply re-exports the packages exported by the org.junit version 4.x bundle.
  
==JUnit Bundle Configuration==
+
== Widened test framework version ranges ==
With the JUnit 3 and 4 bundles both named org.junit, we run into a problem of backwards compatibility. Let's say that our user would like to run his/her tests that depend on org.junit and specify a version less than 4.0.0. This user has also decided to run with a Java 1.5 VM. So now the ETR will attempt to run JUnit 3 tests with the org.junit version 4.x bundle resolved. This shouldn't be a problem as JUnit4 is backwards compatible, except that the user's bundle requires JUnit <4. So we run into an issue of org.junit packages being equal but not identical; ETR is using junit.framework version 4.x but the user's bundle is bound to junit.framework version 3.x. There are two solutions to this: have everyone change their dependencies, or uninstall JUnit4 if a dependency upon JUnit3 is specified. The latter was chosen.
+
 
 +
Test framework bundles have had their dependency on JUnit widened to include both JUnit3 and JUnit4. This will allow the framework to run with either version of JUnit (but not both at once).
 +
 
 +
== Making org.junit a singleton ==
 +
 
 +
Having multiple versions of JUnit running in the same framework instance can easily result in problems when some bundles resolve against JUnit3 and others resolve against JUnit4. We are considering making org.junit a singleton to avoid this kind of runtime error from happening. See {{bug|296104}} for more details and discussion on this proposed change. This change isn't strictly required for the test framework to add support for JUnit4.
 +
 
 +
Update: It was decided not to make these bundles singleton, as it could potentially cause other issues in future, as discussed in in {{bug|296104}}.
 +
 
 +
= Moving to JUnit4 =
 +
 
 +
This section describes steps required by clients who want to switch to using JUnit4 in their tests.
 +
 
 +
Some test bundles will run just fine with either JUnit3 or JUnit4. However in some cases tests have created dependencies on one version of JUnit or another (either deliberately or not). Here are some of the kinds of changes that tests currently running on JUnit3 may have to make in order to run on JUnit4. Feel free to add entries to this section if you encounter other issues when migrating from JUnit3 to JUnit4.
 +
 
 +
== Version ranges ==
 +
 
 +
Some test bundles express a narrow version range on org.junit 3.x, such as "[3.8.0,4.0.0)". Such test bundles won't resolve against JUnit4, and hence require updating the version range. Since JUnit is not an eclipse.org project and doesn't necessarily follow our version evolution semantics, the safest approach is to only specify the lower bound dependency. If you still want to be able to run with JUnit3, use a range of "3.8.0". If you only want to run on JUnit4, you can use a range of "4.8.1".
 +
 
 +
== Dependencies on the shape of the JDT feature ==
 +
 
 +
Some tests make assumptions about the contents of the JDT feature - the number of bundles, the version number of the bundle "org.junit", etc. Such tests need to be updated due to the changes in the JDT feature (new bundle "org.junit" with a 4.x version). Typically these tests are in PDE and JDT and are unlikely to be an issue with tests outside the Eclipse SDK.
 +
 
 +
== Custom subclasses of junit.framework.TestSuite ==
 +
 
 +
Some of the test runners in JUnit4 no longer call TestSuite.tests() to obtain the list of tests. If you have a subclass of TestSuite that overrides the tests() method, you will likely be broken by the move to JUnit4. The test runner in JUnit4 instead calls TestSuite.run(TestResult). If you were previously overriding TestSuite.tests(), you will now also need to override TestSuite.run(TestResult) to make sure your extra tests are called.
 +
 
 +
== Custom subclasses of junit.framework.TestResult ==
 +
 
 +
The JUnit class TestResult changed several of its protected fields from Vector to List. This is a breaking change for any subclass of TestResult that accessed those fields. Such classes will need to update from calling Vector methods to calling List methods. Luckily since Vector implements List, after you make this change you will be able to run on both JUnit3 and JUnit4.
 +
 
 +
== JUnit-4-style test suites ==
 +
If you start converting your JUnit-3-style test suites with a <code>suite()</code> method into [https://github.com/junit-team/junit/wiki/Aggregating-tests-in-suites JUnit-4-style] <code>@RunWith(Suite.class)</code> classes, you may run into [https://github.com/junit-team/junit/issues/1189 issue 1189] if one of the included suite classes is a JUnit-3-style class with a <code>suite()</code> method.
 +
 
 +
The problem is that e.g. the EclipseTestRunner wraps tests in a JUnit4TestAdapter to run them as JUnit-3-style tests. To work around the cited bug, add the <code>@RunWith(org.junit.runners.AllTests.class)</code> annotation to the class that declares the <code>suite()</code> method.
 +
 
 +
= Remaining on JUnit3 =
  
==Loading Tests==
+
Clients that wish to remain with JUnit3 for their tests may do so. However, such clients will need to ensure JUnit4 is not present in their test environment while their tests are running. There are three options for continuing to run tests with JUnit 3:
 +
# Run with a version of the Eclipse test framework from Helios M3 or earlier. Earlier releases of the test framework supported only JUnit3 so clients can continue to use these older versions. This should continue to work for testing throughout the Helios release cycle, but old versions of the test framework may not continue to work with future releases of the platform.
 +
# Run tests with JDK 1.4. This will cause the JUnit4 bundle to be ignored by OSGi and all bundles will be bound to JUnit3.
 +
# Remove the org.junit version 4.x bundle from your test environment completely. This will cause all bundles to resolve and run against JUnit3.
  
= Changes required by test bundles =
+
Clients wishing to continue using JUnit3 should also restrict their org.junit version range to exclude JUnit 4.0.0, to avoid accidentally referencing JUnit4 API in their tests.
= Using the test framework with JUnit3 or Junit4 =
+
== Steps to run with JUnit4 ==
+
# Run the Eclipse Test Framework with a Java 1.5+ VM
+
# Ensure that you specify that your bundle requires org.junit version 4.0.0+, either in the bundle dependencies or package imports.
+
  
== Steps to run with JUnit 3 ==
+
= References =
  
# Run the Eclipse Test Framework with JDK 1.4
+
* Complete details on the move to JUnit 4 can be found in {{bug|153429}}.
# Ensure that you specify that your bundle requires org.junit version less than 4.0.0, either in the bundle dependencies or package imports.
+
* [http://sourceforge.net/projects/junit/files/junit/ JUnit releases]

Latest revision as of 11:25, 12 August 2015

Introduction

JUnit version 4.0 was released in March 2006, and further maintenance on JUnit 3.x stopped around the same time with the 3.8.2 release. Due to its dependency on Java SE 5, JUnit 4 was not adopted by the platform test framework, but there has been strong community interest in running the Eclipse test framework with JUnit4. This page describes the enhancement of the Eclipse Test Framework (ETF) to support JUnit version 4.0. This change is being made in the Helios M4 milestone build.

This will be a transparent change for many users of the test framework, but some tests may be affected by this and will need to react accordingly. This document outlines the steps required to either transition to JUnit4 or to continue using JUnit3 with the Eclipse Test Framework.

The org.apache.ant bundle contains the ant.jar as well as jars for optional tasks, like JUnit. Unfortunately, the manifest file for this bundle doesn't have JUnit and other in its dependency list. To enable the Ant JUnit tasks to work, the Eclipse Test Framework adds a fragment to org.apache.ant which adds the JUnit bundle to the classpath of the Ant bundle. However, since the fragment will only be bound to a single version of JUnit by OSGi, the test framework can only run one version of JUnit for any given invocation of the test framework. As a result of this limitation, clients must select either JUnit3 or JUnit4 to run their tests against.

Clients of the ETF that wish to transition to using JUnit4 may require some migration in their tests due to the change in version number, and some minor incompatibilities between versions of JUnit. Clients that wish to remain on JUnit3 will need to either remove JUnit4 from their test environment, or run with JDK 1.4 which will cause JUnit4 to be ignored by the OSGi runtime.

Test framework changes

JUnit4 Bundle Rename

Because we wanted to retain backwards compatibility of the Framework, having the org.eclipse.test bundle depend on org.junit4 simply wasn't an option. This would mean that users running with a JVM <1.5 would not be able to resolve the org.eclipse.test bundle. So it was decided that by renaming the JUnit4 bundle to "org.junit" (same as the JUnit3 bundle name), the EclipseTestRunner (ETR) could just run with the highest version of org.junit that gets resolved.

org.junit4 is now "empty", and simply re-exports the packages exported by the org.junit version 4.x bundle.

Widened test framework version ranges

Test framework bundles have had their dependency on JUnit widened to include both JUnit3 and JUnit4. This will allow the framework to run with either version of JUnit (but not both at once).

Making org.junit a singleton

Having multiple versions of JUnit running in the same framework instance can easily result in problems when some bundles resolve against JUnit3 and others resolve against JUnit4. We are considering making org.junit a singleton to avoid this kind of runtime error from happening. See bug 296104 for more details and discussion on this proposed change. This change isn't strictly required for the test framework to add support for JUnit4.

Update: It was decided not to make these bundles singleton, as it could potentially cause other issues in future, as discussed in in bug 296104.

Moving to JUnit4

This section describes steps required by clients who want to switch to using JUnit4 in their tests.

Some test bundles will run just fine with either JUnit3 or JUnit4. However in some cases tests have created dependencies on one version of JUnit or another (either deliberately or not). Here are some of the kinds of changes that tests currently running on JUnit3 may have to make in order to run on JUnit4. Feel free to add entries to this section if you encounter other issues when migrating from JUnit3 to JUnit4.

Version ranges

Some test bundles express a narrow version range on org.junit 3.x, such as "[3.8.0,4.0.0)". Such test bundles won't resolve against JUnit4, and hence require updating the version range. Since JUnit is not an eclipse.org project and doesn't necessarily follow our version evolution semantics, the safest approach is to only specify the lower bound dependency. If you still want to be able to run with JUnit3, use a range of "3.8.0". If you only want to run on JUnit4, you can use a range of "4.8.1".

Dependencies on the shape of the JDT feature

Some tests make assumptions about the contents of the JDT feature - the number of bundles, the version number of the bundle "org.junit", etc. Such tests need to be updated due to the changes in the JDT feature (new bundle "org.junit" with a 4.x version). Typically these tests are in PDE and JDT and are unlikely to be an issue with tests outside the Eclipse SDK.

Custom subclasses of junit.framework.TestSuite

Some of the test runners in JUnit4 no longer call TestSuite.tests() to obtain the list of tests. If you have a subclass of TestSuite that overrides the tests() method, you will likely be broken by the move to JUnit4. The test runner in JUnit4 instead calls TestSuite.run(TestResult). If you were previously overriding TestSuite.tests(), you will now also need to override TestSuite.run(TestResult) to make sure your extra tests are called.

Custom subclasses of junit.framework.TestResult

The JUnit class TestResult changed several of its protected fields from Vector to List. This is a breaking change for any subclass of TestResult that accessed those fields. Such classes will need to update from calling Vector methods to calling List methods. Luckily since Vector implements List, after you make this change you will be able to run on both JUnit3 and JUnit4.

JUnit-4-style test suites

If you start converting your JUnit-3-style test suites with a suite() method into JUnit-4-style @RunWith(Suite.class) classes, you may run into issue 1189 if one of the included suite classes is a JUnit-3-style class with a suite() method.

The problem is that e.g. the EclipseTestRunner wraps tests in a JUnit4TestAdapter to run them as JUnit-3-style tests. To work around the cited bug, add the @RunWith(org.junit.runners.AllTests.class) annotation to the class that declares the suite() method.

Remaining on JUnit3

Clients that wish to remain with JUnit3 for their tests may do so. However, such clients will need to ensure JUnit4 is not present in their test environment while their tests are running. There are three options for continuing to run tests with JUnit 3:

  1. Run with a version of the Eclipse test framework from Helios M3 or earlier. Earlier releases of the test framework supported only JUnit3 so clients can continue to use these older versions. This should continue to work for testing throughout the Helios release cycle, but old versions of the test framework may not continue to work with future releases of the platform.
  2. Run tests with JDK 1.4. This will cause the JUnit4 bundle to be ignored by OSGi and all bundles will be bound to JUnit3.
  3. Remove the org.junit version 4.x bundle from your test environment completely. This will cause all bundles to resolve and run against JUnit3.

Clients wishing to continue using JUnit3 should also restrict their org.junit version range to exclude JUnit 4.0.0, to avoid accidentally referencing JUnit4 API in their tests.

References

Back to the top