Eclipse/Testing/Session Tests

Session tests in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner. This FAQ-like document describes how the Platform/Core implemented support for running session tests in the hopes of encouraging adoption of this framework not only by all the Platform/Core team members but also by members of other Eclipse teams interested in developing their own session tests.

1. What is a session test?

   It is a test case that needs to be run isolated in a separate VM instance. Technical details: The VM running a set of test cases (the controller VM) will spawn a second instance (the target VM) for each session test case to be run. The test results are collected (by using a socket) and presented to the test runner as if the test had run in the same VM.

2. Where can I get the framework?

   Check out the org.eclipse.core.tests.harness project from dev.eclipse.org. The infrastructure for session tests is in the org.eclipse.core.tests.session package. It does not hurt mentioning that besides requiring this plug-in, you should have org.junit as a pre-requisite as well.

3. What it is the easiest way to transform my regular plug-in tests into session tests?

   Change the test suite creation to use org.eclipse.core.tests.session.SessionTestSuite instead of junit.framework.TestSuite. Like this:

   Before:

   public class MyTestCase extends TestCase {
   	...
   	public void test1() {
   		...
   	}
   	...
   	public static Test suite() {
   		return new TestSuite(MyTestCase.class);
   	}
   }
   	  

   After:

   public class MyTestCase extends TestCase {
   	...
   	public void test1() {
   		...
   	}
   	...
   	public static Test suite() {
   		return new SessionTestSuite("org.myplugin.tests", MyTestCase.class);
   	}
   }
   	  

   The plug-in id is necessary so the plug-in which the class containing the test case should be loaded from can be determined. Note hat it is possible to build an empty suite and then manually add test cases and/or test suites. When running your tests again, every test case in the suite (included those in nested test suites) will be run in a separate Eclipse session. Note also that if a session test suite contains another session test suite (or any other special types of test suite), the outer session test suite will rule.

4. Why do I get a "...SetupException: No setup descriptions found. Ensure you are specifying the path for an existing setup file..."?

   You are running your session test suite as a regular JUnit test instead of as a Plug-in JUnit test. A default setup (in other words, command line) for launching new Eclipse sessions when running session tests can be computed when running the test suite as a plug-in test, so no further configuration is necessary. It basically contains the same parameters as the ones used to launch the controller instance, with a different instance location (the -data parameter). When launching the controller instance as a regular JUnit test suite, the setup for running session tests has to be configured manually. The framework looks for a default-setup.xml file in the current directory, but a different file name and location can be specified by using the setup.file system property. If not running in Eclipse, and no setup file can be found, the execution fails with this exception. See also next FAQ.

5. Should I run my tests using a JUnit or PDE Junit launch configuration?

   Running with a PDE Junit launch configuration will allow you to test code in your workspace. Also, launching with PDE JUnit frees you from having to provide configuration parameters, since reasonable defaults will be obtained from the controlling Eclipse. If you are still interested in running your tests as a JUnit launch configuration, see the next FAQ.

6. How do I run session tests from a JUnit (not PDE JUnit) launch configuration?

   Ensure you provide a valid set of setup files. Setup files describe option sets that can be used when launching Eclipse. See question about options supported by the framework.

7. How do I run UI-oriented tests?

   Make sure you also set the application when launching the target instance to be SessionTestSuite.UI_TEST_APPLICATION:

   SessionTestSuite suite = new SessionTestSuite("org.myplugin.tests", MyTestCase.class);
   suite.setApplicationId(SessionTestSuite.UI_TEST_APPLICATION);

8. How can I see the command-line used for the Eclipse instance where my session test runs?

   Use the setup.debug boolean system property (check the question about supported options).

9. How do I debug a failing test case?

   The easy way: run the failing test case in isolation as a regular JUnit plug-in test. Ensure the command-line matches the one built by the session test framework when running it as a session test.

   The hard way: you can configure the session test to launch the target VM in debug mode using the following VM argument. Note we are using double equal signs to escape equals on a single line.

      -Dsetup.override.vmArgs=Xdebug;Xrunjdwp:transport==dt_socket,address==8000,server==y,suspend==y
   

   If you are using the setup.files mechanism (see below), then use these VM arguments intead:

      -Dsetup.files=default-setup.xml -Dsetup.options=remote_debug
   

   When the target VM is launched in debug mode, you must connect the debugger to each test individually:

   1. From the "Run" menu, select "Debug..."
   2. Select the "Remote Java Application" category on the left hand side
   3. Click the "New" button to create a new launch configuration
   4. Leave all settings as default and click "Debug"

10. How do I run performance session tests?

    Use the PerformanceSessionTestSuite instead.

11. How do I run session tests that have private instance locations?

    Use the WorkspaceSessionTestSuite instead. Every test suite has its own instance location (workspace), all test cases are run on separate sessions on the same workspace. After the last test case is run, that location is deleted.

12. I just want to try out some examples. Any pointers?

    The org.eclipse.core.tests.session.samples package has some examples. If you load the org.eclipse.core.tests.runtime and org.eclipse.core.tests.resources projects, you will find real-world examples of how to use this framework. Just look for the subclasses of SessionTestSuite.

13. What are all supported options and how to use them?

    The options for the session test framework are specified in the form of system properties. You only have to use these options if running with the default settings (the currently target JRE and Eclipse and default command line options) are not appropriate.

    • setup.options - only valid when setup files are available. Setup files describe option sets, which are collections of command-line options that can be used while running Eclipse. Examples:

      -Dsetup.options=sun_1.5,eclipse_3.1_M1,big_memory

      would run Eclipse 3.1 M1 using Sun JRE 1.5 with whatever amount of memory the big_memory option set specifies. While:

      -Dsetup.options=ibm_1.4,eclipse_3.0,no_jit

      would run Eclipse 3.0 on IBM JRE 1.4 with Just-in-Time compilation disabled

      The meaning of every option set must be specified in setup files (see below).

    • setup.files - setup files provide option set definitions for running the session tests. Setup files can be chained, so a more general setup file (used by the whole team) can be customized with a user specific setup file. This is useful since option sets must specify file system locations, and every user has its own layout for JRE and Eclipse installs. The expected use of this option is:

      -Dsetup.files=/tmp/default-setup.xml,~/user-setup.xml

      Here, the default-setup.xml file is shared by the whole team and defines all option sets that the team may use. The user-setup.xml merely overrides location sensitive options to adapt them to the user specific file system layout. Look here for an example of a default setup file and here for an example of a user custom setup file. These examples are intended to be self-explanatory and are actually useful as they are (the user-specifc setup file has to be tweaked to support your specific file system layout.

    • setup.override.eclipseArgs - allows the user to manually override command-line options corresponding to Eclipse arguments. Example:

      -Dsetup.override.eclipseArgs=data=c:/workspace;debug;showlocation

    • setup.override.vmArgs - allows the user to manually override command-line options corresponding to VM arguments (other than system properties). Example:

      -Dsetup.override.vmArgs=Xint;Xmx256m;hotspot;cp=startup.jar

    • setup.override.systemProperties - allows the user to manually override command-line options corresponding to system properties. Example:

      -Dsetup.override.systemProperties=osgi.checkConfiguration;osgi.locking=none

    • setup.debug - enables the session test framework debug mode. In this mode, among other things, the command-line for every session launched is dumped to the console, so it is useful to verify which option sets are being selected and how they map to the command line.

14. What if I find problems using the session test framework or would like to request/propose an enhancement?

    Please report them in Bugzilla against the Platform/Runtime component.