Jump to: navigation, search

Difference between revisions of "Platform-releng/Eclipse Test Framework"

(initial copy into wiki)
(Interface)
 
(20 intermediate revisions by the same user not shown)
Line 1: Line 1:
= The Eclipse Test Framework =
+
= The Eclipse Test Framework =
  
== Introduction ==
+
== Introduction ==
  
  <p>The testing framework is comprised of the org.eclipse.test
+
This document describes how to make use of Eclipse Test Framework. This framework is used by the Eclipse Project Platform to run their unit and integration tests in an automated way, as described in [[Platform-releng/Automated_Testing]], but may be used by others to automate your testing, if you'd like. Its not intended as a "delivered API" in the normal sense of other Platform deliverables, but if you find it helpful, you are welcome to reuse it.  
    plugin and the org.eclipse.ant.optional.junit fragment.</p>
+
  <p>These two projects are available from the dev.eclipse.org
+
    repository and are included in the
+
    eclipse-test-framework-&lt;buildId&gt;.zip from the eclipse.org
+
    downloads page.</p>
+
  <p>
+
== Building and Installation ==
+
  </p>
+
  <p>Since the org.eclipse.test plugin is stored in the repository
+
    in source-code form, it needs to be compiled before it can be used.
+
    The org.eclipse.ant.optional.junit fragment does not contain any
+
    source and can be used as is from the repository.</p>
+
  <ol>
+
    <li>Turn off automatic builds.
+
      Window-&gt;Preferences-&gt;Workbench and uncheck "Perform build
+
      automatically on resource modification"</li>
+
    <li>Load org.eclipse.test into your workspace.</li>
+
    <li>Right-click on the org.eclipse.test project in either the
+
      Navigator or Packages view. Select 'Rebuild Project' from the
+
      context menu. This will compile the entire org.eclipse.test
+
      plugin.</li>
+
    <li>Finally, copy the org.eclipse.test plugin into your target
+
      Eclipse.</li>
+
    <li>The org.eclipse.ant.optional.junit fragment only needs to
+
      be present in the environment of the Eclipse that is overseeing
+
      the test process. If you are running the test script from within
+
      the Workbench, this means that the fragments need to be present
+
      withinn your development Eclipse. If you are running the tests
+
      from the command line, then the fragments will need to be present
+
      in your target Eclipse.</li>
+
  </ol>
+
  <p>
+
== Setup ==
+
  </p>
+
  <p>Follow the steps given above to build and install all of the
+
    neccessary plugins and fragments. Please note that the current
+
    version of the test framework is not compatible with the PDE notion
+
    of self-hosting. If you want to run the tests, you will need to
+
    setup a full target Eclipse so that the testing framework can detect
+
    everything that is needed.</p>
+
  <p>If you are writing tests for one or more Eclipse plugins, then
+
    you should create a separate plugin for tests. The test plugin will
+
    contain all of the tests that will be run on your development
+
    plugins, as well as defining how those tests get run.</p>
+
  
  <p>If you are not writing tests for an Eclipse plugin, then you
+
Remember, if you simply wish to write and run JUnit tests interactively from an Eclipse workbench, there are easy ways to do that, and they are well described in Eclipse Help:  
    should look into using JUnit on its own to test your work. JUnit is
+
    designed for testing Java code, and by default has no knowledge of
+
    Eclipse. There are separate mechanisms for using JUnit on Java code
+
    in Eclipse. See the documentation provided here:</p>
+
  <p>
+
    <a
+
      href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-ui-home/plugins/org.eclipse.jdt.junit/index.html">
+
      JUnit Support in Eclipse</a><br> <a
+
      href="http://dev.eclipse.org:8080/help/content/help:/org.eclipse.jdt.doc.user/tasks/tasks-207.htm?resultof=%6a%75%6e%69%74%20&toc=%2Forg.eclipse.jdt.doc.user%2Ftoc.xml">
+
      Using JUnit</a>
+
  </p>
+
  
  <p>Once a test plugin has been created, you must then create an
+
* [http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.jdt.doc.user/gettingStarted/qs-junit.htm JUnit Support in Eclipse]
    Ant script that will run all of the tests. Create a file called
+
* [http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tools/launchers/junit_launcher.htm Testing Eclipse Plug-ins with JUnit]
    'test.xml' in the root of your plug-in or bundle. This is an Ant
+
    file that defines how each test is going to be run. The 'test.xml'
+
    file is like a DOS batch file that scripts the entire testing
+
    process. Typically, the 'test.xml' file will contain targets for
+
    setting up the test run, executing the tests, cleaning up
+
    afterwards, and running the entire process.</p>
+
  
== Converting existing Test Suites ==
 
  
  <p>If you are converting an existing set of tests to use the new
+
The Testing Framework described on this page is comprised of the ''org.eclipse.test'' plugin and the ''org.eclipse.ant.optional.junit'' fragment, a ''library.xml'' file, and a ''JUNIT.XSL'' file.  
    framework, the actual tests that have been written should not need
+
    much change.</p>
+
  <p>If you have tests in multiple plugins, move these to a single
+
    test plug-in for your component.</p>
+
  
  <p>Make sure that the tests are defined in a plug-in. This is
+
The plugin and fragment are available from the [http://git.eclipse.org/c/platform/eclipse.platform.releng.git/tree/bundles Eclipse Project Releng Git repository] and are included in the eclipse-test-framework-&lt;buildId&gt;.zip from each [http://download.eclipse.org/eclipse/downloads/ downloads page]. They are also available in our p2 software repository, such as at
    probably the most common cause of confusion in the entire test
+
''<nowiki>http://download.eclipse.org/eclipse/updates/4.3</nowiki>'' so you can install the test framework into an existing
    process. Your tests need to be in a plug-in so that Eclipse can find
+
eclipse platform, with a batch command, similar to the following:
    them when it tries to load them.</p>
+
  
== Creating new Test Suites ==
+
<pre><nowiki>
 +
./eclipse/eclipse -consolelog -nosplash -data workspace-installtests \
 +
  -application org.eclipse.equinox.p2.director \
 +
  -repository http://download.eclipse.org/eclipse/updates/4.3/ \
 +
  -installIU org.eclipse.test.feature.group
 +
</nowiki></pre>
  
  <p>Creating new JUnit tests for an Eclipse plugin should be no
+
The two "text" files are also available from the Git repository, and you may use these "as is", or provide your own customized versions. You can them dynamically directly from the repository using commands such as 'wget' or ant tasks such as <nowiki><get></nowiki>
    more difficult than writing standard JUnit tests. Since the
+
    framework allows tests to be run inside of a working Eclipse, any
+
    tests that you write have available to them any of the methods
+
    supplied by the Eclipse platform, provided that you add the
+
    appropriate dependencies to your tests' manifest.</p>
+
  
== Performance Issues ==
+
* [http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/plain/eclipse.platform.releng.tychoeclipsebuilder/eclipse-junit-tests/src/main/scripts/library.xml library.xml]
 
+
* [http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/plain/eclipse.platform.releng.tychoeclipsebuilder/eclipse-junit-tests/src/main/scripts/JUNIT.XSL JUNIT.XSL]
  <p>You should keep in mind the number of times that Eclipse needs
+
    to be started. Launching Eclipse has a substantial cost in terms of
+
    runtime. To minimize the number of times the platform is started,
+
    you should consider writing a TestSuite called AllTests for each of
+
    your test plugins. AllTests should invoke each of the tests that you
+
    want to run for a particular plugin. The 'test.xml' file can then
+
    run the AllTests class, which will run all of your tests, but the
+
    platform will only ever be started once for each of your test
+
    plugins.</p>
+
  <p>Note: Sometimes tests involve shutting down, restarting, and
+
    testing the state of metadata that was written to disk. These
+
    session tests will require Eclipse to be launched several times in
+
    sequence, which will increase the runtime of your tests. This cannot
+
    be avoided.</p>
+
  <p>&nbsp;</p>
+
  
== Running the Test Suite from the UI ==
+
== Building and Installation  ==
 
+
  <p>Right click on the test.xml file and select 'Run Ant...' from
+
    the pull-down menu. The Running Ant dialog box pops up. From the
+
    list of targets, select the one that runs all of your tests. If you
+
    are using the example file provided below, this target is called
+
    'Run', and will be selected by default. Hit the 'Finish' button to
+
    start the test process.</p>
+
  
== Running the Test Suite from the command line ==
 
  
  <p>When the test suites are invoked automatically, they are run
+
If you decide not to use the "pre-built" versions available on each download page, you can compile your own version of org.eclipse.test. The org.eclipse.ant.optional.junit fragment does not contain any source and can be used as is from the repository (it simply provide a "fragment" for ant to make use of junit).
    from command line. From the ${eclipse-home} directory, the command
+
 
    to use is:</p>
+
#Turn off automatic builds. Window-&gt;Preferences-&gt;Workbench and uncheck "Perform build automatically on resource modification"
 +
#Load org.eclipse.test into your workspace.
 +
#Right-click on the org.eclipse.test project in either the Navigator or Packages view. Select 'Rebuild Project' from the context menu. This will compile the entire org.eclipse.test plugin.
 +
#Finally, copy the org.eclipse.test plugin into your target Eclipse.
 +
#The org.eclipse.ant.optional.junit fragment only needs to be present in the environment of the Eclipse that is overseeing the test process. If you are running the test script from within the Workbench, this means that the fragments need to be present within your development Eclipse. If you are running the tests from the command line, then the fragments will need to be present in your target Eclipse.
 +
 
 +
== Setup  ==
 +
 
 +
Follow the steps given above to build and install all of the necessary plugins and fragments. Please note that the current version of the test framework is not compatible with the PDE notion of self-hosting. If you want to run the tests, you will need to setup a full target Eclipse so that the testing framework can detect everything that is needed.
 +
 
 +
If you are writing tests for one or more Eclipse plugins, then you should create a separate plugin for tests. The test plugin will contain all of the tests that will be run on your development plugins, as well as defining how those tests get run.
 +
 
 +
Once a test plugin has been created, you must then create an Ant script that will run all of the tests. Create a file called 'test.xml' in the root of your plug-in or bundle. This is an Ant file that defines how each test is going to be run. Typically, the 'test.xml' file will contain targets for setting up the test run, executing the tests, cleaning up afterwards, and running the entire process. Be sure it is marked to be "exported" in the build.properties file when your test bundle is built.
 +
 
 +
== Converting existing Test Suites  ==
 +
 
 +
If you are converting an existing set of tests to use the new framework, the actual tests that have been written should not need much change.
 +
 
 +
If you have tests in multiple plugins, move these to a single test plug-in for your component.
 +
 
 +
Make sure that the tests are defined in a plug-in. This is probably the most common cause of confusion in the entire test process. Your tests need to be in a plug-in so that Eclipse can find them when it tries to load them.
 +
 
 +
Also, your test plugin needs to be installed in "directory" form, not as a jar file, since the test framework needs to find the 'test.xml' file on the filesystem. Depending on how you build your tests, you need to use the ''Eclipse-BundleShape: dir'' directive in your plugin's MANIFEST.MF file or do not specify ''unpack="false"'' in your feature.xml.
 +
 
 +
== Creating new Test Suites  ==
 +
 
 +
Creating new JUnit tests for an Eclipse plugin should be no more difficult than writing standard JUnit tests. Since the framework allows tests to be run inside of a working Eclipse, any tests that you write have available to them any of the methods supplied by the Eclipse platform, provided that you add the appropriate dependencies to your tests' manifest.mf file.
 +
 
 +
== Performance Issues  ==
 +
 
 +
You should keep in mind the number of times that Eclipse needs to be started. Launching Eclipse has a substantial cost in terms of runtime. To minimize the number of times the platform is started, you should consider writing a TestSuite called AllTests for each of your test plugins. AllTests should invoke each of the tests that you want to run for a particular plugin. The 'test.xml' file can then run the AllTests class, which will run all of your tests, but the platform will only ever be started once for each of your test plugins.
 +
 
 +
Note: Sometimes tests involve shutting down, restarting, and testing the state of metadata that was written to disk. These session tests will require Eclipse to be launched several times in sequence, which will increase the runtime of your tests. This cannot be avoided.
 +
 
 +
== Running the Test Suite from the Eclipse Workbench  ==
 +
 
 +
Right click on the test.xml file and select 'Run Ant...' from the pull-down menu. The Running Ant dialog box pops up. From the list of targets, select the one that runs all of your tests. If you are using the example file provided below, this target is called 'Run', and will be selected by default. Hit the 'Finish' button to start the test process.
 +
 
 +
== Running the Test Suite from the command line  ==
 +
 
 +
When the test suites are invoked automatically, they are run from command line. From the ${eclipse-home} directory, the command to use is:
 +
 
 
   <code>java -jar
 
   <code>java -jar
    plugins\org.eclipse.equinox.launcher_&lt;version&gt;.jar
+
  plugins\org.eclipse.equinox.launcher_&lt;version&gt;.jar
    -application org.eclipse.ant.core.antRunner -buildfile
+
  -application org.eclipse.ant.core.antRunner -buildfile
    ${test-plugin-path}\test.xml -Declipse-home=${eclipse-home}
+
  ${test-plugin-path}\test.xml -Declipse-home=${eclipse-home}
    -Dos=&lt;operating system&gt; -Dws=&lt;windowing system&gt;
+
  -Dos=&lt;operating system&gt; -Dws=&lt;windowing system&gt;
    -Darch=&lt;architecture&gt;</code>
+
  -Darch=&lt;architecture&gt;</code>
 +
 
 +
Individual tests can also be invoked directly. From the ${eclipse-home} directory, use the command:
  
  <p>Individual tests can also be invoked directly. From the
 
    ${eclipse-home} directory, use the command:</p>
 
 
   <code>java -jar
 
   <code>java -jar
    plugins\org.eclipse.equinox.launcher_&lt;version&gt;.jar
+
  plugins\org.eclipse.equinox.launcher_&lt;version&gt;.jar
    -application ${launcher} -os &lt;operating system&gt; -ws
+
  -application ${launcher} -os &lt;operating system&gt; -ws
    &lt;windowing system&gt; -arch &lt;architecture&gt; -dev bin
+
  &lt;windowing system&gt; -arch &lt;architecture&gt; -dev bin
    -testpluginname ${plugin-name} -classname ${test-classname}
+
  -testpluginname ${plugin-name} -classname ${test-classname}
    formattter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,&lt;path
+
  formattter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,&lt;path
    to output file with .xml extension&gt;</code>
+
  to output file with .xml extension&gt;</code>
  <p>
+
    where ${launcher} is one of: <i>org.eclipse.test.uitestapplication</i>
+
    or <i>org.eclipse.test.coretestapplication</i> depending on whether
+
    or not you want to run your tests within an active Workbench.
+
  </p>
+
  
== Output ==
+
where ${launcher} is one of: ''org.eclipse.test.uitestapplication'' or ''org.eclipse.test.coretestapplication'' depending on whether or not you want to run your tests within an active Workbench.
  
  <p>
+
== Output  ==
    By default, output from each test run is logged as XML. For each
+
    test that is run, the output is logged into the file called
+
    ${classname}.xml. The individual XML files are located in the
+
    ${eclipse-home} directory. When the test run is finished, you can
+
    call the <b>&quot;collect&quot;</b> target in the library file,
+
    which will collect the XML into a single file. See below for an
+
    example of how to use this target correctly.
+
  </p>
+
  
== Other issues ==
+
By default, output from each test run is logged as XML. For each test that is run, the output is logged into the file called ${classname}.xml. The individual XML files are located in the ${eclipse-home} directory. When the test run is finished, you can call the '''"collect"''' target in the library file, which will collect the XML into a single file. See below for an example of how to use this target correctly.  
=== Headless Testing vs. UI testing ===
+
 
+
    Many plugin tests will not need the Workbench active in order to
+
    run. Indeed, only the minimum number of plugins needed to run the
+
    plugin being tested need to be present when testing in a target
+
    Eclipse. There are two different Ant targets provided for running
+
    Eclipse plugin tests. One target starts the entire Workbench. The
+
    other starts Eclipse with the minimum number of plugins needed. It
+
    is up to you to decide which target is most appropriate. For
+
    examples, look at the <b>&quot;ui-test&quot;</b> and <b>&quot;core-test&quot;</b>
+
    targets below.
+
  </p>
+
  
=== Disposing the Display ===
+
== Other issues  ==
  <p>Some low-level tests for the Eclipse platform take actions that
+
    are not normally possible inside of Eclipse. An example of this
+
    behaviour would be disposing the display. While this action can be
+
    performed while running the UI, it will also kill the UI for the
+
    copy of Eclipse that is running, and cause errors when the Workbench
+
    tries to shutdown. If you need to test disposing the display, or
+
    other similar actions, your tests should be running without a UI.</p>
+
  
=== Tests that are not plugins ===
+
=== Headless Testing vs. UI testing  ===
  
  <p>It is very easy to forget to define your tests inside of a
+
Many plugin tests will not need the Workbench active in order to
    plugin. If your tests will not load properly, make sure that a
+
run. Indeed, only the minimum number of plugins needed to run the
    plug-in manifest exists in your test project, and also that the
+
plugin being tested need to be present when testing in a target
    plugin is being loaded by the platform. Make sure that all of the
+
Eclipse. There are two different Ant targets provided for running
    dependencies are satisfied for your test plugin.</p>
+
Eclipse plugin tests. One target starts the entire Workbench. The
 +
other starts Eclipse with the minimum number of plugins needed. It
 +
is up to you to decide which target is most appropriate. For
 +
examples, look at the '''"ui-test"''' and '''"core-test"'''
 +
targets below.
  
== Interface ==
+
=== Disposing the Display  ===
 
+
  <p>The org.eclipse.test plugin defines many useful Ant
+
    tasks/targets to aid developers in writing their test.xml scripts.
+
    Currently, there is only Ant targets defined, which can be called
+
    using Ant's built-in &lt;ant&gt; task. To use these targets, add the
+
    following line to the top of your script, and reference the
+
    ${library-file} property when calling &lt;ant&gt;:</p>
+
  <code>&lt;property name=&quot;library-file&quot;
+
    value=&quot;${eclipse-home}/fragments/org.eclipse.test/library.xml&quot;/&gt;
+
  </code>
+
  
  <p>The targets that are defined are:</p>
+
Some low-level tests for the Eclipse platform take actions that are not normally possible inside of Eclipse. An example of this behaviour would be disposing the display. While this action can be performed while running the UI, it will also kill the UI for the copy of Eclipse that is running, and cause errors when the Workbench tries to shutdown. If you need to test disposing the display, or other similar actions, your tests should be running without a UI.
  <ul>
+
 
    <li><b>ui-test</b> - This target runs a JUnit test suite inside
+
=== Tests that are not plugins  ===
      of an Eclipse Workbench. This target is mainly for testing plugins
+
 
      that use the Eclipse UI and JFace. The output of the test pass is
+
It is very easy to forget to define your tests inside of a plugin. If your tests will not load properly, make sure that a plug-in manifest exists in your test project, and also that the plugin is being loaded by the platform. Make sure that all of the dependencies are satisfied for your test plugin.
      automatically logged in an XML file called ${classname}.xml. It
+
 
      takes four arguments:
+
== Interface  ==
      <ul>
+
 
        <li><i>data-dir</i> - The data directory of the Eclipse
+
The org.eclipse.test plugin and library.xml file defines many useful Ant tasks/targets to aid developers in writing their test.xml scripts. Currently, there is only Ant targets defined, which can be called using Ant's built-in &lt;ant&gt; task. To use these targets, your test.xml needs access to the library.xml file. Depending on your setup, you may be able to add something like the following to the top of your script, and reference the ${library-file} property when calling &lt;ant&gt;:
          that gets run</li>
+
 
        <li><i>plugin-name</i> - The name of the plugin that the
+
<code>&lt;property name="library-file" value="${basedir}/library.xml"/&gt; </code>
          test suite is defined in</li>
+
 
        <li><i>classname</i> - The name of the class that the test
+
The targets that are defined are:
          suite is defined in</li>
+
 
        <li><i>vmargs</i> - An optional argument string to pass to
+
*'''ui-test''' - This target runs a JUnit test suite inside of an Eclipse Workbench. This target is mainly for testing plugins that use the Eclipse UI and JFace. The output of the test pass is automatically logged in an XML file called ${classname}.xml. It takes four arguments:  
          the VM running the tests</li>
+
**''data-dir'' - The data directory of the Eclipse that gets run  
      </ul>
+
**''plugin-name'' - The name of the plugin that the test suite is defined in  
      <p>For example, the following code will run the test
+
**''classname'' - The name of the class that the test suite is defined in  
        org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a
+
**''vmargs'' - An optional argument string to pass to the VM running the tests
        new Eclipse workbench. It passes the string
+
 
        &quot;-Dbaz=true&quot; to the VM. The Eclipse stores its
+
(For a complete list of parameters that can be used in your 'test.xml' file see the documentation in the [http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/eclipse.platform.releng.tychoeclipsebuilder/eclipse-junit-tests/src/main/scripts/library.xml#n16 'library.xml'] file.)
        metadata in the directory &quot;data-folder&quot;.
+
 
      <pre>
+
For example, the following code will run the test org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a new Eclipse workbench. It passes the string "-Dbaz=true" to the VM. The Eclipse stores its metadata in the directory "data-folder".  
        <code>
+
 
        
+
<pre>         
&lt;ant target=&quot;ui-test&quot; antfile=&quot;${library-file}&quot; dir=&quot;${eclipse-home}&quot;&gt;
+
&lt;ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}"&gt;
   &lt;property name=&quot;data-dir&quot; value=&quot;data-folder&quot;/&gt;
+
   &lt;property name="data-dir" value="data-folder"/&gt;
   &lt;property name=&quot;plugin-name&quot; value=&quot;org.eclipse.foo&quot;/&gt;
+
   &lt;property name="plugin-name" value="org.eclipse.foo"/&gt;
   &lt;property name=&quot;classname&quot; value=&quot;org.eclipse.foo.bar.MyTest&quot;/&gt;
+
   &lt;property name="classname" value="org.eclipse.foo.bar.MyTest"/&gt;
   &lt;property name=&quot;vmargs&quot; value=&quot;-Dbaz=true&quot;/&gt;
+
   &lt;property name="vmargs" value="-Dbaz=true"/&gt;
 
&lt;/ant&gt;
 
&lt;/ant&gt;
</code>
+
</pre>
      </pre></li>
+
    <li><b>core-test</b> - This target runs a JUnit test suite
+
*'''core-test''' - This target runs a JUnit test suite inside of an IPlatformRunnable. This target is for testing plugins that use the Eclipse platform, but do not require a UI to be running. The output of the test pass is automatically logged in an XML file called ${classname}.xml. It takes four arguments:  
      inside of an IPlatformRunnable. This target is for testing plugins
+
**''data-dir'' - The data directory of the Eclipse that gets run  
      that use the Eclipse platform, but do not require a UI to be
+
**''plugin-name'' - The name of the plugin that the test suite is defined in  
      running. The output of the test pass is automatically logged in an
+
**''classname'' - The name of the class that the test suite is defined in  
      XML file called ${classname}.xml. It takes four arguments:
+
**''vmargs'' - An optional argument string to pass to the VM running the tests
      <ul>
+
        
        <li><i>data-dir</i> - The data directory of the Eclipse
+
For example, the following code will run the test
          that gets run</li>
+
org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a
        <li><i>plugin-name</i> - The name of the plugin that the
+
headless Eclipse. It passes the string "-Dbaz=true" to
          test suite is defined in</li>
+
the VM. The Eclipse stores its metadata in the directory
        <li><i>classname</i> - The name of the class that the test
+
"data-folder".
          suite is defined in</li>
+
 
        <li><i>vmargs</i> - An optional argument string to pass to
+
<pre>  
          the VM running the tests</li>
+
&lt;ant target="core-test" antfile="${library-file}" dir="${eclipse-home}"&gt;
       </ul>
+
   &lt;property name="data-dir" value="data-folder"/&gt;
      <p>For example, the following code will run the test
+
   &lt;property name="plugin-name" value="org.eclipse.foo"/&gt;
        org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a
+
   &lt;property name="classname" value="org.eclipse.foo.bar.MyTest"/&gt;
        headless Eclipse. It passes the string &quot;-Dbaz=true&quot; to
+
   &lt;property name="vmargs" value="-Dbaz=true"/&gt;
        the VM. The Eclipse stores its metadata in the directory
+
        &quot;data-folder&quot;.
+
      <pre>  
+
        <code>
+
       
+
&lt;ant target=&quot;core-test&quot; antfile=&quot;${library-file}&quot; dir=&quot;${eclipse-home}&quot;&gt;
+
   &lt;property name=&quot;data-dir&quot; value=&quot;data-folder&quot;/&gt;
+
   &lt;property name=&quot;plugin-name&quot; value=&quot;org.eclipse.foo&quot;/&gt;
+
   &lt;property name=&quot;classname&quot; value=&quot;org.eclipse.foo.bar.MyTest&quot;/&gt;
+
   &lt;property name=&quot;vmargs&quot; value=&quot;-Dbaz=true&quot;/&gt;
+
 
&lt;/ant&gt;
 
&lt;/ant&gt;
</code>
+
</pre>
</pre></li>
+
    <li><b>collect</b> - This target collects the XML files that
+
*'''collect''' - This target collects the XML files that are produced over the course of the test script. It takes two arguments:  
      are produced over the course of the test script. It takes two
+
**''includes'' - A pattern matching all XML files to be included in the test report. This argument is typically "org*.xml"
      arguments:
+
**''output-file'' - The filename where the output of the test gets stored. For the automated build process, this file should be ${pluginname}.xml, and be located in the ECLIPSE_HOME directory.  
      <ul>
+
        <li><i>includes</i> - A pattern matching all XML files to
+
For example, the following code collects all of the files
          be included in the test report. This argument is typically
+
matching the pattern "org*.xml" in the directory
          &quot;org*.xml&quot;</li>
+
${eclipse-home}, into the file named "logfile.xml".
        <li><i>output-file</i> - The filename where the output of
+
 
          the test gets stored. For the automated build process, this
+
<pre>   
          file should be ${pluginname}.xml, and be located in the
+
&lt;ant target="collect" antfile="${library-file}" dir="${eclipse-home}"&gt;
          ECLIPSE_HOME directory.</li>
+
  &lt;property name="includes" value="org*.xml"/&gt;
      </ul>
+
  &lt;property name="output-file" value="logfile.xml"/&gt;
      <p>For example, the following code collects all of the files
+
&lt;/ant&gt;
        matching the pattern &quot;org*.xml&quot; in the directory
+
</pre>
        ${eclipse-home}, into the file named &quot;logfile.xml&quot;.
+
 
      <pre>  
+
== Examples  ==
        <code>
+
  
          &lt;ant target=&quot;collect&quot;
+
The following example is the 'test.xml' file from the
          antfile=&quot;${library-file}&quot;
+
org.eclipse.jdt.ui.tests.refactoring plugin. This file controls all
          dir=&quot;${eclipse-home}&quot;&gt; &lt;property
+
of the automated testing that is done for the
          name=&quot;includes&quot; value=&quot;org*.xml&quot;/&gt;
+
org.eclipse.jdt.ui.tests.refactoring plugin. It can be run from
          &lt;property name=&quot;output-file&quot;
+
inside of Eclipse or from the command line. It is intended to serve
          value=&quot;logfile.xml&quot;/&gt; &lt;/ant&gt;
+
as a template file for testing any other plugin.
          </code>
+
      </pre></li>
+
  </ul>
+
  
== Examples ==
+
Notice that the structure of the file roughly mirrors that of a JUnit test. Targets are defined for setting up the tests, defining what needs to be done, cleaning up after the tests, and running everything in the right order.
  
  <p>Included is the 'test.xml' file from the
+
<pre>  
    org.eclipse.jdt.ui.tests.refactoring plugin. This file controls all
+
    of the automated testing that is done for the
+
    org.eclipse.jdt.ui.tests.refactoring plugin. It can be run from
+
    inside of Eclipse or from the command line. It is intended to serve
+
    as a template file for testing any other plugin.</p>
+
  <p>Notice that the structure of the file roughly mirrors that of a
+
    JUnit test. Targets are defined for setting up the tests, defining
+
    what needs to be done, cleaning up after the tests, and running
+
    everything in the right order.</p>
+
  <pre>
+
  <code>
+
   
+
 
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
 
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  
Line 319: Line 206:
 
&lt;property name="plugin-name" value="org.eclipse.jdt.ui.tests.refactoring"/&gt;
 
&lt;property name="plugin-name" value="org.eclipse.jdt.ui.tests.refactoring"/&gt;
 
&lt;property name="library-file"
 
&lt;property name="library-file"
             value="${eclipse-home}/plugins/org.eclipse.test/library.xml"/&gt;
+
             value="${basedir}/library.xml"/&gt;
  
 
&lt;!-- This target holds all initialization code that needs to be done for --&gt;
 
&lt;!-- This target holds all initialization code that needs to be done for --&gt;
Line 358: Line 245:
 
&lt;/ant&gt;
 
&lt;/ant&gt;
 
&lt;/target&gt;
 
&lt;/target&gt;
&lt;/project&gt;
+
&lt;/project&gt;
  </code>
+
</pre>
  </pre>
+
  
 +
== Known Issues  ==
  
== Known Issues ==
+
=== ECLIPSE_HOME  ===
  
=== ECLIPSE_HOME ===
+
The test suites need to know where the root of the eclipse install is on the file system (the ECLIPSE_HOME variable). However, this variable is only defined in JDT. The ${eclipse-home} property can be set to a reasonable default inside of the test.xml script. Then tests can be run from the standard Ant window, without having to specify -Declipse-home=%ECLIPSE_HOME%. If a value for ${eclipse-home} does get passed in, the default (specified in test.xml) gets overridden. The parameter is passed in by the build mechanism. For most cases, the value "${basedir}/../.." is a reasonable default.
  
  <p>The test suites need to know where the root of the eclipse
+
=== No console output  ===
    install is on the file system (the ECLIPSE_HOME variable). However,
+
    this variable is only defined in JDT. The ${eclipse-home} property
+
    can be set to a reasonable default inside of the test.xml script.
+
    Then tests can be run from the standard Ant window, without having
+
    to specify -Declipse-home=%ECLIPSE_HOME%. If a value for
+
    ${eclipse-home} does get passed in, the default (specified in
+
    test.xml) gets overridden. The parameter is passed in by the build
+
    mechanism. For most cases, the value &quot;${basedir}/../..&quot; is
+
    a reasonable default.</p>
+
  
=== No console output ===
+
When you run a TestSuite using the standard JUnit, it normally outputs a series of dots to the console so that you can track the TestSuite's progress. It is not possible to add this feature to the automated testing process at this point in time.
  
  <p>When you run a TestSuite using the standard JUnit, it normally
+
=== Ant java task on Linux  ===
    outputs a series of dots to the console so that you can track the
+
    TestSuite's progress. It is not possible to add this feature to the
+
    automated testing process at this point in time.</p>
+
  
=== Ant java task on Linux ===
+
Ant expects there to be a java executable on the system path.
 
+
Furthermore, the executable must be a real file, not a symbolic
    Ant expects there to be a java executable on the system path.
+
link. If the test framework is throwing an exception
    Furthermore, the executable must be a real file, not a symbolic
+
<code>java.io.IOException: java: not found</code>, ensure that the java executable is on your system path.
    link. If the test framework is throwing an exception
+
    <code>java.io.IOException: java: not found</code>
+
    , ensure that the java executable is on your system path.
+
 
+
=== PDE ===
+
  
  <p>The testing framework currently has no knowledge of PDE. In
+
=== PDE ===
    order to run the automated you must be running a self hosting
+
    environment with a full development and target Eclipse.</p>
+
  
 +
The testing framework currently has no knowledge of PDE. In order to run the automated you must be running a self hosting environment with a full development and target Eclipse.
  
  
 
[[Category:Eclipse_Platform_Releng| Eclipse_Platform_Releng]]
 
[[Category:Eclipse_Platform_Releng| Eclipse_Platform_Releng]]

Latest revision as of 16:42, 6 June 2013

The Eclipse Test Framework

Introduction

This document describes how to make use of Eclipse Test Framework. This framework is used by the Eclipse Project Platform to run their unit and integration tests in an automated way, as described in Platform-releng/Automated_Testing, but may be used by others to automate your testing, if you'd like. Its not intended as a "delivered API" in the normal sense of other Platform deliverables, but if you find it helpful, you are welcome to reuse it.

Remember, if you simply wish to write and run JUnit tests interactively from an Eclipse workbench, there are easy ways to do that, and they are well described in Eclipse Help:


The Testing Framework described on this page is comprised of the org.eclipse.test plugin and the org.eclipse.ant.optional.junit fragment, a library.xml file, and a JUNIT.XSL file.

The plugin and fragment are available from the Eclipse Project Releng Git repository and are included in the eclipse-test-framework-<buildId>.zip from each downloads page. They are also available in our p2 software repository, such as at http://download.eclipse.org/eclipse/updates/4.3 so you can install the test framework into an existing eclipse platform, with a batch command, similar to the following:

 ./eclipse/eclipse -consolelog -nosplash -data workspace-installtests \
   -application org.eclipse.equinox.p2.director \
   -repository http://download.eclipse.org/eclipse/updates/4.3/ \
   -installIU org.eclipse.test.feature.group

The two "text" files are also available from the Git repository, and you may use these "as is", or provide your own customized versions. You can them dynamically directly from the repository using commands such as 'wget' or ant tasks such as <get>

Building and Installation

If you decide not to use the "pre-built" versions available on each download page, you can compile your own version of org.eclipse.test. The org.eclipse.ant.optional.junit fragment does not contain any source and can be used as is from the repository (it simply provide a "fragment" for ant to make use of junit).

  1. Turn off automatic builds. Window->Preferences->Workbench and uncheck "Perform build automatically on resource modification"
  2. Load org.eclipse.test into your workspace.
  3. Right-click on the org.eclipse.test project in either the Navigator or Packages view. Select 'Rebuild Project' from the context menu. This will compile the entire org.eclipse.test plugin.
  4. Finally, copy the org.eclipse.test plugin into your target Eclipse.
  5. The org.eclipse.ant.optional.junit fragment only needs to be present in the environment of the Eclipse that is overseeing the test process. If you are running the test script from within the Workbench, this means that the fragments need to be present within your development Eclipse. If you are running the tests from the command line, then the fragments will need to be present in your target Eclipse.

Setup

Follow the steps given above to build and install all of the necessary plugins and fragments. Please note that the current version of the test framework is not compatible with the PDE notion of self-hosting. If you want to run the tests, you will need to setup a full target Eclipse so that the testing framework can detect everything that is needed.

If you are writing tests for one or more Eclipse plugins, then you should create a separate plugin for tests. The test plugin will contain all of the tests that will be run on your development plugins, as well as defining how those tests get run.

Once a test plugin has been created, you must then create an Ant script that will run all of the tests. Create a file called 'test.xml' in the root of your plug-in or bundle. This is an Ant file that defines how each test is going to be run. Typically, the 'test.xml' file will contain targets for setting up the test run, executing the tests, cleaning up afterwards, and running the entire process. Be sure it is marked to be "exported" in the build.properties file when your test bundle is built.

Converting existing Test Suites

If you are converting an existing set of tests to use the new framework, the actual tests that have been written should not need much change.

If you have tests in multiple plugins, move these to a single test plug-in for your component.

Make sure that the tests are defined in a plug-in. This is probably the most common cause of confusion in the entire test process. Your tests need to be in a plug-in so that Eclipse can find them when it tries to load them.

Also, your test plugin needs to be installed in "directory" form, not as a jar file, since the test framework needs to find the 'test.xml' file on the filesystem. Depending on how you build your tests, you need to use the Eclipse-BundleShape: dir directive in your plugin's MANIFEST.MF file or do not specify unpack="false" in your feature.xml.

Creating new Test Suites

Creating new JUnit tests for an Eclipse plugin should be no more difficult than writing standard JUnit tests. Since the framework allows tests to be run inside of a working Eclipse, any tests that you write have available to them any of the methods supplied by the Eclipse platform, provided that you add the appropriate dependencies to your tests' manifest.mf file.

Performance Issues

You should keep in mind the number of times that Eclipse needs to be started. Launching Eclipse has a substantial cost in terms of runtime. To minimize the number of times the platform is started, you should consider writing a TestSuite called AllTests for each of your test plugins. AllTests should invoke each of the tests that you want to run for a particular plugin. The 'test.xml' file can then run the AllTests class, which will run all of your tests, but the platform will only ever be started once for each of your test plugins.

Note: Sometimes tests involve shutting down, restarting, and testing the state of metadata that was written to disk. These session tests will require Eclipse to be launched several times in sequence, which will increase the runtime of your tests. This cannot be avoided.

Running the Test Suite from the Eclipse Workbench

Right click on the test.xml file and select 'Run Ant...' from the pull-down menu. The Running Ant dialog box pops up. From the list of targets, select the one that runs all of your tests. If you are using the example file provided below, this target is called 'Run', and will be selected by default. Hit the 'Finish' button to start the test process.

Running the Test Suite from the command line

When the test suites are invoked automatically, they are run from command line. From the ${eclipse-home} directory, the command to use is:

 java -jar
  plugins\org.eclipse.equinox.launcher_<version>.jar
  -application org.eclipse.ant.core.antRunner -buildfile
  ${test-plugin-path}\test.xml -Declipse-home=${eclipse-home}
  -Dos=<operating system> -Dws=<windowing system>
  -Darch=<architecture>

Individual tests can also be invoked directly. From the ${eclipse-home} directory, use the command:

 java -jar
  plugins\org.eclipse.equinox.launcher_<version>.jar
  -application ${launcher} -os <operating system> -ws
  <windowing system> -arch <architecture> -dev bin
  -testpluginname ${plugin-name} -classname ${test-classname}
  formattter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,<path
  to output file with .xml extension>

where ${launcher} is one of: org.eclipse.test.uitestapplication or org.eclipse.test.coretestapplication depending on whether or not you want to run your tests within an active Workbench.

Output

By default, output from each test run is logged as XML. For each test that is run, the output is logged into the file called ${classname}.xml. The individual XML files are located in the ${eclipse-home} directory. When the test run is finished, you can call the "collect" target in the library file, which will collect the XML into a single file. See below for an example of how to use this target correctly.

Other issues

Headless Testing vs. UI testing

Many plugin tests will not need the Workbench active in order to run. Indeed, only the minimum number of plugins needed to run the plugin being tested need to be present when testing in a target Eclipse. There are two different Ant targets provided for running Eclipse plugin tests. One target starts the entire Workbench. The other starts Eclipse with the minimum number of plugins needed. It is up to you to decide which target is most appropriate. For examples, look at the "ui-test" and "core-test" targets below.

Disposing the Display

Some low-level tests for the Eclipse platform take actions that are not normally possible inside of Eclipse. An example of this behaviour would be disposing the display. While this action can be performed while running the UI, it will also kill the UI for the copy of Eclipse that is running, and cause errors when the Workbench tries to shutdown. If you need to test disposing the display, or other similar actions, your tests should be running without a UI.

Tests that are not plugins

It is very easy to forget to define your tests inside of a plugin. If your tests will not load properly, make sure that a plug-in manifest exists in your test project, and also that the plugin is being loaded by the platform. Make sure that all of the dependencies are satisfied for your test plugin.

Interface

The org.eclipse.test plugin and library.xml file defines many useful Ant tasks/targets to aid developers in writing their test.xml scripts. Currently, there is only Ant targets defined, which can be called using Ant's built-in <ant> task. To use these targets, your test.xml needs access to the library.xml file. Depending on your setup, you may be able to add something like the following to the top of your script, and reference the ${library-file} property when calling <ant>:

<property name="library-file" value="${basedir}/library.xml"/>

The targets that are defined are:

  • ui-test - This target runs a JUnit test suite inside of an Eclipse Workbench. This target is mainly for testing plugins that use the Eclipse UI and JFace. The output of the test pass is automatically logged in an XML file called ${classname}.xml. It takes four arguments:
    • data-dir - The data directory of the Eclipse that gets run
    • plugin-name - The name of the plugin that the test suite is defined in
    • classname - The name of the class that the test suite is defined in
    • vmargs - An optional argument string to pass to the VM running the tests

(For a complete list of parameters that can be used in your 'test.xml' file see the documentation in the 'library.xml' file.)

For example, the following code will run the test org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a new Eclipse workbench. It passes the string "-Dbaz=true" to the VM. The Eclipse stores its metadata in the directory "data-folder".

        
<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
  <property name="data-dir" value="data-folder"/>
  <property name="plugin-name" value="org.eclipse.foo"/>
  <property name="classname" value="org.eclipse.foo.bar.MyTest"/>
  <property name="vmargs" value="-Dbaz=true"/>
</ant>
  • core-test - This target runs a JUnit test suite inside of an IPlatformRunnable. This target is for testing plugins that use the Eclipse platform, but do not require a UI to be running. The output of the test pass is automatically logged in an XML file called ${classname}.xml. It takes four arguments:
    • data-dir - The data directory of the Eclipse that gets run
    • plugin-name - The name of the plugin that the test suite is defined in
    • classname - The name of the class that the test suite is defined in
    • vmargs - An optional argument string to pass to the VM running the tests

For example, the following code will run the test org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a headless Eclipse. It passes the string "-Dbaz=true" to the VM. The Eclipse stores its metadata in the directory "data-folder".

 
<ant target="core-test" antfile="${library-file}" dir="${eclipse-home}">
  <property name="data-dir" value="data-folder"/>
  <property name="plugin-name" value="org.eclipse.foo"/>
  <property name="classname" value="org.eclipse.foo.bar.MyTest"/>
  <property name="vmargs" value="-Dbaz=true"/>
</ant>
  • collect - This target collects the XML files that are produced over the course of the test script. It takes two arguments:
    • includes - A pattern matching all XML files to be included in the test report. This argument is typically "org*.xml"
    • output-file - The filename where the output of the test gets stored. For the automated build process, this file should be ${pluginname}.xml, and be located in the ECLIPSE_HOME directory.

For example, the following code collects all of the files matching the pattern "org*.xml" in the directory ${eclipse-home}, into the file named "logfile.xml".

    
<ant target="collect" antfile="${library-file}" dir="${eclipse-home}"> 
  <property name="includes" value="org*.xml"/>
  <property name="output-file" value="logfile.xml"/> 
</ant>

Examples

The following example is the 'test.xml' file from the org.eclipse.jdt.ui.tests.refactoring plugin. This file controls all of the automated testing that is done for the org.eclipse.jdt.ui.tests.refactoring plugin. It can be run from inside of Eclipse or from the command line. It is intended to serve as a template file for testing any other plugin.

Notice that the structure of the file roughly mirrors that of a JUnit test. Targets are defined for setting up the tests, defining what needs to be done, cleaning up after the tests, and running everything in the right order.

 
<?xml version="1.0" encoding="UTF-8"?>

<project name="testsuite" default="run" basedir=".">
	<!-- The property ${eclipse-home} should be passed into this script -->
	<!-- Set a meaningful default value for when it is not. -->
	<property name="eclipse-home" value="${basedir}\..\.."/>

	<!-- sets the properties eclipse-home, and library-file -->
	<property name="plugin-name" value="org.eclipse.jdt.ui.tests.refactoring"/>
	<property name="library-file"
            value="${basedir}/library.xml"/>

	<!-- This target holds all initialization code that needs to be done for -->
	<!-- all tests that are to be run. Initialization for individual tests -->
	<!-- should be done within the body of the suite target. -->
	<target name="init">
		<tstamp/>
		<delete>
			<fileset dir="${eclipse-home}" includes="org*.xml"/>
		</delete>
	</target>

	<!-- This target defines the tests that need to be run. -->
	<target name="suite">
		<property name="refactoring-folder" 
              value="${eclipse-home}/refactoring_folder"/>
		<delete dir="${refactoring-folder}" quiet="true"/>
		<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
			<property name="data-dir" value="${refactoring-folder}"/>
			<property name="plugin-name" value="${plugin-name}"/>
			<property name="classname" 
                value="org.eclipse.jdt.ui.tests.refactoring.all.AllAllRefactoringTests"/>
		</ant>
	</target>
	
	<!-- This target holds code to cleanup the testing environment after -->
	<!-- after all of the tests have been run. You can use this target to -->
	<!-- delete temporary files that have been created. -->
	<target name="cleanup">
	</target>

	<!-- This target runs the test suite. Any actions that need to happen -->
	<!-- after all the tests have been run should go here. -->
	<target name="run" depends="init,suite,cleanup">
		<ant target="collect" antfile="${library-file}" dir="${eclipse-home}">
			<property name="includes" value="org*.xml"/>
			<property name="output-file" value="${plugin-name}.xml"/>
		</ant>
	</target>
</project>

Known Issues

ECLIPSE_HOME

The test suites need to know where the root of the eclipse install is on the file system (the ECLIPSE_HOME variable). However, this variable is only defined in JDT. The ${eclipse-home} property can be set to a reasonable default inside of the test.xml script. Then tests can be run from the standard Ant window, without having to specify -Declipse-home=%ECLIPSE_HOME%. If a value for ${eclipse-home} does get passed in, the default (specified in test.xml) gets overridden. The parameter is passed in by the build mechanism. For most cases, the value "${basedir}/../.." is a reasonable default.

No console output

When you run a TestSuite using the standard JUnit, it normally outputs a series of dots to the console so that you can track the TestSuite's progress. It is not possible to add this feature to the automated testing process at this point in time.

Ant java task on Linux

Ant expects there to be a java executable on the system path. Furthermore, the executable must be a real file, not a symbolic link. If the test framework is throwing an exception java.io.IOException: java: not found, ensure that the java executable is on your system path.

PDE

The testing framework currently has no knowledge of PDE. In order to run the automated you must be running a self hosting environment with a full development and target Eclipse.