Jump to: navigation, search

Difference between revisions of "Platform-releng/Automated Testing"

(Initial HTML version added to wiki)
(Initial version edit with "rich text editor" to convert some HTML tags)
Line 1: Line 1:
= Automated Unit and Integration Testing =
+
= Automated Unit and Integration Testing =
  
== Description ==
+
== Description ==
  
  <p>This document describes how to run Eclipse JUnit plugin tests
+
This document describes how to run Eclipse JUnit plugin tests from the command line using Eclipse SDK JUnit test plug-ins and a testing framework. This method of running tests is intended for anyone who wishes to automate their testing process or reproduce the testing mechanism used in Eclipse builds by the Eclipse Platform Release Engineering team.  
    from the command line using Eclipse SDK JUnit test plug-ins and a
+
    testing framework. This method of running tests is intended for
+
    anyone who wishes to automate their testing process or reproduce the
+
    testing mechanism used in Eclipse builds by the Eclipse Platform
+
    Release Engineering team.</p>
+
  <p>
+
    Click <a
+
      href="http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.test/testframework.html?rev=HEAD">here</a>
+
    for a description of the testing framework, and how it can be used
+
    outside of the setup described here.
+
  </p>
+
  <p>If you simply wish to write and run JUnit tests interactively
+
    from an Eclipse workbench, try one of the links below:</p>
+
  <p>
+
    <a
+
      href="http://help.eclipse.org/helios/index.jsp?topic=/org.eclipse.jdt.doc.user/gettingStarted/qs-junit.htm">JUnit
+
      Support in Eclipse</a><br> <a
+
      href="http://help.eclipse.org/helios/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tools/launchers/junit_launcher.htm">Testing
+
      Eclipse Plug-ins with JUnit</a>
+
  </p>
+
  <p>
+
    <font size="+1">Requirements</font>
+
  </p>
+
  <ol>
+
    <li>All test machines will require 1.4 and 1.5 java runtime
+
      environments installed to run all tests. See <A
+
      HREF="#testtargets">test targets</A> for details on which tests
+
      require a 1.5 level jre. All other tests can be executed with
+
      either jre levels.
+
    </li>
+
  
    <li>A properties file containing the following key-value pair:
+
Click &lt;a href="http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.test/testframework.html?rev=HEAD"&gt;here&lt;/a&gt; for a description of the testing framework, and how it can be used outside of the setup described here.
      J2SE-5.0=&lt;path to 1.5 java executable&gt;. On Windows, the path
+
      must be enclosed with double quotes if it contains any space
+
      characters, e.g. J2SE-5.0=&quot;c:/program
+
      files/jdk1.5.0/jre/bin/java&quot;.</li>
+
  
    <li><a href="http://www.info-zip.org/pub/infozip/UnZip.html">Info-ZIP
+
If you simply wish to write and run JUnit tests interactively from an Eclipse workbench, try one of the links below:
        UnZip</a> version 5.41 or later installed and added to the path.</li>
+
  
    <li>The JUnit Plugin Tests <code>eclipse-Automated-Tests-&lt;buildId&gt;.zip</code>
+
&lt;a href="http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.jdt.doc.user/gettingStarted/qs-junit.htm"&gt;JUnit Support in Eclipse&lt;/a&gt;<br> &lt;a href="http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tools/launchers/junit_launcher.htm"&gt;Testing Eclipse Plug-ins with JUnit&lt;/a&gt;
      from an Eclipse build (<a
+
      href="http://download.eclipse.org/eclipse/downloads/">http://download.eclipse.org/eclipse/downloads/</a>).
+
      This ZIP contains this document, scripts to run the tests, the
+
      test framework plug-ins and the JUnit test plug-ins.
+
    </li>
+
  
    <li>An Eclipse SDK distribution (<a
+
<font size="+1">Requirements</font>  
      href="http://download.eclipse.org/eclipse/downloads/">http://download.eclipse.org/eclipse/downloads/</a>)
+
      to test. Usually you will want to test a particular Eclipse build
+
      using a matching Automated Tests feature.
+
    <li>To run the pde build tests you will need the
+
      eclipse-&lt;buildId&gt;delta-pack.zip.</li>
+
  
    <li>To run the Team CVS tests you will need extssh access to a
+
#All test machines will require 1.4 and 1.5 java runtime environments installed to run all tests. See &lt;A HREF="#testtargets"&gt;test targets&lt;/A&gt; for details on which tests require a 1.5 level jre. All other tests can be executed with either jre levels.
      CVS repository. The tests create a lot of junk modules and tags so
+
#A properties file containing the following key-value pair: J2SE-5.0=&lt;path to 1.5 java executable&gt;. On Windows, the path must be enclosed with double quotes if it contains any space characters, e.g. J2SE-5.0="c:/program files/jdk1.5.0/jre/bin/java".
      you will want to keep the test repository seperate from your
+
#&lt;a href="http://www.info-zip.org/pub/infozip/UnZip.html"&gt;Info-ZIP UnZip&lt;/a&gt; version 5.41 or later installed and added to the path.
      development repository. While it is not necessary to clean up the
+
#The JUnit Plugin Tests <code>eclipse-Automated-Tests-&lt;buildId&gt;.zip</code> from an Eclipse build (&lt;a href="http://download.eclipse.org/eclipse/downloads/"&gt;http://download.eclipse.org/eclipse/downloads/&lt;/a&gt;). This ZIP contains this document, scripts to run the tests, the test framework plug-ins and the JUnit test plug-ins.
      repository between each Team test run you will probably want to
+
#An Eclipse SDK distribution (&lt;a href="http://download.eclipse.org/eclipse/downloads/"&gt;http://download.eclipse.org/eclipse/downloads/&lt;/a&gt;) to test. Usually you will want to test a particular Eclipse build using a matching Automated Tests feature.
      clean it up periodically.</li>
+
#To run the pde build tests you will need the eclipse-&lt;buildId&gt;delta-pack.zip.
 +
#To run the Team CVS tests you will need extssh access to a CVS repository. The tests create a lot of junk modules and tags so you will want to keep the test repository seperate from your development repository. While it is not necessary to clean up the repository between each Team test run you will probably want to clean it up periodically.  
 +
#Some of the p2 tests require a copy of the Eclipse Platform Runtime Binary archive for the platform you are testing. More information is available from the &lt;a href="http://wiki.eclipse.org/Equinox/p2/Testing"&gt;Equinox Testing Wiki page&lt;/a&gt;.
  
    <li>Some of the p2 tests require a copy of the Eclipse Platform
+
<font size="+1">Setup</font>  
      Runtime Binary archive for the platform you are testing. More
+
      information is available from the <a
+
      href="http://wiki.eclipse.org/Equinox/p2/Testing">Equinox
+
        Testing Wiki page</a>.
+
    </li>
+
  </ol>
+
  
  <p>
+
#Extract the eclipse-Automated-Tests-&lt;buildId&gt;.zip or tar.gz. On Window, extract this file close to the root of the drive to prevent some path names from exceeding the 255 character limit. This will create a top-level directory <code>eclipse-testing</code>.  
    <font size="+1">Setup</font>
+
#Place the Eclipse SDK distribution to test (and the Eclipse Platform Runtime Binary) in the eclipse-testing directory. Do not extract these files as the test scripts will take care of this.
  </p>
+
#To run the Team CVS tests, edit the test.xml script to provide connection information. Replace the following strings in the '''teamcvs''' target of the <code>test.xml</code> file with the information described in following list. These values are used at test time to automatically replace strings in org.eclipse.team.tests.cvs.core/repository.properties. For more information on how this file is used in the team CVS tests, click &lt;a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.team.tests.cvs.core/readme.html?rev=HEAD"&gt;here&lt;/a&gt;.
  <ol>
+
#*@cvs_user@ - the account used to connect to the cvs repository
    <li>Extract the eclipse-Automated-Tests-&lt;buildId&gt;.zip or
+
#*@cvs_password@ - the account password
      tar.gz. On Window, extract this file close to the root of the
+
#*@cvs_host@- the name of the cvs server
      drive to prevent some path names from exceeding the 255 character
+
#*@cvs_root@ - the repository path
      limit. This will create a top-level directory <code>eclipse-testing</code>.
+
#To Run the PDE build scripts, save the eclipse-&lt;buildId&gt;delta-pack.zip to the eclipse-testing directory.
    </li>
+
  
    <li>Place the Eclipse SDK distribution to test (and the Eclipse
+
<font size="+1">Running Tests</font>  
      Platform Runtime Binary) in the eclipse-testing directory. Do not
+
      extract these files as the test scripts will take care of this.</li>
+
    <li>To run the Team CVS tests, edit the test.xml script to
+
      provide connection information. Replace the following strings in
+
      the <strong>teamcvs</strong> target of the <code>test.xml</code>
+
      file with the information described in following list. These
+
      values are used at test time to automatically replace strings in
+
      org.eclipse.team.tests.cvs.core/repository.properties. For more
+
      information on how this file is used in the team CVS tests, click
+
      <a
+
      href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.team.tests.cvs.core/readme.html?rev=HEAD">here</a>.
+
  
      <ul>
+
From a command prompt or terminal:
        <li>@cvs_user@ - the account used to connect to the cvs
+
          repository</li>
+
        <li>@cvs_password@ - the account password</li>
+
        <li>@cvs_host@- the name of the cvs server</li>
+
        <li>@cvs_root@ - the repository path</li>
+
      </ul>
+
    </li>
+
    <li>To Run the PDE build scripts, save the
+
      eclipse-&lt;buildId&gt;delta-pack.zip to the eclipse-testing
+
      directory.</li>
+
  </ol>
+
  
  <p>
+
#<code>cd</code> to the <code>eclipse-testing</code> directory  
    <font size="+1">Running Tests</font>
+
#Execute the following script:
  </p>
+
 
  <p>From a command prompt or terminal:</p>
+
'''runtests -properties &lt;path to properties file&gt; [-os &lt;operating system&gt;] [-ws &lt;windowing system&gt;] [-arch &lt;architecture&gt;] [-noclean] [-vm &lt;path to java executable&gt;][-Dtest.target=performance][''targetname1 targetname2 targetname3 etc...''] '''
  <ol>
+
 
    <li><code>cd</code> to the <code>eclipse-testing</code>
+
All parameters and targets are optional on windows systems with the exception of "-properties". On *nix systems, the os, ws and arch parameters must be specified in addition to the -properties parameter.  
      directory</li>
+
 
    <li>Execute the following script:
+
The JUnit test results are produced in individual xml files for each test plug-in with a test.xml. These results are copied to the eclipse-testing\results\xml directory. These results can be converted into html by running the "genHtml" target as described in the table below. Please be aware that "Build Successful" printed to the console is NOT indicative of tests passing. &lt;/li&gt; &lt;/ol&gt; &lt;p&gt; <font size="+1">&lt;a name="vm"&gt;&lt;/a&gt;Parameters</font>
      <p>
+
 
        <b>runtests -properties &lt;path to properties file&gt; [-os
+
{| width="90%" border="1"
          &lt;operating system&gt;] [-ws &lt;windowing system&gt;]
+
|-
          [-arch &lt;architecture&gt;] [-noclean] [-vm &lt;path to java
+
| Parameter
          executable&gt;][-Dtest.target=performance][<I>targetname1
+
| Description
            targetname2 targetname3 etc...</I>]
+
|-
        </b>
+
| -os &lt;operating system&gt;
      </p>
+
| The OS the tests are being run on. One of: aix, hpux,
      <p>All parameters and targets are optional on windows systems
+
        linux, macosx, solaris, win32. '''Default win32'''
        with the exception of &quot;-properties&quot;. On *nix systems,
+
 
        the os, ws and arch parameters must be specified in addition to
+
|-
        the -properties parameter.</p>
+
| -ws &lt;windowing system&gt;
      <p>The JUnit test results are produced in individual xml files
+
| The windowing system the tests are being run on. One of:
        for each test plug-in with a test.xml. These results are copied
+
        cocoa, gtk, photon, win32. '''Default win32'''
        to the eclipse-testing\results\xml directory. These results can
+
 
        be converted into html by running the &quot;genHtml&quot; target
+
|-
        as described in the table below. Please be aware that
+
| -arch &lt;architecture&gt;
        &quot;Build Successful&quot; printed to the console is NOT
+
| The architecture the tests are being run on. One of: ppc,
        indicative of tests passing.
+
        PA_RISC, x86, sparc. '''Default x86'''
    </li>
+
 
  </ol>
+
|-
 +
| -noclean
 +
| Run tests without installing a clean Eclipse (in
 +
        "test-eclipse") between test targets. Default is to
 +
    delete an existing test-eclipse directory and re-extract the
 +
 
 +
Eclipse distribution between test targets.
 +
 
 +
|-
 +
| -properties
 +
|
 +
<br>  
  
  <p>
 
    <font size="+1"><a name="vm"></a>Parameters</font>
 
  </p>
 
  <table width="90%" border="1">
 
    <tr>
 
      <td>Parameter</td>
 
      <td>Description</td>
 
    </tr>
 
    <tr>
 
      <td>-os &lt;operating system&gt;</td>
 
      <td>The OS the tests are being run on. One of: aix, hpux,
 
        linux, macosx, solaris, win32. <b>Default win32</b>
 
      </td>
 
    </tr>
 
    <tr>
 
      <td>-ws &lt;windowing system&gt;</td>
 
      <td>The windowing system the tests are being run on. One of:
 
        cocoa, gtk, photon, win32. <b>Default win32</b>
 
      </td>
 
    </tr>
 
    <tr>
 
      <td>-arch &lt;architecture&gt;</td>
 
      <td>The architecture the tests are being run on. One of: ppc,
 
        PA_RISC, x86, sparc. <b>Default x86</b>
 
      </td>
 
    </tr>
 
    <tr>
 
      <td>-noclean</td>
 
      <td>Run tests without installing a clean Eclipse (in
 
        &quot;test-eclipse&quot;) between test targets. Default is to
 
        delete an existing test-eclipse directory and re-extract the
 
        Eclipse distribution between test targets.</td>
 
    </tr>
 
    <tr>
 
      <td>-properties</td>
 
      <td><P>
 
 
           As per requirements noted above, this is used to reference a
 
           As per requirements noted above, this is used to reference a
          properties file containing additional Ant properties used in
+
      properties file containing additional Ant properties used in
          running tests.Can be used to pass additional vm arguments to
+
      running tests.Can be used to pass additional vm arguments to
          the Java virtual machine running the tests by adding the entry
+
      the Java virtual machine running the tests by adding the entry
          &quot;extraVMargs=&lt;the args to pass to the vm&gt;&quot; to
+
      "extraVMargs=&lt;the args to pass to the vm&gt;" to
          the specified properties file.<BR> <BR>At a minimum,
+
      the specified properties file.
          the property file must include a &quot;J2SE-5.0&quot; property
+
          which specifies a path to a 1.5 java executable, for example:<BR>
+
At a minimum,
          <BR> <B>J2SE-5.0=&quot;c:/program
+
      the property file must include a "J2SE-5.0" property
            files/jdk1.5.0_06/jre/bin/java&quot;</B>
+
      which specifies a path to a 1.5 java executable, for example:
        </P>
+
        <P>The path must be enclosed in double quotes if it contains
+
     
          spaces. This key-value pair is required to run JUnit plug-in
+
'''J2SE-5.0="c:/program'''
          tests that require a 1.5 level jvm.</P></td>
+
        files/jdk1.5.0_06/jre/bin/java"
    </tr>
+
 
    <tr>
+
<br>  
      <td>-vm</td>
+
 
      <td><P>
+
The path must be enclosed in double quotes if it contains spaces. This key-value pair is required to run JUnit plug-in tests that require a 1.5 level jvm.  
 +
 
 +
|-
 +
| -vm  
 +
|
 +
<br>  
 +
 
 
           When runtests is executed, two instances of eclipse are
 
           When runtests is executed, two instances of eclipse are
          started: one to invoke the org.eclipse.ant.core.antRunner
+
      started: one to invoke the org.eclipse.ant.core.antRunner
          application to run a test.xml script in a test plug-in (outer
+
      application to run a test.xml script in a test plug-in (outer
          instance) and another in which the JUnit tests are being
+
      instance) and another in which the JUnit tests are being
          executed (inner). By default, both will use the java
+
      executed (inner). By default, both will use the java
          executable on the system path. The -vm property is used to
+
      executable on the system path. The -vm property is used to
          specify an alternate java executable. For example, &quot;-vm
+
      specify an alternate java executable. For example, "-vm
          c:\jdk142\jre\bin\java&quot;.<BR> <BR> However, the
+
      c:\jdk142\jre\bin\java".
          -vm value is ignored by the inner vm for JUnit plug-in tests
+
          requiring a 1.5 JRE to execute (see <A HREF="#testtargets">test
+
However, the
            targets</A>). This property must be used in conjunction with the
+
      -vm value is ignored by the inner vm for JUnit plug-in tests
          -properties parameter described above at all times.
+
      requiring a 1.5 JRE to execute (see &lt;A HREF="#testtargets"&gt;test
        </P></td>
+
        targets&lt;/A&gt;). This property must be used in conjunction with the
    </tr>
+
      -properties parameter described above at all times.
    <tr>
+
 
      <td>-Dtest.target=performance</td>
+
<br>  
      <td>Runs the performance target only in each test plug-in's
+
 
 +
|-
 +
| -Dtest.target=performance  
 +
| Runs the performance target only in each test plug-in's  
 
         test.xml. When running tests on Windows, enclose this parameter
 
         test.xml. When running tests on Windows, enclose this parameter
        in double quotes, ie. runtests
+
    in double quotes, ie. runtests
        &quot;-Dtest.target=performance&quot;.</td>
+
 
    </tr>
+
"-Dtest.target=performance".  
    <tr>
+
 
      <td>targetname1 targetname2 targetname3 etc..</td>
+
|-
      <td><P>
+
| targetname1 targetname2 targetname3 etc..  
           Replace with a list of <A HREF="#testtargets">test targets</A>
+
|
          to execute (space delineated). Each corresponds to an Ant
+
<br>  
          target in the eclipse-testing/test.xml script. <BR> <BR>For
+
 
          example, <BR> <B>runtests -os linux -ws gtk -arch x86
+
           Replace with a list of &lt;A HREF="#testtargets"&gt;test targets&lt;/A&gt;
            ant antui -properties vm.properties</B><BR> This will
+
      to execute (space delineated). Each corresponds to an Ant
          execute the &quot;ant&quot; and &quot;antui&quot; test targets
+
      target in the eclipse-testing/test.xml script.  
          only. If not specified, the &quot;all&quot; target will be
+
          executed.<BR>
+
For
        </P></td>
+
      example,  
    </tr>
+
'''runtests -os linux -ws gtk -arch x86'''
  </table>
+
        ant antui -properties vm.properties
  <p>
+
This will
    <font size="+1"><A NAME="testtargets"></A>Appendix A - Test
+
      execute the "ant" and "antui" test targets
      Targets</font>
+
      only. If not specified, the "all" target will be
  </p>
+
      executed.
  <p>
+
    Ant targets in the main test.xml which execute tests. Typically each
+
 
    corresponds to the execution of a test plug-in's test.xml.<BR>
+
<br>  
  </p>
+
 
  <table width="90%" border="1">
+
|}
    <tr>
+
      <td WIDTH="34%">Target name (special jvm requirements)</td>
+
      <td WIDTH="66%">Description / test plug-in</td>
+
    </tr>
+
    <tr>
+
      <td WIDTH="34%">all (<B>1.4, 1.5 and 1.6 JREs required</B>)
+
      </td>
+
      <td WIDTH="66%">Runs all the targets below. Default.</td>
+
    </tr>
+
    <tr>
+
      <td>ant</td>
+
      <td>org.eclipse.ant.tests.core</td>
+
    </tr>
+
    <tr>
+
      <td>antui</td>
+
      <td>org.eclipse.ant.tests.ui</td>
+
    </tr>
+
    <tr>
+
      <td>compare</td>
+
      <td>org.eclipse.compare.tests</td>
+
    </tr>
+
    <tr>
+
      <td>coreexpressions</td>
+
      <td>org.eclipse.core.expressions.tests</td>
+
    </tr>
+
    <tr>
+
      <td>coreresources</td>
+
      <td>org.eclipse.core.tests.resources</td>
+
    </tr>
+
    <tr>
+
      <td>coreruntime</td>
+
      <td>org.eclipse.core.tests.runtime</td>
+
    </tr>
+
    <tr>
+
      <td>coretestsnet</td>
+
      <td>org.eclipse.core.tests.net</td>
+
    </tr>
+
    <tr>
+
      <td>equinoxsecurity</td>
+
      <td>org.eclipse.equinox.security.tests</td>
+
    </tr>
+
    <tr>
+
      <td>filebuffers</td>
+
      <td>org.eclipse.core.filebuffers.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdtapt <b>(JRE 1.5 required)</b></td>
+
      <td>org.eclipse.jdt.apt.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdtaptpluggable <b>(JRE 1.6 required)</b></td>
+
      <td>org.eclipse.jdt.apt.pluggable.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdtcompilerapt <b>(JRE 1.6 required)</b></td>
+
      <td>org.eclipse.jdt.compiler.apt.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdtcompilertool <b>(JRE 1.6 required)</b></td>
+
      <td>org.eclipse.jdt.compiler.tool.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdtcorebuilder <b>(JRE 1.5 required)</b></td>
+
      <td>org.eclipse.jdt.core.tests.builder</td>
+
    </tr>
+
    <tr>
+
      <td>jdtcorecompiler <b>(JRE 1.5 required)</b></td>
+
      <td>org.eclipse.jdt.core.tests.compiler</td>
+
    </tr>
+
    <tr>
+
      <td>jdtcoremodel <b>(JRE 1.5 required)</b></td>
+
      <td>org.eclipse.jdt.core.tests.model</td>
+
    </tr>
+
    <tr>
+
      <td>jdtcoreperf</td>
+
      <td>org.eclipse.jdt.core.tests.performance</td>
+
    </tr>
+
    <tr>
+
      <td>jdtdebug</td>
+
      <td>org.eclipse.jdt.debug.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdttext</td>
+
      <td>org.eclipse.jdt.text.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdtui</td>
+
      <td>org.eclipse.jdt.ui.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jdtuirefactoring</td>
+
      <td>org.eclipse.jdt.ui.tests.refactoring</td>
+
    </tr>
+
    <tr>
+
      <td>jface</td>
+
      <td>org.eclipse.jface.text.tests</td>
+
    </tr>
+
    <tr>
+
      <td>jfacedatabinding</td>
+
      <td>org.eclipse.jface.tests.databinding</td>
+
    </tr>
+
    <tr>
+
      <td>ltkcorerefactoringtests</td>
+
      <td>org.eclipse.ltk.core.refactoring.tests</td>
+
    </tr>
+
    <tr>
+
      <td>ltkuirefactoringtests</td>
+
      <td>org.eclipse.ltk.ui.refactoring.tests</td>
+
    </tr>
+
    <tr>
+
      <td>osgi</td>
+
      <td>org.eclipse.osgi.tests</td>
+
    </tr>
+
    <tr>
+
      <td>pdeapitooling <b>(JRE 1.5 required)</b></td>
+
      <td>org.eclipse.pde.api.tools.tests</td>
+
    </tr>
+
    <tr>
+
      <td>pdebuild</td>
+
      <td>org.eclipse.pde.build.tests</td>
+
    </tr>
+
    <tr>
+
      <td>pdeui</td>
+
      <td>org.eclipse.pde.ui.tests</td>
+
    </tr>
+
    <tr>
+
      <td>relEng</td>
+
      <td>org.eclipse.releng.tests</td>
+
    </tr>
+
    <tr>
+
      <td>swt</td>
+
      <td>org.eclipse.swt.tests</td>
+
    </tr>
+
    <tr>
+
      <td>teamcore</td>
+
      <td>org.eclipse.team.tests.core</td>
+
    </tr>
+
    <tr>
+
      <td>teamcvs</td>
+
      <td>org.eclipse.team.tests.cvs.core</td>
+
    </tr>
+
    <tr>
+
      <td>text</td>
+
      <td>org.eclipse.text.tests</td>
+
    </tr>
+
    <tr>
+
      <td>ua</td>
+
      <td>org.eclipse.ua.tests</td>
+
    </tr>
+
    <tr>
+
      <td>ui</td>
+
      <td>org.eclipse.ui.tests</td>
+
    </tr>
+
    <tr>
+
      <td>uieditors</td>
+
      <td>org.eclipse.ui.editors.tests</td>
+
    </tr>
+
    <tr>
+
      <td>uiforms</td>
+
      <td>org.eclipse.ui.tests.forms</td>
+
    </tr>
+
    <tr>
+
      <td>uinavigator</td>
+
      <td>org.eclipse.ui.tests.navigator</td>
+
    </tr>
+
    <tr>
+
      <td>uiperformance</td>
+
      <td>org.eclipse.ui.tests.performance</td>
+
    </tr>
+
    <tr>
+
      <td>uircp</td>
+
      <td>org.eclipse.ui.tests.rcp</td>
+
    </tr>
+
    <tr>
+
      <td>uiviews</td>
+
      <td>org.eclipse.ui.tests.views.properties.tabbed</td>
+
    </tr>
+
    <tr>
+
      <td>uiworkbenchtexteditor</td>
+
      <td>org.eclipse.ui.workbench.texteditor.tests</td>
+
    </tr>
+
  
 +
<font size="+1">&lt;A NAME="testtargets"&gt;&lt;/A&gt;Appendix A - Test Targets</font>
  
  </table>
+
Ant targets in the main test.xml which execute tests. Typically each corresponds to the execution of a test plug-in's test.xml.<br>  
  <p>
+
    <br>
+
  </p>
+
  <p>
+
    <font size="+1"><a name="appendixb"></a>Appendix B -
+
      Supported os, ws, and arch value combinations</font>
+
  </p>
+
  <p>The following table contains the values that can be used with
+
    the -os, -ws, and -arch switches.</p>
+
  <table width="75%" border="1">
+
    <tr>
+
      <td>Operating System (os)</td>
+
      <td>Windowing System (ws)</td>
+
      <td>Architecture (arch)</td>
+
    </tr>
+
    <tr>
+
      <td>hpux</td>
+
      <td>motif</td>
+
      <td>PA_RISC</td>
+
    </tr>
+
    <tr>
+
      <td>linux</td>
+
      <td>gtk</td>
+
      <td>x86</td>
+
    </tr>
+
    <tr>
+
      <td>linux</td>
+
      <td>gtk</td>
+
      <td>x86_64</td>
+
    </tr>
+
    <tr>
+
      <td>macosx</td>
+
      <td>carbox</td>
+
      <td>ppc</td>
+
    </tr>
+
    <tr>
+
      <td>macosx</td>
+
      <td>cocoa</td>
+
      <td>x86</td>
+
    </tr>
+
    <tr>
+
      <td>macosx</td>
+
      <td>cocoa</td>
+
      <td>x86_64</td>
+
    </tr>
+
    <tr>
+
      <td>solaris</td>
+
      <td>gtk</td>
+
      <td>sparc</td>
+
    </tr>
+
    <tr>
+
      <td>solaris</td>
+
      <td>gtk</td>
+
      <td>x86</td>
+
    </tr>
+
    <tr>
+
      <td>solaris</td>
+
      <td>motif</td>
+
      <td>sparc</td>
+
    </tr>
+
    <tr>
+
      <td>win32</td>
+
      <td>win32</td>
+
      <td>x86</td>
+
    </tr>
+
  </table>
+
  
 +
{| width="90%" border="1"
 +
|-
 +
| width="34%" | Target name (special jvm requirements)
 +
| width="66%" | Description / test plug-in
 +
|-
 +
| width="34%" | all ('''1.4, 1.5 and 1.6 JREs required''')
 +
| width="66%" | Runs all the targets below. Default.
 +
|-
 +
| ant
 +
| org.eclipse.ant.tests.core
 +
|-
 +
| antui
 +
| org.eclipse.ant.tests.ui
 +
|-
 +
| compare
 +
| org.eclipse.compare.tests
 +
|-
 +
| coreexpressions
 +
| org.eclipse.core.expressions.tests
 +
|-
 +
| coreresources
 +
| org.eclipse.core.tests.resources
 +
|-
 +
| coreruntime
 +
| org.eclipse.core.tests.runtime
 +
|-
 +
| coretestsnet
 +
| org.eclipse.core.tests.net
 +
|-
 +
| equinoxsecurity
 +
| org.eclipse.equinox.security.tests
 +
|-
 +
| filebuffers
 +
| org.eclipse.core.filebuffers.tests
 +
|-
 +
| jdtapt '''(JRE 1.5 required)'''
 +
| org.eclipse.jdt.apt.tests
 +
|-
 +
| jdtaptpluggable '''(JRE 1.6 required)'''
 +
| org.eclipse.jdt.apt.pluggable.tests
 +
|-
 +
| jdtcompilerapt '''(JRE 1.6 required)'''
 +
| org.eclipse.jdt.compiler.apt.tests
 +
|-
 +
| jdtcompilertool '''(JRE 1.6 required)'''
 +
| org.eclipse.jdt.compiler.tool.tests
 +
|-
 +
| jdtcorebuilder '''(JRE 1.5 required)'''
 +
| org.eclipse.jdt.core.tests.builder
 +
|-
 +
| jdtcorecompiler '''(JRE 1.5 required)'''
 +
| org.eclipse.jdt.core.tests.compiler
 +
|-
 +
| jdtcoremodel '''(JRE 1.5 required)'''
 +
| org.eclipse.jdt.core.tests.model
 +
|-
 +
| jdtcoreperf
 +
| org.eclipse.jdt.core.tests.performance
 +
|-
 +
| jdtdebug
 +
| org.eclipse.jdt.debug.tests
 +
|-
 +
| jdttext
 +
| org.eclipse.jdt.text.tests
 +
|-
 +
| jdtui
 +
| org.eclipse.jdt.ui.tests
 +
|-
 +
| jdtuirefactoring
 +
| org.eclipse.jdt.ui.tests.refactoring
 +
|-
 +
| jface
 +
| org.eclipse.jface.text.tests
 +
|-
 +
| jfacedatabinding
 +
| org.eclipse.jface.tests.databinding
 +
|-
 +
| ltkcorerefactoringtests
 +
| org.eclipse.ltk.core.refactoring.tests
 +
|-
 +
| ltkuirefactoringtests
 +
| org.eclipse.ltk.ui.refactoring.tests
 +
|-
 +
| osgi
 +
| org.eclipse.osgi.tests
 +
|-
 +
| pdeapitooling '''(JRE 1.5 required)'''
 +
| org.eclipse.pde.api.tools.tests
 +
|-
 +
| pdebuild
 +
| org.eclipse.pde.build.tests
 +
|-
 +
| pdeui
 +
| org.eclipse.pde.ui.tests
 +
|-
 +
| relEng
 +
| org.eclipse.releng.tests
 +
|-
 +
| swt
 +
| org.eclipse.swt.tests
 +
|-
 +
| teamcore
 +
| org.eclipse.team.tests.core
 +
|-
 +
| teamcvs
 +
| org.eclipse.team.tests.cvs.core
 +
|-
 +
| text
 +
| org.eclipse.text.tests
 +
|-
 +
| ua
 +
| org.eclipse.ua.tests
 +
|-
 +
| ui
 +
| org.eclipse.ui.tests
 +
|-
 +
| uieditors
 +
| org.eclipse.ui.editors.tests
 +
|-
 +
| uiforms
 +
| org.eclipse.ui.tests.forms
 +
|-
 +
| uinavigator
 +
| org.eclipse.ui.tests.navigator
 +
|-
 +
| uiperformance
 +
| org.eclipse.ui.tests.performance
 +
|-
 +
| uircp
 +
| org.eclipse.ui.tests.rcp
 +
|-
 +
| uiviews
 +
| org.eclipse.ui.tests.views.properties.tabbed
 +
|-
 +
| uiworkbenchtexteditor
 +
| org.eclipse.ui.workbench.texteditor.tests
 +
|}
  
 +
<br>
  
 +
<font size="+1">&lt;a name="appendixb"&gt;&lt;/a&gt;Appendix B - Supported os, ws, and arch value combinations</font>
  
 +
The following table contains the values that can be used with the -os, -ws, and -arch switches.
  
 +
{| width="75%" border="1"
 +
|-
 +
| Operating System (os)
 +
| Windowing System (ws)
 +
| Architecture (arch)
 +
|-
 +
| hpux
 +
| motif
 +
| PA_RISC
 +
|-
 +
| linux
 +
| gtk
 +
| x86
 +
|-
 +
| linux
 +
| gtk
 +
| x86_64
 +
|-
 +
| macosx
 +
| carbox
 +
| ppc
 +
|-
 +
| macosx
 +
| cocoa
 +
| x86
 +
|-
 +
| macosx
 +
| cocoa
 +
| x86_64
 +
|-
 +
| solaris
 +
| gtk
 +
| sparc
 +
|-
 +
| solaris
 +
| gtk
 +
| x86
 +
|-
 +
| solaris
 +
| motif
 +
| sparc
 +
|-
 +
| win32
 +
| win32
 +
| x86
 +
|}
  
[[Category:Eclipse_Platform_Releng| ]]
+
<br> [[Category:Eclipse_Platform_Releng|Eclipse_Platform_Releng]]

Revision as of 23:37, 26 April 2013

Automated Unit and Integration Testing

Description

This document describes how to run Eclipse JUnit plugin tests from the command line using Eclipse SDK JUnit test plug-ins and a testing framework. This method of running tests is intended for anyone who wishes to automate their testing process or reproduce the testing mechanism used in Eclipse builds by the Eclipse Platform Release Engineering team.

Click <a href="http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.test/testframework.html?rev=HEAD">here</a> for a description of the testing framework, and how it can be used outside of the setup described here.

If you simply wish to write and run JUnit tests interactively from an Eclipse workbench, try one of the links below:

<a href="http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.jdt.doc.user/gettingStarted/qs-junit.htm">JUnit Support in Eclipse</a>
<a href="http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tools/launchers/junit_launcher.htm">Testing Eclipse Plug-ins with JUnit</a>

Requirements

  1. All test machines will require 1.4 and 1.5 java runtime environments installed to run all tests. See <A HREF="#testtargets">test targets</A> for details on which tests require a 1.5 level jre. All other tests can be executed with either jre levels.
  2. A properties file containing the following key-value pair: J2SE-5.0=<path to 1.5 java executable>. On Windows, the path must be enclosed with double quotes if it contains any space characters, e.g. J2SE-5.0="c:/program files/jdk1.5.0/jre/bin/java".
  3. <a href="http://www.info-zip.org/pub/infozip/UnZip.html">Info-ZIP UnZip</a> version 5.41 or later installed and added to the path.
  4. The JUnit Plugin Tests eclipse-Automated-Tests-<buildId>.zip from an Eclipse build (<a href="http://download.eclipse.org/eclipse/downloads/">http://download.eclipse.org/eclipse/downloads/</a>). This ZIP contains this document, scripts to run the tests, the test framework plug-ins and the JUnit test plug-ins.
  5. An Eclipse SDK distribution (<a href="http://download.eclipse.org/eclipse/downloads/">http://download.eclipse.org/eclipse/downloads/</a>) to test. Usually you will want to test a particular Eclipse build using a matching Automated Tests feature.
  6. To run the pde build tests you will need the eclipse-<buildId>delta-pack.zip.
  7. To run the Team CVS tests you will need extssh access to a CVS repository. The tests create a lot of junk modules and tags so you will want to keep the test repository seperate from your development repository. While it is not necessary to clean up the repository between each Team test run you will probably want to clean it up periodically.
  8. Some of the p2 tests require a copy of the Eclipse Platform Runtime Binary archive for the platform you are testing. More information is available from the <a href="http://wiki.eclipse.org/Equinox/p2/Testing">Equinox Testing Wiki page</a>.

Setup

  1. Extract the eclipse-Automated-Tests-<buildId>.zip or tar.gz. On Window, extract this file close to the root of the drive to prevent some path names from exceeding the 255 character limit. This will create a top-level directory eclipse-testing.
  2. Place the Eclipse SDK distribution to test (and the Eclipse Platform Runtime Binary) in the eclipse-testing directory. Do not extract these files as the test scripts will take care of this.
  3. To run the Team CVS tests, edit the test.xml script to provide connection information. Replace the following strings in the teamcvs target of the test.xml file with the information described in following list. These values are used at test time to automatically replace strings in org.eclipse.team.tests.cvs.core/repository.properties. For more information on how this file is used in the team CVS tests, click <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.team.tests.cvs.core/readme.html?rev=HEAD">here</a>.
    • @cvs_user@ - the account used to connect to the cvs repository
    • @cvs_password@ - the account password
    • @cvs_host@- the name of the cvs server
    • @cvs_root@ - the repository path
  4. To Run the PDE build scripts, save the eclipse-<buildId>delta-pack.zip to the eclipse-testing directory.

Running Tests

From a command prompt or terminal:

  1. cd to the eclipse-testing directory
  2. Execute the following script:

runtests -properties <path to properties file> [-os <operating system>] [-ws <windowing system>] [-arch <architecture>] [-noclean] [-vm <path to java executable>][-Dtest.target=performance][targetname1 targetname2 targetname3 etc...]

All parameters and targets are optional on windows systems with the exception of "-properties". On *nix systems, the os, ws and arch parameters must be specified in addition to the -properties parameter.

The JUnit test results are produced in individual xml files for each test plug-in with a test.xml. These results are copied to the eclipse-testing\results\xml directory. These results can be converted into html by running the "genHtml" target as described in the table below. Please be aware that "Build Successful" printed to the console is NOT indicative of tests passing. </li> </ol> <p> <a name="vm"></a>Parameters

Parameter Description
-os <operating system> The OS the tests are being run on. One of: aix, hpux,
       linux, macosx, solaris, win32. Default win32
-ws <windowing system> The windowing system the tests are being run on. One of:
       cocoa, gtk, photon, win32. Default win32
-arch <architecture> The architecture the tests are being run on. One of: ppc,
       PA_RISC, x86, sparc. Default x86
-noclean Run tests without installing a clean Eclipse (in
       "test-eclipse") between test targets. Default is to
    delete an existing test-eclipse directory and re-extract the

Eclipse distribution between test targets.

-properties


         As per requirements noted above, this is used to reference a
      properties file containing additional Ant properties used in
      running tests.Can be used to pass additional vm arguments to
      the Java virtual machine running the tests by adding the entry
      "extraVMargs=<the args to pass to the vm>" to
      the specified properties file.

At a minimum,
      the property file must include a "J2SE-5.0" property
      which specifies a path to a 1.5 java executable, for example:

      
J2SE-5.0="c:/program
        files/jdk1.5.0_06/jre/bin/java"


The path must be enclosed in double quotes if it contains spaces. This key-value pair is required to run JUnit plug-in tests that require a 1.5 level jvm.

-vm


         When runtests is executed, two instances of eclipse are
      started: one to invoke the org.eclipse.ant.core.antRunner
      application to run a test.xml script in a test plug-in (outer
      instance) and another in which the JUnit tests are being
      executed (inner). By default, both will use the java
      executable on the system path. The -vm property is used to
      specify an alternate java executable. For example, "-vm
      c:\jdk142\jre\bin\java".

However, the
      -vm value is ignored by the inner vm for JUnit plug-in tests
      requiring a 1.5 JRE to execute (see <A HREF="#testtargets">test
        targets</A>). This property must be used in conjunction with the
      -properties parameter described above at all times.


-Dtest.target=performance Runs the performance target only in each test plug-in's
       test.xml. When running tests on Windows, enclose this parameter
    in double quotes, ie. runtests

"-Dtest.target=performance".

targetname1 targetname2 targetname3 etc..


         Replace with a list of <A HREF="#testtargets">test targets</A>
      to execute (space delineated). Each corresponds to an Ant
      target in the eclipse-testing/test.xml script. 

For
      example, 
runtests -os linux -ws gtk -arch x86
        ant antui -properties vm.properties
This will
      execute the "ant" and "antui" test targets
      only. If not specified, the "all" target will be
      executed.


<A NAME="testtargets"></A>Appendix A - Test Targets

Ant targets in the main test.xml which execute tests. Typically each corresponds to the execution of a test plug-in's test.xml.

Target name (special jvm requirements) Description / test plug-in
all (1.4, 1.5 and 1.6 JREs required) Runs all the targets below. Default.
ant org.eclipse.ant.tests.core
antui org.eclipse.ant.tests.ui
compare org.eclipse.compare.tests
coreexpressions org.eclipse.core.expressions.tests
coreresources org.eclipse.core.tests.resources
coreruntime org.eclipse.core.tests.runtime
coretestsnet org.eclipse.core.tests.net
equinoxsecurity org.eclipse.equinox.security.tests
filebuffers org.eclipse.core.filebuffers.tests
jdtapt (JRE 1.5 required) org.eclipse.jdt.apt.tests
jdtaptpluggable (JRE 1.6 required) org.eclipse.jdt.apt.pluggable.tests
jdtcompilerapt (JRE 1.6 required) org.eclipse.jdt.compiler.apt.tests
jdtcompilertool (JRE 1.6 required) org.eclipse.jdt.compiler.tool.tests
jdtcorebuilder (JRE 1.5 required) org.eclipse.jdt.core.tests.builder
jdtcorecompiler (JRE 1.5 required) org.eclipse.jdt.core.tests.compiler
jdtcoremodel (JRE 1.5 required) org.eclipse.jdt.core.tests.model
jdtcoreperf org.eclipse.jdt.core.tests.performance
jdtdebug org.eclipse.jdt.debug.tests
jdttext org.eclipse.jdt.text.tests
jdtui org.eclipse.jdt.ui.tests
jdtuirefactoring org.eclipse.jdt.ui.tests.refactoring
jface org.eclipse.jface.text.tests
jfacedatabinding org.eclipse.jface.tests.databinding
ltkcorerefactoringtests org.eclipse.ltk.core.refactoring.tests
ltkuirefactoringtests org.eclipse.ltk.ui.refactoring.tests
osgi org.eclipse.osgi.tests
pdeapitooling (JRE 1.5 required) org.eclipse.pde.api.tools.tests
pdebuild org.eclipse.pde.build.tests
pdeui org.eclipse.pde.ui.tests
relEng org.eclipse.releng.tests
swt org.eclipse.swt.tests
teamcore org.eclipse.team.tests.core
teamcvs org.eclipse.team.tests.cvs.core
text org.eclipse.text.tests
ua org.eclipse.ua.tests
ui org.eclipse.ui.tests
uieditors org.eclipse.ui.editors.tests
uiforms org.eclipse.ui.tests.forms
uinavigator org.eclipse.ui.tests.navigator
uiperformance org.eclipse.ui.tests.performance
uircp org.eclipse.ui.tests.rcp
uiviews org.eclipse.ui.tests.views.properties.tabbed
uiworkbenchtexteditor org.eclipse.ui.workbench.texteditor.tests


<a name="appendixb"></a>Appendix B - Supported os, ws, and arch value combinations

The following table contains the values that can be used with the -os, -ws, and -arch switches.

Operating System (os) Windowing System (ws) Architecture (arch)
hpux motif PA_RISC
linux gtk x86
linux gtk x86_64
macosx carbox ppc
macosx cocoa x86
macosx cocoa x86_64
solaris gtk sparc
solaris gtk x86
solaris motif sparc
win32 win32 x86