Jump to: navigation, search

Difference between revisions of "Eclipse/Testing"

(added information about Squish)
Line 1: Line 1:
The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the [http://junit.org JUnit] test framework. This page contains links to information about how to run and create automated tests for the platform.
+
The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the [http://junit.org JUnit] test framework. This page contains links to information about how to run and create automated tests for the platform.  
  
== Correctness tests ==
+
== Correctness tests ==
  
The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests. These tests are run every build on Windows, Linux, and Mac OSX.
+
The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests. These tests are run every build on Windows, Linux, and Mac OSX.  
  
=== Running tests from within Eclipse ===
+
=== Running tests from within Eclipse ===
  
Correctness tests can be run manually from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:
+
Correctness tests can be run manually from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:  
  
# Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:
+
#Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:  
#* org.junit - The JUnit test framework
+
#*org.junit - The JUnit test framework  
#* org.eclipse.test - The basic infrastructure for running Eclipse tests
+
#*org.eclipse.test - The basic infrastructure for running Eclipse tests  
#* org.eclipse.core.tests.harness - Various utility pieces used by many tests
+
#*org.eclipse.core.tests.harness - Various utility pieces used by many tests  
#* org.eclipse.core.tests.runtime - Tests for the runtime component
+
#*org.eclipse.core.tests.runtime - Tests for the runtime component  
#* org.eclipse.core.tests.resources - Tests for the resources component
+
#*org.eclipse.core.tests.resources - Tests for the resources component  
#* org.eclipse.ui.tests.harness - Various utility pieces used by UI tests
+
#*org.eclipse.ui.tests.harness - Various utility pieces used by UI tests  
#* org.eclipse.ui.tests - Tests for the Eclipse UI
+
#*org.eclipse.ui.tests - Tests for the Eclipse UI  
# In the Navigator or Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:
+
#In the Navigator or Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:  
#* org.eclipse.ui.tests.UiTestSuite - runs all UI tests
+
#*org.eclipse.ui.tests.UiTestSuite - runs all UI tests  
#* org.eclipse.core.tests.runtime.AutomatedTests - runs all runtime tests
+
#*org.eclipse.core.tests.runtime.AutomatedTests - runs all runtime tests  
#* org.eclipse.core.tests.resources.AutomatedTests - runs all resource tests
+
#*org.eclipse.core.tests.resources.AutomatedTests - runs all resource tests  
# Select Run > Run...
+
#Select Run > Run...  
# Choose the "JUnit Plug-in Test" category, and click the button to create a new test
+
#Choose the "JUnit Plug-in Test" category, and click the button to create a new test  
# On the "Main" tab, select the appropriate application for that test. Here are some important applications to know about:
+
#On the "Main" tab, select the appropriate application for that test. Here are some important applications to know about:  
#* [No Application] - Headless Mode - for running headless core (runtime, osgi, resource) tests
+
#*[No Application] - Headless Mode - for running headless core (runtime, osgi, resource) tests  
#* org.eclipse.ui.ide.workbench - for running UI tests
+
#*org.eclipse.ui.ide.workbench - for running UI tests  
# Click Run.
+
#Click Run.
  
=== Using EasyMock ===
+
=== Using EasyMock ===
  
In the Eclipse [[Galileo]] release the utility [http://easymock.org EasyMock] was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.
+
In the Eclipse [[Galileo]] release the utility [http://easymock.org EasyMock] was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.  
  
To use EasyMock in your tests, simply load the org.easymock bundle from the [[Orbit]] repository. There is a convenience [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/test/org.eclipse.test/easymock.psf?root=Tools_Project&view=log|''easymock.psf''] [[PSF|project set file]] in the org.eclipse.test bundle to facilitate loading of this bundle.  
+
To use EasyMock in your tests, simply load the org.easymock bundle from the [[Orbit]] repository. There is a convenience [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/test/org.eclipse.test/easymock.psf?root=Tools_Project&view=log| ''easymock.psf''] [[PSF|project set file]] in the org.eclipse.test bundle to facilitate loading of this bundle.  
  
Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.
+
Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.  
  
=== Support for JUnit4 ===
+
=== Support for JUnit4 ===
  
During the [[Helios]] release the platform is moving to support JUnit4 in the test framework in addition to JUnit3. For more details on this transition see [[Eclipse/Testing/JUnit4 Changes]].
+
During the [[Helios]] release the platform is moving to support JUnit4 in the test framework in addition to JUnit3. For more details on this transition see [[Eclipse/Testing/JUnit4 Changes]].  
  
== Session tests ==
+
== Session tests ==
  
[[Session Tests]] in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.
+
[[Session Tests]] in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.  
  
== Performance tests ==
+
== Performance tests ==
  
See the [http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html Performance Tests How-to]
+
See the [http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html Performance Tests How-to]  
  
=== Profiling performance tests ===
+
=== Profiling performance tests ===
  
It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:
+
It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:  
  
# Download the tests for the build you are interested in (available from same [http://download.eclipse.org/eclipse download page] as the build itself).
+
#Download the tests for the build you are interested in (available from same [http://download.eclipse.org/eclipse download page] as the build itself).  
# Unzip the test framework, and follow the instructions in the readme.html to configure your tests
+
#Unzip the test framework, and follow the instructions in the readme.html to configure your tests  
# Create a properties file (let's say profile.properties) with the following contents:
+
#Create a properties file (let's say profile.properties) with the following contents:
  
 
   extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot
 
   extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot
  
<ol start="4">
+
#Invoke the performance test and specify the properties file location:  
<li>Invoke the performance test and specify the properties file location:</li>
+
</ol>
+
  
   runtests "-Dtest.target=performance" -properties profile.properties <yourtarget>
+
   runtests "-Dtest.target=performance" -properties profile.properties &lt;yourtarget&gt;
  
If your performance test is a session test, a more complex spell is needed in your profile.properties file:
+
If your performance test is a session test, a more complex spell is needed in your profile.properties file:  
  
 
   extraVMargs=-Dsetup.override.vmArgs=agentlib:yjpagent==sampling,onexit==snapshot
 
   extraVMargs=-Dsetup.override.vmArgs=agentlib:yjpagent==sampling,onexit==snapshot
  
This instructs the [[Session Tests]] framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.
+
This instructs the [[Session Tests]] framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.  
  
The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various [http://www.yourkit.com/docs/75/help/getting_started/running_with_profiler/additional_agent_options.jsp other options] for configuring the headless profile agent.
+
The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various [http://www.yourkit.com/docs/75/help/getting_started/running_with_profiler/additional_agent_options.jsp other options] for configuring the headless profile agent.  
  
 
== UI tests  ==
 
== UI tests  ==
Line 88: Line 86:
 
! [http://www.instantiations.com/windowtester/ Window Tester]  
 
! [http://www.instantiations.com/windowtester/ Window Tester]  
 
|  
 
|  
Commercial, Eclipse committers are allowed to use it for free, [http://www.instantiations.com/company/opensource.html free license available for Open Source projects]  
+
<strike>Commercial, Eclipse committers are allowed to use it for free, </strike>[http://www.instantiations.com/company/opensource.html <strike>free license available for Open Source projects</strike>]
 +
 
 +
[http://code.google.com/javadevtools/download-wintester.html Available for free from Google]<strike></strike>
  
 
|  
 
|  
Line 224: Line 224:
  
 
|  
 
|  
*lots of nice features
+
*lots of nice features  
*tests SWT, Swing, Web, and a lot more
+
*tests SWT, Swing, Web, and a lot more  
 
*scripts in various scripting languages (JavaScript, Python, Perl or Tcl)  
 
*scripts in various scripting languages (JavaScript, Python, Perl or Tcl)  
*support for native dialogs
+
*support for native dialogs  
*good documentation
+
*good documentation  
*responsive technical support
+
*responsive technical support  
 
*Eclipse and Ant integration
 
*Eclipse and Ant integration
 +
 
|  
 
|  
*component recognition doesn't seem to handle JFace widgets very well; getting any script to work involves a lot of trial and error
+
*component recognition doesn't seem to handle JFace widgets very well; getting any script to work involves a lot of trial and error  
*their symbolic object naming strategy is very unclear; component names can even change in the same script
+
*their symbolic object naming strategy is very unclear; component names can even change in the same script  
 
*expensive: [http://www.froglogic.com/products/licensing.php 2400 euro] for single edition license
 
*expensive: [http://www.froglogic.com/products/licensing.php 2400 euro] for single edition license
 +
 
| <br>
 
| <br>
 
|}
 
|}

Revision as of 09:55, 13 November 2010

The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the JUnit test framework. This page contains links to information about how to run and create automated tests for the platform.

Correctness tests

The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests. These tests are run every build on Windows, Linux, and Mac OSX.

Running tests from within Eclipse

Correctness tests can be run manually from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:

  1. Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:
    • org.junit - The JUnit test framework
    • org.eclipse.test - The basic infrastructure for running Eclipse tests
    • org.eclipse.core.tests.harness - Various utility pieces used by many tests
    • org.eclipse.core.tests.runtime - Tests for the runtime component
    • org.eclipse.core.tests.resources - Tests for the resources component
    • org.eclipse.ui.tests.harness - Various utility pieces used by UI tests
    • org.eclipse.ui.tests - Tests for the Eclipse UI
  2. In the Navigator or Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:
    • org.eclipse.ui.tests.UiTestSuite - runs all UI tests
    • org.eclipse.core.tests.runtime.AutomatedTests - runs all runtime tests
    • org.eclipse.core.tests.resources.AutomatedTests - runs all resource tests
  3. Select Run > Run...
  4. Choose the "JUnit Plug-in Test" category, and click the button to create a new test
  5. On the "Main" tab, select the appropriate application for that test. Here are some important applications to know about:
    • [No Application] - Headless Mode - for running headless core (runtime, osgi, resource) tests
    • org.eclipse.ui.ide.workbench - for running UI tests
  6. Click Run.

Using EasyMock

In the Eclipse Galileo release the utility EasyMock was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.

To use EasyMock in your tests, simply load the org.easymock bundle from the Orbit repository. There is a convenience easymock.psf project set file in the org.eclipse.test bundle to facilitate loading of this bundle.

Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.

Support for JUnit4

During the Helios release the platform is moving to support JUnit4 in the test framework in addition to JUnit3. For more details on this transition see Eclipse/Testing/JUnit4 Changes.

Session tests

Session Tests in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.

Performance tests

See the Performance Tests How-to

Profiling performance tests

It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:

  1. Download the tests for the build you are interested in (available from same download page as the build itself).
  2. Unzip the test framework, and follow the instructions in the readme.html to configure your tests
  3. Create a properties file (let's say profile.properties) with the following contents:
 extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot
  1. Invoke the performance test and specify the properties file location:
 runtests "-Dtest.target=performance" -properties profile.properties <yourtarget>

If your performance test is a session test, a more complex spell is needed in your profile.properties file:

 extraVMargs=-Dsetup.override.vmArgs=agentlib:yjpagent==sampling,onexit==snapshot

This instructs the Session Tests framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.

The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various other options for configuring the headless profile agent.

UI tests

The table below gathers pros and cons of tools making UI testing easier. It should help developers to choose the most appropriate tool.

UI testing tools
Tool License Recorder Pros Cons Remarks/Comments
Window Tester

Commercial, Eclipse committers are allowed to use it for free, free license available for Open Source projects

Available for free from Google

Yes

  • looks stable
  • creates JUnit tests, can be launched along with other tests
  • easy API, good documentation
  • very easy to modify generated code
  • API looks similar to SWTBot API, tests could be perhaps migrated to SWTBot when it's stable
  • large footprint when creating tests (around 60MB for Window Tester plug-ins)
  • sensitive for focus changing, very hard to debug these UI tests (i.e. changing focus to the host Eclipse causes exceptions)
  • we would need a license to use libraries on our test servers

QF-Test

Commercial, free license available for Open Source projects

Yes

  • very robust test automation based on "soft" events at Windows/Gtk level
  • support for native dialogs like FileDialog
  • excellent component recognition, including automatic naming for Eclipse/RCP components
  • supports testing combined SWT/Swing applications
  • tests are batch and thus ant-runnable producing a detailed run-log and XML and HTML reports
  • Jython or Groovy scripting available
  • comprehensive manual and tutorial available online; mailing list with searchable archive
  • very responsive support
  • Eclipse/SWT support limited to Windows and Linux, both x86 and x86_64, Solaris Gtk support available but no Mac
  • QF-Test is Java application but not integrated into Eclipse
  • tests don't compile to JUnit
  • replaces the org.eclipse.swt plugin (corresponding patches are available, could be applied automatically as part of daily builds)
  • also supports GEF/GMF
  • also supports web testing
GUI Dancer

Commercial

No, "not a capture/replay tool"

  • tests Java (Swing, SWT/RCP) and HTML GUI's
  • extensible using Java
  • quite expensive 4641 euro for a license for one GUI toolkit (SWT/RCP)
  • tests cannot be run as JUnits
  • store tests in a database
  • this tool doesn't make writing tests easier ;) it is pretty hard to start working with it (especially comparing to tools like SWT Bot)

SWT Bot

EPL

No

  • eclipse project in incubator, still being improved
  • easy API, simple to get started with
  • open source, hosted at eclipse.org
  • small tool, less than 1 MB (including source)
  • very modular, good test coverage, easy to contribute fixes and enhancements
  • poor documentation, one example on wiki, only javadocs
  • some major loss of function, e.g. 259908, some have workarounds
  • long responses on important bugs e.g. 259908 and 271132
  • cannot test native dialogs (File Dialogs, Color Dialogs, etc) see 164192
  • Leverages SWT/Eclipse capabilities to "perform" most operations. Support for new things is highly dependent on capabilities offered by SWT/Eclipse bug 164192
Automated GUI Recorder

EPL

Yes

  • ability to provide your own Widget Resolver
  • open source
  • seems pretty stable, used by TPTP to do UI tests
  • large footprint when creating tests (probably around 50MB - TPTP Platform and TPTP Test Tools)
  • test descriptions only in XML, verification points in Java can be added though
  • didn't find any way to test absence of a dialog
  • can't run tests in debug mode
  • provided as is (because IBM pulled resources out of TPTP)

TestComplete

Commercial

?

  • looks nice and powerful
  • got nice features
  • is implemented using COM technology
  • works only on Windows

Squish

Commercial

Yes

  • lots of nice features
  • tests SWT, Swing, Web, and a lot more
  • scripts in various scripting languages (JavaScript, Python, Perl or Tcl)
  • support for native dialogs
  • good documentation
  • responsive technical support
  • Eclipse and Ant integration
  • component recognition doesn't seem to handle JFace widgets very well; getting any script to work involves a lot of trial and error
  • their symbolic object naming strategy is very unclear; component names can even change in the same script
  • expensive: 2400 euro for single edition license

The table is in progress. Feel free to update it, if you know useful features or serious limitations of the mentioned tools. You are also welcome to add new tools.

More we know about those tools, easier to choose the best one.

Notes on what you should be evaluating on

  • CI support -- ability to run tests from command line/ant
  • Cross platform support -- developers use a variety of platforms -- windows, linux, mac
  • Scripting capabilities -- vendor script/Java/jruby ?
  • Reporting capabilities -- html/xml/push to database ?

Back to Eclipse Project home