Skip to main content
Jump to: navigation, search

Difference between revisions of "Eclipse/Testing"

(UI tests)
(Importing a coverage session in the Eclipse IDE with EclEmma)
 
(66 intermediate revisions by 33 users not shown)
Line 1: Line 1:
The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the [http://junit.org JUnit] test framework. This page contains links to information about how to run and create automated tests for the platform.
+
The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the [http://junit.org JUnit] test framework. This page contains links to information about how to run and create automated tests for the platform.  
  
== Correctness tests ==
+
== Correctness tests ==
  
The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests.  These tests are run every build on Windows, Linux, and Mac OSX.
+
The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests.  
  
*See [[ETF_JUnit4_Changes|this page]] for information about running JUnit4 tests within the Automated Testing Framework.
+
=== Running tests from within Eclipse  ===
  
=== Running tests from within Eclipse ===
+
Correctness tests can be run from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:
  
Correctness tests can be run manually from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:
+
#In the Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:
 +
#*org.eclipse.ui.tests.UiTestSuite - runs all UI tests
 +
#*org.eclipse.core.tests.runtime.AutomatedTests - runs all runtime tests
 +
#*org.eclipse.core.tests.resources.AutomatedTests - runs all resource tests
 +
#Select Run > Run...
 +
#Choose the "JUnit Plug-in Test" category, and click the button to create a new test
 +
#On the "Main" tab, select the appropriate application for that test. Here are some important applications to know about:  
 +
#*[No Application] - Headless Mode - for running headless core (runtime, osgi, resource) tests
 +
#*org.eclipse.ui.ide.workbench - for running UI tests
 +
#Click Run.
  
# Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:
+
=== Running with Tycho ===
#* org.junit - The JUnit test framework
+
#* org.eclipse.test - The basic infrastructure for running Eclipse tests
+
#* org.eclipse.core.tests.harness - Various utility pieces used by many tests
+
#* org.eclipse.core.tests.runtime - Tests for the runtime component
+
#* org.eclipse.core.tests.resources - Tests for the resources component
+
#* org.eclipse.ui.tests.harness - Various utility pieces used by UI tests
+
#* org.eclipse.ui.tests - Tests for the Eclipse UI
+
# In the Navigator or Package Explorer, select the test or test suite you want to run.  Each test package typically contains a TestSuite class that contains all the tests in that package.  Suites from multiple packages are then aggregated into higher level suites.  Here are some useful suites to know about:
+
#* org.eclipse.ui.tests.UiTestSuite - runs all UI tests
+
#* org.eclipse.core.tests.runtime.AutomatedTests - runs all runtime tests
+
#* org.eclipse.core.tests.resources.AutomatedTests - runs all resource tests
+
# Select Run > Run...
+
# Choose the "JUnit Plug-in Test" category, and click the button to create a new test
+
# On the "Main" tab, select the appropriate application for that test. Here are some important applications to know about:
+
#* [No Application] - Headless Mode - for running headless core (runtime, osgi, resource) tests
+
#* org.eclipse.ui.ide.workbench - for running UI tests
+
# Click Run.
+
  
=== Using EasyMock ===
+
Since adoption of Tycho, [https://bugs.eclipse.org/bugs/show_bug.cgi?id=416904 some tests have been enabled] as part of the integration-test step of the Tycho build, relying on the [https://eclipse.org/tycho/sitedocs/tycho-surefire/tycho-surefire-plugin/test-mojo.html tycho-surefire-plugin]. Those tests are run automatically when launching <tt>mvn clean verify</tt> command on the whole platform code, or <tt>mvn clean verify -Pbuild-individual-bundles</tt> if you're building a subset of Platform. They can be skipped appending <tt>-DskipTests</tt> to the command-line.
  
In the Eclipse [[Galileo]] release the utility [http://easymock.org EasyMock] was added to the platform test framework.  EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.
+
=== Getting the required bundles ===
  
To use EasyMock in your tests, simply load the org.easymock bundle from the [[Orbit]] repository. There is a convenience ''easymock.psf'' [[PSF|project set file]] in the org.eclipse.test bundle to facilitate loading of this bundle.  
+
Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:
  
Note that EasyMock requires Java 5 or greater. At this time, we run tests on Java 1.4.2 and Java 5. This means you need to add the following to the Ant target for your tests (contact releng for details):
+
*org.junit - The JUnit test framework
 +
*org.eclipse.test - The basic infrastructure for running Eclipse tests  
 +
*org.eclipse.core.tests.harness - Various utility pieces used by many tests
 +
*org.eclipse.core.tests.runtime - Tests for the runtime component
 +
*org.eclipse.core.tests.resources - Tests for the resources component
 +
*org.eclipse.ui.tests.harness - Various utility pieces used by UI tests
 +
*org.eclipse.ui.tests - Tests for the Eclipse UI
  
<target name="YourTestTarget" depends="setJVMProperties">
+
=== Using EasyMock ===
  <property name="jvm" value="${J2SE-5.0}" />
+
<!--only run test if J2SE-5.0 property  set-->
+
<condition property="skip.test">
+
<not>
+
  <isset property="J2SE-5.0" />
+
</not>
+
</condition>
+
... remaining target definition as usual
+
  
== Session tests ==
+
In the Eclipse [[Galileo]] release the utility [http://easymock.org EasyMock] was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.
  
[[Session Tests]] in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.
+
To use EasyMock in your tests, simply load the org.easymock bundle from the [[Orbit]] repository. There is a convenience [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/test/org.eclipse.test/easymock.psf?root=Tools_Project&view=log| ''easymock.psf''] [[PSF|project set file]] in the org.eclipse.test bundle to facilitate loading of this bundle.  
  
== Performance tests ==
+
Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.
  
See the [http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html Performance Tests How-to]
+
=== Support for JUnit4  ===
  
=== Profiling performance tests ===
+
During the [[Helios]] release the platform is moving to support JUnit4 in the test framework in addition to JUnit3. For more details on this transition see [[Eclipse/Testing/JUnit4 Changes]].
  
It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:
+
=== Testing on Java 7 ===
  
# Download the tests for the build you are interested in (available from same [http://download.eclipse.org/eclipse download page] as the build itself).
+
In some Java 7 implementations, the order of methods returned by java.lang.Class#getDeclaredMethods() is not consistent across runs. This method is used by JUnit to discover test methods, which means test order may change across executions. Generally each test method should be coded to not depend on test execution order, which also makes it easier for people to run individual tests when debugging and get consistent results. In some rare cases separate tests rely on each other, in which case you can use the test suite to hard-code a test execution order. There is also a convenience class [http://git.eclipse.org/c/platform/eclipse.platform.releng.git/tree/bundles/org.eclipse.test.performance/src/org/eclipse/test/OrderedTestSuite.java OrderedTestSuite.java] that you can use or copy for this purpose.
# Unzip the test framework, and follow the instructions in the readme.html to configure your tests
+
 
# Create a properties file (let's say profile.properties) with the following contents:
+
== Code coverage ==
 +
 
 +
Platform uses [http://eclemma.org/jacoco/ JaCoCo] as code-coverage tool. JaCoCo doesn't require additional configuration than setting up a Java agent, and doesn't require to modify any class file on the filesystem. There's a Maven plugin to enable it, which works perfectly with tycho-surefire.
 +
 
 +
=== Generate JaCoCo coverage data with Maven/Tycho ===
 +
 
 +
Parent pom for Platform defines a <tt>coverage</tt> profile that enables and configures the [http://eclemma.org/jacoco/trunk/doc/maven.html jacoco-maven-plugin] for tests. So you can simply run <tt>mvn clean verify -Pjacoco ...</tt> and you'll get the jacoco.exec report file in the <tt>tests/target</tt> folder (or just <tt>target/</tt> if there is no <tt>tests</tt> module).
 +
 
 +
The location where this .exec file is generated can be overriden by command line settings: <tt>-Djacoco.destFile=/some/better/path</tt>. As the coverage data gets appended to the output file, you should usually make sure that there is no such file already before running tests to ensure it's only going to contain data from your current build, not a previous one. Concretely, that means that you should delete the jacoco file before running any build.
 +
 
 +
=== CI builds with JaCoCo and SonarQube ===
 +
 
 +
A bunch of module builds are configured to report code quality hints to [[Sonar|SonarQube]], they are visible at https://hudson.eclipse.org/platform/view/SonarQube/ . For each of these jobs, a SonarQube dashboard is created. You can see it by clicking on the <tt>SonarQube</tt> link. Then, among other, you'll be able to browse the coverage data in the Web UI of SonarQube.
 +
 
 +
Example - Platfrom resources:
 +
* Job: https://hudson.eclipse.org/platform/job/eclipse.platform.resources-SonarQube/
 +
* Coverage report file: https://hudson.eclipse.org/platform/job/eclipse.platform.resources-SonarQube/lastSuccessfulBuild/artifact/jacoco.exec
 +
* SonarQube (linked from Job page): https://dev.eclipse.org/sonar/dashboard/index/33441 . Coverage reports are on the right column, classes that have 100% coverage do not appear in drill-down details.
 +
 
 +
=== Importing a coverage session in the Eclipse IDE with EclEmma ===
 +
 
 +
{{warning|This approach is currently not working, see [[https://bugs.eclipse.org/bugs/show_bug.cgi?id=388956#c39 bug]] for details}}
 +
 
 +
[http://eclemma.org/installation.html EclEmma] provides a very good integration of JaCoCo reports into the Eclipse IDE, with metrics view, browsable coverage and highlighting lines according to whether they were covered or not. You can either let EclEmma run your tests from your IDE and it will enable coverage for this test, or reuse a previous <tt>jacoco.exec</tt> to analyze it in the IDE. This <tt>jacoco.exec</tt> can be one you generated locally by running tests with coverage enabled, or one that is archived in the [https://hudson.eclipse.org/platform/view/SonarQube/ SonarQube job of your choice] containing data from the last execution of that job. Once you've chosen a jacoco.exec, simply Use <tt>File > Import...</tt> in the IDE, and import it as <tt>Coverage session</tt>. EclEmma will guide you to enjoy coverage.
 +
 
 +
=== Taking advantage of coverage ===
 +
 
 +
There are several ways of taking advantage of coverage. Here is one which seems relevant for Eclipse development: rather than focusing on the coverage metrics, coverage can be used to detect '''not covered code''' and decide on '''what tests to add'''. This uses coverage as a way to identify which critical pieces of code seem to be missing test coverage. If those are pretty critical, difficult, error-prone, frequently changes... that are not covered, it most likely means that in order to provide efficient error detection for the project, some effort could be spent on writing a test that verify this critical part to avoid regressions later. Gor example, if you're about to make some refactoring, it's interesting to check whether the code you're about to change is covered, in order to detect potential issues with your change, and if the code isn't covered then it's worth considering the addition of a test as a "preparation" of your refactoring.
 +
 
 +
Focusing on the percentage may be wrong: indeed, in Java and UI development, it's pretty frequent that some code is boilerplate, or even is never meant to be invoked in realistic scenarios. Things such as Exception handling are sometimes pure defensive programming and there is not much profit in making extra-effort to cover those lines.
 +
 
 +
== Session tests  ==
 +
 
 +
[[Session Tests]] in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.
 +
 
 +
== Performance tests  ==
 +
 
 +
See the [[Performance/Automated Tests|Performance Tests How-to]]
 +
 
 +
=== Profiling performance tests  ===
 +
 
 +
It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:
 +
 
 +
#Download the tests for the build you are interested in (available from same [http://download.eclipse.org/eclipse download page] as the build itself).  
 +
#Unzip the test framework, and follow the instructions in the readme.html to configure your tests  
 +
#Create a properties file (let's say profile.properties) with the following contents:
  
 
   extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot
 
   extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot
  
<ol start="4">
+
#Invoke the performance test and specify the properties file location:  
<li>Invoke the performance test and specify the properties file location:</li>
+
</ol>
+
  
   runtests "-Dtest.target=performance" -properties profile.properties <yourtarget>
+
   runtests "-Dtest.target=performance" -properties profile.properties &lt;yourtarget&gt;
  
If your performance test is a session test, a more complex spell is needed in your profile.properties file:
+
If your performance test is a session test, a more complex spell is needed in your profile.properties file:  
  
 
   extraVMargs=-Dsetup.override.vmArgs=agentlib:yjpagent==sampling,onexit==snapshot
 
   extraVMargs=-Dsetup.override.vmArgs=agentlib:yjpagent==sampling,onexit==snapshot
  
This instructs the [[Session Tests]] framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.
+
This instructs the [[Session Tests]] framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.  
  
 
The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various [http://www.yourkit.com/docs/75/help/getting_started/running_with_profiler/additional_agent_options.jsp other options] for configuring the headless profile agent.
 
The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various [http://www.yourkit.com/docs/75/help/getting_started/running_with_profiler/additional_agent_options.jsp other options] for configuring the headless profile agent.
  
== UI tests ==
+
== API tests ==
  
The table below gathers pros and cons of tools making UI testing easier. It should help developers to choose
+
The Eclipse project also runs an suite of API tests during builds. These tests capture problems such as breaking API changes, and correct evolution of bundle [[Version Numbering|version numbers]]. For more details on these tests, refer to [[PDE/API_Tools]].
the most appropriate tool.
+
  
{| border="1" cellpadding="5"
+
== UI tests  ==
|+ UI testing tools
+
 
! Tool !! License !! Recorder !! Pros !! Cons
+
The table below gathers pros and cons of tools making UI testing easier. It should help developers to choose the most appropriate tool.
 +
 
 +
{| {{BMTableStyle}}
 +
|+ UI testing tools  
 +
|- {{BMTHStyle}}
 +
! Tool  
 +
! License  
 +
! Recorder  
 +
! Pros  
 +
! Cons  
 +
! Remarks/Comments
 
|-
 
|-
! [http://www.instantiations.com/windowtester/ Window Tester]
+
! [https://developers.google.com/java-dev-tools/wintester/html/index Window Tester]  
|
+
 
Commercial, Eclipse committers are allowed to use it
+
|  
|
+
[https://github.com/google/windowtester Open Source]
 +
 
 +
|
 +
Yes
 +
 
 +
|
 +
*Stable
 +
*creates JUnit tests, can be launched along with other tests
 +
*easy API
 +
*good documentation
 +
*OpenSource
 +
*very easy to modify generated code
 +
 
 +
|
 +
*large footprint when creating tests (around 60MB for WindowTester plug-ins)
 +
*sensitive for focus changing, very hard to debug these UI tests (i.e. changing focus to the host Eclipse causes exceptions)
 +
 
 +
| <br>
 +
|-
 +
! [https://www.eclipse.org/rcptt/ RCPTT]
 +
|
 +
* Free Open Source IDE and Runner
 +
* [http://www.xored.com/products/rcptt/rcptt-professional-support/ Commercial support]
 +
 
 +
|  
 
Yes
 
Yes
|
+
 
*looks stable
+
|  
*creates JUnit tests, can be launched along with other tests
+
* Designed specially to test Eclipse-based apps - plugins
*easy API, good documentation
+
* Supports SWT/JFace, Platform, IDE, GEF, and GMF
*very easy to modify generated code
+
* Contexts for AUT state modeling
*API looks similar to SWTBot API, tests could be perhaps migrated to SWTBot when it's stable
+
* Let you to run any subset of your tests in any order
|
+
* Realtime Eclipse code instrumentation and Runtime Intelligence
*large footprint when creating tests (around 60MB for Window Tester plug-ins)
+
* Modern Eclipse-based IDE
*sensitive for focus changing, very hard to debug these UI tests (i.e. changing focus to the host Eclipse causes exceptions)
+
* BDD-style scripting language ([https://www.eclipse.org/rcptt/documentation/userguide/ecl/ Eclipse Command Language])
*we would need a license to use libraries on our test servers
+
* Maven/Tycho plugin, Ant/CLI tool, and RCPTT Server to distribute test runs in the Cloud
 +
* Works on OSX 64, Linux 32/64, Window 32/64
 +
* Eclipse 3.5-4.5 and upcoming releases.
 +
|
 +
 
 +
|
 +
* Part of Eclipse Foundation.
 
|-
 
|-
! [http://www.bredex.de/en/guidancer/first.html GUI Dancer]
+
! [http://www.qftest.com QF-Test]  
|
+
|  
Commercial
+
Commercial, free license available for Open Source projects
|
+
 
No, "not a capture/replay tool"
+
|  
|
+
Yes
*platform independent
+
 
*extensible using Java
+
|
|
+
*very robust test automation based on "soft" events at Windows/Gtk level
*quite expensive [https://cgi.bredex.de/GUIdancerShop/prices.do 4641 euro] for a license for one GUI toolkin (SWT/RCP)
+
*support for native dialogs like FileDialog
*tests cannot be run as JUnits
+
*excellent component recognition, including automatic naming for Eclipse/RCP components
*store tests in a database
+
*supports testing combined SWT/Swing applications
*this tool doesn't make writing tests easier ;) I've found it pretty hard to start working with (especially comparing to tools like swtbot)
+
*tests are batch and thus ant-runnable producing a detailed run-log and XML and HTML reports
 +
*Jython or Groovy scripting available
 +
*comprehensive [http://www.qfs.de/en/qftest/manual.html manual] and [http://www.qfs.de/en/qftest/tutorial.html tutorial] available online; mailing list with [http://www.qftest.com/archive/qftest-list/ searchable archive]
 +
*[https://www.qfs.de/en/qf-test-videos.html Videos] (for beginners and advanced users)
 +
*very responsive support
 +
 
 +
|
 +
*Eclipse/SWT support limited to Windows and Linux, both x86 and x86_64, Solaris Gtk support available but no Mac
 +
*QF-Test is Java application but not integrated into Eclipse<br>
 +
*tests don't compile to JUnit
 +
 
 +
|
 +
*Developers can integrate QF-Test in their launch configuration.
 +
*replaces the org.eclipse.swt plugin (corresponding patches are available, could be applied automatically as part of daily builds)<br>
 +
*also supports GEF/GMF
 +
*also supports [https://www.qfs.de/en/qf-test/web-testing.html web testing] (including AJAX-toolkits like GWT, Vaadin etc. & HTML5)
 +
*also supports [https://www.qfs.de/en/qf-test/java-testing.html Java] Swing, JavaFX and SWT testing
 +
*Jenkins Plugin
 +
 
 
|-
 
|-
! [http://wiki.eclipse.org/SWTBot SWT Bot]
+
! [http://www.eclipse.org/jubula/ Jubula]
|
+
|
EPL
+
Open Source, EPL
|
+
 
No
+
|
|
+
Tests specified by drag &amp; drop without need to record. <br> Recording is possible as well.
*eclipse project in incubator, still being improved
+
 
*easy API
+
|
*open source
+
*Supports Swing, SWT/RCP/GEF, JavaFX, HTML and iOS applications
*small tool, less than 1 MB
+
*Works on Windows, Unix/Linux and Mac
*regarding its size, it would be easy to contribute to it (fixes, new enhancements)
+
*Black-box testing - no source code needed to execute tests
|
+
*Robust, high level test automation without any coding effort
*poor documentation, one example on wiki, only javadocs
+
*Incorporates best practices of software development into tests
*some major loss of function, e.g. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=259908 259908], some have workarounds
+
*Suitable for agile processes because tests can be written before an app is available
*long responses on important bugs e.g. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=259908 259908] and [https://bugs.eclipse.org/bugs/show_bug.cgi?id=271132 271132]
+
*Robust object recognition heuristic
 +
*Tests can be run from command line for continuous integration
 +
*Since 2015 there is also a programmatic API for writing tests using Java to be executed via e.g. JUnit
 +
*Extensible to add support for own components / actions
 +
 
 +
|
 +
*tests don't compile to JUnit
 +
*does not support combined SWT/Swing applications
 +
*not extendable without knowledge about RCP-Programming
 +
*few elements missing, like loops and if-conditions
 +
*Tests are written into XML format at back-end.
 +
 
 +
| Eclipse Project
 +
 
 +
|-
 +
! [[SWTBot|SWT Bot]]  
 +
|  
 +
EPL  
 +
 
 +
|  
 +
[http://wiki.eclipse.org/SWTBot/Recorder Yes ]
 +
 
 +
|  
 +
*eclipse project, still being improved  
 +
*easy API, simple to get started with
 +
*open source, hosted at eclipse.org
 +
*small tool, less than 1 MB (including source)
 +
*very modular, good test coverage, easy to contribute fixes and enhancements
 +
 
 +
|  
 +
*long responses on important bugs e.g. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=271132 271132]  
 
*cannot test native dialogs (File Dialogs, Color Dialogs, etc) see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=164192 164192]
 
*cannot test native dialogs (File Dialogs, Color Dialogs, etc) see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=164192 164192]
*tests are run  as SWTBot tests, not PDE-JUnit tests ([http://wiki.eclipse.org/SWTBot/FAQ#Why_do_tests_run_on_a_non-UI_thread.3F run in non-UI thread])
+
 
 +
|
 +
*Leverages SWT/Eclipse capabilities to perform most operations. Support for new things is highly dependent on capabilities offered by SWT/Eclipse {{bug|164192}}
 +
 
 
|-
 
|-
! [http://www.eclipse.org/tptp/test/documents/userguides/Intro-Auto-GUI.html Automated GUI Recorder]
+
! [http://www.eclipse.org/tptp/test/documents/userguides/Intro-Auto-GUI.html Automated GUI Recorder]  
|
+
|  
EPL
+
EPL  
|
+
 
Yes
+
|  
|
+
Yes  
*ability to provide your own Widget Resolver
+
 
*open source
+
|  
 +
*ability to provide your own Widget Resolver  
 +
*open source  
 
*seems pretty stable, used by TPTP to do UI tests
 
*seems pretty stable, used by TPTP to do UI tests
|
+
 
*large footprint when creating tests (probably around 50MB - TPTP Platform and TPTP Test Tools)
+
|  
*test descriptions only in XML, verification points in Java can be added though
+
*large footprint when creating tests (probably around 50MB - TPTP Platform and TPTP Test Tools)  
*didn't find any way to test absence of a dialog
+
*test descriptions only in XML, verification points in Java can be added though  
*can't run tests in debug mode
+
*didn't find any way to test absence of a dialog  
 +
*can't run tests in debug mode  
 
*provided as is (because IBM pulled resources out of TPTP)
 
*provided as is (because IBM pulled resources out of TPTP)
 +
 +
| <br>
 
|-
 
|-
! [http://www.automatedqa.com/products/testcomplete/ TestComplete]
+
! [http://www.automatedqa.com/products/testcomplete/ TestComplete]  
|
+
|  
Commercial
+
Commercial  
|
+
 
?
+
|  
|
+
Yes
*looks nice and powerful
+
 
 +
|  
 +
*looks nice and powerful  
 
*got nice features
 
*got nice features
|
+
*excellent screenshot visualiser identifies and highlights both mapped and unmapped UI components with mouse-over
*is implemented using COM technology
+
*scripts in various languages (VBScript, JScript, DelphiScript, C++Script, C#Script)
*works only on Windows  
+
*command-line support, see [http://smartbear.com/support/viewarticle/10998 TestComplete Command Line]
 +
|  
 +
*is implemented using COM technology  
 +
*works only on Windows
 +
*RCP application launched from TestComplete test did not exit cleanly, zombie process remained.
 +
| <br>
 +
|-
 +
! [http://www.froglogic.com/pg Squish]
 +
|
 +
Commercial
 +
 
 +
|
 +
Yes, free trial available
 +
 
 +
|
 +
*lots of nice features
 +
*fully cross-platform
 +
*tests RCP, SWT, Swing, JavaFx, Web, Qt, Android, iOS, native Windows, native macOS and more
 +
*full access to all Java APIs via introspection
 +
*powerful object recognition mechanism
 +
*scripts in various scripting languages (JavaScript, Python, Perl, Ruby or Tcl)
 +
*powerful Eclipse-based IDE
 +
*fully integrated BDD (Behavior Driven Development) support 100% compatible to Cucumber
 +
*support for native dialogs
 +
*good documentation
 +
*responsive technical support
 +
*Eclipse, Jenkins, Ant integration just to mention a few
 +
*Pricing starts at 2400 EUR per named user (see [https://www.froglogic.com/squish/pricing/ Pricing] for some examples
 +
|
 +
 
 +
| <br>
 
|}
 
|}
 +
 +
The table is in progress. Feel free to update it, if you know useful features or serious limitations of the mentioned tools. You are also welcome to add new tools.
 +
 +
More we know about those tools, easier to choose the best one.
 +
 +
==== Notes on what you should be evaluating on  ====
 +
 +
*CI support -- ability to run tests from command line/ant
 +
*Cross platform support -- developers use a variety of platforms -- windows, linux, mac
 +
*Scripting capabilities -- vendor script/Java/jruby&nbsp;?
 +
*Reporting capabilities -- html/xml/push to database&nbsp;?
  
 
----
 
----
Back to [[Eclipse Project]] home
+
 
 +
Back to [[Eclipse Project]] home  
  
 
[[Category:Performance]]
 
[[Category:Performance]]

Latest revision as of 06:07, 22 March 2017

The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the JUnit test framework. This page contains links to information about how to run and create automated tests for the platform.

Correctness tests

The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests.

Running tests from within Eclipse

Correctness tests can be run from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:

  1. In the Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:
    • org.eclipse.ui.tests.UiTestSuite - runs all UI tests
    • org.eclipse.core.tests.runtime.AutomatedTests - runs all runtime tests
    • org.eclipse.core.tests.resources.AutomatedTests - runs all resource tests
  2. Select Run > Run...
  3. Choose the "JUnit Plug-in Test" category, and click the button to create a new test
  4. On the "Main" tab, select the appropriate application for that test. Here are some important applications to know about:
    • [No Application] - Headless Mode - for running headless core (runtime, osgi, resource) tests
    • org.eclipse.ui.ide.workbench - for running UI tests
  5. Click Run.

Running with Tycho

Since adoption of Tycho, some tests have been enabled as part of the integration-test step of the Tycho build, relying on the tycho-surefire-plugin. Those tests are run automatically when launching mvn clean verify command on the whole platform code, or mvn clean verify -Pbuild-individual-bundles if you're building a subset of Platform. They can be skipped appending -DskipTests to the command-line.

Getting the required bundles

Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:

  • org.junit - The JUnit test framework
  • org.eclipse.test - The basic infrastructure for running Eclipse tests
  • org.eclipse.core.tests.harness - Various utility pieces used by many tests
  • org.eclipse.core.tests.runtime - Tests for the runtime component
  • org.eclipse.core.tests.resources - Tests for the resources component
  • org.eclipse.ui.tests.harness - Various utility pieces used by UI tests
  • org.eclipse.ui.tests - Tests for the Eclipse UI

Using EasyMock

In the Eclipse Galileo release the utility EasyMock was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.

To use EasyMock in your tests, simply load the org.easymock bundle from the Orbit repository. There is a convenience easymock.psf project set file in the org.eclipse.test bundle to facilitate loading of this bundle.

Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.

Support for JUnit4

During the Helios release the platform is moving to support JUnit4 in the test framework in addition to JUnit3. For more details on this transition see Eclipse/Testing/JUnit4 Changes.

Testing on Java 7

In some Java 7 implementations, the order of methods returned by java.lang.Class#getDeclaredMethods() is not consistent across runs. This method is used by JUnit to discover test methods, which means test order may change across executions. Generally each test method should be coded to not depend on test execution order, which also makes it easier for people to run individual tests when debugging and get consistent results. In some rare cases separate tests rely on each other, in which case you can use the test suite to hard-code a test execution order. There is also a convenience class OrderedTestSuite.java that you can use or copy for this purpose.

Code coverage

Platform uses JaCoCo as code-coverage tool. JaCoCo doesn't require additional configuration than setting up a Java agent, and doesn't require to modify any class file on the filesystem. There's a Maven plugin to enable it, which works perfectly with tycho-surefire.

Generate JaCoCo coverage data with Maven/Tycho

Parent pom for Platform defines a coverage profile that enables and configures the jacoco-maven-plugin for tests. So you can simply run mvn clean verify -Pjacoco ... and you'll get the jacoco.exec report file in the tests/target folder (or just target/ if there is no tests module).

The location where this .exec file is generated can be overriden by command line settings: -Djacoco.destFile=/some/better/path. As the coverage data gets appended to the output file, you should usually make sure that there is no such file already before running tests to ensure it's only going to contain data from your current build, not a previous one. Concretely, that means that you should delete the jacoco file before running any build.

CI builds with JaCoCo and SonarQube

A bunch of module builds are configured to report code quality hints to SonarQube, they are visible at https://hudson.eclipse.org/platform/view/SonarQube/ . For each of these jobs, a SonarQube dashboard is created. You can see it by clicking on the SonarQube link. Then, among other, you'll be able to browse the coverage data in the Web UI of SonarQube.

Example - Platfrom resources:

Importing a coverage session in the Eclipse IDE with EclEmma

Warning2.png
This approach is currently not working, see [bug] for details


EclEmma provides a very good integration of JaCoCo reports into the Eclipse IDE, with metrics view, browsable coverage and highlighting lines according to whether they were covered or not. You can either let EclEmma run your tests from your IDE and it will enable coverage for this test, or reuse a previous jacoco.exec to analyze it in the IDE. This jacoco.exec can be one you generated locally by running tests with coverage enabled, or one that is archived in the SonarQube job of your choice containing data from the last execution of that job. Once you've chosen a jacoco.exec, simply Use File > Import... in the IDE, and import it as Coverage session. EclEmma will guide you to enjoy coverage.

Taking advantage of coverage

There are several ways of taking advantage of coverage. Here is one which seems relevant for Eclipse development: rather than focusing on the coverage metrics, coverage can be used to detect not covered code and decide on what tests to add. This uses coverage as a way to identify which critical pieces of code seem to be missing test coverage. If those are pretty critical, difficult, error-prone, frequently changes... that are not covered, it most likely means that in order to provide efficient error detection for the project, some effort could be spent on writing a test that verify this critical part to avoid regressions later. Gor example, if you're about to make some refactoring, it's interesting to check whether the code you're about to change is covered, in order to detect potential issues with your change, and if the code isn't covered then it's worth considering the addition of a test as a "preparation" of your refactoring.

Focusing on the percentage may be wrong: indeed, in Java and UI development, it's pretty frequent that some code is boilerplate, or even is never meant to be invoked in realistic scenarios. Things such as Exception handling are sometimes pure defensive programming and there is not much profit in making extra-effort to cover those lines.

Session tests

Session Tests in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.

Performance tests

See the Performance Tests How-to

Profiling performance tests

It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:

  1. Download the tests for the build you are interested in (available from same download page as the build itself).
  2. Unzip the test framework, and follow the instructions in the readme.html to configure your tests
  3. Create a properties file (let's say profile.properties) with the following contents:
 extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot
  1. Invoke the performance test and specify the properties file location:
 runtests "-Dtest.target=performance" -properties profile.properties <yourtarget>

If your performance test is a session test, a more complex spell is needed in your profile.properties file:

 extraVMargs=-Dsetup.override.vmArgs=agentlib:yjpagent==sampling,onexit==snapshot

This instructs the Session Tests framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.

The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various other options for configuring the headless profile agent.

API tests

The Eclipse project also runs an suite of API tests during builds. These tests capture problems such as breaking API changes, and correct evolution of bundle version numbers. For more details on these tests, refer to PDE/API_Tools.

UI tests

The table below gathers pros and cons of tools making UI testing easier. It should help developers to choose the most appropriate tool.

UI testing tools
Tool License Recorder Pros Cons Remarks/Comments
Window Tester

Open Source

Yes

  • Stable
  • creates JUnit tests, can be launched along with other tests
  • easy API
  • good documentation
  • OpenSource
  • very easy to modify generated code
  • large footprint when creating tests (around 60MB for WindowTester plug-ins)
  • sensitive for focus changing, very hard to debug these UI tests (i.e. changing focus to the host Eclipse causes exceptions)

RCPTT

Yes

  • Designed specially to test Eclipse-based apps - plugins
  • Supports SWT/JFace, Platform, IDE, GEF, and GMF
  • Contexts for AUT state modeling
  • Let you to run any subset of your tests in any order
  • Realtime Eclipse code instrumentation and Runtime Intelligence
  • Modern Eclipse-based IDE
  • BDD-style scripting language (Eclipse Command Language)
  • Maven/Tycho plugin, Ant/CLI tool, and RCPTT Server to distribute test runs in the Cloud
  • Works on OSX 64, Linux 32/64, Window 32/64
  • Eclipse 3.5-4.5 and upcoming releases.
  • Part of Eclipse Foundation.
QF-Test

Commercial, free license available for Open Source projects

Yes

  • very robust test automation based on "soft" events at Windows/Gtk level
  • support for native dialogs like FileDialog
  • excellent component recognition, including automatic naming for Eclipse/RCP components
  • supports testing combined SWT/Swing applications
  • tests are batch and thus ant-runnable producing a detailed run-log and XML and HTML reports
  • Jython or Groovy scripting available
  • comprehensive manual and tutorial available online; mailing list with searchable archive
  • Videos (for beginners and advanced users)
  • very responsive support
  • Eclipse/SWT support limited to Windows and Linux, both x86 and x86_64, Solaris Gtk support available but no Mac
  • QF-Test is Java application but not integrated into Eclipse
  • tests don't compile to JUnit
  • Developers can integrate QF-Test in their launch configuration.
  • replaces the org.eclipse.swt plugin (corresponding patches are available, could be applied automatically as part of daily builds)
  • also supports GEF/GMF
  • also supports web testing (including AJAX-toolkits like GWT, Vaadin etc. & HTML5)
  • also supports Java Swing, JavaFX and SWT testing
  • Jenkins Plugin
Jubula

Open Source, EPL

Tests specified by drag & drop without need to record.
Recording is possible as well.

  • Supports Swing, SWT/RCP/GEF, JavaFX, HTML and iOS applications
  • Works on Windows, Unix/Linux and Mac
  • Black-box testing - no source code needed to execute tests
  • Robust, high level test automation without any coding effort
  • Incorporates best practices of software development into tests
  • Suitable for agile processes because tests can be written before an app is available
  • Robust object recognition heuristic
  • Tests can be run from command line for continuous integration
  • Since 2015 there is also a programmatic API for writing tests using Java to be executed via e.g. JUnit
  • Extensible to add support for own components / actions
  • tests don't compile to JUnit
  • does not support combined SWT/Swing applications
  • not extendable without knowledge about RCP-Programming
  • few elements missing, like loops and if-conditions
  • Tests are written into XML format at back-end.
Eclipse Project
SWT Bot

EPL

Yes

  • eclipse project, still being improved
  • easy API, simple to get started with
  • open source, hosted at eclipse.org
  • small tool, less than 1 MB (including source)
  • very modular, good test coverage, easy to contribute fixes and enhancements
  • long responses on important bugs e.g. 271132
  • cannot test native dialogs (File Dialogs, Color Dialogs, etc) see 164192
  • Leverages SWT/Eclipse capabilities to perform most operations. Support for new things is highly dependent on capabilities offered by SWT/Eclipse bug 164192
Automated GUI Recorder

EPL

Yes

  • ability to provide your own Widget Resolver
  • open source
  • seems pretty stable, used by TPTP to do UI tests
  • large footprint when creating tests (probably around 50MB - TPTP Platform and TPTP Test Tools)
  • test descriptions only in XML, verification points in Java can be added though
  • didn't find any way to test absence of a dialog
  • can't run tests in debug mode
  • provided as is (because IBM pulled resources out of TPTP)

TestComplete

Commercial

Yes

  • looks nice and powerful
  • got nice features
  • excellent screenshot visualiser identifies and highlights both mapped and unmapped UI components with mouse-over
  • scripts in various languages (VBScript, JScript, DelphiScript, C++Script, C#Script)
  • command-line support, see TestComplete Command Line
  • is implemented using COM technology
  • works only on Windows
  • RCP application launched from TestComplete test did not exit cleanly, zombie process remained.

Squish

Commercial

Yes, free trial available

  • lots of nice features
  • fully cross-platform
  • tests RCP, SWT, Swing, JavaFx, Web, Qt, Android, iOS, native Windows, native macOS and more
  • full access to all Java APIs via introspection
  • powerful object recognition mechanism
  • scripts in various scripting languages (JavaScript, Python, Perl, Ruby or Tcl)
  • powerful Eclipse-based IDE
  • fully integrated BDD (Behavior Driven Development) support 100% compatible to Cucumber
  • support for native dialogs
  • good documentation
  • responsive technical support
  • Eclipse, Jenkins, Ant integration just to mention a few
  • Pricing starts at 2400 EUR per named user (see Pricing for some examples

The table is in progress. Feel free to update it, if you know useful features or serious limitations of the mentioned tools. You are also welcome to add new tools.

More we know about those tools, easier to choose the best one.

Notes on what you should be evaluating on

  • CI support -- ability to run tests from command line/ant
  • Cross platform support -- developers use a variety of platforms -- windows, linux, mac
  • Scripting capabilities -- vendor script/Java/jruby ?
  • Reporting capabilities -- html/xml/push to database ?

Back to Eclipse Project home

Back to the top