Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Eclipse/Testing"
(→UI tests) |
|||
Line 129: | Line 129: | ||
*tests are batch and thus ant-runnable; Jython or Groovy scripting available<br> | *tests are batch and thus ant-runnable; Jython or Groovy scripting available<br> | ||
*comprehensive [http://www.qfs.de/en/qftest/manual.html manual] and [http://www.qfs.de/en/qftest/tutorial.html tutorial] available online; mailing list with [http://www.qftest.com/archive/qftest-list/ searchable archive] | *comprehensive [http://www.qfs.de/en/qftest/manual.html manual] and [http://www.qfs.de/en/qftest/tutorial.html tutorial] available online; mailing list with [http://www.qftest.com/archive/qftest-list/ searchable archive] | ||
+ | *very responsive support | ||
| | | | ||
Line 136: | Line 137: | ||
| | | | ||
− | *replaces the org.eclipse.swt plugin (corresponding patches | + | *replaces the org.eclipse.swt plugin (corresponding patches are available, could be applied automatically as part of daily builds)<br> |
*also supports web testing | *also supports web testing | ||
Revision as of 05:31, 17 July 2009
The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the JUnit test framework. This page contains links to information about how to run and create automated tests for the platform.
Contents
Correctness tests
The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests. These tests are run every build on Windows, Linux, and Mac OSX.
- See this page for information about running JUnit4 tests within the Automated Testing Framework.
Running tests from within Eclipse
Correctness tests can be run manually from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:
- Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:
- org.junit - The JUnit test framework
- org.eclipse.test - The basic infrastructure for running Eclipse tests
- org.eclipse.core.tests.harness - Various utility pieces used by many tests
- org.eclipse.core.tests.runtime - Tests for the runtime component
- org.eclipse.core.tests.resources - Tests for the resources component
- org.eclipse.ui.tests.harness - Various utility pieces used by UI tests
- org.eclipse.ui.tests - Tests for the Eclipse UI
- In the Navigator or Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:
- org.eclipse.ui.tests.UiTestSuite - runs all UI tests
- org.eclipse.core.tests.runtime.AutomatedTests - runs all runtime tests
- org.eclipse.core.tests.resources.AutomatedTests - runs all resource tests
- Select Run > Run...
- Choose the "JUnit Plug-in Test" category, and click the button to create a new test
- On the "Main" tab, select the appropriate application for that test. Here are some important applications to know about:
- [No Application] - Headless Mode - for running headless core (runtime, osgi, resource) tests
- org.eclipse.ui.ide.workbench - for running UI tests
- Click Run.
Using EasyMock
In the Eclipse Galileo release the utility EasyMock was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.
To use EasyMock in your tests, simply load the org.easymock bundle from the Orbit repository. There is a convenience easymock.psf project set file in the org.eclipse.test bundle to facilitate loading of this bundle.
Note that EasyMock requires Java 5 or greater. At this time, we run tests on Java 1.4.2 and Java 5. This means you need to add the following to the Ant target for your tests (contact releng for details):
<target name="YourTestTarget" depends="setJVMProperties"> <property name="jvm" value="${J2SE-5.0}" /> <condition property="skip.test"> <not> <isset property="J2SE-5.0" /> </not> </condition> ... remaining target definition as usual
Session tests
Session Tests in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.
Performance tests
See the Performance Tests How-to
Profiling performance tests
It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:
- Download the tests for the build you are interested in (available from same download page as the build itself).
- Unzip the test framework, and follow the instructions in the readme.html to configure your tests
- Create a properties file (let's say profile.properties) with the following contents:
extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot
- Invoke the performance test and specify the properties file location:
runtests "-Dtest.target=performance" -properties profile.properties <yourtarget>
If your performance test is a session test, a more complex spell is needed in your profile.properties file:
extraVMargs=-Dsetup.override.vmArgs=agentlib:yjpagent==sampling,onexit==snapshot
This instructs the Session Tests framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.
The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various other options for configuring the headless profile agent.
UI tests
The table below gathers pros and cons of tools making UI testing easier. It should help developers to choose the most appropriate tool.
Tool | License | Recorder | Pros | Cons | Remarks/Comments |
---|---|---|---|---|---|
Window Tester |
Commercial, Eclipse committers are allowed to use it for free |
Yes |
|
|
|
QF-Test |
Commercial, free license available for Open Source projects |
Yes |
|
|
|
GUI Dancer |
Commercial |
No, "not a capture/replay tool" |
|
|
|
SWT Bot |
EPL |
No |
|
| |
Automated GUI Recorder |
EPL |
Yes |
|
|
|
TestComplete |
Commercial |
? |
|
|
|
Squish |
Commercial |
Yes |
Waiting for an evaluation copy, please update if you already use it |
Waiting for an evaluation copy, please update if you already use it |
|
The table is in progress. Feel free to update it, if you know useful features or serious limitations of the mentioned tools. You are also welcome to add new tools.
More we know about those tools, easier to choose the best one.
Notes on what you should be evaluating on
- CI support -- ability to run tests from command line/ant
- Cross platform support -- developers use a variety of platforms -- windows, linux, mac
- Scripting capabilities -- vendor script/Java/jruby ?
- Reporting capabilities -- html/xml/push to database ?
Back to Eclipse Project home