Jump to: navigation, search

Difference between revisions of "SWTBot/Automate test execution"

(With Ant or Command-Line)
(15 intermediate revisions by the same user not shown)
Line 6: Line 6:
 
On Linux, you need to have a X session started with a Window Manager. GTK must be installed.
 
On Linux, you need to have a X session started with a Window Manager. GTK must be installed.
  
== Locally on Linux: use another DISPLAY to save time ==
+
== Locally ==
  
SWTBot requires tested application to have focus in order to work well. If it happens on the DISPLAY you're working on, then you have to wait for tests to end before you can go on working. It's a waste of time. On Linux, you can run your test on another DISPLAY, so you can keep on working while tests are running. You can use '''Xephyr''' for that, since it allows you to see what is happening in another window. You'll also need to have a Window Manager started on this Xephyr. we recommand to run Metacity, but any other WM should run (twm is successfully used too).
+
=== Linux hints ===
 +
 
 +
==== use another DISPLAY to save time ====
 +
SWTBot requires tested application to have focus in order to work well. If it happens on the DISPLAY you're working on, then you have to wait for tests to end before you can go on working. It's a waste of time. On Linux, you can run your test on another DISPLAY, so you can keep on working while tests are running. You can use '''Xephyr''' for that, since it allows you to see what is happening in another window. You'll also need to have a Window Manager started on this Xephyr. we recommand to run '''metacity'''.
 
<pre>
 
<pre>
 
$ Xephyr -screen 1024x768 :51 &
 
$ Xephyr -screen 1024x768 :51 &
Line 15: Line 18:
 
$ ...command that runs SWTBot tests...
 
$ ...command that runs SWTBot tests...
 
</pre>
 
</pre>
 +
 +
The DISPLAY can be set from Eclipse Run configuration, in the Environment tab.
 +
 +
=== Windows hints ===
 +
 +
==== Use Cygwin/X ====
 +
 +
Never tested...
  
 
== On Jenkins ==
 
== On Jenkins ==
  
Use [https://wiki.jenkins-ci.org/display/JENKINS/Xvnc+Plugin Xvnc] or [https://wiki.jenkins-ci.org/display/JENKINS/Xvfb+Plugin Xvfb plugin| in you job to start on X session on a new DISPLAY. Then invoke a "Shell build step" to start a window manager (metacity or twm are working fine, others should be fine too)
+
=== Linux hints ===
 +
* Use [https://wiki.jenkins-ci.org/display/JENKINS/Xvnc+Plugin Xvnc] or [https://wiki.jenkins-ci.org/display/JENKINS/Xvfb+Plugin Xvfb] Jenkins plugin in you job to start on X session on a new DISPLAY.
 +
* Invoke a "Shell build step" to start a window manager: ''metacity'' is recommended, some other may work too, but the oldest ones (such as twm) are known to make some tests failing
 +
 
 +
=== Windows hints ===
 +
* Do Not install Jenkins as a service: it causes focus issues that prevent SWTBot from working well.
  
 
= With Maven and Tycho =  
 
= With Maven and Tycho =  
  
Tycho tests can be run in SWTBot as normal surefire test. The only difference is SWTBot requires '''<useUIHarness>true</useUIHarness> and <useUIThread>false</useUIThread>'''
+
Tycho tests can be run in SWTBot as normal surefire test. First. you need to make sure SWTBot bundles are available in your [[Tycho/Target Platform#Target platform configuration|target platform]]. Then, the only difference is SWTBot requires '''<useUIHarness>true</useUIHarness>''' and '''<useUIThread>false</useUIThread>'''
  
 
<source lang="xml">
 
<source lang="xml">
Line 42: Line 58:
 
</source>
 
</source>
  
 
+
Debug can be enabled using the tycho-surefire-plugin [http://www.eclipse.org/tycho/sitedocs/tycho-surefire/tycho-surefire-plugin/test-mojo.html#debugPort -DdebugPort] flag on command-line.
  
 
Examples:
 
Examples:
Line 50: Line 66:
 
** [http://egit.eclipse.org/w/?p=egit.git;a=blob;f=pom.xml;h=df22dd38eb89f40d05d01f7caf9233bda5706d0f;hb=da053e043f3224c714774746f1f3aa9f639e3fe1 egit parent pom.xml]
 
** [http://egit.eclipse.org/w/?p=egit.git;a=blob;f=pom.xml;h=df22dd38eb89f40d05d01f7caf9233bda5706d0f;hb=da053e043f3224c714774746f1f3aa9f639e3fe1 egit parent pom.xml]
  
= With Ant or Command-Line =
+
= On an existing application, with Ant or Command-Line =
  
  
 
SWTBot allows for tests to be executed from ant or command-line. Here are some steps that you need to perform in order to run tests in a 'headless' way.
 
SWTBot allows for tests to be executed from ant or command-line. Here are some steps that you need to perform in order to run tests in a 'headless' way.
 +
 +
In order to run test from command-line, here is the generic process:
 +
# Install SWTBot in the application you want to test (see [[#Installing headless Testing Framework]])
 +
# Install your tests into the application you want to test (see [[#Installing your SWTBot test plugin]])
 +
# Running the test on the application (see [[#Executing]])
  
 
== A Screencast ==
 
== A Screencast ==
Line 62: Line 83:
  
 
== Installing headless Testing Framework ==
 
== Installing headless Testing Framework ==
 +
 +
* If you're using PDE/Build, those bundles must be installed in your target-platform. If you don't use a target-platform definition, then it has to be installed in the Eclipse which hosts your PDE/Build using the following instructions.
 +
* If you're trying to run tests on an existing application, you'll need to install SWTBot stuff directly in the application (from installer UI or using the following instructions
  
 
=== With p2 (recommanded) ===
 
=== With p2 (recommanded) ===
Line 71: Line 95:
 
</pre>
 
</pre>
  
See [[Equinox/p2/Director_application|Director app wiki page]] for details.
+
See [[Equinox/p2/Director_application|Director app wiki page]] for details and this [http://wiki.eclipse.org/Equinox_p2_director_application/Examples/Install_into_eclipse_using_SDKProfile examples for installing with Ant].
 +
 
 +
=== or unzipping manually (deprecated) ===
  
=== or Manually ===
+
This approach is deprecated because installing bundles just by unzipping does not ensure a successful installation. Using p2 is much better as it will generally work, and if it does not, it tells you why. Please also note that this approach is not supported/tested since SWTBot 2.1.0.
  
 
* Download the '''Headless Testing Framework''' for running tests from within ant. This file is called named something like org.eclipse.swtbot.eclipse.test-2.0.0.187-dev.zip
 
* Download the '''Headless Testing Framework''' for running tests from within ant. This file is called named something like org.eclipse.swtbot.eclipse.test-2.0.0.187-dev.zip
Line 83: Line 109:
 
** org.apache.log4j
 
** org.apache.log4j
  
=== Setup ===
+
== Installing your SWTBot test plugin ==
  
Your folder structure should look like the following:
+
* If you're using PDE/Build, then your test bundle needs to be either available in the workspace then built and installed at the same time as your other bundle, or installed in the target-platform.
 
+
* If you're trying to run tests on an existing application, you'll need to install SWTBot stuff directly in the application (from installer UI or using the following instructions. This can be achieved using the p2.director [http://help.eclipse.org/indigo/index.jsp?topic=/org.eclipse.platform.doc.isv/guide/p2_director.html cammand-line] or [http://wiki.eclipse.org/Equinox_p2_director_application/Examples/Install_into_eclipse_using_SDKProfile Ant task].
<pre>
+
ECLIPSE_HOME
+
+ plugins
+
  | - com.yourplugins
+
  | - ...
+
  | - ...
+
  | - org.eclipse.swtbot.swt.finder
+
  | - org.eclipse.swtbot.eclipse.finder
+
  | - org.hamcrest (and other dependencies)
+
  | - ...
+
  | - ...
+
  | - org.eclipse.swtbot.ant.optional.junit4 (or junit3, but not both) (from the Headless Testing Framework)
+
  | - org.eclipse.swtbot.eclipse.junit4.headless (or junit3.headless, but not both) (from the Headless Testing Framework)
+
  | - org.junit4 (or org.junit, but not both)
+
+ features
+
</pre>
+
  
 
== Executing ==
 
== Executing ==
Line 111: Line 121:
 
Here's a command line argument that's used to execute the tests:
 
Here's a command line argument that's used to execute the tests:
  
<pre>
+
<source lang="bash">
  
$ ECLIPSE_HOME=/path/to/your/application
+
## Where to run the tests?
$ TEST_CLASS=com.yourcompany.product.test.AllTessts # see http://github.com/ketan/swtbot/blob/master/org.eclipse.swtbot.eclipse.finder.test/src/org/eclipse/swtbot/eclipse/finder/AllTests.java for an example
+
# /path/to/your/application must be replaced by the path to your application, which contains the plugins/ folder
$ TEST_APPLICATION_ID=com.yourcompany.example.application # the id of the application that needs to be tested
+
# /path/to/your/workspace is the location of the workspace that will be used to run tests. It can be a non-existing/empty folder, so a new workspace will be created; or an existing workspace folder, which will be reused for the test execution
$ WORKSPACE=/path/to/your/workspace
+
## Which test to run?
$ TEST_PRODUCT_ID=com.yourcompany.product # optional, if you're not using a product, please do not add the -product argument in the command below
+
# com.yourcompany.test.bundle must be replaced by the Bundle-SymbolicName of the bundle which contains the test you want to run
$ TEST_PLUGIN_ID=com.yourcompany.product.test # the id of the plugin containing SWTBot tests
+
# com.yourcompany.product.test.AllTests must be replaced by the name of the Test class you want to run
 +
## On what application should the test run?
 +
# com.yourcompany.product (optional) is the RCP product you want to start while running testss, If you are not using a product, please do not add the -product argument in the command below. Examples: org.eclipse.sdk, com.jboss.jbds.product...
 +
# com.yourcompany.application is the id of the application that needs to be tested. Example: org.eclipse.ui.ide.workbench
  
$ OS=[macosx | win32 | linux]
+
java \
$ WS=[[cocoa|carbon] | gtk | win32] # for macosx, linux, and windows respectively.
+
  -Xms256M -Xmx768M -XX:MaxPermSize=512M #Memory settings and other VM args such as System properties settings \
$ ARCH=[x86|x86_64] # for 32 bit and 64 bit swt binaries.
+
  -jar /path/to/your/application/plugins/org.eclipse.equinox.launcher_*.jar \
 
+
$ $JAVA_HOME/bin/java \
+
  -Xms256M -Xmx768M -XX:MaxPermSize=512M \
+
  -classpath $ECLIPSE_HOME/plugins/org.eclipse.equinox.launcher_1.0.200.v20090128-1500.jar \
+
org.eclipse.core.launcher.Main \
+
 
  -application org.eclipse.swtbot.eclipse.junit4.headless.swtbottestapplication \
 
  -application org.eclipse.swtbot.eclipse.junit4.headless.swtbottestapplication \
-testApplication $TEST_APPLICATION_ID
+
  -product com.yourcompany.product # optional, only if you are working with a product \
  -product $TEST_PRODUCT_ID \ # optional, only if you're working with a product
+
-testApplication com.yourcompany.application \
  -data $WORKSPACE \
+
  -data /path/to/your/workspace \
  formatter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,$ECLIPSE_HOME/$TEST_CLASS.xml \
+
-testPluginName com.yourcompany.test.bundle \
 +
  formatter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,/path/to/reports/com.yourcompany.product.test.AllTests.xml \
 
  formatter=org.apache.tools.ant.taskdefs.optional.junit.PlainJUnitResultFormatter \
 
  formatter=org.apache.tools.ant.taskdefs.optional.junit.PlainJUnitResultFormatter \
-testPluginName $TEST_PLUGIN_ID \
+
  -className com.yourcompany.product.test.AllTests \
  -className $TEST_CLASS \
+
  -os linux -ws gtk -arch x86_64 #Adapt to your os/ws/arch \
  -os $OS -ws $WS -arch $ARCH \
+
 
  -consoleLog -debug
 
  -consoleLog -debug
</pre>
+
</source>
  
 
This will run the test specified by TEST_CLASS and generate the test output in ECLIPSE_HOME/TEST_CLASS.xml. This output can be parsed using junitreport ant task.
 
This will run the test specified by TEST_CLASS and generate the test output in ECLIPSE_HOME/TEST_CLASS.xml. This output can be parsed using junitreport ant task.
Line 174: Line 182:
  
 
The ${library-file} is part of the '''Headless Test Framework''' and can be viewed at http://github.com/ketan/swtbot/blob/master/org.eclipse.swtbot.eclipse.junit4.headless/library.xml. This ant script performs the actual launch for the tests using the command line above.
 
The ${library-file} is part of the '''Headless Test Framework''' and can be viewed at http://github.com/ketan/swtbot/blob/master/org.eclipse.swtbot.eclipse.junit4.headless/library.xml. This ant script performs the actual launch for the tests using the command line above.
 +
 +
=== Debugging from Eclipse ===
 +
 +
You can enable debug using standard JVM debug flags on your test process ''-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y'' and then using Eclipse debug "Remote Java Application", bind it to port 8000 (as specified by ''socket'' in command-line).
 +
 +
= On an already packaged RCP application =
 +
 +
== Alter an RCP application to install and run your tests ==
 +
 +
# Your tests must be available in a p2 repository/update-site
 +
# Use p2 UI or [http://help.eclipse.org/indigo/index.jsp?topic=/org.eclipse.platform.doc.isv/guide/p2_director.html p2 director] in install SWTBot + your test bundles from update-sites into you application
 +
# Use the command-line described earlier to start SWTBot test execution on your application (think about setting the '''-product''' and/or '''-application''' parameters)
 +
 +
You can debug this from Eclipse using the command-line arguments described earlier.
 +
 +
== Or, use your IDE to run tests on an existing application ==
 +
 +
# Install SWTBot in your IDE to access the SWTBot configuration menus
 +
# Create a Target Platform made of your application + SWTBot runtime.
 +
# Import your test bundles as projects in your workspace
 +
# Then use the "SWTBot tests" wizard to configure the execution: think about including all Target Platform + SWTBot + your tests bundles; and configure product to refer to your application.
 +
You'll also be able to use Debug.

Revision as of 04:23, 29 August 2013


SWTBot
Website
Update Sites
Community
Mailing List
Forums/Newsgroups
IRC
Contribute
Open Bugzilla tickets
Open Gerrit reviews
Browse Source
Continuous Integration


Environment requirements

On Mac and Windows, you can only run SWTBot tests in the main UI session, and you have to wait for tests to finish before you can keep on working. On Linux, you need to have a X session started with a Window Manager. GTK must be installed.

Locally

Linux hints

use another DISPLAY to save time

SWTBot requires tested application to have focus in order to work well. If it happens on the DISPLAY you're working on, then you have to wait for tests to end before you can go on working. It's a waste of time. On Linux, you can run your test on another DISPLAY, so you can keep on working while tests are running. You can use Xephyr for that, since it allows you to see what is happening in another window. You'll also need to have a Window Manager started on this Xephyr. we recommand to run metacity.

$ Xephyr -screen 1024x768 :51 &
$ export DISPLAY=:51
$ metacity --sm-disable --replace &
$ ...command that runs SWTBot tests...

The DISPLAY can be set from Eclipse Run configuration, in the Environment tab.

Windows hints

Use Cygwin/X

Never tested...

On Jenkins

Linux hints

  • Use Xvnc or Xvfb Jenkins plugin in you job to start on X session on a new DISPLAY.
  • Invoke a "Shell build step" to start a window manager: metacity is recommended, some other may work too, but the oldest ones (such as twm) are known to make some tests failing

Windows hints

  • Do Not install Jenkins as a service: it causes focus issues that prevent SWTBot from working well.

With Maven and Tycho

Tycho tests can be run in SWTBot as normal surefire test. First. you need to make sure SWTBot bundles are available in your target platform. Then, the only difference is SWTBot requires <useUIHarness>true</useUIHarness> and <useUIThread>false</useUIThread>

<build>
    <plugins>
        <plugin>
            <groupId>org.eclipse.tycho</groupId>
            <artifactId>tycho-surefire-plugin</artifactId>
            <version>${tycho.version}</version>
            <configuration>
              <useUIHarness>true</useUIHarness>
              <useUIThread>false</useUIThread>
              <product>org.eclipse.sdk.ide</product>
              <application>org.eclipse.ui.ide.workbench</application>
            </configuration>
        </plugin>
    </plugins>
</build>

Debug can be enabled using the tycho-surefire-plugin -DdebugPort flag on command-line.

Examples:

On an existing application, with Ant or Command-Line

SWTBot allows for tests to be executed from ant or command-line. Here are some steps that you need to perform in order to run tests in a 'headless' way.

In order to run test from command-line, here is the generic process:

  1. Install SWTBot in the application you want to test (see #Installing headless Testing Framework)
  2. Install your tests into the application you want to test (see #Installing your SWTBot test plugin)
  3. Running the test on the application (see #Executing)

A Screencast

Videos speak louder than pictures and words put together:

Installing headless Testing Framework

  • If you're using PDE/Build, those bundles must be installed in your target-platform. If you don't use a target-platform definition, then it has to be installed in the Eclipse which hosts your PDE/Build using the following instructions.
  • If you're trying to run tests on an existing application, you'll need to install SWTBot stuff directly in the application (from installer UI or using the following instructions

With p2 (recommanded)

From your favourite RCP application using p2, run, by replacing SWTBot site by the SWTBot update site of the version you want to use

SWTBOT_SITE=...
java -jar plugins/org.eclipse.equinox.laucher_*.jar -application org.eclipse.p2.director -consolelog -repository $SWTBOT_SITE -installIU org.eclipse.swtbot.eclipse.test.junit.feature.group -installIU org.hamcrest

See Director app wiki page for details and this examples for installing with Ant.

or unzipping manually (deprecated)

This approach is deprecated because installing bundles just by unzipping does not ensure a successful installation. Using p2 is much better as it will generally work, and if it does not, it tells you why. Please also note that this approach is not supported/tested since SWTBot 2.1.0.

  • Download the Headless Testing Framework for running tests from within ant. This file is called named something like org.eclipse.swtbot.eclipse.test-2.0.0.187-dev.zip
  • You'll need the following plugins from swtbot (you don't need all the swtbot packages)
    • org.eclipse.swtbot.swt.finder
    • org.eclipse.swtbot.eclipse.finder
    • org.hamcrest
    • org.junit4(recommended) you may also use org.junit
    • org.apache.log4j

Installing your SWTBot test plugin

  • If you're using PDE/Build, then your test bundle needs to be either available in the workspace then built and installed at the same time as your other bundle, or installed in the target-platform.
  • If you're trying to run tests on an existing application, you'll need to install SWTBot stuff directly in the application (from installer UI or using the following instructions. This can be achieved using the p2.director cammand-line or Ant task.

Executing

The key is to run the org.eclipse.swtbot.eclipse.junit4.headless.swtbottestapplication

Command-line

Here's a command line argument that's used to execute the tests:

## Where to run the tests?
# /path/to/your/application must be replaced by the path to your application, which contains the plugins/ folder
# /path/to/your/workspace is the location of the workspace that will be used to run tests. It can be a non-existing/empty folder, so a new workspace will be created; or an existing workspace folder, which will be reused for the test execution
## Which test to run?
# com.yourcompany.test.bundle must be replaced by the Bundle-SymbolicName of the bundle which contains the test you want to run
# com.yourcompany.product.test.AllTests must be replaced by the name of the Test class you want to run
## On what application should the test run?
# com.yourcompany.product (optional) is the RCP product you want to start while running testss, If you are not using a product, please do not add the -product argument in the command below. Examples: org.eclipse.sdk, com.jboss.jbds.product...
# com.yourcompany.application is the id of the application that needs to be tested. Example: org.eclipse.ui.ide.workbench
 
java \
 -Xms256M -Xmx768M -XX:MaxPermSize=512M #Memory settings and other VM args such as System properties settings \
 -jar /path/to/your/application/plugins/org.eclipse.equinox.launcher_*.jar \
 -application org.eclipse.swtbot.eclipse.junit4.headless.swtbottestapplication \
 -product com.yourcompany.product # optional, only if you are working with a product \
 -testApplication com.yourcompany.application \
 -data /path/to/your/workspace \
 -testPluginName com.yourcompany.test.bundle \
 formatter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,/path/to/reports/com.yourcompany.product.test.AllTests.xml \
 formatter=org.apache.tools.ant.taskdefs.optional.junit.PlainJUnitResultFormatter \
 -className com.yourcompany.product.test.AllTests \
 -os linux -ws gtk -arch x86_64 #Adapt to your os/ws/arch \
 -consoleLog -debug

This will run the test specified by TEST_CLASS and generate the test output in ECLIPSE_HOME/TEST_CLASS.xml. This output can be parsed using junitreport ant task.

Ant

<echo>SWTBot test</echo>
<java dir="${eclipse.home}" fork="true" output="${eclipse.test.home}/output.txt" logError="true" classname="org.eclipse.core.launcher.Main" failonerror="false">
	<classpath>
		<fileset dir="${eclipse.home}/plugins">
			<include name="org.eclipse.equinox.launcher_*.jar"/>
		</fileset>
	</classpath>
	<jvmarg line="-Xms256M -Xmx768M -XX:MaxPermSize=512M"/>
	<arg line="-application org.eclipse.swtbot.eclipse.junit4.headless.swtbottestapplication"/>
	<arg line="ARGUMENTS"/>
	<arg line="FROM"/>
	<arg line="COMMAND LINE"/>
	<arg line="EXAMPLE ABOVE"/>
</java>

Pre-prepared ant tasks

The script at http://github.com/ketan/swtbot/blob/master/org.eclipse.swtbot.releng/test-sandbox/org.eclipse.swtbot.eclipse.finder.test.xml shows an example of how you can use the standard ant task available with SWTBot:

<ant target="swtbot-test" antfile="${library-file}" dir="${eclipse-home}">
	<property name="data-dir" value="${temp-workspace}" />
	<property name="plugin-name" value="${plugin-name}" />
	<property name="classname" value="org.eclipse.swtbot.eclipse.finder.AllTests" />
	<property name="vmargs" value=" -Xms128M -Xmx368M -XX:MaxPermSize=256M ${jvmOption}" />
</ant>

The ${library-file} is part of the Headless Test Framework and can be viewed at http://github.com/ketan/swtbot/blob/master/org.eclipse.swtbot.eclipse.junit4.headless/library.xml. This ant script performs the actual launch for the tests using the command line above.

Debugging from Eclipse

You can enable debug using standard JVM debug flags on your test process -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y and then using Eclipse debug "Remote Java Application", bind it to port 8000 (as specified by socket in command-line).

On an already packaged RCP application

Alter an RCP application to install and run your tests

  1. Your tests must be available in a p2 repository/update-site
  2. Use p2 UI or p2 director in install SWTBot + your test bundles from update-sites into you application
  3. Use the command-line described earlier to start SWTBot test execution on your application (think about setting the -product and/or -application parameters)

You can debug this from Eclipse using the command-line arguments described earlier.

Or, use your IDE to run tests on an existing application

  1. Install SWTBot in your IDE to access the SWTBot configuration menus
  2. Create a Target Platform made of your application + SWTBot runtime.
  3. Import your test bundles as projects in your workspace
  4. Then use the "SWTBot tests" wizard to configure the execution: think about including all Target Platform + SWTBot + your tests bundles; and configure product to refer to your application.

You'll also be able to use Debug.