Jump to: navigation, search

WTP Performance Tests/How To

How to create a performance test case?

The Performance Tests HowTo document describes well how to create a performance test case. Useful tips:

  • Start with a clean Debry DB instance. Do not reuse a DB that was used for other purposes.
  • Always make an archive of the Derby DB before each run. In case of wrongly confugured runs of the performance tests you may "polute" your DB with unneeded data. In such case you can easily restore the DB from the archive.
  • Make sure that you are not connected to the Derby DB during test execution. An active connection may prevent the establishment of new connections and this can result in NullPointerException.
  • Use the following pattern for the -Declipse.perf.config property:
-Declipse.perf.config=build=<build_id>;host=<host_where_the_tests_are_run>;jvm=<jvm_used>

Examples:

-Declipse.perf.config=build=3.0_200808010000_200808010000;host=my_host;jvm=sun
-Declipse.perf.config=build=I20080915-0000;host=my_host;jvm=sun

How to create a baseline?

The baseline is called a perfromance test run, which results are used for comparision with future runs. A baseline can be easily created by running the performance test suite as a JUnit Plug-in Test. Just right-click on the test suite's Java class and call Run As > Run JUnit Plug-in Test. Create the run configuration with Program arguments:

-os ${target.os} -ws ${target.ws} -arch ${target.arch} -nl ${target.nl}

and VM arguments that could look like:

-Xms256M -Xmx256M
-Declipse.perf.dbloc=C:\eclipse\perf\wtp-perf-db
-Declipse.perf.config=build=3.0_200808010000_200808010000;host=my_host;jvm=sun

It is very important that the build part in the eclipse.perf.config VM argument has the format X.Y_YYYYMMDDHHMM_YYYYMMDDHHMM, where

  • X.Y is the version of the build, and
  • YYYYMMDDHHMM is the timestamp of the build.

This format will be later required by the graph generation tool.

How to run tests compared to a baseline?

Here we have another run of the performance test suite, but this time compared to an already created baseline in the DB. The run configuration is similar to that of creating a baseline - a JUnit Plug-in Test configuration with Program arguments:

-os ${target.os} -ws ${target.ws} -arch ${target.arch} -nl ${target.nl}

and VM arguments that could look like:

-Xms256M -Xmx256M
-Declipse.perf.dbloc=C:\eclipse\perf\wtp-perf-db
-Declipse.perf.config=build=I20080915-0000;host=my_host;jvm=sun
-Declipse.perf.assertAgainst=build=3.0_200808010000_200808010000;host=my_host;jvm=sun

Note that this time the format of the buildpart in the eclipse.perf.config VM argument is different: TYYYYMMDD-HHMM where

  • T is the build type: R, M, S, I or N, and
  • YYYYMMDD-HHMM is the timestamp of the build.

This format is again important for the graph generation tool.

The value of the -Declipse.perf.assertAgainst VM argument must point to the value of -Declipse.perf.config on the run configution of the baseline we want to compare against.

How to create graph results?

After we have created the baseline and run the performance tests against the baseline, we can now run the graph generation tool to create the comparision graphics. The tool documentation does not provide enough information how to successfully run the tool. There is also a limitation in the tool to generate results for the WTP tests. The below info feels the gap.

First, the org.eclipse.test.performance plug-in must be checked out and the patch from bug 244986 needs to be applied.

Second, the org.eclipse.releng.basebuilder project must be checked out from CVS. It contains the graph generetion tool - under /plugins/org.eclipse.test.performance.ui.

Finally, the tool can be executed as an Eclipse application. The run configuration must have the following parameters.

Program to Run:

Run as application: org.eclipse.test.performance.ui.resultGenerator

Program arguments that look like:

-baseline 3.0_200808010000_200808010000 
-current I20080915-0000 
-jvm sun 
-config my_host 
-scenario.pattern org.eclipse.% 
-output C:\eclipse\perf/perfRoot/results/graph 
-print

VM arguments that look like:

-Xms40m
-Xmx512m
-XX:MaxPermSize=256M
-Dosgi.ws=win32
-Dosgi.os=win32
-Dosgi.arch=x86
-Declipse.perf.dbloc=C:\eclipse\perf/wtp-perf-db
-classpath C:\eclipse\perf\perfRoot\eclipse\plugins\org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar

The tool generates graphics for the comparision between the builds specified by the -baseline and -current program arguments related to the VM, specified by -jvm, and host, specified by -config. Graphics are generated only for scenarios filetered in the -scenario.pattern argument - it is very important to give org.eclipse.% here. The tool looks for data in the DB specified by the -Declipse.perf.dbloc argument and writes the result in the location specified by the -output argument.

How to run performance tests automatically?

Setting up the environment

A couple of plugins need to be checked out from CVS:

  • org.eclipse.wtp.releng.wtpbuilder - contains the Java code for executing the performance test process
  • releng.wtpbuilder - contains Ant scripts and other XML descriptors that define the performance test process

The performance test results are uploaded to the WTP download web site. The file transfer is done over SCP. Therefore, it is needed to install an SSH client on your system and configure "public key authentication" with the build.eclipse.org SSH server using your committer account. There are many guidelines on Google how to do this. Here is the guideline in the OpenSSH documentation.

Running the NullBuilder

The performance test process runs against the builds declared on the WTP download page. The builds, that the tests are already executed against, are described in a special property file - perf_completed_builds.properties. Therefore, to run the peformance tests against a certain declared build, you should have all other, but that one, listed in the property file.

The easiest way to generate the property file with all the declared builds is by using the NullBuilder. The run configuration should have the following parameters.

Project:

org.eclipse.wtp.releng.wtpbuilder

Main class:

org.eclipse.wtp.releng.wtpbuilder.NullBuilder

Program arguments:

-baseos win32
-basews win32
-basearch x86
-build.home C:\eclipse\perf

Environment variables to set:

BASEARCH=x86
BASEOS=win32
BASEWS=win32
BUILD_HOME=C:\eclipse\perf
RELENG=webtools.maps/releng
RELENGWTPBUILDER=releng.wtpbuilder

Note: Some of the above settings may not be needed, but it is save to have them.

The result of the NullBuilder is a completed_builds.properties in the folder specified by the BUILD_HOME variable. Rename this file to perf_completed_builds.properties. Then open it in an editor at comment (with # char) the build that you want to run the performance tests against.

Note: The NullBuilder script will never finish by itself. You need to terminate it manually. You have to first check if the completed_builds.properties already contains all declared builds and then to kill the process.

Running the PerfBuilder

Once we have the perf_completed_builds.properties file in the BUILD_HOME folder, make sure that only the one build is commented in this file - the one we want to execute the performance tests against.

Now we are ready to run the PerfBuilder, which triggers the automated performance test procedure. The run configuration should have the following parameters.

Project:

org.eclipse.wtp.releng.wtpbuilder

Main class:

org.eclipse.wtp.releng.wtpbuilder.PerfBuilder

Program arguments:

-baseos win32
-basews win32
-basearch x86
-login <your_committer_login_username>
-build.home C:\eclipse\perf

VM arguments:

-Xms256m
-Xmx512m
-XX:MaxPermSize=256m

Environment variables to set:

BASEARCH=x86
BASEOS=win32
BASEWS=win32
BUILD_HOME=C:\eclipse\perf
DBPASSWD=guest
DBUSER=guest
LOCAL_PREREQS_CACHE=C:\eclipse\perf\cache
RELENG=webtools.maps/releng
RELENG_BUILDER=releng.wtpbuilder
RELENGWTPBUILDER=releng.wtpbuilder

Note: Some of the above settings may not be needed, but it is save to have them.

The PerfBuilder executes in several steps.

  1. MAIN - executes the performance tests
    1. Cleans up old logs and working directory: BUILD_HOME/perfRoot and BUILD_HOME/workDir folders.
    2. Downloads Eclipse binaries in the LOCAL_PREREQS_CACHE folder. The download is done from the main Eclipse.org download site, so this step could be very slow. You can hurry up the process if you download the binaries manually - this way you have the choice to use a faster mirror site.
    3. Extract the binaries in the BUILD_HOME/perfRoot folder.
    4. Execute the performance tests.
      • Results are generated in the BUILD_HOME/perfRoot/results folder.
      • Results are also written in the Derby DB - location: BUILD_HOME/wtp-perf-db/perfDB. This location can be changed in the perf.properties file (see the next section).
  2. GRAPH - generates comparison graphs
    1. Checks out the org.eclipse.releng.basebuilder project from the CVS. The checkout tag can be specified in the build.xml file (see the next section).
    2. Execute the org.eclipse.test.performance.ui.resultGenerator application.
      • Results are generated in the BUILD_HOME/perfRoot/results/graph folder.
  3. UPLOAD - uploads the results to the WTP download page
    1. Packs the content of the BUILD_HOME/perfRoot/results folder and writes the archive in the BUILD_HOME/archives folder.
    2. Uploads the content of the BUILD_HOME/perfRoot/results to the WTP download site.
  4. CLEAN - cleans logs, working directories and WTP binaries

Note: A manual step is needed for the results to appear on the WTP download site. The UPLOAD step uploads them on the download server:

build.eclipse.org/downloads/webtools/downloads/drops/R<X.Y>/R-<X.Y.Z>-<YYYYMMDDHHMMSS>/perfresults

But, in order to appear on the web page the file permission must be changed. The following command must be executed on the above folder:

chmod -R g+rw,o+rx perfresults

After this is done, an undefined amount of time should pass before the results appear on the web site - a Performance Results link appear on the bottom of the web page for the specific build.

Note: The CLEAN step cleans the BUILD_HOME/perfRoot and BUILD_HOME/workDir folders, but also the WTP binaries downloaded in the LOCAL_PREREQS_CACHE folder. This means that after the procedure finishes there aren't any logs to look at (except in the produced archive) and the WTP binaries have to be downloaded again on the next execution on the same WTP build. Therefore, it is highly recommended to comment this step (see next section for howto) when debugging the performance test procedure.

Important files for configuration and customization

PerfBuilder

Location:

/org.eclipse.wtp.releng.wtpbuilder/src/org/eclipse/wtp/releng/wtpbuilder/perf/PerfBuilder.java

In this file you can comment execution step if you don't need to run them. This can be done in the build(Build) method.

You can also enable the debug mode of Ant for a certain step. This is done by adding the "-d" option flag. For example, to enable the debug mode for the GRAPH step, change line:

Main.main(new String[] { "-f", buildScript, GRAPH}); //$NON-NLS-1$

to:

Main.main(new String[] { "-d", "-f", buildScript, GRAPH}); //$NON-NLS-1$

build.xml

Location:

/releng.wtpbuilder/distribution/wtp.perf/build.xml

This the main Ant script file that defines the execution of the performance tests. There are lots of customizations that can be done here. The most important ones a for the GRAPH step. This step is executed in the graph target.

You can change the tag the org.eclipse.releng.basebuilder project is checked out to:

<cvs
    cvsRoot=":pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse"
    package="org.eclipse.releng.basebuilder"
    dest="${perfRoot}"
    command="export"
    tag="r34x_v20080904" />   <-- here

You can also change the arguments which are used for the execution of the org.eclipse.test.performance.ui.resultGenerator application. See the java Ant task that is around line 524.

perf.properties

Location:

/releng.wtpbuilder/distribution/wtp.perf/perf.properties

Here you can set various properies regarding the Derby DB and how the results of the test are written there.

perf.xml

Location:

/releng.wtpbuilder/distribution/wtp.perf/perf.xml

This XML file defines the execution of the MAIN step: which test suites are executed, how the HTML results are generated, etc.

Here you can define which test suites to be executed: to remove suites, add new ones and modify existing.

performance-tests.xml

Location:

/releng.wtpbuilder/distribution/wtp.perf/performance-tests.xml

This XML file defines how each of the test suites defined in perf.xml are executed. Various parameters can be customized here.