Jump to: navigation, search

Difference between revisions of "WTP Performance Tests/How To"

(How to create graph results?)
Line 56: Line 56:
 
Second, the '''org.eclipse.releng.basebuilder''' project must be checked out from CVS. It contains the graph generetion tool - under '''/plugins/org.eclipse.test.performance.ui'''.  
 
Second, the '''org.eclipse.releng.basebuilder''' project must be checked out from CVS. It contains the graph generetion tool - under '''/plugins/org.eclipse.test.performance.ui'''.  
  
Finally, the tool can be executed as an Eclipse application. The run configuration must have '''Program arguments''' that look like:
+
Finally, the tool can be executed as an Eclipse application. The run configuration must have the following parameters.
 +
 
 +
'''Program to Run''':
 +
 
 +
Run as application: org.eclipse.test.performance.ui.resultGenerator
 +
 
 +
'''Program arguments''' that look like:
 
  -baseline 3.0_200808010000_200808010000  
 
  -baseline 3.0_200808010000_200808010000  
 
  -current I20080915-0000  
 
  -current I20080915-0000  
Line 65: Line 71:
 
  -print
 
  -print
  
and '''VM arguments''' that look like:
+
'''VM arguments''' that look like:
 
  -Xms40m
 
  -Xms40m
 
  -Xmx512m
 
  -Xmx512m

Revision as of 03:38, 7 October 2008

How to create a performance test case?

The Performance Tests HowTo document describes well how to create a performance test case. Useful tips:

  • Start with a clean Debry DB instance. Do not reuse a DB that was used for other purposes.
  • Always make an archive of the Derby DB before each run. In case of wrongly confugured runs of the performance tests you may "polute" your DB with unneeded data. In such case you can easily restore the DB from the archive.
  • Make sure that you are not connected to the Derby DB during test execution. An active connection may prevent the establishment of new connections and this can result in NullPointerException.
  • Use the following pattern for the -Declipse.perf.config property:
-Declipse.perf.config=build=<build_id>;host=<host_where_the_tests_are_run>;jvm=<jvm_used>

Examples:

-Declipse.perf.config=build=3.0_200808010000_200808010000;host=my_host;jvm=sun
-Declipse.perf.config=build=I20080915-0000;host=my_host;jvm=sun

How to create a baseline?

The baseline is called a perfromance test run, which results are used for comparision with future runs. A baseline can be easily created by running the performance test suite as a JUnit Plug-in Test with Program arguments:

-os ${target.os} -ws ${target.ws} -arch ${target.arch} -nl ${target.nl}

and VM arguments that could look like:

-Xms256M -Xmx256M
-Declipse.perf.dbloc=C:\eclipse\perf\wtp-perf-db
-Declipse.perf.config=build=3.0_200808010000_200808010000;host=my_host;jvm=sun

It is very important that the build part in the eclipse.perf.config VM argument has the format X.Y_YYYYMMDDHHMM_YYYYMMDDHHMM, where

  • X.Y is the version of the build, and
  • YYYYMMDDHHMM is the timestamp of the build.

This format will be later required by the graph generation tool.

How to run tests compared to a baseline?

Here we have another run of the performance test suite, but this time compared to an already created baseline in the DB. The run configuration is similar to that of creating a baseline - a JUnit Plug-in Test configuration with Program arguments:

-os ${target.os} -ws ${target.ws} -arch ${target.arch} -nl ${target.nl}

and VM arguments that could look like:

-Xms256M -Xmx256M
-Declipse.perf.dbloc=C:\eclipse\perf\wtp-perf-db
-Declipse.perf.config=build=I20080915-0000;host=my_host;jvm=sun
-Declipse.perf.assertAgainst=build=3.0_200808010000_200808010000;host=my_host;jvm=sun

Note that this time the format of the buildpart in the eclipse.perf.config VM argument is different: TYYYYMMDD-HHMM where

  • T is the build type: R, M, S, I or N, and
  • YYYYMMDD-HHMM is the timestamp of the build.

This format is again important for the graph generation tool.

The value of the -Declipse.perf.assertAgainst VM argument must point to the value of -Declipse.perf.config on the run configution of the baseline we want to compare against.

How to create graph results?

After we have created the baseline and run the performance tests against the baseline, we can now run the graph generation tool to create the comparision graphics. The tool documentation does not provide enough information how to successfully run the tool. There is also a limitation in the tool to generate results for the WTP tests. The below info feels the gap.

First, the org.eclipse.test.performance plug-in must be checked out and the patch from bug 244986 needs to be applied.

Second, the org.eclipse.releng.basebuilder project must be checked out from CVS. It contains the graph generetion tool - under /plugins/org.eclipse.test.performance.ui.

Finally, the tool can be executed as an Eclipse application. The run configuration must have the following parameters.

Program to Run:

Run as application: org.eclipse.test.performance.ui.resultGenerator

Program arguments that look like:

-baseline 3.0_200808010000_200808010000 
-current I20080915-0000 
-jvm sun 
-config my_host 
-scenario.pattern org.eclipse.% 
-output C:\eclipse\perf/perfRoot/results/graph 
-print

VM arguments that look like:

-Xms40m
-Xmx512m
-XX:MaxPermSize=256M
-Dosgi.ws=win32
-Dosgi.os=win32
-Dosgi.arch=x86
-Declipse.perf.dbloc=C:\eclipse\perf/wtp-perf-db
-classpath C:\eclipse\perf\perfRoot\eclipse\plugins\org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar

The tool generates graphics for the comparision between the builds specified by the -baseline and -current program arguments related to the VM, specified by -jvm, and host, specified by -config. Graphics are generated only for scenarios filetered in the -scenario.pattern argument - it is very important to give org.eclipse.% here. The tool looks for data in the DB specified by the -Declipse.perf.dbloc argument and writes the result in the location specified by the -output argument.

How to run performance tests automatically?

Setting up the environment

A couple of plugins need to be checked out from CVS:

  • org.eclipse.wtp.releng.wtpbuilder - contains the Java code for executing the performance test process
  • releng.wtpbuilder - contains Ant scripts and other XML descriptors that define the performance test process

Running the NullBuilder

The performance test process runs against the builds declared on the WTP download page. The builds, that the tests are already executed against, are described in a special property file - perf_completed_builds.properties. Therefore, to run the peformance tests against a certain declared build, you should have all other, but that one, listed in the property file.

The easiest way to generate the property file with all the declared builds is by using the NullBuilder. The run configuration should have the following parameters.

Project:

org.eclipse.wtp.releng.wtpbuilder

Main class:

org.eclipse.wtp.releng.wtpbuilder.NullBuilder

Program arguments:

-baseos win32
-basews win32
-basearch x86
-build.home C:\eclipse\perf

Environment variables to set:

BASEARCH=x86
BASEOS=win32
BASEWS=win32
BUILD_HOME=C:\eclipse\perf
RELENG=webtools.maps/releng
RELENGWTPBUILDER=releng.wtpbuilder

Note: Some of the above settings may not be needed, but it is save to have them.

The result of the NullBuilder is a completed_builds.properties in the folder specified by the BUILD_HOME variable. Rename this file to perf_completed_builds.properties. Then open it in an editor at comment (with # char) the build that you want to run the performance tests against.

Note: The NullBuilder script will never finish by itself. You need to terminate it manually. You have to first check if the completed_builds.properties already contains all declared builds and then to kill the process.

Running the PerfBuilder

To be done