Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "RAP/LoadTesting"

< RAP
m (→‎Running a test)
(→‎Configuration and Tuning Tips)
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
| [[RAP|RAP wiki home]] | [http://eclipse.org/rap RAP project home] |
+
This article describes load testing / stress testing of RAP applications with Apache [http://jakarta.apache.org/jmeter/ JMeter].
 +
JMeter lets you record all HTTP(S) requests of a browser session and replay these requests in multiple parallel sessions.
  
−
=DRAFT Load testing / stress testing of RAP applications=
+
This [http://jakarta.apache.org/jmeter/usermanual/jmeter_proxy_step_by_step.pdf Introduction (PDF)] is very useful for getting started with JMeter.
  
−
This is a brief example on how to do load testing / stress testing of RAP applications with [http://jakarta.apache.org/jmeter/ JMeter].
+
== Prepare a Test Plan ==
  
−
==Recording a test==
+
* Download a recent version of JMeter and start it.
−
JMeter allows to record tests by adding a proxy to your browser and simply record the user interactions with the server. This [http://jakarta.apache.org/jmeter/usermanual/jmeter_proxy_step_by_step.pdf Introduction (PDF)] is very useful for getting started with JMeter.
+
  
−
Here is:
+
[[File:JMeter-TestPlan.png]]
−
* a sample proxy configuration (download and unzip [[Image:HTTP_Proxy_Server.zip]]), or
+
−
* a sample test recording (download and unzip [[Image:demo-testplan.zip]])
+
−
that you can load into JMeter 2.3.2.
+
  
−
There are a couple of hints that you need to take into account for recording RAP application tests:
+
There are two elements in the tree, called ''Test Plan'' and ''WorkBench''. Let's start with creating the Test Plan.
−
* Re-Start the browser before recording a test (or make otherwise sure that you are starting with a NEW session, e.g. by deleting the cookies)
+
−
* RAP uses an increasing <tt>requestCounter</tt> to keep track of the requests. This can lead to problems when replaying requests when the number does not begin at zero or is not increasing.
+
−
* To get around this problem go to the Testplan and add a parameter '<tt>test</tt>' with the value 'requestCounter' before recording the test. When you record the test the <tt>requestCounter</tt> value will be replaced by the parameter <tt>${test}</tt>
+
−
* When you have finished recording, just change the value of the testplan parameter '<tt>test</tt>' to the value 'test'. This way you just send an additional parameter which does not influence the RAP internals. Change it back to 'requestCounter' if you do some additional recording
+
−
* After recording your test you need to eliminate duplicate subsequent requests to the UICallbackServiceHandler (so that there are never two or more consecutive requests of this type). The callback requests do not have the ?nocache directive as url parameter.
+
−
* <font color="red">WRONG: </font>
+
−
  [[Image:UICallback.png]]
+
−
* <font color="green">RIGHT:</font>
+
−
  [[Image:OneCallback.png]]
+
−
* It does not matter which callback request(s) you delete, as long as exactly one is remaining. Subsequent requests with a ?nocache directive have to remain untouched.
+
  
−
==Running a test==
+
=== Add a Thread Group ===
−
* When running the tests with multiple threads you need to include an HTTP Cookie Manager (Please note that there is a bug in JMeter 2.3.1 for clearing cookies for multiple runs / iterations; please use either JMeter 2.3.0 or 2.3.2 see [http://markmail.org/message/bis6om3wnpqtqin5])
+
  
−
This is how the test plan should look like:
+
The Thread Group will contain your recorded requests.
  
−
[[Image:JMeterCookieManager.png]]
+
* Select ''Add > Threads (Users) > Thread Group'' from the context menu.
 +
* Configure at least these settings:
 +
** ''Number of Threads (users)'': this is the number of sessions that will in run in parallel
 +
** ''Ramp-up Period (in second)'': the time it takes until the last session is started
  
−
If cookies are not working every request will create a new session, this is not what we want to test (and leads to massive server load).  
+
[[File:JMeter-ThreadGroup.png]]
  
−
Also note the Gaussian Timer that has been added. A timer adds wait time between the requests, leading to a better approximation of a real users behaviour. If you don't add a timer you are putting a much higher load on the server, as every response is immediately followed by the next request, something a real user is not able to do.
+
=== Add HTTP Request Defaults ===
  
−
'''Generating <tt>requestCounter</tt>'''
+
* ''Add > Config Element > HTTP Request Data''
 +
* Configure at least these settings:
 +
** ''Server Name or IP'': the host name or IP address of the server you run the tests against.
 +
** ''Path'': the base path of your application.
  
−
To make sure that we don't get the erros related to a wrong '<tt>requestCounter</tt>' value, add the following
+
[[File:JMeter-RequestDefaults.png]]
−
* A 'Response Assertion in the Thread Group for 'Main sample and sub-samples' with a Pattern Matching Rule 'contains not'
+
−
Multiple browser-instances
+
  
−
To generate the '<tt>requestCounter</tt>' parameter when replaying the request you have to add the following:
+
=== Add a Recording Controller ===
−
* A 'User Defined Variable' in the Thread Group with the name '<tt>request_counter</tt>' and the value '0'
+
−
* A 'Bean Shell Pre-Processor' in the Thread Group containing the following script
+
−
  var sampler = ctx.getCurrentSampler();
+
−
  if(sampler.getPath().contains("nocache")){
+
−
    var doCount = vars.get("startCounter");
+
−
    if(doCount == "OK") {
+
−
      oldCounter = Integer.parseInt(vars.get("request_counter"));
+
−
      sampler.addArgument("requestCounter", Integer.toString(newCounter));
+
−
      newCounter = oldCounter + 1;
+
−
      vars.put("request_counter",Integer.toString(newCounter));
+
−
    }
+
−
  }
+
−
* Underneath the first request with a <tt>?nocache</tt> directive add a 'Bean Shell PostProcessor' with the following script
+
−
  vars.put("startCounter", "OK");
+
  
−
This is what it does:
+
The recording controller will contain the recorded requests
−
* After the first post we set the flag 'startCounter' to 'OK' to start incrementing the <tt>requestCounter</tt> with every subsequent request.
+
−
* The Pre-Processor is called before every request. If it contains the <tt>?nocache</tt> directive it is a POST and must contain the '<tt>requestCounter</tt>' parameter
+
−
* The parameter ist added to the request with the <tt>sampler.addArgument</tt> call.
+
−
* The user variable is incremented for the next request
+
  
−
'''Check Playback'''
+
* ''Add > Logic Controller > Recording Controller''
  
−
This is how we check if the playback of the session is working.
+
[[File:JMeter-RecordingController.png]]
−
* Start testing with only ONE session and two loops (configured in the Thread Group)
+
−
* Add a "Save Responses to a file" listener
+
−
* [[Image:JMeterSaveResponses.png]]
+
−
* Examine the response files, the patter should look like this
+
−
** The first response is an HTML file (expanded size is approxametly one megabyte, the compressed size that is going over the wire is about 200KB)
+
−
** The second response should be a .js or .javascript file with a size above 20KB, it contains the instructions for setting up the first user interface
+
−
** You may have responses that are of type "unknown" as responses to the UICallBack requests (can also appear as .js), in our example the third response
+
−
** The fourth response and all consecutive "non-UICallback" requests should have a response that contains some JavaScript instructions that do set some properties (e.g. w.setSpace or w.setItems in the example below)
+
−
<pre>org.eclipse.swt.EventUtil.suspendEventHandling();
+
−
var req = org.eclipse.swt.Request.getInstance();req.setRequestCounter( "7" );
+
−
var wm = org.eclipse.swt.WidgetManager.getInstance();
+
−
var w = wm.findWidgetById( "w44" );
+
−
w.setSpace( 0, 796, 19, 20 );
+
−
var w = wm.findWidgetById( "w81" );
+
−
w.setItems( [ "View I / Locate in browser view", "View I / Root", "View I / null"]);
+
−
qx.ui.core.Widget.flushGlobalQueues();
+
−
org.eclipse.swt.EventUtil.resumeEventHandling();</pre>
+
−
** if all responses contain only code like this, the test is not properly executed
+
−
<pre>org.eclipse.swt.Request.getInstance().send();
+
−
org.eclipse.swt.Request.getInstance().enableUICallBack( "rap",
+
−
"custom_service_handler",
+
−
"org.eclipse.rwt.internal.lifecycle.UICallBackServiceHandler" );
+
−
</pre>
+
−
** The second pass of the the test should lead to the EXACT same pattern of responses as the first one
+
−
** If this is not the case you need to turn on "Custom IDs" for the RAP widgets, an instruction will follow shortly
+
−
** After you verified the result pattern you should save only failed responses and turn up the volume of your load tests.
+
  
−
==Configuration and Tuning Tips ==
+
=== Add a Cookie Manager ===
  
−
* Tomcat / JVM
+
The cookie manager is required to ensure the same HTTP session in each request.
−
** For best results monitor the memory consumption of your application. Turn on the garbage collector statistics to find out what is going on. If your application is leaking memory, performance will decrease very quickly!
+
If cookies are not working every request will create a new HTTP session and the server will return an HTTP error code.
−
** Provide your application with enough heap. This depends on the number of active sessions and the memory consumption per session of your application. For realistic results, use an appropriate session time out setting
+
 
−
** Choose an appropriate garbage collection strategy. We've found <tt>-XX:+UseConcMarkSweepGC</tt> to work best (on dual-core machines)
+
* ''Add > Config Element > HTTP Cookie Manager''
−
** Start the JVM in server mode (<tt>-server</tt>)
+
 
−
** RAP can ensure that widget id's are generated in the exact same way from session to session. This comes at a slight performance cost. It is useful for load testing because, if id's are different between sessions, this could result in a server response that does not match what the test expects. To activate the 'repeatable id numbering scheme', download the <tt>org.eclipse.rap.widgetidgenerator</tt> plug-in from the RAP Sandbox (/cvsroot/rt/org.eclipse.rap/sandbox/org.eclipse.rap.widgetidgenerator) and deploy it with your application. Use <tt>-Dorg.eclipse.rwt.enableUITests=true</tt> in tomcat's launch arguments
+
The defaults are fine.
−
** If you use tomcat, this servlet <tt>http://yourserver/manager/status/all</tt> provides interesting information
+
 
−
** Obviously - don't run JMeter and the Tomcat on the same machine
+
[[File:JMeter-CookieManager.png]]
 +
 
 +
Please note that there is a bug in JMeter 2.3.1 for clearing cookies for multiple runs / iterations; please use either JMeter 2.3.2 or later see [http://markmail.org/message/bis6om3wnpqtqin5]).
 +
 
 +
=== Add a Timer ===
 +
 
 +
In order to configure the request interval, you have to add a timer.
 +
Without a timer, every response is immediately followed by the next request, something a real user is not able to do.
 +
 
 +
You might want to use a ''random'' timer to add some deviation to the interval. This will lead to a better approximation of a real users behavior.
 +
We recommend the Gaussian random timer.
 +
 
 +
* ''Add > Timer > Gaussian Random Timer''
 +
* Configure:
 +
** ''Constant Delay Offset (in milliseconds)'': the time between two requests
 +
** ''Deviation (in milliseconds)'': the deviation of this interval
 +
 
 +
[[File:JMeter-RandomTimer.png]]
 +
 
 +
== Prepare the Recording ==
 +
 
 +
=== Add Test Script Recorder ===
 +
 
 +
JMeter contains a test script recorder. This is basically a proxy server that listens on a local port (8080 by default).
 +
Since the recorder is not part of the test plan, it is added to the ''Workbench'' node.
 +
 
 +
* Open the context menu on ''WorkBench'', and select ''Add > Non-Test Elements > HTTP(S) Test Script Recorder'' (in older versions this was called ''HTTP Proxy Server'').
 +
* Things to configure:
 +
** ''Global Settings > Port'': the port that the proxy server should listen to.
 +
** ''URL Patterns to exclude'': to exclude static resources served by RAP, use the exclude pattern <code>.*\/rwt-resources\/.*</code>
 +
 
 +
Whether or not to exclude requests for static resources (images, script files, etc.) is up to you.
 +
These resources are cached by the browser and only contribute a small portion to the entire server load.
 +
However, you can safely keep static resources in the recording if you prefer.
 +
 
 +
[[File:JMeter-ScriptRecorder.png]]
 +
 
 +
=== Configure Browser ===
 +
 
 +
* Configure your browser to use a proxy. Use ''localhost'' and the port you configured in the test recorder (8080 by default).
 +
 
 +
== Record a test ==
 +
 
 +
* Press the ''Start'' button at the bottom of the ''HTTP(S) Test Script Recorder'' page to start the recorder.
 +
* Open a '''new browser tab''' and load the URL of your application.
 +
* Now click through the application to record the test session.
 +
 
 +
* Note: In RAP versions prior to 2.1, you had to re-start the browser before recording a test in order to ensure the parameter ''requestCounter'' started from zero.
 +
 
 +
=== Remove ServerPush Requests ===
 +
 
 +
After recording your test, you need to eliminate all requests with parameter ''servicehandler'' = ''org.eclipse.rap.serverpush''. Usually, these are all requests without a ''cid'' parameter in the URL.
 +
 
 +
These requests are not answered by the server and will lead to blocking.
 +
 
 +
=== Save Test Plan ===
 +
 
 +
At this point, you should save the recorded test plan to a file.
 +
 
 +
== Running a test ==
 +
 
 +
Once the test plan is configured and session is recorded, you can start the test by pressing the green ''Start'' button in the toolbar.
 +
 
 +
In the top right corner, you can observe the number of parallel sessions that are currently running.
 +
 
 +
[[File:JMeter-AggregateReport.png]]
 +
 
 +
== Checking the Results ==
 +
 
 +
To ensure that your recording is OK, you may want to start the test with only ONE session (configured in the Thread Group).
 +
 
 +
=== Save Responses to Files ===
 +
 
 +
By adding this element to your test plan you can ensure that the responses contain real JSON (not just errors).
 +
The first response will be HTML, all other responses JSON.
 +
 
 +
''Add > Listener > Safe Responses to a File''
 +
 
 +
[[File:JMeter-SaveResponses.png]]
 +
 
 +
Note that the target directory has to be present, it will not be created by JMeter.
 +
 
 +
If you run the recording twice, the second pass should lead to the same pattern of responses as the first one.
 +
 
 +
== Configuration and Tuning Tips ==
 +
 
 +
* Run JMeter and the server on different machines. JMeter causes significant CPU load as well.
 +
 
 +
* Provide your application with enough heap. This depends on the number of active sessions and the memory consumption per session of your application.
 +
 
 +
* Choose an appropriate garbage collection strategy. We've found <tt>-XX:+UseConcMarkSweepGC</tt> to work best (on multi-core machines). The default garbage collector may lead to longer GC periods and can delay some responses drastically.
 +
 
 +
* Start the JVM in server mode (<tt>-server</tt>)
 +
 
 +
* If you use Tomcat, this servlet <tt>http://yourserver/manager/status/all</tt> provides interesting information
 +
 
 +
* When running JMeter with many worker threads, you must increase its heap size. It is advisable to start with a large initial heap size, to avoid delays resulting from additional heap allocation. Refer to the <tt>HEAP</tt> parameter in the <tt>jmeter</tt> script.
 +
 
 +
=== Running JMeter in headless mode ===
 +
 
 +
Running JMeter with a GUI could slow it down. You can run JMeter in headless mode, using the command below.
 +
This runs <tt>testplan.jmx</tt> and saves the results to <tt>result.jtl</tt>.
  
−
* JMeter
 
−
** When running JMeter with many worker threads, you must increase it's heap size. It is advisable to start with a large initial heap size, to avoid delays resulting from additional heap allocation. Refer to the <tt>HEAP</tt> parameter in the <tt>jmeter / jmeter.bat</tt> scripts
 
−
** Running JMeter with a GUI could slow it down, since updating the GUI takes away CPU cycles. You can run jMeter without a UI, using the commandline below. It starts jmeter without a UI, runs testplan.jmx and saves the results to result.jtl. The result file can be opened with jmeter after the test run is finished. A large result file takes some time and memory to open - be patient and increase the heap. Warning: interrupting a test run in progress, will create an incomplete result file that cannot be opened!
 
 
<pre>./jmeter -n -t /home/elias/testplan.jmx -l /tmp/result.jtl</pre>
 
<pre>./jmeter -n -t /home/elias/testplan.jmx -l /tmp/result.jtl</pre>
 +
 +
The result file can be opened with jmeter after the test run is finished. A large result file takes some time and memory to open - be patient and increase the heap. Warning: interrupting a test run in progress, will create an incomplete result file that cannot be opened!
 +
Note also that the command will immediately return and then jmeter will be executing in the background (as a separate javaw process). Wait for this to finish before opening the results file.
  
 
=== URL rewriting instead of cookies ===
 
=== URL rewriting instead of cookies ===
  
−
From [http://www.eclipse.org/forums/index.php/mv/msg/261197/754685/#msg_754685]:
+
We don't recommend using URL rewriting. However, if you rely on it, [http://www.eclipse.org/forums/index.php/mv/msg/261197/754685/#msg_754685 this forum post] might help:
 
<blockquote>
 
<blockquote>
 
I was able to run the test with the url rewriting. In JMeter there is something called regular expression extracter. I used this to extract the jsessionid from the first response from the server and then pass the same id in the all the other requests.
 
I was able to run the test with the url rewriting. In JMeter there is something called regular expression extracter. I used this to extract the jsessionid from the first response from the server and then pass the same id in the all the other requests.
 
</blockquote>
 
</blockquote>
−
 
−
==Troubleshooting==
 
−
* Multiple Browser instances
 
−
** Problem: The only response data you get looks like this:
 
−
<pre>
 
−
qx.core.Init.getInstance().getApplication().reload( "Multiple browser-instances or
 
−
browser-tabs per session are not\nsupported. You may click OK for restarting the session." )
 
−
</pre>
 
−
** Solution: You likely didn't get a new session on recording but rather reused an old one. This reflects in the parameter <tt>requestCounter</tt> that is sent with each HTTP request. When recording, instead of reloading the application in an existing browser tab you need to load the page in a new browser tab or window.
 
−
** Follow the advices in the Recording section to be able to generate a fresh <tt>requestCounter</tt> value for each request
 
  
 
[[Category:RAP]]
 
[[Category:RAP]]

Latest revision as of 07:57, 28 February 2014

This article describes load testing / stress testing of RAP applications with Apache JMeter. JMeter lets you record all HTTP(S) requests of a browser session and replay these requests in multiple parallel sessions.

This Introduction (PDF) is very useful for getting started with JMeter.

Prepare a Test Plan

  • Download a recent version of JMeter and start it.

JMeter-TestPlan.png

There are two elements in the tree, called Test Plan and WorkBench. Let's start with creating the Test Plan.

Add a Thread Group

The Thread Group will contain your recorded requests.

  • Select Add > Threads (Users) > Thread Group from the context menu.
  • Configure at least these settings:
    • Number of Threads (users): this is the number of sessions that will in run in parallel
    • Ramp-up Period (in second): the time it takes until the last session is started

JMeter-ThreadGroup.png

Add HTTP Request Defaults

  • Add > Config Element > HTTP Request Data
  • Configure at least these settings:
    • Server Name or IP: the host name or IP address of the server you run the tests against.
    • Path: the base path of your application.

JMeter-RequestDefaults.png

Add a Recording Controller

The recording controller will contain the recorded requests

  • Add > Logic Controller > Recording Controller

JMeter-RecordingController.png

Add a Cookie Manager

The cookie manager is required to ensure the same HTTP session in each request. If cookies are not working every request will create a new HTTP session and the server will return an HTTP error code.

  • Add > Config Element > HTTP Cookie Manager

The defaults are fine.

JMeter-CookieManager.png

Please note that there is a bug in JMeter 2.3.1 for clearing cookies for multiple runs / iterations; please use either JMeter 2.3.2 or later see [1]).

Add a Timer

In order to configure the request interval, you have to add a timer. Without a timer, every response is immediately followed by the next request, something a real user is not able to do.

You might want to use a random timer to add some deviation to the interval. This will lead to a better approximation of a real users behavior. We recommend the Gaussian random timer.

  • Add > Timer > Gaussian Random Timer
  • Configure:
    • Constant Delay Offset (in milliseconds): the time between two requests
    • Deviation (in milliseconds): the deviation of this interval

JMeter-RandomTimer.png

Prepare the Recording

Add Test Script Recorder

JMeter contains a test script recorder. This is basically a proxy server that listens on a local port (8080 by default). Since the recorder is not part of the test plan, it is added to the Workbench node.

  • Open the context menu on WorkBench, and select Add > Non-Test Elements > HTTP(S) Test Script Recorder (in older versions this was called HTTP Proxy Server).
  • Things to configure:
    • Global Settings > Port: the port that the proxy server should listen to.
    • URL Patterns to exclude: to exclude static resources served by RAP, use the exclude pattern .*\/rwt-resources\/.*

Whether or not to exclude requests for static resources (images, script files, etc.) is up to you. These resources are cached by the browser and only contribute a small portion to the entire server load. However, you can safely keep static resources in the recording if you prefer.

JMeter-ScriptRecorder.png

Configure Browser

  • Configure your browser to use a proxy. Use localhost and the port you configured in the test recorder (8080 by default).

Record a test

  • Press the Start button at the bottom of the HTTP(S) Test Script Recorder page to start the recorder.
  • Open a new browser tab and load the URL of your application.
  • Now click through the application to record the test session.
  • Note: In RAP versions prior to 2.1, you had to re-start the browser before recording a test in order to ensure the parameter requestCounter started from zero.

Remove ServerPush Requests

After recording your test, you need to eliminate all requests with parameter servicehandler = org.eclipse.rap.serverpush. Usually, these are all requests without a cid parameter in the URL.

These requests are not answered by the server and will lead to blocking.

Save Test Plan

At this point, you should save the recorded test plan to a file.

Running a test

Once the test plan is configured and session is recorded, you can start the test by pressing the green Start button in the toolbar.

In the top right corner, you can observe the number of parallel sessions that are currently running.

JMeter-AggregateReport.png

Checking the Results

To ensure that your recording is OK, you may want to start the test with only ONE session (configured in the Thread Group).

Save Responses to Files

By adding this element to your test plan you can ensure that the responses contain real JSON (not just errors). The first response will be HTML, all other responses JSON.

Add > Listener > Safe Responses to a File

JMeter-SaveResponses.png

Note that the target directory has to be present, it will not be created by JMeter.

If you run the recording twice, the second pass should lead to the same pattern of responses as the first one.

Configuration and Tuning Tips

  • Run JMeter and the server on different machines. JMeter causes significant CPU load as well.
  • Provide your application with enough heap. This depends on the number of active sessions and the memory consumption per session of your application.
  • Choose an appropriate garbage collection strategy. We've found -XX:+UseConcMarkSweepGC to work best (on multi-core machines). The default garbage collector may lead to longer GC periods and can delay some responses drastically.
  • Start the JVM in server mode (-server)
  • When running JMeter with many worker threads, you must increase its heap size. It is advisable to start with a large initial heap size, to avoid delays resulting from additional heap allocation. Refer to the HEAP parameter in the jmeter script.

Running JMeter in headless mode

Running JMeter with a GUI could slow it down. You can run JMeter in headless mode, using the command below. This runs testplan.jmx and saves the results to result.jtl. 

./jmeter -n -t /home/elias/testplan.jmx -l /tmp/result.jtl

The result file can be opened with jmeter after the test run is finished. A large result file takes some time and memory to open - be patient and increase the heap. Warning: interrupting a test run in progress, will create an incomplete result file that cannot be opened! Note also that the command will immediately return and then jmeter will be executing in the background (as a separate javaw process). Wait for this to finish before opening the results file.

URL rewriting instead of cookies

We don't recommend using URL rewriting. However, if you rely on it, this forum post might help:

I was able to run the test with the url rewriting. In JMeter there is something called regular expression extracter. I used this to extract the jsessionid from the first response from the server and then pass the same id in the all the other requests.

Retrieved from "https://wiki.eclipse.org/index.php?title=RAP/LoadTesting&oldid=357070"

This page was last modified 07:57, 28 February 2014 by Ralf Sternberg. Based on work by Ivan Furnadjiev, Marcus Kalthoff and Elias Volanakis and others.

Copyright © Eclipse Foundation, Inc. All Rights Reserved.