# Difference between revisions of "Mylyn/Performance Testing"

(→A performance test example for Mylyn repository) |
(→The result after running the performance test) |
||

Line 38: | Line 38: | ||

== The result after running the performance test == | == The result after running the performance test == | ||

+ | <pre> | ||

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10 samples): | Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10 samples): | ||

Used Java Heap: 8.96K (95% in [-353.26K, 371.18K]) Measurable effect: 640.47K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) | Used Java Heap: 8.96K (95% in [-353.26K, 371.18K]) Measurable effect: 640.47K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) | ||

Line 48: | Line 49: | ||

CPU Time: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) | CPU Time: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) | ||

GDI Objects: 0 (95% in [0, 0]) | GDI Objects: 0 (95% in [0, 0]) | ||

+ | </pre> | ||

== Explanations for the printouts == | == Explanations for the printouts == |

## Revision as of 04:57, 12 February 2008

## Contents

## Note: all performance test data are applicable only to Windows XP server pack 2

## A performance test example for Mylyn repository

public class TaskContainerTest extends PerformanceTestCase {
int counter;
private void addChildren(AbstractTask parent, int[] childCount, int depth) {
for (int i = 0; i < childCount[depth]; i++) {
MockRepositoryTask task = new MockRepositoryTask("task", ++counter + "");
parent.internalAddChild(task);
if (depth < childCount.length - 1) {
addChildren(task, childCount, depth + 1);
}
}
}
public void testContains() {
MockRepositoryTask task = new MockRepositoryTask(++counter + "");
addChildren(task, new int[] { 1000, 10, 2 }, 0);
for (int i = 0; i < 30000; i++) {
startMeasuring();
task.contains("handle");
stopMeasuring();
}
commitMeasurements();
assertPerformance();
}
}

## The result after running the performance test

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10 samples): Used Java Heap: 8.96K (95% in [-353.26K, 371.18K]) Measurable effect: 640.47K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set: 108K (95% in [-136.32K, 352.32K]) Measurable effect: 432K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Committed: -2.4K (95% in [-7.83K, 3.03K]) Measurable effect: 9.6K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set Peak: 108K (95% in [-121.63K, 337.63K]) Measurable effect: 406.03K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Kernel time: 0ms (95% in [0ms, 0ms]) Page Faults: 42 (95% in [-18, 103]) Measurable effect: 108 (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) CPU Time: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) GDI Objects: 0 (95% in [0, 0])

## Explanations for the printouts

**Printout format:**

Scenario ‘*<test case name>*#*<test method name>*’ (average over *<sample size>* samples)
*<dimension>* : *<Scalar>* (95% in *<confidence interval>*) Measurable effect: *<effect size>* (*<SD value>* SDs) (required sample size for an effect of 5% of *<standard deviation/mean>* : *<smallest sample size needed>*)

Sample size: the number of observations. For example:

for (int i = 0; i < N; i++) { startMeasuring(); task.contains("handle"); stopMeasuring(); }

N is a sample size.

Dimension: below are the predefined dimensions for Windows:

Used Java Heap Runtime.totalMemory() - Runtime.freeMemory() Working Set The working set (or on Linux the resident set). "Working Set" COMITTED "Committed" Working Set Peak "Working Set Peak" Elapsed Process "Elapsed Process" Kernel time The amount of elapsed kernel time. "Kernel time" Page Faults "Page Faults" CPU Time The amount of CPU time we have used so far. "CPU Time" GDI Objects "GDI Objects"

Scalar: a variable that only has magnitude. Four units are defined: Second, Byte, Cardinal, and Invocation.

Confidence interval: a percentage of confidence level that the probability will lay between the confidence interval. In our example, we are 95% (default percentage set in the performance plug-in, see “StatisticsUtil.T95”) sure that a certain scalar will be in the confidence interval, e.g. [-353.26K, 371.18K]. Smaller interval indicates more precise result.

Effect size: the size of any observed effects that satisfy the scalar number. In our example, we have to have the effect size of 640.47K before we can realize the number of scalar. Smaller effect size indicates more accurate result.

SD value: relative size of the difference between observed value and reference mean. Here it equals to effect_size/standard_deviation. The larger the value, the farther the observation from the reference population.

Smallest sample size needed: the smallest sample size to prove the scalar number.

## Tests data for the TaskContainerTest.testContains() in Windows XP server pack 2

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10 samples):

Used Java Heap: 8.96K (95% in [-353.26K, 371.18K]) Measurable effect: 640.47K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set: 108K (95% in [-136.32K, 352.32K]) Measurable effect: 432K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Committed: -2.4K (95% in [-7.83K, 3.03K]) Measurable effect: 9.6K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set Peak: 108K (95% in [-121.63K, 337.63K]) Measurable effect: 406.03K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Kernel time: 0ms (95% in [0ms, 0ms]) Page Faults: 42 (95% in [-18, 103]) Measurable effect: 108 (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) CPU Time: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) GDI Objects: 0 (95% in [0, 0])

___________________________________________________________________________________________________________________________________

Used Java Heap: 8.32K (95% in [-375.61K, 392.24K]) Measurable effect: 678.85K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set: 104.4K (95% in [-123.84K, 332.64K]) Measurable effect: 403.58K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Committed: 1.6K (95% in [-1.16K, 4.36K]) Measurable effect: 4.89K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set Peak: 101.2K (95% in [-113.08K, 315.48K]) Measurable effect: 378.88K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Kernel time: 0ms (95% in [0ms, 0ms]) Page Faults: 41 (95% in [-15, 98]) Measurable effect: 100 (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) CPU Time: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) GDI Objects: 0 (95% in [0, 0])

___________________________________________________________________________________________________________________________________

Used Java Heap: 3.43K (95% in [-363.68K, 370.55K]) Measurable effect: 649.13K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set: 104.8K (95% in [-125.33K, 334.93K]) Measurable effect: 406.91K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Committed: 1.6K (95% in [-1.16K, 4.36K]) Measurable effect: 4.89K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set Peak: 102K (95% in [-114.08K, 318.08K]) Measurable effect: 382.08K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Kernel time: 0ms (95% in [0ms, 0ms]) Page Faults: 41 (95% in [-16, 98]) Measurable effect: 101 (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) CPU Time: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) GDI Objects: 0 (95% in [0, 0])

___________________________________________________________________________________________________________________________________

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 6300 samples):

Used Java Heap: 1.74K (95% in [-18.32K, 21.81K]) Measurable effect: 40.45K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set: 1.81K (95% in [894, 2.75K]) Measurable effect: 1.89K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Committed: 3.45K (95% in [-5.34K, 12.24K]) Measurable effect: 17.72K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set Peak: 1.88K (95% in [538, 3.23K]) Measurable effect: 2.72K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 5ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6400) Page Faults: 1 (95% in [0, 1]) Measurable effect: 0 (0.1 SDs) (required sample size for an effect of 5% of stdev: 6400) CPU Time: 6ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 6401 samples):

Used Java Heap: 1.15K (95% in [-18.79K, 21.1K]) Measurable effect: 40.22K (0.0 SDs) Working Set: 2.26K (95% in [1.26K, 3.25K]) Measurable effect: 2.01K (0.0 SDs) Committed: 4.38K (95% in [-5.41K, 14.18K]) Measurable effect: 19.75K (0.0 SDs) Working Set Peak: 2.23K (95% in [709, 3.77K]) Measurable effect: 3.1K (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 1 (95% in [0, 1]) Measurable effect: 1 (0.0 SDs) CPU Time: 6ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10000 samples):

Used Java Heap: 787 (95% in [-15.28K, 16.82K]) Measurable effect: 32.36K (0.0 SDs) Working Set: 1.63K (95% in [773, 2.51K]) Measurable effect: 1.77K (0.0 SDs) Committed: 2.67K (95% in [-2.36K, 7.69K]) Measurable effect: 10.13K (0.0 SDs) Working Set Peak: 1.6K (95% in [656, 2.57K]) Measurable effect: 1.94K (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 0 (95% in [0, 1]) Measurable effect: 0 (0.0 SDs) CPU Time: 5ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 20000 samples):

Used Java Heap: 873 (95% in [-10.56K, 12.27K]) Measurable effect: 23.02K (0.0 SDs) Working Set: 1.34K (95% in [946, 1.76K]) Measurable effect: 867 (0.0 SDs) Committed: 1.48K (95% in [-1.21K, 4.18K]) Measurable effect: 5.43K (0.0 SDs) Working Set Peak: 1.42K (95% in [875, 1.99K]) Measurable effect: 1.14K (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 0 (95% in [0, 0]) Measurable effect: 0 (0.0 SDs) CPU Time: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 20000 samples):

Used Java Heap: 834 (95% in [-10.6K, 12.23K]) Measurable effect: 23.01K (0.0 SDs) Working Set: 1.34K (95% in [1.05K, 1.63K]) Measurable effect: 596 (0.0 SDs) Committed: 1.29K (95% in [-1.34K, 3.91K]) Measurable effect: 5.29K (0.0 SDs) Working Set Peak: 1.44K (95% in [978, 1.92K]) Measurable effect: 991 (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 0 (95% in [0, 0]) Measurable effect: 0 (0.0 SDs) CPU Time: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])