# Difference between revisions of "Mylyn/Performance Testing"

(→Explanations for the printout format) |
(→Explanations for the printout format) |
||

(11 intermediate revisions by the same user not shown) | |||

Line 1: | Line 1: | ||

== Note: all performance test data are applicable only to Windows XP server pack 2 == | == Note: all performance test data are applicable only to Windows XP server pack 2 == | ||

− | The org.eclipse.test.performance plug-in only | + | The org.eclipse.test.performance plug-in only supports Windows and Linux. And the test data discussed below are only applicable to Windows. |

− | -- | + | --Owen 04:23, 12 February 2008 (EST) |

== A performance test example for Mylyn repository == | == A performance test example for Mylyn repository == | ||

Line 26: | Line 26: | ||

addChildren(task, new int[] { 1000, 10, 2 }, 0); | addChildren(task, new int[] { 1000, 10, 2 }, 0); | ||

− | for (int i = 0; i < | + | for (int i = 0; i < 10; i++) { |

startMeasuring(); | startMeasuring(); | ||

task.contains("handle"); | task.contains("handle"); | ||

Line 39: | Line 39: | ||

</pre></code> | </pre></code> | ||

− | -- | + | --Owen 04:24, 12 February 2008 (EST) |

== The result after running the performance test == | == The result after running the performance test == | ||

Line 55: | Line 55: | ||

</pre> | </pre> | ||

− | -- | + | --Owen 04:24, 12 February 2008 (EST) |

== Explanations for the printout format == | == Explanations for the printout format == | ||

Line 61: | Line 61: | ||

'''Printout format:''' | '''Printout format:''' | ||

− | Scenario | + | <pre>Scenario <test case name> # <test method name> (average over <sample size> samples) |

− | + | <dimension>: <scalar> (95% in <confidence interval>) | |

− | Measurable effect: | + | Measurable effect: <effect size> (<SD value> SDs) |

− | (required sample size for an effect of 5% of | + | (required sample size for an effect of 5% of <standard deviation/mean> : <smallest sample size needed>) |

+ | </pre> | ||

[[Sample size:]] the number of observations. For example: | [[Sample size:]] the number of observations. For example: | ||

Line 101: | Line 102: | ||

[[Smallest sample size needed:]] the smallest sample size to prove the scalar number. | [[Smallest sample size needed:]] the smallest sample size to prove the scalar number. | ||

− | -- | + | --Owen 04:24, 12 February 2008 (EST) |

== More test data for the TaskContainerTest.testContains() in Windows XP server pack 2== | == More test data for the TaskContainerTest.testContains() in Windows XP server pack 2== | ||

Line 166: | Line 167: | ||

GDI Objects: 0 (95% in [0, 0]) | GDI Objects: 0 (95% in [0, 0]) | ||

</pre> | </pre> | ||

+ | |||

+ | --Owen 04:25, 12 February 2008 (EST) |

## Latest revision as of 05:44, 12 February 2008

## Contents

- 1 Note: all performance test data are applicable only to Windows XP server pack 2
- 2 A performance test example for Mylyn repository
- 3 The result after running the performance test
- 4 Explanations for the printout format
- 5 More test data for the TaskContainerTest.testContains() in Windows XP server pack 2

## Note: all performance test data are applicable only to Windows XP server pack 2

The org.eclipse.test.performance plug-in only supports Windows and Linux. And the test data discussed below are only applicable to Windows.

--Owen 04:23, 12 February 2008 (EST)

## A performance test example for Mylyn repository

public class TaskContainerTest extends PerformanceTestCase {
int counter;
private void addChildren(AbstractTask parent, int[] childCount, int depth) {
for (int i = 0; i < childCount[depth]; i++) {
MockRepositoryTask task = new MockRepositoryTask("task", ++counter + "");
parent.internalAddChild(task);
if (depth < childCount.length - 1) {
addChildren(task, childCount, depth + 1);
}
}
}
public void testContains() {
MockRepositoryTask task = new MockRepositoryTask(++counter + "");
addChildren(task, new int[] { 1000, 10, 2 }, 0);
for (int i = 0; i < 10; i++) {
startMeasuring();
task.contains("handle");
stopMeasuring();
}
commitMeasurements();
assertPerformance();
}
}

--Owen 04:24, 12 February 2008 (EST)

## The result after running the performance test

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10 samples): Used Java Heap: 8.96K (95% in [-353.26K, 371.18K]) Measurable effect: 640.47K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set: 108K (95% in [-136.32K, 352.32K]) Measurable effect: 432K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Committed: -2.4K (95% in [-7.83K, 3.03K]) Measurable effect: 9.6K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set Peak: 108K (95% in [-121.63K, 337.63K]) Measurable effect: 406.03K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) Kernel time: 0ms (95% in [0ms, 0ms]) Page Faults: 42 (95% in [-18, 103]) Measurable effect: 108 (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400) CPU Time: 7ms (95% in [1ms, 13ms]) Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401) GDI Objects: 0 (95% in [0, 0])

--Owen 04:24, 12 February 2008 (EST)

## Explanations for the printout format

**Printout format:**

Scenario <test case name> # <test method name> (average over <sample size> samples) <dimension>: <scalar> (95% in <confidence interval>) Measurable effect: <effect size> (<SD value> SDs) (required sample size for an effect of 5% of <standard deviation/mean> : <smallest sample size needed>)

Sample size: the number of observations. For example:

for (int i = 0; i < N; i++) {
startMeasuring();
task.contains("handle");
stopMeasuring();
}

N is a sample size.

Dimension: below are the predefined dimensions for Windows:

Used Java Heap Runtime.totalMemory() - Runtime.freeMemory() Working Set The working set (or on Linux the resident set). "Working Set" COMITTED "Committed" Working Set Peak "Working Set Peak" Elapsed Process "Elapsed Process" Kernel time The amount of elapsed kernel time. "Kernel time" Page Faults "Page Faults" CPU Time The amount of CPU time we have used so far. "CPU Time" GDI Objects "GDI Objects"

Scalar: a variable that only has magnitude. Four units are defined: Second, Byte, Cardinal, and Invocation.

Confidence interval: a percentage of confidence level that the probability will lay between the confidence interval. In our example, we are 95% (default percentage set in the performance plug-in, see “StatisticsUtil.T95”) sure that a certain scalar will be in the confidence interval, e.g. [-353.26K, 371.18K]. Smaller interval indicates more precise result.

Effect size: the size of any observed effects that satisfy the scalar number. In our example, we have to have the effect size of 640.47K before we can realize the number of scalar. Smaller effect size indicates more accurate result.

SD value: relative size of the difference between observed value and reference mean. Here it equals to effect_size/standard_deviation. The larger the value, the farther the observation from the reference population.

Smallest sample size needed: the smallest sample size to prove the scalar number.

--Owen 04:24, 12 February 2008 (EST)

## More test data for the TaskContainerTest.testContains() in Windows XP server pack 2

If the sample size is more than the one suggested, it will no longer complain about the inadequate sample data. And the effect size intend to stabilize in order to observe the scalar number.

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 6300 samples): Used Java Heap: 1.74K (95% in [-18.32K, 21.81K]) Measurable effect: 40.45K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set: 1.81K (95% in [894, 2.75K]) Measurable effect: 1.89K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Committed: 3.45K (95% in [-5.34K, 12.24K]) Measurable effect: 17.72K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set Peak 1.88K (95% in [538, 3.23K]) Measurable effect: 2.72K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process 5ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) Page Faults: 1 (95% in [0, 1]) Measurable effect: 0 (0.1 SDs) (required sample size for an effect of 5% of stdev: 6400) CPU Time: 6ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 6401 samples): Used Java Heap: 1.15K (95% in [-18.79K, 21.1K]) Measurable effect: 40.22K (0.0 SDs) Working Set: 2.26K (95% in [1.26K, 3.25K]) Measurable effect: 2.01K (0.0 SDs) Committed: 4.38K (95% in [-5.41K, 14.18K]) Measurable effect: 19.75K (0.0 SDs) Working Set Peak: 2.23K (95% in [709, 3.77K]) Measurable effect: 3.1K (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 1 (95% in [0, 1]) Measurable effect: 1 (0.0 SDs) CPU Time: 6ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10000 samples): Used Java Heap: 787 (95% in [-15.28K, 16.82K]) Measurable effect: 32.36K (0.0 SDs) Working Set: 1.63K (95% in [773, 2.51K]) Measurable effect: 1.77K (0.0 SDs) Committed: 2.67K (95% in [-2.36K, 7.69K]) Measurable effect: 10.13K (0.0 SDs) Working Set Peak: 1.6K (95% in [656, 2.57K]) Measurable effect: 1.94K (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 0 (95% in [0, 1]) Measurable effect: 0 (0.0 SDs) CPU Time: 5ms (95% in [5ms, 6ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 20000 samples): Used Java Heap: 873 (95% in [-10.56K, 12.27K]) Measurable effect: 23.02K (0.0 SDs) Working Set: 1.34K (95% in [946, 1.76K]) Measurable effect: 867 (0.0 SDs) Committed: 1.48K (95% in [-1.21K, 4.18K]) Measurable effect: 5.43K (0.0 SDs) Working Set Peak: 1.42K (95% in [875, 1.99K]) Measurable effect: 1.14K (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 0 (95% in [0, 0]) Measurable effect: 0 (0.0 SDs) CPU Time: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 20000 samples): Used Java Heap: 834 (95% in [-10.6K, 12.23K]) Measurable effect: 23.01K (0.0 SDs) Working Set: 1.34K (95% in [1.05K, 1.63K]) Measurable effect: 596 (0.0 SDs) Committed: 1.29K (95% in [-1.34K, 3.91K]) Measurable effect: 5.29K (0.0 SDs) Working Set Peak: 1.44K (95% in [978, 1.92K]) Measurable effect: 991 (0.0 SDs) Elapsed Process: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) Kernel time: 0ms (95% in [0ms, 0ms]) Measurable effect: 0ms (0.0 SDs) Page Faults: 0 (95% in [0, 0]) Measurable effect: 0 (0.0 SDs) CPU Time: 5ms (95% in [5ms, 5ms]) Measurable effect: 0ms (0.0 SDs) GDI Objects: 0 (95% in [0, 0])

--Owen 04:25, 12 February 2008 (EST)