Skip to main content
Jump to: navigation, search

Mylyn/Performance Testing

< Mylyn
Revision as of 05:23, 12 February 2008 by Jingweno.cs.ubc.ca (Talk | contribs) (Note: all performance test data are applicable only to Windows XP server pack 2)

Note: all performance test data are applicable only to Windows XP server pack 2

The org.eclipse.test.performance plug-in only support Windows and Linux. And the test data discussed below only are only applicable to Windows.

--Owen 04:23, 12 February 2008 (EST)

A performance test example for Mylyn repository

public class TaskContainerTest extends PerformanceTestCase {

	int counter;

	private void addChildren(AbstractTask parent, int[] childCount, int depth) {
		for (int i = 0; i < childCount[depth]; i++) {
			MockRepositoryTask task = new MockRepositoryTask("task", ++counter + "");
			parent.internalAddChild(task);
			if (depth < childCount.length - 1) {
				addChildren(task, childCount, depth + 1);
			}
		}
	}

	public void testContains() {
		MockRepositoryTask task = new MockRepositoryTask(++counter + "");
		addChildren(task, new int[] { 1000, 10, 2 }, 0);

		for (int i = 0; i < 30000; i++) {
			startMeasuring();
			task.contains("handle");
			stopMeasuring();
		}
		
		commitMeasurements();
		assertPerformance();
	}

}

--Jingweno.cs.ubc.ca 04:21, 12 February 2008 (EST)

The result after running the performance test

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10 samples):
  Used Java Heap:   8.96K   (95% in [-353.26K, 371.18K])  Measurable effect: 640.47K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Working Set:      108K    (95% in [-136.32K, 352.32K])  Measurable effect: 432K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Committed:        -2.4K   (95% in [-7.83K, 3.03K])      Measurable effect: 9.6K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Working Set Peak:  108K   (95% in [-121.63K, 337.63K])  Measurable effect: 406.03K (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Elapsed Process:   7ms    (95% in [1ms, 13ms])          Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Kernel time:       0ms    (95% in [0ms, 0ms])           
  Page Faults:       42     (95% in [-18, 103])           Measurable effect: 108 (1.3 SDs) (required sample size for an effect of 5% of stdev: 6400)
  CPU Time:          7ms    (95% in [1ms, 13ms])          Measurable effect: 10ms (1.3 SDs) (required sample size for an effect of 5% of stdev: 6401)
  GDI Objects:       0      (95% in [0, 0])

--Jingweno.cs.ubc.ca 04:21, 12 February 2008 (EST)

Explanations for the printout format

Printout format:

Scenario ‘<test case name> # <test method name>’ (average over <sample size> samples) <dimension> : <Scalar> (95% in <confidence interval>) Measurable effect: <effect size> (<SD value> SDs) (required sample size for an effect of 5% of <standard deviation/mean> : <smallest sample size needed>)

Sample size: the number of observations. For example:

for (int i = 0; i < N; i++) {
	startMeasuring();
	task.contains("handle");
	stopMeasuring();
}

N is a sample size.

Dimension: below are the predefined dimensions for Windows:

Used Java Heap		Runtime.totalMemory() - Runtime.freeMemory()
Working Set		The working set (or on Linux the resident set). "Working Set"
COMITTED		"Committed"
Working Set Peak	"Working Set Peak"
Elapsed Process		"Elapsed Process"
Kernel time		The amount of elapsed kernel time. "Kernel time"
Page Faults		"Page Faults"
CPU Time		The amount of CPU time we have used so far. "CPU Time"
GDI Objects		"GDI Objects"

Scalar: a variable that only has magnitude. Four units are defined: Second, Byte, Cardinal, and Invocation.

Confidence interval: a percentage of confidence level that the probability will lay between the confidence interval. In our example, we are 95% (default percentage set in the performance plug-in, see “StatisticsUtil.T95”) sure that a certain scalar will be in the confidence interval, e.g. [-353.26K, 371.18K]. Smaller interval indicates more precise result.

Effect size: the size of any observed effects that satisfy the scalar number. In our example, we have to have the effect size of 640.47K before we can realize the number of scalar. Smaller effect size indicates more accurate result.

SD value: relative size of the difference between observed value and reference mean. Here it equals to effect_size/standard_deviation. The larger the value, the farther the observation from the reference population.

Smallest sample size needed: the smallest sample size to prove the scalar number.

--Jingweno.cs.ubc.ca 04:22, 12 February 2008 (EST)

More test data for the TaskContainerTest.testContains() in Windows XP server pack 2

If the sample size is more than the one suggested, it will no longer complain about the inadequate sample data. And the effect size intend to stabilize in order to observe the scalar number.

Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 6300 samples):
  Used Java Heap:   1.74K   (95% in [-18.32K, 21.81K])  Measurable effect: 40.45K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Working Set:      1.81K   (95% in [894, 2.75K])       Measurable effect: 1.89K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Committed:        3.45K   (95% in [-5.34K, 12.24K])   Measurable effect: 17.72K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Working Set Peak  1.88K   (95% in [538, 3.23K])       Measurable effect: 2.72K (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Elapsed Process   5ms     (95% in [5ms, 6ms])         Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Kernel time:      0ms     (95% in [0ms, 0ms])         Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Page Faults:      1       (95% in [0, 1])             Measurable effect: 0 (0.1 SDs) (required sample size for an effect of 5% of stdev: 6400)
  CPU Time:         6ms     (95% in [5ms, 6ms])         Measurable effect: 0ms (0.1 SDs) (required sample size for an effect of 5% of stdev: 6401)
  GDI Objects:      0       (95% in [0, 0])               
Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 6401 samples):
  Used Java Heap:         1.15K         (95% in [-18.79K, 21.1K])      Measurable effect: 40.22K (0.0 SDs)
  Working Set:            2.26K         (95% in [1.26K, 3.25K])        Measurable effect: 2.01K (0.0 SDs)
  Committed:              4.38K         (95% in [-5.41K, 14.18K])      Measurable effect: 19.75K (0.0 SDs)
  Working Set Peak:       2.23K         (95% in [709, 3.77K])          Measurable effect: 3.1K (0.0 SDs)
  Elapsed Process:           5ms        (95% in [5ms, 5ms])            Measurable effect: 0ms (0.0 SDs)
  Kernel time:               0ms        (95% in [0ms, 0ms])            Measurable effect: 0ms (0.0 SDs)
  Page Faults:               1          (95% in [0, 1])                Measurable effect: 1 (0.0 SDs)
  CPU Time:                  6ms        (95% in [5ms, 6ms])            Measurable effect: 0ms (0.0 SDs)
  GDI Objects:               0          (95% in [0, 0])               
Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 10000 samples):
  Used Java Heap:          787          (95% in [-15.28K, 16.82K])     Measurable effect: 32.36K (0.0 SDs)
  Working Set:            1.63K         (95% in [773, 2.51K])          Measurable effect: 1.77K (0.0 SDs)
  Committed:              2.67K         (95% in [-2.36K, 7.69K])       Measurable effect: 10.13K (0.0 SDs)
  Working Set Peak:        1.6K         (95% in [656, 2.57K])          Measurable effect: 1.94K (0.0 SDs)
  Elapsed Process:           5ms        (95% in [5ms, 5ms])            Measurable effect: 0ms (0.0 SDs)
  Kernel time:               0ms        (95% in [0ms, 0ms])            Measurable effect: 0ms (0.0 SDs)
  Page Faults:               0          (95% in [0, 1])                Measurable effect: 0 (0.0 SDs)
  CPU Time:                  5ms        (95% in [5ms, 6ms])            Measurable effect: 0ms (0.0 SDs)
  GDI Objects:               0          (95% in [0, 0])      
Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 20000 samples):
  Used Java Heap:          873          (95% in [-10.56K, 12.27K])     Measurable effect: 23.02K (0.0 SDs)
  Working Set:            1.34K         (95% in [946, 1.76K])          Measurable effect: 867 (0.0 SDs)
  Committed:              1.48K         (95% in [-1.21K, 4.18K])       Measurable effect: 5.43K (0.0 SDs)
  Working Set Peak:       1.42K         (95% in [875, 1.99K])          Measurable effect: 1.14K (0.0 SDs)
  Elapsed Process:           5ms        (95% in [5ms, 5ms])            Measurable effect: 0ms (0.0 SDs)
  Kernel time:               0ms        (95% in [0ms, 0ms])            Measurable effect: 0ms (0.0 SDs)
  Page Faults:               0          (95% in [0, 0])                Measurable effect: 0 (0.0 SDs)
  CPU Time:                  5ms        (95% in [5ms, 5ms])            Measurable effect: 0ms (0.0 SDs)
  GDI Objects:               0          (95% in [0, 0])    
Scenario 'org.eclipse.mylyn.tasks.tests.performance.TaskContainerTest#testContains()' (average over 20000 samples):
  Used Java Heap:          834          (95% in [-10.6K, 12.23K])      Measurable effect: 23.01K (0.0 SDs)
  Working Set:            1.34K         (95% in [1.05K, 1.63K])        Measurable effect: 596 (0.0 SDs)
  Committed:              1.29K         (95% in [-1.34K, 3.91K])       Measurable effect: 5.29K (0.0 SDs)
  Working Set Peak:       1.44K         (95% in [978, 1.92K])          Measurable effect: 991 (0.0 SDs)
  Elapsed Process:           5ms        (95% in [5ms, 5ms])            Measurable effect: 0ms (0.0 SDs)
  Kernel time:               0ms        (95% in [0ms, 0ms])            Measurable effect: 0ms (0.0 SDs)
  Page Faults:               0          (95% in [0, 0])                Measurable effect: 0 (0.0 SDs)
  CPU Time:                  5ms        (95% in [5ms, 5ms])            Measurable effect: 0ms (0.0 SDs)
  GDI Objects:               0          (95% in [0, 0])

--Jingweno.cs.ubc.ca 04:22, 12 February 2008 (EST)

Back to the top