Hudson-ci/features/Memory Performance

The basic idea behind performance analysis and fix is to free up memory that is not actively being utilized. When builds are running there is some inevitable pinning of memory, but when a build is finished, the data for that build need not be held in memory but can be read from disk when requested. A summary information may be cached in some data structure to service frequently used queries. Similarly, a Job definition is mostly static information that need not be pinned, it can also be loaded from disk and some key attributes can be cached.

Basic considerations:

1. There should be no change in APIs, plugins should be minimally impacted if at all.
2. We should not trade one problem for another. If nothing is held in memory, I/O could spike when views are rendered.
3. Changes should be incrementally testable.
4. Changes need to be published back to Hudson repository.

Design Considerations

Current plan is to replace some of the model objects with ‘Lazy’ versions that act like a proxy for the real object and only loads it from disk as needed. The real object is weakly-held and thus available for GC. The higher the object in the model hierarchy( i.e. the closer it is to the Hudson object), the larger the size of the graph that is freed but conversely the higher the I/O impact when it has to be loaded. This approach takes us from a situation where ‘everything is cached’ to ‘nothing is cached’, another extreme.

This is where a strongly-referenced cache comes in, that holds onto frequently used objects based on an LRU list or a timed eviction policy (or both). The parameters of the cache can then be fine tuned to adjust the setting between the two extremes of the scale. Multiple caches may be constructed for each type of model object.

Design Details

We will start with the Project or Job object, put a Lazy decorator on that, and then move down to its children, viz Builds or Runs and so no, till we get down to the build artifacts like log files and console output. This means even when a ‘real’ Job object is constructed, it won’t construct a ‘real’ Build, but will instead hold onto a Build proxy that is light-weight and does not trigger creation of its children and grandchildren.

Tuning Cache parameters

There is a trade off in this memory saving scheme. There is a performance hit in loading the main dashboard, because Hudson releases the Job and Build cache if the server is idle for more than 60 seconds. Hudson 3.1.2 introduces tuning parameters to this cache purging scheme. If you have enough memory and you don't want the aggressive memory reclaiming, then you can tune the cache parameters. The following system parameters can be set to tune the Jobs and Builds caches. The builds cache is maintained for each job, when the job is released, all builds for that job are also released.

Eviction stopwatch resets on last access.

hudson.jobs.cache.evict_in_seconds ( default=60 ) hudson.jobs.cache.initial_capacity ( default=1024 ) hudson.jobs.cache.max_entries ( default=1024)

// For each job hudson.job.builds.cache.evict_in_seconds ( default=60 ) hudson.job.builds.cache.initial_capacity" ( default=512 ) hudson.job.builds.cache.max_entries ( default=10240 )

Here is an example on how to change the cache eviction time to 5 minutes

java -jar hudson-war-3.1.2-SNAPSHOT.war -Dhudson.jobs.cache.evict_in_seconds=300 -Dhudson.job.builds.cache.evict_in_seconds=300

Some Results

ci.hudson-ci.org
jmap -histo:live 24967 | grep FreeStyle
 

  80:          1049         117488  hudson.model.FreeStyleBuild
 310:            64          10752  hudson.model.FreeStyleProject
2914:             1             32  hudson.model.FreeStyleProject$DescriptorImpl

ci-hudson-ci.org/perf/  ( at 1/2 min intervals)
./jmap.sh | grep FreeStyle
JVM pid is 6062
 157:            90          15840  hudson.model.FreeStyleProject
1213:             2            224  hudson.model.FreeStyleBuild
2594:             1             32  hudson.model.FreeStyleProject$DescriptorImpl
./jmap.sh | grep FreeStyle
JVM pid is 6062
 157:            90          15840  hudson.model.FreeStyleProject
1215:             2            224  hudson.model.FreeStyleBuild
2594:             1             32  hudson.model.FreeStyleProject$DescriptorImpl

<<<Did some browser navigation at this point>>>

./jmap.sh | grep FreeStyle
JVM pid is 6062
  35:          2008         224896  hudson.model.FreeStyleBuild
 185:            90          15840  hudson.model.FreeStyleProject
2621:             1             32  hudson.model.FreeStyleProject$DescriptorImpl
[hudson@ahumv0002 perf-sandbox]$ ./jmap.sh | grep FreeStyle
JVM pid is 6062
  35:          2008         224896  hudson.model.FreeStyleBuild
 185:            90          15840  hudson.model.FreeStyleProject
2621:             1             32  hudson.model.FreeStyleProject$DescriptorImpl
./jmap.sh | grep FreeStyle
JVM pid is 6062
  33:          2246         251552  hudson.model.FreeStyleBuild
 150:           135          23760  hudson.model.FreeStyleProject
2628:             1             32  hudson.model.FreeStyleProject$DescriptorImpl

<<<Stopped using browser >>>>

./jmap.sh | grep FreeStyle
JVM pid is 6062
  34:          2166         242592  hudson.model.FreeStyleBuild
 149:           135          23760  hudson.model.FreeStyleProject
2625:             1             32  hudson.model.FreeStyleProject$DescriptorImpl
./jmap.sh | grep FreeStyle
JVM pid is 6062
  38:          1934         216608  hudson.model.FreeStyleBuild
 148:           135          23760  hudson.model.FreeStyleProject
2625:             1             32  hudson.model.FreeStyleProject$DescriptorImpl
./jmap.sh | grep FreeStyle
JVM pid is 6062
 162:            90          15840  hudson.model.FreeStyleProject
1232:             2            224  hudson.model.FreeStyleBuild
2604:             1             32  hudson.model.FreeStyleProject$DescriptorImpl

Before Screen Refresh

[hudson@ahumv0002 perf-sandbox]$ ./jmap.sh
JVM pid is 15974
Running gc
 214:           322          10304  org.sonatype.guice.bean.locators.LazyBeanEntry
 536:            46           1472  hudson.model.LazyTopLevelItem
 537:            46           1472  hudson.model.LazyTopLevelItem$Key
 639:            41            984  sun.swing.SwingLazyValue
 852:            31            496  javax.swing.UIDefaults$LazyInputMap
1232:             7            168  javax.swing.plaf.metal.MetalLookAndFeel$MetalLazyValue
2219:             1             32  hudson.model.FreeStyleProject$DescriptorImpl

After Screen Refresh

[hudson@ahumv0002 perf-sandbox]$ ./jmap.sh
JVM pid is 15974
Running gc
  36:          5345         256560  hudson.model.RunMap$LazyRunValue
  44:          5345         171040  hudson.model.RunMap$LazyRunValue$Key
 225:           322          10304  org.sonatype.guice.bean.locators.LazyBeanEntry
 583:            46           1472  hudson.model.LazyTopLevelItem
 584:            46           1472  hudson.model.LazyTopLevelItem$Key
 703:            41            984  sun.swing.SwingLazyValue
 792:            49            784  hudson.model.RunMap$LazyRunValueCache
 793:            49            784  hudson.model.RunMap$LazyRunValueCache$1
 952:            31            496  javax.swing.UIDefaults$LazyInputMap
1375:             7            168  javax.swing.plaf.metal.MetalLookAndFeel$MetalLazyValue
 249:            45           7920  hudson.model.FreeStyleProject
2377:             1             32  hudson.model.FreeStyleProject$DescriptorImpl

[hudson@ahumv0002 perf-sandbox]$ jmap -heap 15974
Attaching to process ID 15974, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.6-b04

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 134217728 (128.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 85983232 (82.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 40960000 (39.0625MB)
   used     = 4214608 (4.0193634033203125MB)
   free     = 36745392 (35.04313659667969MB)
   10.2895703125% used
From Space:
   capacity = 1835008 (1.75MB)
   used     = 0 (0.0MB)
   free     = 1835008 (1.75MB)
   0.0% used
To Space:
   capacity = 1835008 (1.75MB)
   used     = 0 (0.0MB)
   free     = 1835008 (1.75MB)
   0.0% used
PS Old Generation
   capacity = 89522176 (85.375MB)
   used     = 64062824 (61.095069885253906MB)
   free     = 25459352 (24.279930114746094MB)
   71.56084320381132% used
PS Perm Generation
   capacity = 63963136 (61.0MB)
   used     = 63503000 (60.561180114746094MB)
   free     = 460136 (0.43881988525390625MB)
   99.28062313892802% used

22330 interned Strings occupying 2045080 bytes.

  	       Heap 	  PermGen         Jobs            Builds            Total Builds
                                       (In Memory)      (in Memory)
	 
Production 	83 MB 	67MB               45 	           900 	                  900
Perf Test
(After Start) 	29 MB 	45MB 	           120 	            584 	          2700
Perf Test
(After 2 days) 	65 MB 	60MB               45 	            1394 	          5275
Perf Test
(After 
1 month)        133 MB  61MB 	            45 	            1128                  40459 

Blog

More details about this implementation can be found in this blog by Roy Varghese