Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Hudson-ci/features/Memory Performance"
(New page: The basic idea behind performance analysis and fix is to free up memory that is not actively being utilized. When builds are running there is some inevitable pinning of memory, but when a ...) |
|||
| Line 11: | Line 11: | ||
This is where a strongly-referenced cache comes in, that holds onto frequently used objects based on an LRU list or a timed eviction policy (or both). The parameters of the cache can then be fine tuned to adjust the setting between the two extremes of the scale. Multiple caches may be constructed for each type of model object. | This is where a strongly-referenced cache comes in, that holds onto frequently used objects based on an LRU list or a timed eviction policy (or both). The parameters of the cache can then be fine tuned to adjust the setting between the two extremes of the scale. Multiple caches may be constructed for each type of model object. | ||
| − | === Design Details === | + | === Design Details === |
| − | We will start with the Project or Job object, put a Lazy decorator on that, and then move down to its children, viz Builds or Runs and so no, till we get down to the build artifacts like log files and console output. This means even when a ‘real’ Job object is constructed, it won’t construct a ‘real’ Build, but will instead hold onto a Build proxy that is light-weight and does not trigger creation of its children and grandchildren. | + | We will start with the Project or Job object, put a Lazy decorator on that, and then move down to its children, viz Builds or Runs and so no, till we get down to the build artifacts like log files and console output. This means even when a ‘real’ Job object is constructed, it won’t construct a ‘real’ Build, but will instead hold onto a Build proxy that is light-weight and does not trigger creation of its children and grandchildren. |
| − | Some Results< | + | Some Results |
| + | |||
| + | ci.hudson-ci.org | ||
| + | jmap -histo:live 24967 | grep FreeStyle | ||
| + | 80: 1049 117488 hudson.model.FreeStyleBuild | ||
| + | 310: 64 10752 hudson.model.FreeStyleProject | ||
| + | 2914: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | |||
| + | ci-hudson-ci.org/perf/ ( at 1/2 min intervals) | ||
| + | ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 157: 90 15840 hudson.model.FreeStyleProject | ||
| + | 1213: 2 224 hudson.model.FreeStyleBuild | ||
| + | 2594: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 157: 90 15840 hudson.model.FreeStyleProject | ||
| + | 1215: 2 224 hudson.model.FreeStyleBuild | ||
| + | 2594: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | |||
| + | <<<Did some browser navigation at this point>>> | ||
| + | |||
| + | ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 35: 2008 224896 hudson.model.FreeStyleBuild | ||
| + | 185: 90 15840 hudson.model.FreeStyleProject | ||
| + | 2621: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | [hudson@ahumv0002 perf-sandbox]$ ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 35: 2008 224896 hudson.model.FreeStyleBuild | ||
| + | 185: 90 15840 hudson.model.FreeStyleProject | ||
| + | 2621: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 33: 2246 251552 hudson.model.FreeStyleBuild | ||
| + | 150: 135 23760 hudson.model.FreeStyleProject | ||
| + | 2628: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | |||
| + | <<<Stopped using browser >>>> | ||
| + | |||
| + | ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 34: 2166 242592 hudson.model.FreeStyleBuild | ||
| + | 149: 135 23760 hudson.model.FreeStyleProject | ||
| + | 2625: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 38: 1934 216608 hudson.model.FreeStyleBuild | ||
| + | 148: 135 23760 hudson.model.FreeStyleProject | ||
| + | 2625: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
| + | ./jmap.sh | grep FreeStyle | ||
| + | JVM pid is 6062 | ||
| + | 162: 90 15840 hudson.model.FreeStyleProject | ||
| + | 1232: 2 224 hudson.model.FreeStyleBuild | ||
| + | 2604: 1 32 hudson.model.FreeStyleProject$DescriptorImpl | ||
Revision as of 13:44, 1 August 2013
The basic idea behind performance analysis and fix is to free up memory that is not actively being utilized. When builds are running there is some inevitable pinning of memory, but when a build is finished, the data for that build need not be held in memory but can be read from disk when requested. A summary information may be cached in some data structure to service frequently used queries. Similarly, a Job definition is mostly static information that need not be pinned, it can also be loaded from disk and some key attributes can be cached.
Basic considerations:
1. There should be no change in APIs, plugins should be minimally impacted if at all.
2. We should not trade one problem for another. If nothing is held in memory, I/O could spike when views are rendered.
3. Changes should be incrementally testable.
4. Changes need to be published back to Hudson repository.
Design Considerations
Current plan is to replace some of the model objects with ‘Lazy’ versions that act like a proxy for the real object and only loads it from disk as needed. The real object is weakly-held and thus available for GC. The higher the object in the model hierarchy( i.e. the closer it is to the Hudson object), the larger the size of the graph that is freed but conversely the higher the I/O impact when it has to be loaded. This approach takes us from a situation where ‘everything is cached’ to ‘nothing is cached’, another extreme.
This is where a strongly-referenced cache comes in, that holds onto frequently used objects based on an LRU list or a timed eviction policy (or both). The parameters of the cache can then be fine tuned to adjust the setting between the two extremes of the scale. Multiple caches may be constructed for each type of model object.
Design Details
We will start with the Project or Job object, put a Lazy decorator on that, and then move down to its children, viz Builds or Runs and so no, till we get down to the build artifacts like log files and console output. This means even when a ‘real’ Job object is constructed, it won’t construct a ‘real’ Build, but will instead hold onto a Build proxy that is light-weight and does not trigger creation of its children and grandchildren.
Some Results
ci.hudson-ci.org jmap -histo:live 24967 | grep FreeStyle
80: 1049 117488 hudson.model.FreeStyleBuild 310: 64 10752 hudson.model.FreeStyleProject
2914: 1 32 hudson.model.FreeStyleProject$DescriptorImpl
ci-hudson-ci.org/perf/ ( at 1/2 min intervals) ./jmap.sh | grep FreeStyle JVM pid is 6062
157: 90 15840 hudson.model.FreeStyleProject
1213: 2 224 hudson.model.FreeStyleBuild 2594: 1 32 hudson.model.FreeStyleProject$DescriptorImpl ./jmap.sh | grep FreeStyle JVM pid is 6062
157: 90 15840 hudson.model.FreeStyleProject
1215: 2 224 hudson.model.FreeStyleBuild 2594: 1 32 hudson.model.FreeStyleProject$DescriptorImpl
<<<Did some browser navigation at this point>>>
./jmap.sh | grep FreeStyle JVM pid is 6062
35: 2008 224896 hudson.model.FreeStyleBuild 185: 90 15840 hudson.model.FreeStyleProject
2621: 1 32 hudson.model.FreeStyleProject$DescriptorImpl [hudson@ahumv0002 perf-sandbox]$ ./jmap.sh | grep FreeStyle JVM pid is 6062
35: 2008 224896 hudson.model.FreeStyleBuild 185: 90 15840 hudson.model.FreeStyleProject
2621: 1 32 hudson.model.FreeStyleProject$DescriptorImpl ./jmap.sh | grep FreeStyle JVM pid is 6062
33: 2246 251552 hudson.model.FreeStyleBuild 150: 135 23760 hudson.model.FreeStyleProject
2628: 1 32 hudson.model.FreeStyleProject$DescriptorImpl
<<<Stopped using browser >>>>
./jmap.sh | grep FreeStyle JVM pid is 6062
34: 2166 242592 hudson.model.FreeStyleBuild 149: 135 23760 hudson.model.FreeStyleProject
2625: 1 32 hudson.model.FreeStyleProject$DescriptorImpl ./jmap.sh | grep FreeStyle JVM pid is 6062
38: 1934 216608 hudson.model.FreeStyleBuild 148: 135 23760 hudson.model.FreeStyleProject
2625: 1 32 hudson.model.FreeStyleProject$DescriptorImpl ./jmap.sh | grep FreeStyle JVM pid is 6062
162: 90 15840 hudson.model.FreeStyleProject
1232: 2 224 hudson.model.FreeStyleBuild 2604: 1 32 hudson.model.FreeStyleProject$DescriptorImpl