Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "CI best practices"
(→Build history) |
(→Continuous Integration - best practices) |
||
(6 intermediate revisions by the same user not shown) | |||
Line 10: | Line 10: | ||
** build artifacts in general should be seen as ephemeral | ** build artifacts in general should be seen as ephemeral | ||
** if a build artifact is considered a milestone, release candidate, release, etc. it should be copied to a storage server (eg. downloads.eclipse.org) | ** if a build artifact is considered a milestone, release candidate, release, etc. it should be copied to a storage server (eg. downloads.eclipse.org) | ||
− | ** public announcements like "You can get the latest release from our | + | ** public announcements like "You can get the latest release from our Jenkins instance" are considered to be bad practice |
* [[Eclipse/Testing | General tips about testing]] | * [[Eclipse/Testing | General tips about testing]] | ||
− | == | + | == Jenkins == |
− | == Job configuration smells == | + | === Job configuration smells === |
* build logic leaks to job | * build logic leaks to job | ||
* too many build steps | * too many build steps | ||
Line 24: | Line 24: | ||
Source: https://narkisr.github.io/jenkins-scaling/#configuration-smells | Source: https://narkisr.github.io/jenkins-scaling/#configuration-smells | ||
− | == Deal with branches == | + | === Deal with branches === |
* managing branches from UI results in duplication, create a template project and clone, use programmatic API instead | * managing branches from UI results in duplication, create a template project and clone, use programmatic API instead | ||
− | == Disk usage == | + | === Job timeouts === |
+ | * reasonable timeouts should be set to avoid long blocking builds (several hours or days) | ||
+ | ** Freestyle Job | ||
+ | *** Job config -> Build Environment -> Abort the build if it's stuck<br>[[Image:Hudson_AbortBuildIfStuck.png]] | ||
+ | ** Pipeline Job | ||
+ | *** | ||
+ | <source lang="groovy" style="border:1px solid;padding: 5px; margin: 5px;"> | ||
+ | pipeline { | ||
+ | options { | ||
+ | timeout(time: 30, unit: 'MINUTES') | ||
+ | } | ||
+ | stages { .. } | ||
+ | // .. | ||
+ | } | ||
+ | </source> | ||
+ | |||
+ | === Periodical builds (nightly builds) === | ||
+ | * instead of building a job every night at 12am whether or not code has changed, it's recommended to poll the SCM for changes instead (or even better, trigger jobs from webhooks) | ||
+ | ** Job config -> Build Triggers -> Poll SCM | ||
+ | *** to avoid load spikes and distribute load more evenly, it's recommended to use a cron definition like <code>H H(0-3) * * *</code> (build is triggered sometime between 12AM and 3AM) | ||
+ | |||
+ | === Disk usage === | ||
Builds and tests can consume large amounts of data while checking out Git repositories, downloading Maven artifacts and compiling multi-OS, multi-arch, multi-bitness artifacts, etc.<br /> | Builds and tests can consume large amounts of data while checking out Git repositories, downloading Maven artifacts and compiling multi-OS, multi-arch, multi-bitness artifacts, etc.<br /> | ||
'''Disk space is cheap, but not free (yet)'''. Therefore disk usage should be reasonable. | '''Disk space is cheap, but not free (yet)'''. Therefore disk usage should be reasonable. | ||
− | Here is a rough guideline for | + | Here is a rough guideline for JIPPs: |
* up to 50GB of disk space is considered to be fine. | * up to 50GB of disk space is considered to be fine. | ||
* 50-100GB is mostly considered to be OK, if it has no impact on other projects. | * 50-100GB is mostly considered to be OK, if it has no impact on other projects. | ||
* upwards of 100GB is considered to be excessive and you should check your job structure/configurations | * upwards of 100GB is considered to be excessive and you should check your job structure/configurations | ||
− | As always there are exceptions to that rule. If a project has | + | As always there are exceptions to that rule. If a project has '''good reasons''' to use a lot of disk space this will be respected.<br /> |
If not, you will be notified and ultimately blamed publicly. | If not, you will be notified and ultimately blamed publicly. | ||
Line 43: | Line 64: | ||
** it's nice to see the last 500 builds, but in many cases 10-20 builds have nearly the same value to see trends | ** it's nice to see the last 500 builds, but in many cases 10-20 builds have nearly the same value to see trends | ||
** in most cases there is no benefit in keeping more than a few builds with their artifacts, especially if the size is in the 500MB+ range | ** in most cases there is no benefit in keeping more than a few builds with their artifacts, especially if the size is in the 500MB+ range | ||
− | *** The relevant setting in | + | *** The relevant setting in Jenkins job configurations is '''Max # of builds to keep with artifacts''', this should be kept between 2 and 5, depending on the size of the artifacts (bigger artifacts -> keep less builds) |
=== Console logs === | === Console logs === | ||
* console logs are important, but if all debug levels are dialed to 11, logs can become quite big (100s of MB) | * console logs are important, but if all debug levels are dialed to 11, logs can become quite big (100s of MB) | ||
− | |||
** browsers in general don't like huge console logs | ** browsers in general don't like huge console logs | ||
** try to find a balance between huge logs and time spent to run a build again with different logging levels if something failed (maybe a log level parameter can help) | ** try to find a balance between huge logs and time spent to run a build again with different logging levels if something failed (maybe a log level parameter can help) | ||
− | === | + | === Workspace === |
− | * | + | * workspaces should be cleaned '''before the build starts''' to avoid old/changed files influencing new builds |
+ | ** Job config -> Build Environment -> Delete workspace before build starts | ||
+ | * workspaces can also be cleaned when a build is done | ||
+ | ** this is mostly useful for jobs with huge workspaces, where accessing the workspace for debugging purpose is only rarely required | ||
+ | ** Job config -> Post-build Actions -> Delete workspace when build is done | ||
+ | |||
+ | {{Note|Concurrent builds| Please keep in mind, that if you use the option "Execute concurrent builds if necessary", each build runs in its own workspace. This will multiply the disk usage.}} | ||
− | === | + | === Keep this build forever === |
− | * | + | * this option should only be used in special circumstances and only for a very limited period of time |
+ | * this option '''should not''' be used to store the build artifacts (e.g. of a special build, like a release) for an extended period of time | ||
+ | ** instead the build artifacts should be copied/moved to the project's directory on download.eclipse.org | ||
=== Old build jobs === | === Old build jobs === | ||
* best case scenario: all your build configurations are stored in a SCM and you don't have to keep extra job configurations | * best case scenario: all your build configurations are stored in a SCM and you don't have to keep extra job configurations | ||
* if a build job is deprecated or archived, consider deleting the workspace to save disk space | * if a build job is deprecated or archived, consider deleting the workspace to save disk space |
Latest revision as of 05:54, 10 May 2021
Continuous Integration - best practices
General tips
- keep the builds and tests fast!
- true to the XP spirit, turnaround time between check in and feedback (build successful/failed) should be kept to a minimum
- long running builds and tests should be split up
- eg. into staged tests: test a few basic things first and only if they are successful, run more detailed tests
- a CI server is not a build artifact repository or storage server!
- build artifacts in general should be seen as ephemeral
- if a build artifact is considered a milestone, release candidate, release, etc. it should be copied to a storage server (eg. downloads.eclipse.org)
- public announcements like "You can get the latest release from our Jenkins instance" are considered to be bad practice
Jenkins
Job configuration smells
- build logic leaks to job
- too many build steps
- shell steps
- not under SCM
Source: https://narkisr.github.io/jenkins-scaling/#configuration-smells
Deal with branches
- managing branches from UI results in duplication, create a template project and clone, use programmatic API instead
Job timeouts
- reasonable timeouts should be set to avoid long blocking builds (several hours or days)
pipeline { options { timeout(time: 30, unit: 'MINUTES') } stages { .. } // .. }
Periodical builds (nightly builds)
- instead of building a job every night at 12am whether or not code has changed, it's recommended to poll the SCM for changes instead (or even better, trigger jobs from webhooks)
- Job config -> Build Triggers -> Poll SCM
- to avoid load spikes and distribute load more evenly, it's recommended to use a cron definition like
H H(0-3) * * *
(build is triggered sometime between 12AM and 3AM)
- to avoid load spikes and distribute load more evenly, it's recommended to use a cron definition like
- Job config -> Build Triggers -> Poll SCM
Disk usage
Builds and tests can consume large amounts of data while checking out Git repositories, downloading Maven artifacts and compiling multi-OS, multi-arch, multi-bitness artifacts, etc.
Disk space is cheap, but not free (yet). Therefore disk usage should be reasonable.
Here is a rough guideline for JIPPs:
- up to 50GB of disk space is considered to be fine.
- 50-100GB is mostly considered to be OK, if it has no impact on other projects.
- upwards of 100GB is considered to be excessive and you should check your job structure/configurations
As always there are exceptions to that rule. If a project has good reasons to use a lot of disk space this will be respected.
If not, you will be notified and ultimately blamed publicly.
Build history
- a build history is a good indicator to track the health/stability/test coverage, etc. of your project
- since the build history can take up considerable large amounts of disk space it should be kept at bay
- it's nice to see the last 500 builds, but in many cases 10-20 builds have nearly the same value to see trends
- in most cases there is no benefit in keeping more than a few builds with their artifacts, especially if the size is in the 500MB+ range
- The relevant setting in Jenkins job configurations is Max # of builds to keep with artifacts, this should be kept between 2 and 5, depending on the size of the artifacts (bigger artifacts -> keep less builds)
Console logs
- console logs are important, but if all debug levels are dialed to 11, logs can become quite big (100s of MB)
- browsers in general don't like huge console logs
- try to find a balance between huge logs and time spent to run a build again with different logging levels if something failed (maybe a log level parameter can help)
Workspace
- workspaces should be cleaned before the build starts to avoid old/changed files influencing new builds
- Job config -> Build Environment -> Delete workspace before build starts
- workspaces can also be cleaned when a build is done
- this is mostly useful for jobs with huge workspaces, where accessing the workspace for debugging purpose is only rarely required
- Job config -> Post-build Actions -> Delete workspace when build is done
Keep this build forever
- this option should only be used in special circumstances and only for a very limited period of time
- this option should not be used to store the build artifacts (e.g. of a special build, like a release) for an extended period of time
- instead the build artifacts should be copied/moved to the project's directory on download.eclipse.org
Old build jobs
- best case scenario: all your build configurations are stored in a SCM and you don't have to keep extra job configurations
- if a build job is deprecated or archived, consider deleting the workspace to save disk space