Skip to main content
Jump to: navigation, search

CI best practices

Continuous Integration - best practices

General tips

  • keep the builds and tests fast!
    • true to the XP spirit, turnaround time between check in and feedback (build successful/failed) should be kept to a minimum
    • long running builds and tests should be split up
      • eg. into staged tests: test a few basic things first and only if they are successful, run more detailed tests
  • a CI server is not a build artifact repository or storage server!
    • build artifacts in general should be seen as ephemeral
    • if a build artifact is considered a milestone, release candidate, release, etc. it should be copied to a storage server (eg. downloads.eclipse.org)
    • public announcements like "You can get the latest release from our Hudson/Jenkins instance" are considered to be bad practice

Jenkins

Job configuration smells

  • build logic leaks to job
  • too many build steps
  • shell steps
  • not under SCM

Source: https://narkisr.github.io/jenkins-scaling/#configuration-smells

Deal with branches

  • managing branches from UI results in duplication, create a template project and clone, use programmatic API instead

Job timeouts

  • reasonable timeouts should be set to avoid long blocking builds (several hours)
    • Job config -> Build Environment -> Abort the build if it's stuck

Periodical builds (nightly builds)

  • instead of building a job every night at 12am whether or not code has changed, it's recommended to poll the SCM for changes instead (or even better, trigger jobs from webhooks)
    • Job config -> Build Triggers -> Poll SCM
      • to avoid load spikes and distribute load more evenly, it's recommended to use a cron definition like H H(0-3) * * * (build is triggered sometime between 12AM and 3AM)

Disk usage

Builds and tests can consume large amounts of data while checking out Git repositories, downloading Maven artifacts and compiling multi-OS, multi-arch, multi-bitness artifacts, etc.
Disk space is cheap, but not free (yet). Therefore disk usage should be reasonable.

Here is a rough guideline for HIPPs/JIPPs:

  • up to 50GB of disk space is considered to be fine.
  • 50-100GB is mostly considered to be OK, if it has no impact on other projects.
  • upwards of 100GB is considered to be excessive and you should check your job structure/configurations

As always there are exceptions to that rule. If a project has good reasons to use a lot of disk space this will be respected.
If not, you will be notified and ultimately blamed publicly.

Build history

  • a build history is a good indicator to track the health/stability/test coverage, etc. of your project
  • since the build history can take up considerable large amounts of disk space it should be kept at bay
    • it's nice to see the last 500 builds, but in many cases 10-20 builds have nearly the same value to see trends
    • in most cases there is no benefit in keeping more than a few builds with their artifacts, especially if the size is in the 500MB+ range
      • The relevant setting in Hudson/Jenkins job configurations is Max # of builds to keep with artifacts, this should be kept between 2 and 5, depending on the size of the artifacts (bigger artifacts -> keep less builds)

Console logs

  • console logs are important, but if all debug levels are dialed to 11, logs can become quite big (100s of MB)
    • Hudson is not good at dealing with huge console logs, Jenkins is better
    • browsers in general don't like huge console logs
    • try to find a balance between huge logs and time spent to run a build again with different logging levels if something failed (maybe a log level parameter can help)

Workspace

  • workspaces should be cleaned before the build starts to avoid old/changed files influencing new builds
    • Job config -> Build Environment -> Delete workspace before build starts
  • workspaces can also be cleaned when a build is done
    • this is mostly useful for jobs with huge workspaces, where accessing the workspace for debugging purpose is only rarely required
    • Job config -> Post-build Actions -> Delete workspace when build is done


Keep this build forever

  • this option should only be used in special circumstances and only for a very limited period of time
  • this option should not be used to store the build artifacts (e.g. of a special build, like a release) for an extended period of time
    • instead the build artifacts should be copied/moved to the project's directory on download.eclipse.org

Old build jobs

  • best case scenario: all your build configurations are stored in a SCM and you don't have to keep extra job configurations
  • if a build job is deprecated or archived, consider deleting the workspace to save disk space

Back to the top