Skip to main content
Jump to: navigation, search

Difference between revisions of "Papyrus-RT/Developer/Developer Guide/Builds"

(Initial documentation of Papyrus-RT builds design)
 
(Add some detail about job triggering)
Line 95: Line 95:
 
<tr><td>.../Papyrus-RT-Master-xtUML-RT/lastSuccessful/archive/repository/</td></tr>
 
<tr><td>.../Papyrus-RT-Master-xtUML-RT/lastSuccessful/archive/repository/</td></tr>
 
</table>
 
</table>
 +
 +
==== Triggering ====
 +
 +
Both the continuous integration (nightly) and Gerrit build jobs for each component are triggered by changes only in specific locations in the Git repository. Triggering paths for the Git poll trigger for the continuous integration jobs are specified as inclusion and exclusion patterns. Only regular expression patterns are supported, so for consistency (promoting correct alignment of the jobs) the Gerrit triggers will also specify triggering paths via regular expressions.
 +
 +
The '''Papyrus-RT-Gerrit-<code>&lt;Component&gt;</code>''' and '''Papyrus-RT-<code>&lt;Stream&gt;</code>-<code>&lt;Component&gt;</code>''' jobs are triggered by changes in paths matching patterns of the form
 +
 +
<pre>pom.xml
 +
(releng|features|plugins|tests)/&lt;component&gt;/.*
 +
(plugins|tests)/umlrt/&lt;component&gt;/.*</pre>
 +
 +
Some components use additional directories, also, for example the '''Papyrus-RT-Gerrit-Codegen''' job:
 +
 +
<pre>models/.*
 +
scripts/codegen/.*</pre>
 +
 +
All jobs are triggered by changes in the root POM, which naturally affects them all.
 +
 
== P2 Repositories ==
 
== P2 Repositories ==
  

Revision as of 12:21, 9 June 2016

Papyrus-RT Build System

This page presents documentation of the design and implementation of the Papyrus-RT builds in the project's Eclipse HIPP instance.

Overview

The Papyrus-RT builds serve at least three important audiences:

  • developers: for validation of contributions, automated testing and quality reporting
  • testers: for evaluation of features and bug fixes, quality assurance
  • end users: for all use cases that the software is designed to support

These different audiences typically have many common requirements of the build, but also some different requirements:

  • developers typically look for quick turn-around of compile and test phases; don't need packaging because they work in a development environment with run-time workbenches in debug mode
  • testers need frequent packaging builds, typically throughout the day, in order to be able to evaluate fixes and new feature deliveries in a timely manner
  • end users usually look for occasional milestone builds that are assured of some stability and tested quality; these builds should be properly signed to verify the origin and integrity of the download. Bleeding-edge users may want weekly integration builds but typically don't follow updates day-to-day

The combination of Hudson jobs for triggering and publishing builds with Maven/Tycho for performing builds offers all the flexibility that is required to serve the various needs of these disparate audiences. The following subsections present a high-level description of how build jobs are organized on the Papyrus-RT HIPP instance to satisfy these requirements.

Stream Model

One important consideration is that the development of Papyrus usually progresses on multiple parallel streams, targeting different feature (x.y version) releases. Accordingly, the build system needs to be able to handle this parallel-stream development. In particular,

  • Gerrit builds need to be able to be triggered to verify contributions on all active streams
  • packaging and weekly integration builds are required to publish p2 repositories and, where appropriate, RCP packages, for all active streams
  • continuous integration builds are required to provide up-to-the minute binaries for the developer environment's PDE target, plus automated test/quality reports, for all active streams

Therefore, a degree of parameterization of the builds is required to account for:

  • branches in the Git source repository from which to build the sources
  • repositories from which to obtain binaries of dependencies (e.g., Eclipse, Papyrus, GMF, other Papyrus-RT components)
  • repository URLs at which to publish artifacts

Terminology

The descriptions below make references to certain variables that are placeholders for names of streams, components, and other elements of variability in the build configurations. Those variables are listed and defined here. These variables are referenced in angle brackets, e.g. <name> where "name" (without the quotation marks) is the variable name. Variable names are not case-sensitive, so <name> and <Name> both reference the same variable "name". However, where a variable name reference is capitalized, the text that is substituted for it is rendered in Title Case (or CamelCase, depending on its context). Otherwise, it is rendered in lower case.

  • <stream> — the development stream, corresponding to an Eclipse simultaneous release. e.g., mars, neon, oxygen in reference to maintenance or advanced development of releases other than the current release; or master in reference to the current release
  • <component> — a component of the Papyrus-RT product, more or less corresponding to a module in the Git source repository. e.g., core, tooling, codegen; all or product for the entirety of Papyrus-RT
  • <build> — a kind of build: nightly, integration, milestone, release
  • <stamp> — a 12-digit build timestamp, of the form YYYYMMDDhhmm
  • <dependent> — placeholder for any or each of the <component>s that depend on a given component

Hudson Build Jobs

The diagram below shows the organization of build jobs on the Hudson server for the Master stream (the Neon release stream at the time of writing). As you can see, there are quite a few jobs: one per component in the Gerrit and continuous (nightly) jobs. Solid arrows show triggering relationships between jobs, dashed arrows show target-platform inputs, and candy arrows show publication of artifacts to a p2 repository (update site) on the downloads server. All job names are prefixed by "Papyrus-RT" (as indicated by the leading hyphen) although this is, strictly speaking, redundant on the Papyrus-RT HIPP instance.

Hudson Build Jobs
caption Hudson build jobs flow diagram

The Gerrit jobs all are triggered independently by changes in the portions of the Git source tree that they build:

  • plugins/<component> (which in many cases includes an umlrt/ path segment)
  • tests/<component> (again, sometimes with an umlrt/ path segment)
  • releng/<component>

Each Gerrit job builds the sources for its components and runs tests for all of that component's dependency tree with a target platform based on the latest nightly continuous build produced by the Papyrus-RT-<Stream>-Product build (in the example of this diagram, the Papyrus-RT-Master-Product build) plus, of course, Papyrus, Eclipse, and the rest of the stack. So,

  • the -Gerrit-<Stream>-Profile build runs absolutely all tests
  • the -Gerrit-<Stream>-Common build runs all tests except profile
  • the -Gerrit-<Stream>-Core build runs all tests except profile and common
  • the -Gerrit-<Stream>-Tooling build runs the tooling tests
  • the -Gerrit-<Stream>-xtUML-RT build runs the xtUML-RT and codegen tests
  • the -Gerrit-<Stream>-Codegen build runs the codegen tests

As a special case, changes to the target definitions in the targetplatform/ tree trigger the Papyrus-RT-<Stream>-Tooling build as being the one most likely to have the most dependencies on Papyrus and the rest of the stack beneath Papyrus-RT.

Target Platforms<a name="tps"/>

To ensure isolation and concurrent execution of the builds, they use private Maven repositories and execute only the clean and verify goals; they do not install. Accordingly, the target platform for each build must include its dependencies. For the Gerrit builds, this is simple: the Gerrit builds all include the

$HOME/.hudson/jobs/Papyrus-RT-<Stream>-Product/lastSuccessful/archive/repository/

in their target platform (thus, all can share a target-platform definition) because they are triggered independently and build by overlaying their individual component's sources on the common target.

In the cases of the continuous integration (nightly) builds, each has its own target platform that adds the last successful archived repositories of its dependencies

$HOME/.hudson/jobs/Papyrus-RT-<Stream>-<Component>/lastSuccessful/archive/repository/

to the shared platform comprising Papyrus, Eclipse, and the rest of the stack below Papyrus-RT. So, for example, in the master stream:

ComponentTarget Platform
Profile
Common.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
Core.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
Tooling.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Core/lastSuccessful/archive/repository/
xtUML-RT.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Core/lastSuccessful/archive/repository/
Codegen.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Core/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-xtUML-RT/lastSuccessful/archive/repository/

Triggering

Both the continuous integration (nightly) and Gerrit build jobs for each component are triggered by changes only in specific locations in the Git repository. Triggering paths for the Git poll trigger for the continuous integration jobs are specified as inclusion and exclusion patterns. Only regular expression patterns are supported, so for consistency (promoting correct alignment of the jobs) the Gerrit triggers will also specify triggering paths via regular expressions.

The Papyrus-RT-Gerrit-<Component> and Papyrus-RT-<Stream>-<Component> jobs are triggered by changes in paths matching patterns of the form

pom.xml
(releng|features|plugins|tests)/<component>/.*
(plugins|tests)/umlrt/<component>/.*

Some components use additional directories, also, for example the Papyrus-RT-Gerrit-Codegen job:

models/.*
scripts/codegen/.*

All jobs are triggered by changes in the root POM, which naturally affects them all.

P2 Repositories

Individual Components Nightly builds

The output of each Papyrus-RT-<Stream>-<Component> build job for individual components is simply archived so that it appears at a URL of the form

https://hudson.eclipse.org/papyrus-rt/job/Papyrus-RT-<Stream>-<Component>/lastSuccessfulBuild/artifact/repository/

These repositories serve as inputs to downstream builds that need to consume their binaries. The continuous output of the Product pseudo-component build is similarly archived (see below).

Individual Components Integration, Milestone, and Release Builds

There are no integration, milestone, nor release builds of individual components. A component on its own is by definition not integrated, so these builds do not make sense for the components.

Product Nightly Build

The output of the Papyrus-RT-<Stream>-Product build job for the Product pseudo-component is published on the Eclipse Download Server at a URL of the form

https://download.eclipse.org/papyrus-rt/updates/<stream>/nightly/

but this is nothing more than a composite p2 repository that references a single nested repository, that being the continuous-integration archive for the Product pseudo-component:

https://hudson.eclipse.org/papyrus-rt/job/Papyrus-RT-<Stream>-Product/lastSuccessfulBuild/artifact/repository/

In this way, the nightly (continuous integration) build for any given stream needs only be set up once and is effectively replaced by each new successful build.

Product Integration Build

The weekly integration build of the Papyrus-RT-<Stream>-Product-Integration build job is simply a promotion of latest continuous (nightly) build at the time that the job runs to an actual stable (though not permanent) repository on the download server at

https://download.eclipse.org/papyrus-rt/updates/<stream>/integration/I<stamp>/

This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: integration builds are deleted from the download server when the following milestone or release build is published.

Product Milestone Build

The (roughly) six-weekly milestone build of the Papyrus-RT product is a manual invocation of the Papyrus-RT-<Stream>-Product-Milestone job that signs the content and copies it to an actual stable (though not permanent) repository on the download server at

https://download.eclipse.org/papyrus-rt/updates/<stream>/milestones/M<stamp>/

This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: milestone builds are deleted from the download server when the following release build is published.

Product Release Build

Towards the end of the release, release candidate builds (RC1, RC2, etc.) are the final milestone builds, produced on a tighter schedule than the usual six weeks. The last of these is manually copied to a permanent repository on the download server at

https://download.eclipse.org/papyrus-rt/updates/<stream>/releases/R<stamp>/

where the <stamp> is the same as the RC milestone that was copied. Thus, the binaries are identical to the last RC milestone and can be trusted as having passed those quality controls.

Testing

Because a part of the definition of the build system is on the Hudson build server in the job configurations, which information is not managed in Git, it is vital that the initial development of the build system and maintenance thereafter be done in a local test environment before deployment to the Papyrus-RT HIPP on the Eclipse server. This is best accomplished in a Linux virtual machine on the developer desktop. In order to make the build as portable as possible between Hudson instances, it is necessary to parameterize certain URLs and paths in the target definitions, Maven POMs, and shell scripts:

  • p2 repository locations in upstream job archives: the filesystem path of Hudson jobs differs on each host system
  • p2 repository locations for Papyrus-RT dependencies (Papyrus, Eclipse, etc.): these are accessible in the shared downloads mount point on the Hudson server but only via HTTP in the test environment
  • Git repository URLs for clone/checkout: in the test environment, this needs to be the developer's local clone in order to access committed changes in the POMs and scripts before they are pushed to git.eclipse.org

Maven

As mentioned above, every build job uses a private Maven repository. This has a few advantages:

  • the repository is automatically cleaned out when wiping the job's workspace
  • multiple concurrent builds don't have to worry about concurrent writes to the repository
  • there is no single shared repository that will grow without bounds over time as new JARs and new versions are added to it

and a few disadvantages:

  • each build will have a distinct copy of all of the same maven plug-ins
  • there is no shared repository in which builds can install their outputs to communicate to other builds

This latter limitation is easily overcome by the target-platform definitions including the archived p2 repositories of upstream build jobs. And because each build's Maven repository is independent of all others, there is no need to perform the install goal.

Profiles

All builds start at the root POM:

mvn -f source/pom.xml clean verify

The determination of which modules to build is specified by profiles. For each component, there are a few different profiles, as summarized below. Note that the stream is implicit in the Git branch checked out by the build; the Maven POMs are all maintained in the same Git repository and branches as the sources that they build.

  • <component>-gerrit profile: includes the releng/<component>/, tests/junit/../<component>, and tests/junit/../<dependent>/ modules (recall that each Gerrit build runs the tests of all of the components that depend on it)
  • <component> profile: includes the releng/<component>/ and tests/junit/../<component> modules and the releng/org.eclipse.papyrusrt.<component>.p2/ module for building the repository that is later copied to $WORKSPACE/repository by the post-build.sh script and then archived by Hudson in the build's artifacts

Back to the top