Difference between revisions of "Papyrus-RT/Developer/Developer Guide/Builds"
(Initial documentation of Papyrus-RT builds design)
(Add some detail about job triggering)
|Line 95:||Line 95:|
== P2 Repositories ==
== P2 Repositories ==
Revision as of 11:21, 9 June 2016
- 1 Papyrus-RT Build System
- 2 Overview
- 3 P2 Repositories
- 4 Testing
- 5 Maven
Papyrus-RT Build System
This page presents documentation of the design and implementation of the Papyrus-RT builds in the project's Eclipse HIPP instance.
The Papyrus-RT builds serve at least three important audiences:
- developers: for validation of contributions, automated testing and quality reporting
- testers: for evaluation of features and bug fixes, quality assurance
- end users: for all use cases that the software is designed to support
These different audiences typically have many common requirements of the build, but also some different requirements:
- developers typically look for quick turn-around of compile and test phases; don't need packaging because they work in a development environment with run-time workbenches in debug mode
- testers need frequent packaging builds, typically throughout the day, in order to be able to evaluate fixes and new feature deliveries in a timely manner
- end users usually look for occasional milestone builds that are assured of some stability and tested quality; these builds should be properly signed to verify the origin and integrity of the download. Bleeding-edge users may want weekly integration builds but typically don't follow updates day-to-day
The combination of Hudson jobs for triggering and publishing builds with Maven/Tycho for performing builds offers all the flexibility that is required to serve the various needs of these disparate audiences. The following subsections present a high-level description of how build jobs are organized on the Papyrus-RT HIPP instance to satisfy these requirements.
One important consideration is that the development of Papyrus usually progresses on multiple parallel streams, targeting different feature (x.y version) releases. Accordingly, the build system needs to be able to handle this parallel-stream development. In particular,
- Gerrit builds need to be able to be triggered to verify contributions on all active streams
- packaging and weekly integration builds are required to publish p2 repositories and, where appropriate, RCP packages, for all active streams
- continuous integration builds are required to provide up-to-the minute binaries for the developer environment's PDE target, plus automated test/quality reports, for all active streams
Therefore, a degree of parameterization of the builds is required to account for:
- branches in the Git source repository from which to build the sources
- repositories from which to obtain binaries of dependencies (e.g., Eclipse, Papyrus, GMF, other Papyrus-RT components)
- repository URLs at which to publish artifacts
The descriptions below make references to certain variables that are placeholders for names of streams, components, and other elements of variability in the build configurations. Those variables are listed and defined here. These variables are referenced in angle brackets, e.g.
<name> where "name" (without the quotation marks) is the variable name. Variable names are not case-sensitive, so
<Name> both reference the same variable "name". However, where a variable name reference is capitalized, the text that is substituted for it is rendered in Title Case (or CamelCase, depending on its context). Otherwise, it is rendered in lower case.
<stream>— the development stream, corresponding to an Eclipse simultaneous release. e.g.,
oxygenin reference to maintenance or advanced development of releases other than the current release; or
masterin reference to the current release
<component>— a component of the Papyrus-RT product, more or less corresponding to a module in the Git source repository. e.g.,
productfor the entirety of Papyrus-RT
<build>— a kind of build:
<stamp>— a 12-digit build timestamp, of the form
<dependent>— placeholder for any or each of the
<component>s that depend on a given component
Hudson Build Jobs
The diagram below shows the organization of build jobs on the Hudson server for the Master stream (the Neon release stream at the time of writing). As you can see, there are quite a few jobs: one per component in the Gerrit and continuous (nightly) jobs. Solid arrows show triggering relationships between jobs, dashed arrows show target-platform inputs, and candy arrows show publication of artifacts to a p2 repository (update site) on the downloads server. All job names are prefixed by "Papyrus-RT" (as indicated by the leading hyphen) although this is, strictly speaking, redundant on the Papyrus-RT HIPP instance.
The Gerrit jobs all are triggered independently by changes in the portions of the Git source tree that they build:
plugins/<component>(which in many cases includes an
tests/<component>(again, sometimes with an
Each Gerrit job builds the sources for its components and runs tests for all of that component's dependency tree with a target platform based on the latest nightly continuous build produced by the Papyrus-RT-
<Stream>-Product build (in the example of this diagram, the Papyrus-RT-Master-Product build) plus, of course, Papyrus, Eclipse, and the rest of the stack. So,
- the -Gerrit-
<Stream>-Profile build runs absolutely all tests
- the -Gerrit-
<Stream>-Common build runs all tests except profile
- the -Gerrit-
<Stream>-Core build runs all tests except profile and common
- the -Gerrit-
<Stream>-Tooling build runs the tooling tests
- the -Gerrit-
<Stream>-xtUML-RT build runs the xtUML-RT and codegen tests
- the -Gerrit-
<Stream>-Codegen build runs the codegen tests
As a special case, changes to the target definitions in the
targetplatform/ tree trigger the Papyrus-RT-
<Stream>-Tooling build as being the one most likely to have the most dependencies on Papyrus and the rest of the stack beneath Papyrus-RT.
Target Platforms<a name="tps"/>
To ensure isolation and concurrent execution of the builds, they use private Maven repositories and execute only the
verify goals; they do not install. Accordingly, the target platform for each build must include its dependencies. For the Gerrit builds, this is simple: the Gerrit builds all include the
in their target platform (thus, all can share a target-platform definition) because they are triggered independently and build by overlaying their individual component's sources on the common target.
In the cases of the continuous integration (nightly) builds, each has its own target platform that adds the last successful archived repositories of its dependencies
to the shared platform comprising Papyrus, Eclipse, and the rest of the stack below Papyrus-RT. So, for example, in the master stream:
Both the continuous integration (nightly) and Gerrit build jobs for each component are triggered by changes only in specific locations in the Git repository. Triggering paths for the Git poll trigger for the continuous integration jobs are specified as inclusion and exclusion patterns. Only regular expression patterns are supported, so for consistency (promoting correct alignment of the jobs) the Gerrit triggers will also specify triggering paths via regular expressions.
<Component> and Papyrus-RT-
<Component> jobs are triggered by changes in paths matching patterns of the form
pom.xml (releng|features|plugins|tests)/<component>/.* (plugins|tests)/umlrt/<component>/.*
Some components use additional directories, also, for example the Papyrus-RT-Gerrit-Codegen job:
All jobs are triggered by changes in the root POM, which naturally affects them all.
Individual Components Nightly builds
The output of each Papyrus-RT-
<Component> build job for individual components is simply archived so that it appears at a URL of the form
These repositories serve as inputs to downstream builds that need to consume their binaries. The continuous output of the
Product pseudo-component build is similarly archived (see below).
Individual Components Integration, Milestone, and Release Builds
There are no integration, milestone, nor release builds of individual components. A component on its own is by definition not integrated, so these builds do not make sense for the components.
Product Nightly Build
The output of the Papyrus-RT-
<Stream>-Product build job for the
Product pseudo-component is published on the Eclipse Download Server at a URL of the form
but this is nothing more than a composite p2 repository that references a single nested repository, that being the continuous-integration archive for the
In this way, the nightly (continuous integration) build for any given stream needs only be set up once and is effectively replaced by each new successful build.
Product Integration Build
The weekly integration build of the Papyrus-RT-
<Stream>-Product-Integration build job is simply a promotion of latest continuous (nightly) build at the time that the job runs to an actual stable (though not permanent) repository on the download server at
This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: integration builds are deleted from the download server when the following milestone or release build is published.
Product Milestone Build
The (roughly) six-weekly milestone build of the Papyrus-RT product is a manual invocation of the Papyrus-RT-
<Stream>-Product-Milestone job that signs the content and copies it to an actual stable (though not permanent) repository on the download server at
This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: milestone builds are deleted from the download server when the following release build is published.
Product Release Build
Towards the end of the release, release candidate builds (RC1, RC2, etc.) are the final milestone builds, produced on a tighter schedule than the usual six weeks. The last of these is manually copied to a permanent repository on the download server at
<stamp> is the same as the RC milestone that was copied. Thus, the binaries are identical to the last RC milestone and can be trusted as having passed those quality controls.
Because a part of the definition of the build system is on the Hudson build server in the job configurations, which information is not managed in Git, it is vital that the initial development of the build system and maintenance thereafter be done in a local test environment before deployment to the Papyrus-RT HIPP on the Eclipse server. This is best accomplished in a Linux virtual machine on the developer desktop. In order to make the build as portable as possible between Hudson instances, it is necessary to parameterize certain URLs and paths in the target definitions, Maven POMs, and shell scripts:
- p2 repository locations in upstream job archives: the filesystem path of Hudson jobs differs on each host system
- p2 repository locations for Papyrus-RT dependencies (Papyrus, Eclipse, etc.): these are accessible in the shared downloads mount point on the Hudson server but only via HTTP in the test environment
- Git repository URLs for clone/checkout: in the test environment, this needs to be the developer's local clone in order to access committed changes in the POMs and scripts before they are pushed to git.eclipse.org
As mentioned above, every build job uses a private Maven repository. This has a few advantages:
- the repository is automatically cleaned out when wiping the job's workspace
- multiple concurrent builds don't have to worry about concurrent writes to the repository
- there is no single shared repository that will grow without bounds over time as new JARs and new versions are added to it
and a few disadvantages:
- each build will have a distinct copy of all of the same maven plug-ins
- there is no shared repository in which builds can install their outputs to communicate to other builds
This latter limitation is easily overcome by the target-platform definitions including the archived p2 repositories of upstream build jobs. And because each build's Maven repository is independent of all others, there is no need to perform the
All builds start at the root POM:
mvn -f source/pom.xml clean verify
The determination of which modules to build is specified by profiles. For each component, there are a few different profiles, as summarized below. Note that the stream is implicit in the Git branch checked out by the build; the Maven POMs are all maintained in the same Git repository and branches as the sources that they build.
<component>-gerritprofile: includes the
tests/junit/../<dependent>/modules (recall that each Gerrit build runs the tests of all of the components that depend on it)
<component>profile: includes the
tests/junit/../<component>modules and the
releng/org.eclipse.papyrusrt.<component>.p2/module for building the repository that is later copied to
post-build.shscript and then archived by Hudson in the build's artifacts