Difference between revisions of "Papyrus-RT/Developer/Developer Guide/Builds"
m (Charles.zeligsoft.com moved page Papyrus-RT/Developer Guide/Builds to Papyrus-RT/Developer/Developer Guide/Builds: Wiki reorganisation)
Revision as of 13:03, 17 November 2016
This page presents documentation of the design and implementation of the Papyrus-RT builds in the project's Eclipse HIPP instance.
- 1 Overview
- 2 P2 Repositories
- 3 Testing
- 4 Maven
- 5 Installer
The Papyrus-RT builds serve at least three important audiences:
- developers: for validation of contributions, automated testing and quality reporting
- testers: for evaluation of features and bug fixes, quality assurance
- end users: for all use cases that the software is designed to support
These different audiences typically have many common requirements of the build, but also some different requirements:
- developers typically look for quick turn-around of compile and test phases; don't need packaging because they work in a development environment with run-time workbenches in debug mode
- testers need frequent packaging builds, typically throughout the day, in order to be able to evaluate fixes and new feature deliveries in a timely manner
- end users usually look for occasional milestone builds that are assured of some stability and tested quality; these builds should be properly signed to verify the origin and integrity of the download. Bleeding-edge users may want weekly integration builds but typically don't follow updates day-to-day
The combination of Hudson jobs for triggering and publishing builds with Maven/Tycho for performing builds offers all the flexibility that is required to serve the various needs of these disparate audiences. The following subsections present a high-level description of how build jobs are organized on the Papyrus-RT HIPP instance to satisfy these requirements.
Apart from all of this is a separate corner of the releng system that builds the Papyrus-RT Installer. This is a customization of the Eclipse Installer product from the Oomph project and is entirely independent of the Papyrus-RT product and its source code. It is described in some detail, below
One important consideration is that the development of Papyrus usually progresses on multiple parallel streams, targeting different feature (x.y version) releases. Accordingly, the build system needs to be able to handle this parallel-stream development. In particular,
- Gerrit builds need to be able to be triggered to verify contributions on all active streams
- packaging and weekly integration builds are required to publish p2 repositories and, where appropriate, RCP packages, for all active streams
- continuous integration builds are required to provide up-to-the minute binaries for the developer environment's PDE target, plus automated test/quality reports, for all active streams
Therefore, a degree of parameterization of the builds is required to account for:
- branches in the Git source repository from which to build the sources
- repositories from which to obtain binaries of dependencies (e.g., Eclipse, Papyrus, GMF, other Papyrus-RT components)
- repository URLs at which to publish artifacts
The descriptions below make references to certain variables that are placeholders for names of streams, components, and other elements of variability in the build configurations. Those variables are listed and defined here. These variables are referenced in angle brackets, e.g.
<name> where "name" (without the quotation marks) is the variable name. Variable names are not case-sensitive, so
<Name> both reference the same variable "name". However, where a variable name reference is capitalized, the text that is substituted for it is rendered in Title Case (or CamelCase, depending on its context). Otherwise, it is rendered in lower case.
<stream>— the development stream, corresponding to an Eclipse simultaneous release. e.g.,
oxygenin reference to maintenance or advanced development of releases other than the current release; or
masterin reference to the current release
<component>— a component of the Papyrus-RT product, more or less corresponding to a module in the Git source repository. e.g.,
productfor the entirety of Papyrus-RT
<build>— a kind of build:
<stamp>— a 12-digit build timestamp, of the form
<dependent>— placeholder for any or each of the
<component>s that depend on a given component
Hudson Build Jobs
The diagram below shows the organization of build jobs on the Hudson server for the Master stream (the Neon release stream at the time of writing). As you can see, there are quite a few jobs: one per component in the Gerrit and one per main group (Core, Tooling, Codegen) in the continuous (nightly) jobs. Solid arrows show triggering relationships between jobs, dashed arrows show target-platform inputs, and candy arrows show publication of artifacts to a p2 repository (update site) on the downloads server. All job names are prefixed by "Papyrus-RT" (as indicated by the leading hyphen) although this is, strictly speaking, redundant on the Papyrus-RT HIPP instance.
The Gerrit jobs all are triggered independently by changes in the portions of the Git source tree that they build:
plugins/<component>(which in many cases includes an
tests/<component>(again, sometimes with an
Each Gerrit job builds the sources for its components and runs tests for all of that component's dependency tree with a target platform based on the latest nightly continuous build produced by the Papyrus-RT-
<Stream>-Product build (in the example of this diagram, the Papyrus-RT-Master-Product build) plus, of course, Papyrus, Eclipse, and the rest of the stack. So,
- the -Gerrit-
<Stream>-Profile build runs absolutely all tests
- the -Gerrit-
<Stream>-Common build runs all tests except profile
- the -Gerrit-
<Stream>-Core build runs all tests except profile and common
- the -Gerrit-
<Stream>-Tooling build runs the tooling tests
- the -Gerrit-
<Stream>-xtUML-RT build runs the xtUML-RT and codegen tests
- the -Gerrit-
<Stream>-Codegen build runs the codegen tests
As a special case, changes to the target definitions in the
targetplatform/ tree trigger the Papyrus-RT-
<Stream>-Tooling build as being the one most likely to have the most dependencies on Papyrus and the rest of the stack beneath Papyrus-RT.
There are several Continuous jobs (nightly): Core, Tooling and Codegen. Each one builds the sub components and a sub repository without running the tests (ran in the gerrit)
<Stream>-Core build runs all the Core plugins (common, core, profile, cpp, ..) and a local repo oeprt.core.p2
<Stream>-Tooling build runs all the Tooling plugins (tooling, migration, ..) and a local repo oeprt.tooling.p2
<Stream>-Codegen build runs all the Codegen plugins (codegen, xtumlrt, rts, ...) and a local repo oeprt.codegen.p2
<Stream>-Product builds all the release engineering modules based on the previous nightly pseudo-p2: oeprt.
Additional Jobs are in charge of the release engineering part. Building:
The integration jobs ensure that the integration tests (RCPTT) pass. They are also in charge of publishing and archiving the different produced artefacts:
<Stream>-Product-Integration publishes the p2 but does not archive
<Stream>-Product-Milestones publishes and archives
<Stream>-Product-Releases does not publish and does not archive. These are manual steps that occur only after verification has certified the release
For more information see the section P2 Repositories.
To ensure isolation and concurrent execution of the builds, they use private Maven repositories and execute only the
verify goals; they do not install. Accordingly, the target platform for each build must include its dependencies. For the Gerrit builds, this is simple: the Gerrit builds all include the
in their target platform (thus, all can share a target-platform definition) because they are triggered independently and build by overlaying their individual component's sources on the common target.
In the cases of the continuous integration (nightly) builds, each has its own target platform that adds the last successful archived repositories of its dependencies
to the shared platform comprising Papyrus, Eclipse, and the rest of the stack below Papyrus-RT. So, for example, in the master stream:
Both the continuous integration (nightly) and Gerrit build jobs for each component are triggered by changes only in specific locations in the Git repository. Triggering paths for the Git poll trigger for the continuous integration jobs are specified as inclusion and exclusion patterns. Only regular expression patterns are supported, so for consistency (promoting correct alignment of the jobs) the Gerrit triggers will also specify triggering paths via regular expressions.
<Component> jobs are triggered by changes in paths matching patterns of the form
pom.xml (releng|features|plugins|tests)/<component>/.* (plugins|tests)/umlrt/<component>/.*
Some components use additional directories, also, for example the Papyrus-RT-Gerrit-Codegen job:
All jobs are triggered by changes in the root POM, which naturally affects them all.
<Component> jobs are triggered in cascade such as:
<Stream>-Core is triggered every 15 minutes
<Stream>-Tooling is triggered by the-
<Stream>-Codegen is triggered by the-
<Stream>-Product is triggered by the-
<Stream>-Tooling and -
<Stream>-Product-Integration is triggered every week
<Stream>-Product-Milestones is triggered manually
<Stream>-Product-Releases is triggered manually
Currently Implemented Components
All of the preceding discussion refers to the idealized componentization of the Papyrus-RT code into the most fine-grained plausibly self-contained units. But for practical purposes, the initial implementation of the build system takes a coarser-grained approach. The following table illustrates the grouping of logical components into larger components in the build system. Once the current build architecture has been refined and proven, as the need arises, these components may be refactored (which exercise is itself a good test of the build architecture).
|Aggregate Component||Component||Git Path(s)|
Individual Components Nightly builds
The output of each Papyrus-RT-
<Component> build job for individual components is simply archived so that it appears at a URL of the form
These repositories serve as inputs to downstream builds that need to consume their binaries. The continuous output of the
Product pseudo-component build is similarly archived (see below).
Individual Components Integration, Milestone, and Release Builds
There are no integration, milestone, nor release builds of individual components. A component on its own is by definition not integrated, so these builds do not make sense for the components.
Product Nightly Build
The output of the Papyrus-RT-
<Stream>-Product build job for the
Product pseudo-component is published on the Eclipse Download Server at a URL of the form
but this is nothing more than a composite p2 repository that references a single nested repository, that being the continuous-integration archive for the
In this way, the nightly (continuous integration) build for any given stream needs only be set up once and is effectively replaced by each new successful build.
Also, when the content of the RCP package is decided, then that will be published as an archive on an appropriate URL to be determined on the download server, in the format most appropriate for each OS platform. If there should be multiple different RCP packages defined, then it may be more appropriate to off-load the cost of the RCP packaging into a separate Papyrus-RT-
Product Integration Build
The weekly integration build of the Papyrus-RT-
<Stream>-Product-Integration build job is simply a promotion of latest continuous (nightly) build at the time that the job runs to an actual stable (though not permanent) repository on the download server at
This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: integration builds are deleted from the download server when the following milestone or release build is published.
Product Milestone Build
The (roughly) six-weekly milestone build of the Papyrus-RT product is a manual invocation of the Papyrus-RT-
<Stream>-Product-Milestone job that signs the content and copies it to an actual stable (though not permanent) repository on the download server at
This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: milestone builds are deleted from the download server when the following release build is published.
Product Release Build
Towards the end of the release, release candidate builds (RC1, RC2, etc.) are the final milestone builds, produced on a tighter schedule than the usual six weeks. The last of these is manually copied to a permanent repository on the download server at
<stamp> is the same as the RC milestone that was copied. Thus, the binaries are identical to the last RC milestone and can be trusted as having passed those quality controls.
Because a part of the definition of the build system is on the Hudson build server in the job configurations, which information is not managed in Git, it is vital that the initial development of the build system and maintenance thereafter be done in a local test environment before deployment to the Papyrus-RT HIPP on the Eclipse server. This is best accomplished in a Linux virtual machine on the developer desktop. In order to make the build as portable as possible between Hudson instances, it is necessary to parameterize certain URLs and paths in the target definitions, Maven POMs, and shell scripts:
- p2 repository locations in upstream job archives: the filesystem path of Hudson jobs differs on each host system
- p2 repository locations for Papyrus-RT dependencies (Papyrus, Eclipse, etc.): these are accessible in the shared downloads mount point on the Hudson server but only via HTTP in the test environment
- Git repository URLs for clone/checkout: in the test environment, this needs to be the developer's local clone in order to access committed changes in the POMs and scripts before they are pushed to git.eclipse.org
As mentioned above, every build job uses a private Maven repository. This has a few advantages:
- the repository is automatically cleaned out when wiping the job's workspace
- multiple concurrent builds don't have to worry about concurrent writes to the repository
- there is no single shared repository that will grow without bounds over time as new JARs and new versions are added to it
and a few disadvantages:
- each build will have a distinct copy of all of the same maven plug-ins
- there is no shared repository in which builds can install their outputs to communicate to other builds
This latter limitation is easily overcome by the target-platform definitions including the archived p2 repositories of upstream build jobs. And because each build's Maven repository is independent of all others, there is no need to perform the
All builds start at the root POM:
mvn -f source/pom.xml clean verify
The determination of which modules to build is specified by profiles. For each component, there are a few different profiles, as summarized below. Note that the stream is implicit in the Git branch checked out by the build; the Maven POMs are all maintained in the same Git repository and branches as the sources that they build.
<component>-gerritprofile: includes the
tests/junit/../<dependent>/modules (recall that each Gerrit build runs the tests of all of the components that depend on it)
<component>profile: includes the
tests/junit/../<component>modules and the
releng/org.eclipse.papyrusrt.<component>.p2/module for building the repository that is later copied to
post-build.shscript and then archived by Hudson in the build's artifacts
As indicated in the top POM, there are three key properties that determine the selection of the Target Platform for any given build. Each of the profiles discussed above sets these properties (overriding one or more of the default values), but some build jobs also need to override these in the Hudson configuration. In some cases, that is a temporary measure, such as the Core job selecting the
releng target kind instead of 'core' as long as its tests require Tooling bundles. In other cases, it is a more long-term thing, such as selection of the
papyrusnightly Papyrus build kind.
eclipse.release— the Eclipse release stream on which to base the build. Values are
papyrus.kind— the kind of Papyrus build to use of the stream selected by the
eclipse.release. Values are
target.kind— the kind of Target Platform to use. There are target platforms specific to each component's continuous-integration build, a TP tailored for the Gerrit builds, another for the Product packaging build, etc.
The Papyrus-RT Installer is an RCP product build based on the Eclipse Installer from the Oomph project. The few Eclipse plug-in sources and the build system are entirely contained within the
releng/installer/ tree in the Git repository.
The installer product is an Eclipse RCP product like any other. The
org.eclipse.papyrusrt.installer plug-in is the product branding plug-in, with application name, window images, and other branding details in the usual way associated the the
org.eclipse.papyrusrt.installer.product product identifier. The application for the product is the Eclipse Installer,
org.eclipse.oomph.setup.installer.application. This is re-used as is, because it draws its branding from the active Eclipse Product definition and (as required, possibly not yet) other extensibility mechanisms built into the installer UI. This plug-in is packaged via the
org.eclipse.papyrusrt.installer.feature feature in the feature-based product definition in the
Apart from the usual RCP product definition things in the
org.eclipse.papyrusrt.installer.product directory are some that are peculiar to the Oomph installer framework. Most notable is the product descriptor file:
product-descriptor-64.txt for the 64-bit product builds and
product-descriptor-32.txt for the 32-bit product builds. This is used by the self-extracting installer executable (currently only available for Windows platform) to determine:
- the minimum Java execution environment version (e.g., J2SE 1.7)
- whether a full JDK is required or just a JRE
- in case these requirements are not met by the host system, the product name and a branding image that the self-extractor can inject into the web page to which it directs the user to download the required Java EE
Most of the build process for the installer is accomplished by the usual Tycho plug-ins for Maven as for any RCP product. The interesting bits are in the scripts used to build the final installer packages and to publish them on the Eclipse downloads server. These scripts are located in the
releng/installer/scripts directory. These scripts are executed by the Hudson build jobs, not by Maven.
repackage.sh script is responsible for re-packaging the product archives produced by Tycho. In particular, what it does is
- update the launcher INI file to tell the installer where its p2 update site is, according to the kind of build. The installer is thus pre-configured for self-updates from
- http://download.eclipse.org/papyrus-rt/installer/updates/repository for release builds or
- http://download.eclipse.org/papyrus-rt/installer/updates/nightly/repository for nightly builds
- rebuild the archive in a form optimized for the platform:
exe(a self-extracting executable) for Windows
- sign the build, if the
ECLIPSE_SIGNenvironment variable is
- only applies to Mac and Windows platforms
The self-extracting executable is built by extracting the required pieces from the latest Eclipse Installer build (of release or nightly kind according to the current build parameters) and re-assembling them, together with the appropriate Papyrus-RT Installer product descriptor file as described above, according to the procedure outlined in the Oomph wiki.
The final step in the build process is publication to the Eclipse downloads server. This is the job of the
Builds are published to
- nightly builds: http://download.eclipse.org/papyrus-rt/installer/updates/nightly/
- release builds: http://download.eclipse.org/papyrus-rt/installer/updates/
The downloadable packages are stored in each of the above folders folders:
- for Linux:
- for Mac:
- for Windows:
The p2 repository containing the update site for the previous version to use to update itself is stored in a
repository/ folder of both of these locations. This structure follows the same pattern as the download site of the Eclipse Installer.
The current nightly or release build always replaces the previous. Versions are not maintained on the download server, as the only product that it is interesting to version is what the installer provisions on the user's system. But it is useful to maintain separation of nightly builds of the installer from publicly available releases while the installer, itself, is undergoing development. The nightly build is pre-configured to update itself from the nightly update site and the release build from the release update site.
Two Hudson build jobs implement the build and publication phases of the installer build:
- Papyrus-RT-Installer builds the installer product and, for nightly builds only, publishes it to the download server
- Papyrus-RT-Installer-Promotion publishes the last successful build of the Papyrus-RT-Installer job
The latter is really only useful for publishing the release build, because a nightly build is automatically published when it is complete. The separation of these steps for the release build is important to provide an opportunity to test the build before publishing it.
Installer Release Process
By default, the Papyrus-RT-Installer build is configured to build and publish a nightly installer build, with signing. To disable signing, uncheck the
ECLIPSE_SIGN build parameter.
To create a release build, change the value of the
BUILD_TYPE parameter to
release (the milestone and integration options aren't really interesting for installer development). This kind of build is not automatically published. All of the installer packages should be downloaded and tested on each platform prior to publication; find them in the build artifact archive. At a minimum, it is required to verify in the About dialog that the correct p2 repository URL is shown from which the installer will update itself (see discussion above of nightly and release repositories).
Once the release build is validated, run the Papyrus-RT-Installer-Promotion build to publish it. The default value of the
BUILD_TYPE parameter in this build is
release, so just make sure that it is so.
From time to time, as needs warrant, it will be necessary to make changes to the installer, perhaps to support new customizability or other features of the Eclipse Installer on which it based. The nightly builds are a good means to iterate on and test such changes. Whenever possible, it is best to build always on the latest official release of the Eclipse Installer. This is ensured by the
papyrus.kind=papyrusrelease property set in the Maven build configuration (so named because this build uses the same target-platform selection mechanism as the Papyrus-RT builds). If you need to build on the latest Eclipse Installer nightlies, set this property value to
papyrusnightly but don't forget to set it back to the release kind before building the installer release! The Papyrus-RT Installer must not release a build that is based on a non-release build of the Eclipse Installer.