Jump to: navigation, search

Papyrus-RT/Developer/Developer Guide/Builds

PapyrusForRealTime-Logo-Icon.png




Build System


This page presents documentation of the design and implementation of the Papyrus-RT builds in the project's Eclipse HIPP instance.

Overview

The Papyrus-RT builds serve at least three important audiences:

  • developers: for validation of contributions, automated testing and quality reporting
  • testers: for evaluation of features and bug fixes, quality assurance
  • end users: for all use cases that the software is designed to support

These different audiences typically have many common requirements of the build, but also some different requirements:

  • developers typically look for quick turn-around of compile and test phases; don't need packaging because they work in a development environment with run-time workbenches in debug mode
  • testers need frequent packaging builds, typically throughout the day, in order to be able to evaluate fixes and new feature deliveries in a timely manner
  • end users usually look for occasional milestone builds that are assured of some stability and tested quality; these builds should be properly signed to verify the origin and integrity of the download. Bleeding-edge users may want weekly integration builds but typically don't follow updates day-to-day

The combination of Hudson jobs for triggering and publishing builds with Maven/Tycho for performing builds offers all the flexibility that is required to serve the various needs of these disparate audiences. The following subsections present a high-level description of how build jobs are organized on the Papyrus-RT HIPP instance to satisfy these requirements.

Apart from all of this is a separate corner of the releng system that builds the Papyrus-RT Installer. This is a customization of the Eclipse Installer product from the Oomph project and is entirely independent of the Papyrus-RT product and its source code. It is described in some detail, below

Stream Model

One important consideration is that the development of Papyrus usually progresses on multiple parallel streams, targeting different feature (x.y version) releases. Accordingly, the build system needs to be able to handle this parallel-stream development. In particular,

  • Gerrit builds need to be able to be triggered to verify contributions on all active streams
  • packaging and weekly integration builds are required to publish p2 repositories and, where appropriate, RCP packages, for all active streams
  • continuous integration builds are required to provide up-to-the minute binaries for the developer environment's PDE target, plus automated test/quality reports, for all active streams

Therefore, a degree of parameterization of the builds is required to account for:

  • branches in the Git source repository from which to build the sources
  • repositories from which to obtain binaries of dependencies (e.g., Eclipse, Papyrus, GMF, other Papyrus-RT components)
  • repository URLs at which to publish artifacts

Terminology

The descriptions below make references to certain variables that are placeholders for names of streams, components, and other elements of variability in the build configurations. Those variables are listed and defined here. These variables are referenced in angle brackets, e.g. <name> where "name" (without the quotation marks) is the variable name. Variable names are not case-sensitive, so <name> and <Name> both reference the same variable "name". However, where a variable name reference is capitalized, the text that is substituted for it is rendered in Title Case (or CamelCase, depending on its context). Otherwise, it is rendered in lower case.

  • <stream> — the development stream, corresponding to an Eclipse simultaneous release. e.g., mars, neon, oxygen in reference to maintenance or advanced development of releases other than the current release; or master in reference to the current release
  • <component> — a component of the Papyrus-RT product, more or less corresponding to a module in the Git source repository. e.g., core, tooling, codegen; all or product for the entirety of Papyrus-RT
  • <build> — a kind of build: nightly, integration, milestone, release
  • <stamp> — a 12-digit build timestamp, of the form YYYYMMDDhhmm
  • <dependent> — placeholder for any or each of the <component>s that depend on a given component

Hudson Build Jobs

The diagram below shows the organization of build jobs on the Hudson server for the Master stream (the Neon release stream at the time of writing). As you can see, there are quite a few jobs: one per component in the Gerrit and one per main group (Core, Tooling, Codegen) in the continuous (nightly) jobs. Solid arrows show triggering relationships between jobs, dashed arrows show target-platform inputs, and candy arrows show publication of artifacts to a p2 repository (update site) on the downloads server. All job names are prefixed by "Papyrus-RT" (as indicated by the leading hyphen) although this is, strictly speaking, redundant on the Papyrus-RT HIPP instance.

Hudson Build Jobs
Hudson build jobs flow diagram


Gerrit Jobs

The Gerrit jobs all are triggered independently by changes in the portions of the Git source tree that they build:

  • plugins/<component> (which in many cases includes an umlrt/ path segment)
  • tests/<component> (again, sometimes with an umlrt/ path segment)
  • releng/<component>

Each Gerrit job builds the sources for its components and runs tests for all of that component's dependency tree with a target platform based on the latest nightly continuous build produced by the Papyrus-RT-<Stream>-Product build (in the example of this diagram, the Papyrus-RT-Master-Product build) plus, of course, Papyrus, Eclipse, and the rest of the stack. So,

  • the -Gerrit-<Stream>-Profile build runs absolutely all tests
  • the -Gerrit-<Stream>-Common build runs all tests except profile
  • the -Gerrit-<Stream>-Core build runs all tests except profile and common
  • the -Gerrit-<Stream>-Tooling build runs the tooling tests
  • the -Gerrit-<Stream>-xtUML-RT build runs the xtUML-RT and codegen tests
  • the -Gerrit-<Stream>-Codegen build runs the codegen tests

As a special case, changes to the target definitions in the targetplatform/ tree trigger the Papyrus-RT-<Stream>-Tooling build as being the one most likely to have the most dependencies on Papyrus and the rest of the stack beneath Papyrus-RT.


Continuous Jobs

There are several Continuous jobs (nightly): Profile, Core, Tooling and Codegen. Each one builds the sub components and a sub repository without running the tests (ran in the gerrit)

  • Papyrus-RT-<Stream>-Profile build runs all the Profile plugins (profile & uml2 specific implementation) and a local repo oeprt.profile.p2
  • Papyrus-RT-<Stream>-Core build runs all the Core plugins (common, core, cpp, ..) and a local repo oeprt.core.p2
  • Papyrus-RT-<Stream>-Tooling build runs all the Tooling plugins (tooling, migration, ..) and a local repo oeprt.tooling.p2
  • Papyrus-RT-<Stream>-Codegen build runs all the Codegen plugins (codegen, xtumlrt, rts, ...) and a local repo oeprt.codegen.p2
  • Papyrus-RT-<Stream>-Product builds all the release engineering modules based on the previous nightly pseudo-p2: oeprt.<Component>.p2

Integration Jobs

Additional Jobs are in charge of the release engineering part. Building:

  • org.eclipse.papyrusrt.p2
  • org.eclipse.papyrusrt.rcp
  • org.eclipse.papyrusrt.feature
  • org.eclipse.papyrusrt.product
  • org.eclipse.papyrusrt.rcptt

The integration jobs ensure that the integration tests (RCPTT) pass. They are also in charge of publishing and archiving the different produced artefacts:

  • Papyrus-RT-<Stream>-Product-Integration publishes the p2 but does not archive
  • Papyrus-RT-<Stream>-Product-Milestones publishes and archives
  • Papyrus-RT-<Stream>-Product-Releases does not publish and does not archive. These are manual steps that occur only after verification has certified the release

For more information see the section P2 Repositories.

Target Platforms

To ensure isolation and concurrent execution of the builds, they use private Maven repositories and execute only the clean and verify goals; they do not install. Accordingly, the target platform for each build must include its dependencies. For the Gerrit builds, this is simple: the Gerrit builds all include the

$HOME/.hudson/jobs/Papyrus-RT-<Stream>-Product/lastSuccessful/archive/repository/

in their target platform (thus, all can share a target-platform definition) because they are triggered independently and build by overlaying their individual component's sources on the common target.

In the cases of the continuous integration (nightly) builds, each has its own target platform that adds the last successful archived repositories of its dependencies

$HOME/.hudson/jobs/Papyrus-RT-<Stream>-<Component>/lastSuccessful/archive/repository/

to the shared platform comprising Papyrus, Eclipse, and the rest of the stack below Papyrus-RT. So, for example, in the master stream:

ComponentTarget Platform
Profile
Common.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
Core.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
Tooling.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Core/lastSuccessful/archive/repository/
xtUML-RT.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Core/lastSuccessful/archive/repository/
Codegen.../Papyrus-RT-Master-Profile/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Common/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-Core/lastSuccessful/archive/repository/
.../Papyrus-RT-Master-xtUML-RT/lastSuccessful/archive/repository/

Triggering

Both the continuous integration (nightly) and Gerrit build jobs for each component are triggered by changes only in specific locations in the Git repository. Triggering paths for the Git poll trigger for the continuous integration jobs are specified as inclusion and exclusion patterns. Only regular expression patterns are supported, so for consistency (promoting correct alignment of the jobs) the Gerrit triggers will also specify triggering paths via regular expressions.

The Papyrus-RT-Gerrit-<Component> jobs are triggered by changes in paths matching patterns of the form

pom.xml
(releng|features|plugins|tests)/<component>/.*
(plugins|tests)/umlrt/<component>/.*

Some components use additional directories, also, for example the Papyrus-RT-Gerrit-Codegen job:

models/.*
scripts/codegen/.*

All jobs are triggered by changes in the root POM, which naturally affects them all.

The Papyrus-RT-<Stream>-<Component> jobs are triggered in cascade such as:

  • Papyrus-RT-<Stream>-Core is triggered every 15 minutes
  • Papyrus-RT-<Stream>-Tooling is triggered by the-<Stream>-Core
  • Papyrus-RT-<Stream>-Codegen is triggered by the-<Stream>-Core
  • Papyrus-RT-<Stream>-Product is triggered by the-<Stream>-Tooling and -<Stream>-Codegen
  • Papyrus-RT-<Stream>-Product-Integration is triggered every week
  • Papyrus-RT-<Stream>-Product-Milestones is triggered manually
  • Papyrus-RT-<Stream>-Product-Releases is triggered manually

Currently Implemented Components

All of the preceding discussion refers to the idealized componentization of the Papyrus-RT code into the most fine-grained plausibly self-contained units. But for practical purposes, the initial implementation of the build system takes a coarser-grained approach. The following table illustrates the grouping of logical components into larger components in the build system. Once the current build architecture has been refined and proven, as the need arises, these components may be refactored (which exercise is itself a good test of the build architecture).

Aggregate ComponentComponentGit Path(s)
ProfileProfileplugins/umlrt/profile/
Core
Commonplugins/umlrt/common/
Coreplugins/umlrt/core/
CPPplugins/umlrt/cpp/
CodegenCodegenplugins/umlrt/codegen/
Xtumlrtplugins/xtumlrt/
Runtimeplugins/umlrt/runtime/
ToolingToolingplugins/umlrt/tooling/
Compareplugins/umlrt/compare/
Migrationplugins/umlrt/migration/

P2 Repositories

Individual Components Nightly builds

The output of each Papyrus-RT-<Stream>-<Component> build job for individual components is simply archived so that it appears at a URL of the form

https://hudson.eclipse.org/papyrus-rt/job/Papyrus-RT-<Stream>-<Component>/lastSuccessfulBuild/artifact/repository/

These repositories serve as inputs to downstream builds that need to consume their binaries. The continuous output of the Product pseudo-component build is similarly archived (see below).

Individual Components Integration, Milestone, and Release Builds

There are no integration, milestone, nor release builds of individual components. A component on its own is by definition not integrated, so these builds do not make sense for the components.

Product Nightly Build

The output of the Papyrus-RT-<Stream>-Product build job for the Product pseudo-component is published on the Eclipse Download Server at a URL of the form

https://download.eclipse.org/papyrus-rt/updates/nightly/<stream>/

but this is nothing more than a composite p2 repository that references a single nested repository, that being the continuous-integration archive for the Product pseudo-component:

https://hudson.eclipse.org/papyrus-rt/job/Papyrus-RT-<Stream>-Product/lastSuccessfulBuild/artifact/repository/

In this way, the nightly (continuous integration) build for any given stream needs only be set up once and is effectively replaced by each new successful build.

Also, when the content of the RCP package is decided, then that will be published as an archive on an appropriate URL to be determined on the download server, in the format most appropriate for each OS platform. If there should be multiple different RCP packages defined, then it may be more appropriate to off-load the cost of the RCP packaging into a separate Papyrus-RT-<Stream>-RCP build.

Product Integration Build

The weekly integration build of the Papyrus-RT-<Stream>-Product-Integration build job is simply a promotion of latest continuous (nightly) build at the time that the job runs to an actual stable (though not permanent) repository on the download server at

https://download.eclipse.org/papyrus-rt/updates/integration/<stream>/I<stamp>/

This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: integration builds are deleted from the download server when the following milestone or release build is published.

Product Milestone Build

The (roughly) six-weekly milestone build of the Papyrus-RT product is a manual invocation of the Papyrus-RT-<Stream>-Product-Milestone job that signs the content and copies it to an actual stable (though not permanent) repository on the download server at

https://download.eclipse.org/papyrus-rt/updates/milestones/<stream>/M<stamp>/

This is a copy of the repository folder from the last successful build's archive, so that it may persist after that build has been purged by Hudson. However, it is not permanent: milestone builds are deleted from the download server when the following release build is published.

Product Release Build

Towards the end of the release, release candidate builds (RC1, RC2, etc.) are the final milestone builds, produced on a tighter schedule than the usual six weeks. The last of these is manually copied to a permanent repository on the download server at

https://download.eclipse.org/papyrus-rt/updates/releases/<stream>/R<stamp>/

where the <stamp> is the same as the RC milestone that was copied. Thus, the binaries are identical to the last RC milestone and can be trusted as having passed those quality controls.

Testing

Because a part of the definition of the build system is on the Hudson build server in the job configurations, which information is not managed in Git, it is vital that the initial development of the build system and maintenance thereafter be done in a local test environment before deployment to the Papyrus-RT HIPP on the Eclipse server. This is best accomplished in a Linux virtual machine on the developer desktop. In order to make the build as portable as possible between Hudson instances, it is necessary to parameterize certain URLs and paths in the target definitions, Maven POMs, and shell scripts:

  • p2 repository locations in upstream job archives: the filesystem path of Hudson jobs differs on each host system
  • p2 repository locations for Papyrus-RT dependencies (Papyrus, Eclipse, etc.): these are accessible in the shared downloads mount point on the Hudson server but only via HTTP in the test environment
  • Git repository URLs for clone/checkout: in the test environment, this needs to be the developer's local clone in order to access committed changes in the POMs and scripts before they are pushed to git.eclipse.org

Maven

As mentioned above, every build job uses a private Maven repository. This has a few advantages:

  • the repository is automatically cleaned out when wiping the job's workspace
  • multiple concurrent builds don't have to worry about concurrent writes to the repository
  • there is no single shared repository that will grow without bounds over time as new JARs and new versions are added to it

and a few disadvantages:

  • each build will have a distinct copy of all of the same maven plug-ins
  • there is no shared repository in which builds can install their outputs to communicate to other builds

This latter limitation is easily overcome by the target-platform definitions including the archived p2 repositories of upstream build jobs. And because each build's Maven repository is independent of all others, there is no need to perform the install goal.

Profiles

All builds start at the root POM:

mvn -f source/pom.xml clean verify

The determination of which modules to build is specified by profiles. For each component, there are a few different profiles, as summarized below. Note that the stream is implicit in the Git branch checked out by the build; the Maven POMs are all maintained in the same Git repository and branches as the sources that they build.

  • <component>-gerrit profile: includes the releng/<component>/, tests/junit/../<component>, and tests/junit/../<dependent>/ modules (recall that each Gerrit build runs the tests of all of the components that depend on it)
  • <component> profile: includes the releng/<component>/ and tests/junit/../<component> modules and the releng/org.eclipse.papyrusrt.<component>.p2/ module for building the repository that is later copied to $WORKSPACE/repository by the post-build.sh script and then archived by Hudson in the build's artifacts

Properties

As indicated in the top POM, there are three key properties that determine the selection of the Target Platform for any given build. Each of the profiles discussed above sets these properties (overriding one or more of the default values), but some build jobs also need to override these in the Hudson configuration. In some cases, that is a temporary measure, such as the Core job selecting the releng target kind instead of 'core' as long as its tests require Tooling bundles. In other cases, it is a more long-term thing, such as selection of the papyrusnightly Papyrus build kind.

  • eclipse.release — the Eclipse release stream on which to base the build. Values are mars, neon, oxygen, etc.
  • papyrus.kind — the kind of Papyrus build to use of the stream selected by the eclipse.release. Values are papyrusnightly, papyrusmilestone, or papyrusrelease
  • target.kind — the kind of Target Platform to use. There are target platforms specific to each component's continuous-integration build, a TP tailored for the Gerrit builds, another for the Product packaging build, etc.

Installer

The Papyrus-RT Installer is an RCP product build based on the Eclipse Installer from the Oomph project. The few Eclipse plug-in sources and the build system are entirely contained within the releng/installer/ tree in the Git repository.

Installer Product

The installer product is an Eclipse RCP product like any other. The org.eclipse.papyrusrt.installer plug-in is the product branding plug-in, with application name, window images, and other branding details in the usual way associated the the org.eclipse.papyrusrt.installer.product product identifier. The application for the product is the Eclipse Installer, org.eclipse.oomph.setup.installer.application. This is re-used as is, because it draws its branding from the active Eclipse Product definition and (as required, possibly not yet) other extensibility mechanisms built into the installer UI. This plug-in is packaged via the org.eclipse.papyrusrt.installer.feature feature in the feature-based product definition in the org.eclipse.papyrusrt.installer.product directory.

Apart from the usual RCP product definition things in the org.eclipse.papyrusrt.installer.product directory are some that are peculiar to the Oomph installer framework. Most notable is the product descriptor file: product-descriptor-64.txt for the 64-bit product builds and product-descriptor-32.txt for the 32-bit product builds. This is used by the self-extracting installer executable (currently only available for Windows platform) to determine:

  • the minimum Java execution environment version (e.g., J2SE 1.7)
  • whether a full JDK is required or just a JRE
  • in case these requirements are not met by the host system, the product name and a branding image that the self-extractor can inject into the web page to which it directs the user to download the required Java EE

Scripts

Most of the build process for the installer is accomplished by the usual Tycho plug-ins for Maven as for any RCP product. The interesting bits are in the scripts used to build the final installer packages and to publish them on the Eclipse downloads server. These scripts are located in the releng/installer/scripts directory. These scripts are executed by the Hudson build jobs, not by Maven.

Packaging

The repackage.sh script is responsible for re-packaging the product archives produced by Tycho. In particular, what it does is

The self-extracting executable is built by extracting the required pieces from the latest Eclipse Installer build (of release or nightly kind according to the current build parameters) and re-assembling them, together with the appropriate Papyrus-RT Installer product descriptor file as described above, according to the procedure outlined in the Oomph wiki.

Publication

The final step in the build process is publication to the Eclipse downloads server. This is the job of the publish.sh script.

Builds are published to

The downloadable packages are stored in each of the above folders folders:

  • for Linux: Papyrus-RT-Installer-linux64.tar.gz
  • for Mac: Papyrus-RT-Installer-mac64.zip
  • for Windows: Papyrus-RT-Installer-win64.exe

The p2 repository containing the update site for the previous version to use to update itself is stored in a repository/ folder of both of these locations. This structure follows the same pattern as the download site of the Eclipse Installer.

The current nightly or release build always replaces the previous. Versions are not maintained on the download server, as the only product that it is interesting to version is what the installer provisions on the user's system. But it is useful to maintain separation of nightly builds of the installer from publicly available releases while the installer, itself, is undergoing development. The nightly build is pre-configured to update itself from the nightly update site and the release build from the release update site.

Hudson Builds

Two Hudson build jobs implement the build and publication phases of the installer build:

The latter is really only useful for publishing the release build, because a nightly build is automatically published when it is complete. The separation of these steps for the release build is important to provide an opportunity to test the build before publishing it.

Installer Release Process

By default, the Papyrus-RT-Installer build is configured to build and publish a nightly installer build, with signing. To disable signing, uncheck the ECLIPSE_SIGN build parameter.

To create a release build, change the value of the BUILD_TYPE parameter to release (the milestone and integration options aren't really interesting for installer development). This kind of build is not automatically published. All of the installer packages should be downloaded and tested on each platform prior to publication; find them in the build artifact archive. At a minimum, it is required to verify in the About dialog that the correct p2 repository URL is shown from which the installer will update itself (see discussion above of nightly and release repositories).

Once the release build is validated, run the Papyrus-RT-Installer-Promotion build to publish it. The default value of the BUILD_TYPE parameter in this build is release, so just make sure that it is so.

From time to time, as needs warrant, it will be necessary to make changes to the installer, perhaps to support new customizability or other features of the Eclipse Installer on which it based. The nightly builds are a good means to iterate on and test such changes. Whenever possible, it is best to build always on the latest official release of the Eclipse Installer. This is ensured by the papyrus.kind=papyrusrelease property set in the Maven build configuration (so named because this build uses the same target-platform selection mechanism as the Papyrus-RT builds). If you need to build on the latest Eclipse Installer nightlies, set this property value to papyrusnightly but don't forget to set it back to the release kind before building the installer release! The Papyrus-RT Installer must not release a build that is based on a non-release build of the Eclipse Installer.