Papyrus/Papyrus Developer Guide/Build Process
- 1 Requirements to manage a build
- 2 Description of the build
- 3 How to add a plugin and or a set of plugins in the build process of Papyrus
- 4 Build Best practices
- 5 Special Thanks
Requirements to manage a build
- be a Papyrus commiter (of course)
- have ssh right on build.eclipse.org (log a bug to Community/Server to get it, see bug 366699)
- have right on dev.eclipse.org:/cvsroot/callisto (log a bug to Community/CVS to get it, see bug 366727)
- use extssh and not pserver for this connection
- have right to lauch this job https://hudson.eclipse.org/hudson/view/Repository%20Aggregation/job/juno.runAggregator/ (should comes with the previous right)
- have ssh right on download.eclipse.org (should comes with commiter right)
Description of the build
Papyrus MDT project is currently build using the Common Build Infrastructure/Managing Hudson; 9 hudson jobs are currently dedicated to Papyrus on the server here? :
build on the branch 0.9.X
- papyrus-trunk-nightly builds Papyrus from the SVN repository using the trunk version of the code. It produces the nightly build (N) of the tool, which is provided as a p2 repository
- papyrus-trunk-nightly-tests launches tests on the result of the previous job using an Eclipse 4.2
- papyrus-trunk-nightly-3.8-tests launches tests on the result of papyrus-trunk-nightly, but on an Eclipse 3.8
- papyrus-trunk-extra-nightly builds extra plugins for Papyrus trunk. It is dedicated for experimental tools, sub-components of the tool that should not be placed in the current distribution, etc.
- papyrus-trunk-extra-nightly-tests (not yet used)
builds on the branch 0.8.X
- papyrus-0.8-maintenance-nightly builds Papyrus from the SVN repository using the branch/0.8.X version of the code. It produces the nightly build (N) of the tool, which is provided as a p2 repository
- papyrus-0.8-maintenance-nightly-tests launches tests on the result of the previous job
- papyrus-0.8-maintenance-extra-nightly builds extra plugins for Papyrus branch 0.8.X. It is dedicated for experimental tools, sub-components of the tool that should not be placed in the current distribution, etc.
- papyrus-0.8-maintenance-extra-nightly-tests launches tests on the previous build (not yet used)
Hudson is only a continuous integration build server. That means that it only control the launch of build actions. It does not defined what is inside the job itself.
Hudson also provides a nice web interface in order to:
- manage the configuration,
- launch the builds manually
- have nice reports about last builds: junit test reports, change notes, build time and size
- web interface corresponding to the build workspace on the server, to easilly navigate the workspace from the web.
Several tools exists to monitore Hudson build status. I use the following one on Windows: Hudson Tray Tracker. This displays notifications as soon as a build crashes and display a nice window on Project build status.
The Papyrus jobs are located here.
The configuration of the job (accessible from the main job page > configure) is divided into 3 parts:
- The begining describes the job itself, the rights attribution on the job, etc.
- The second section describes the various parameters for the build. The parameters can have different types: String, Boolean and Choice.
When mutli-valued, the first value is selected by default when launching the job automatically.
- The third section describes the management of source code: which repository to watch, when the build should be triggered, etc.
- The fourth section is the section that "does" the job: call to ant tasks mainly for Papyrus job. It is detailed in the next chapter
- The last section contains the actions post-build: how to publish the reports, which results should be available, who should receive a mail when build is unstable or broken.
4 subtasks are describing the steps of the job
- The first step is a script shell that produces some environment variables, mainly build ids. It also cleans the previous results.
- The second step is the main step, relying on buckminster tool to produce the build target platform, download the sources from SVN, compiling plugins and gather all results in a p2 repository. It is configured by the variable BUILD_TARGET, specified as an choice value when launching the build. The 2 values are:
|| consisting on producing target platform, downloading sources and compiling them, gathering results in p2 repository|
| test (default value)
|| all site.p2 tasks and then launching junit tests on the results.|
- The third and fourth steps gathers all results (build logs, p2 repository, junit results) to be send to the download servers.
Configuration of the build task
The buckminster gathers itself sources from the Eclipse SVN repository. The configuration of the build is placed in the project org.eclipse.mdt.papyrus.releng.buckminster. This project contains all artifacts required by the build. The content of the plugins and features to be build is defined in the feature org.eclipse.papyrus.build-feature.
This project contains:
- build.xml: the ant file realizing the main build task. It relies mainly on buckminster engine.
- build.properties: the basic configuration for variables for the build.xml file.
- build.mspec: configuration file for buckminster. This file should not be modified. It mainly points to the cquery file (see more information on buckminster reference book)
- build.cquery: configuration file for buckminster. This file should not be modified also. For example, it indicates that the papyrus source plugins should not be computed, as they are directly handled and generated by buckminster itself. It also points to the build.rmap file
- build.rmap: it indicates where to find the various artifacts required for the build. It indicates where the sources can be found for Papyrus from SVN, using locator patterns and svn providers. It also indicates where to get the p2 repositories used to download IUs and produce the build target platform. This file should be modified as soon as the structure of the svn evolve (moving the build from branch to trunk, new folder, etc.) and when a new dependency is required by the papyrus tool.
The project also contains one folder 'xsl', used to generate the psf files from the Bill of Material generated by buckminster engine.
This feature is the master feature for the build. It indicates which features will be present on the Papyrus update site, the categories proposed by the update site, the plugins and fragments for junit tests, etc.
Publising results to the build server
This part is a bit tricky, as the build server (https://hudson.eclipse.org) and the download server are 2 separated servers, with different access rights. The download server can be accessed by all commiters, the hudson server jobs are launched by a user that does not have write access to the download server. Thus, publishing results must be done by 2 different tasks:
- the hudson task gathers all results and sends a signal to the download server (basically, writing in a signal file on the download server, giving the build id).
- A cron task on the download server area, running each 5 minutes, is comparing the timestamp of the signal file to a reference file. When the signal file is more recent than the reference file, the cron task proceeds to some actions: cleaning the download area (leaving for example only the last 5 nightly builds for each version of the plugins), and then getting the results provided by the hudson job to copy them in the right places in the download area. The cron task launches the script cronPromote located in the area /opt/public/modeling/mdt/papyrus/ of the download server.
How to add a plugin and or a set of plugins in the build process of Papyrus
First of all, you have to add a bugin the papyrus bugzilla, under the Add Releng task. So we can have a trace of which plugin has been added, by who and when. See the following bug for example
Adding a plugin
- To add a plugin in the build process, it must be accessible from the rmap file of the releng project. The entries in this file indicates where the sources and the required binaries can be found.
- The second step is to add it as a packaged plugin to an existing feature or create a new feature that will be added to the build process (See Adding a feature below).
- Commit your modifications to the SVN repository (map file in the releng project, feature referencing this packaged plugin, and your plugin itself)
- You can then manually lauch the build to test the new configuration using the "Launch a build" button on the left side of the window. If you do not see the button, you are not connected as a papyrus commiter.
Adding a feature
- To add a feature in the build process, it also has to be accessible from the rmap file of the releng project. Do not forget to add also the plugins packaged in this feature, following previous chapter.
- You must then reference this feature in the main papyrus feature or the feature requiring your new feature.
- Commit your modifications to the SVN repository (map file in the releng project, feature referencing this new feature, and your feature itself)
- Finally, manually launch a build to test the configuration.
If the build works, do not forget to close the bugzilla task created in the first point.
Build Best practices
- Setting a build for a personal branch (bugs/** or committers/**)
- Should not use a daily periodical build
- add a link (in the description of the job) to the wiki page of the work done in the branch
I would like to thank Nick Boldt and the Modisco team for their help during the initialization of the build process. You can find details about the Athena Build Process in the Modisco Releng part in the Eclipse Wiki