Revision as of 10:50, 30 June 2011

Graphiti Build Process

This page describes the Graphiti build setup. Thanks to the Teneo and CDO guys for providing a template for setting up a first version of our build and an excellent description of it (the Teneo build description was also used as a template for this page). Graphiti uses Buckminster and the Eclipse Hudson for its continous build. The job gmp-graphiti-nightly is used to build Graphiti's head version. To understand the content of this page it is necessary to understand the main Buckminster concepts. These are described in detail in the Bucky Book. Some Buckminster concepts can seem fairly abstract initially, the 'Understanding Buckminster' section below may help here.

Local Execution

It should be possible to do a local Graphiti build. [TODO: describe the necessary steps]

Understanding Buckminster

This section may help to make it a bit clearer how Buckminster works. The main functionality (for use within the Graphiti build):

• Buckminster creates workspaces and target platforms for running the build and test. These workspaces are the same or similar to what you see in your IDE.
• Buckminster uses the includes and dependency definitions in the MANIFEST.MF (for a plugin project) and the feature.xml (for a feature project) to download related projects or binaries. Additional dependencies are defined in a buckminster.cspex file. The build and buckminster files only refer to so-called root feature projects. The includes and dependencies of those projects then automatically drive the download of all other relevant projects and binaries into the workspace and target platform.
• Buckminster can search many types of locations for dependencies. The locations and their search order are defined in a configuration file (the graphiti.rmap files in the build and test subfolders). Development projects are downloaded from cvs while binary dependencies are coming from update sites.
• When downloading dependencies, Buckminster determines which dependency to place in the workspace (as a development project) and which dependencies to place in the target platform. The choice where to place what is defined in the graphiti.rmap file using the attribute 'source' with the value 'true', which signals a dependency to be a source (so should be placed in the workspace).
• Buckminster has several commands which can be executed against a workspace/target platform, for example run tests, build a p2.site etc.

A short description on the meaning of some files in the Graphiti build:

• rmap file: Defines where and how to download dependencies and development projects, contains cvs locations, update site url's and specifies an order in which these locations are searched. Also identifies which sources are considered to contain sources (i.e. development projects) and which are used for binary dependencies.
• cquery file: Defines which rmap file to use and defines the root (feature) project, the content (defined via the feature.xml) and the dependencies of the content are downloaded.
• mspec file: This file is the starting point. It specifies which cquery file to use and also how the plugins/development projects downloaded as part of resolving the cquery file should be treated, or materialized. Materialized means: where to place the downloaded plugins/development projects, in the workspace or in the target platform.

<h4>Main Build Steps</h4>
<p>The Graphiti build process is executed through an ant script
which calls Buckminster for the main build steps. The ant script goes
through the following steps.</p>
<ol>
<li>Download the p2 director and using the director, download
buckminster</li>
<li>Download the development projects from CVS and use them to
create a workspace with a target platform</li>
<li>Download/retrieve the dependencies and set them in the target
platform</li>
<li>Build the software (including the test plugins)</li>
<li>Build the update site</li>
<li>Create downloadable zips</li>
<li>Run the test cases and convert the test results to a format
which can be read by Hudson</li>
</ol>

<p>The following sections describe the necessary files for the
build: the ant xml files and the buckminster files.</p>

<h4>Structure of build projects/features</h4>
<p>The build infrastructure for Graphiti is centered around a releng
project containing the ant build.xml file and the main Buckminster
files. A number of other feature projects are used to define which
plugins are part of the build, part of the update site and which are
part of the tests. The releng and feature projects can be found in the
CVS location
dev.eclipse.org/cvsroot/modeling/org.eclipse.gmp/org.eclipse.gmp.graphiti
in the folders releng (releng project) and features (feature projects).</p>

<p>The org.eclipse.graphiti.releng project contains the ant and the
buckminster files for the build and test steps. The content of this
project and the buckminster setup is described in more detail below.
This project is the most important one to look at when trying to
understand Graphiti's build setup. The feature projects are used to
structure the output of the build in the correct way, i.e. which
features/plugins are made available via an update site, which
features/plugins are involved in testing etc. Here is a summary of these
feature projects:</p>
<ul>
<li><b>org.eclipse.graphiti.site</b> - Defines the update site and
the main features. This project contains a buckminster.cspex file to
define extra dependencies for stuff added via package imports (this is
described in more detail below). The included features are:
<ul>
<li>org.eclipse.graphiti.feature - The Graphiti runtime feature</li>
<li>org.eclipse.graphiti.feature.examples - The documentation and
the tutorial plugin for Graphiti</li>
</ul>
<li><b>org.eclipse.graphiti.tests-feature</b> - Contains the test
plugins and their dependencies. The test plugins contain test launch
configurations (see below running test cases).</li>
</ul>

<h4>Main build project: org.eclipse.graphiti.releng</h4>
<p>After downloading the main build project from cvs you can see
that it consists of several files in the root and the xsl subfolder.
Here is a summary for the meaning of different files:</p>
<ul>
<li><b>build.properties:</b> defines several properties related to
how to sign plugins and where to download the director and buckminster
applications. Used within the central build.</li>
<li><b>local.properties:</b> defines several properties related to
how to sign plugins and where to download the director and buckminster
applications. Used within the local build.</li>
<li><b>build.xml:</b> the main build file which drives the build
and test phases. Its content is described in more detail below.</li>
</ul>
<p>The Graphiti build consists of three steps: 1) build the Graphiti
plugins and features, 2) use these build results to set up a P2 update
site and 3) run the tests. The build.xml file has 3 main targets:</p>
<ul>
<li>test: is the main target, builds the core plugins, creates an
update site and runs the test cases</li>
<li>promote.sites: copies the created update site to another
location where it is available for everyone, again uses Buckminster to
do so</li>
<li>clean.all: cleans the tools and output directories, uses
standard ant remove directory commands</li>
</ul>
<p>The interesting one is the test command, this one is explained in
more detail.</p>

<p>This test target depends on the target site.p2 and triggers the
test runs for Graphiti (there are 3 of them: a JUnit test run for each
the plugins org.eclipse.graphiti and org.eclipse.graphiti.ui and a
SWTBot-based test).</p>

<p>Before the site.p2 command is run which triggers a build using
Buckminster, creates an update site again by using Buckminster and
providing human readable content descriptions.</p>

<p>This step again depends on the "provision" step which sets up all
the relevant variables (e.g. from the properties files), installs the
Buckminster Director application, the actual Buckminster installation
and collects the target platform and workspace to do the build.</p>

<h5><b>Build:</b></h5>
<p>The build setup consists of the following files:</p>
<ul>
<li><b>build.cquery:</b> this files defines the root request
project. It is the releng project that again defines the dependencies
which projects to retrieve from cvs; this is mainly the site feature
with its complete dependency tree. The content of the root feature
project in the feature.xml (and an additional dependency definition
file, a cspex file, see below) specifies which development projects to
download.</li>
<li><b>build.mspec:</b> this file specifies that dependencies
which are source (source="true") should be materialized in the
workspace, while the other dependencies are materialized in the target
platform.</li>
<li><b>build.rmap:</b> defines where and how development projects
and dependencies can be downloaded. This file for example lists cvs
locations and several update sites.</li>
</ul>
<p>The mspec file is the starting point for Buckminster. Its
content:</p>
<pre>
<code>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;mspec:mspec xmlns:mspec="http://www.eclipse.org/buckminster/MetaData-1.0" installLocation="" materializer="p2" name="build.mspec" url="build.cquery"&gt;
&lt;mspec:property key="target.os" value="*"/&gt;
&lt;mspec:property key="target.ws" value="*"/&gt;
&lt;mspec:property key="target.arch" value="*"/&gt;
&lt;mspec:mspecNode materializer="workspace" filter="(buckminster.source=true)"/&gt;
&lt;/mspec:mspec&gt;
</code>
</pre>

<p>This file specifies two things: firstly, which query file to use,
the cquery file is explained below, it defines which root feature
project to download as a starting point for the build and secondly, how
to materialize: as a default materialize to the target platform
(materializer="p2"), only if source is "true" materialize to the
workspace. As noted above the mspec file defines which cquery file to
use; here's the content of the cquery file:</p>
<pre>
<code>
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="build.rmap"&gt;
&lt;cq:rootRequest name="org.eclipse.graphiti.releng" componentType="buckminster"/&gt;
&lt;cq:advisorNode namePattern="^org\.eclipse\.graphiti(\..+)?\.source*" skipComponent="true"/&gt;
&lt;cq:advisorNode namePattern=".*" useMaterialization="false" useTargetPlatform="false"/&gt;
&lt;/cq:componentQuery&gt;
</code>
</pre>
<p>The cquery file defines which rmap file to use, the rmap file
specifies the download locations (update sites) and cvs locations to
download dependencies. Buckminster needs a root project to use as a
starting point which is defined by the rootRequest tag. The advisorNode
tag is used to tell Buckminster to ignore dependencies (defined in
feature.xml files) which have a pattern starting with
org.eclipse.graphiti and ending with source. This is needed because the
SDK feature projects contain these dependencies. However, these *.source
plugins/features do not exist in cvs but are created by
Buckminster/Eclipse during the build. So they should be ignored when
downloading dependencies. See <a
href="http://www.eclipse.org/forums/index.php?t=tree&th=162136&">here</a>
for a thread on the Buckminster newsgroup on this.</p>

<p>The rmap file describes where Buckminster can download the
development projects and dependencies. For example the following snippet
defines the cvs location for the Graphiti plugins (note also the
source="true" attribute which signals Buckminster to treat these
dependencies as source and put them in the workspace (see the mspec file
above)).</p>
<pre>
<code>
&lt;rm:searchPath name="releng"&gt;
&lt;rm:provider componentTypes="buckminster,osgi.bundle" readerType="cvs"&gt;
&lt;rm:uri format="{0},org.eclipse.gmp/org.eclipse.gmp.graphiti/releng/{1}"&gt;
&lt;bc:propertyRef key="cvs.repository"/&gt;
&lt;bc:propertyRef key="buckminster.component"/&gt;
&lt;/rm:uri&gt;
&lt;/rm:provider&gt;
&lt;rm:provider componentTypes="eclipse.feature" readerType="cvs"&gt;
&lt;rm:uri format="{0},org.eclipse.gmp/org.eclipse.gmp.graphiti/features/{1}"&gt;
&lt;bc:propertyRef key="cvs.repository"/&gt;
&lt;bc:propertyRef key="buckminster.component"/&gt;
&lt;/rm:uri&gt;
&lt;/rm:provider&gt;
&lt;/rm:searchPath&gt;
</code>
</pre>

<p>Then create the p2 site, note that the top feature used for the
site is not included in the site (as a default, see an option a few
lines further down).</p>
<pre>
<code>
&lt;buckminster command="perform" workspace="${workspacePath}"&gt;
&lt;cmdargs&gt;
&lt;arg value="org.eclipse.graphiti.site#site.p2" /&gt;
&lt;/cmdargs&gt;
&lt;/buckminster&gt;
</code>
</pre>

<p>This concludes the build step. The next step is testing using the
output from the build process.</p>

<h5>Test:</h5>
<p>The test task finally call the tests on the previously build
plugins.</p>

<p>The build.xml has a "test" target which performs the necessary
steps to run the test cases. The emma command will run junit with code
coverage checking. See a later section on code coverage setup. Instead
of the emma command, the command 'junit' could also have been used (but
then no code coverage report is created of course). The junit test run
itself consists of calling a launch configuration and defining the
output of the test results. The launch configuration has been created
manually and checked into cvs.</p>
<pre>
<code>
&lt;buckminster command="emma" workspace="${workspacePath}"&gt;
&lt;cmdargs&gt;
&lt;arg value="-l" /&gt;
&lt;arg value="org.eclipse.graphiti.tests/AllJunitTests.launch" /&gt;
&lt;arg value="-o" /&gt;
&lt;arg value="${testResultsPath}/output/JUnit-graphiti-results.xml" /&gt;
&lt;arg value="--stdout" /&gt;
&lt;arg value="${testResultsPath}/output/JUnit-graphiti-stdout.txt" /&gt;
&lt;arg value="--stderr" /&gt;
&lt;arg value="${testResultsPath}/output/JUnit-graphiti-stderr.txt" /&gt;
&lt;arg value="--xml" /&gt;
&lt;arg value="${testResultsPath}/output/JUnit-graphiti-coverageReport.xml" /&gt;
&lt;arg value="--flatXML"/&gt;
&lt;/cmdargs&gt;
&lt;/buckminster&gt;
</code>
</pre>
<p>The flatXML argument is used to convert the output from the tests
to a result which can be read by Hudson.</p>

<h4>Defining dependencies: buckminster.cspex</h4>
<p>Buckminster uses the explicit plugin dependencies in the
MANIFEST.MF to download related project and binary dependencies.
Buckminster can however not directly/automatically resolve dependencies
defined by the import-packages directive. This is logical as Buckminster
can not identify the plugins to download, just on the basis of the
package name. To solve this, it is possible to define additional
dependencies using a so-called buckminster.cspex file. The Graphiti
build has a buckminster.cspex file in two feature projects and in the
releng project:</p>
<ul>
<li>org.eclipse.graphiti.site-feature</li>
<li>org.eclipse.graphiti.tests-feature</li>
<li>org.eclipse.graphiti.releng</li>
</ul>
<p>The releng project is used in therootRequest of the cquery file
and delegates to the two feature projects. Note, the file containing the
additional dependencies must be called buckminster.cspex.</p>
<p>This cspex file defines dependencies on several eclipse features.
The rmap file define where to find these features (i.e. which update
site for example).</p>

<h4>Testing</h4>
<p>Testcases are run through Buckminster using a launch
configuration (so Buckminster calls a launch configuration). Graphiti
has 3 launch configurations one for running JUnit tests for the core
plugin (org.eclipse.graphiti), one for running JUnit tests for the UI
plugin (org.eclipse.graphiti.ui) and one for running the SWTBot tests
for the complete framework (UI tests).</p>
<p>The launch configuration is easy to create, after running your
test suite in Eclipse, a run configuration should be present in the
window Run > Run Configurations. To store the run/launch configuration
in a file: in the last tab of the configuration (Common), the save as
location is set to a shared file in a project. The file is checked-in
into cvs. This shared file is then used as the input to the test run
through buckminster, see the call for running the tests above. The
output location is specified with the -o parameter. The xml in the
output location is used by Hudson to create a test report.</p>

<h4>Defining and creating an update site</h4>
<p>The Graphiti build creates an update site, which site is defined
through a feature project: the org.eclipse.graphiti.site project. It has
one important file:</p>
<ul>
<li>feature.xml: defines which features are part of the update
site</li>
</ul>
<p>The feature.xml contains 2 features:</p>
<ul>
<li>org.eclipse.graphiti.feature</li>
<li>org.eclipse.graphiti.feature.examples</li>
</ul>
<p>To build the update site the build xml file contains the
following snippet:</p>
<pre>
<code>
&lt;buckminster command="perform" workspace="${workspacePath}">
&lt;cmdargs>
&lt;arg value="org.eclipse.graphiti.site:eclipse.feature#site.p2" />
&lt;/cmdargs>
&lt;/buckminster>
</code>
</pre>
<p>Buckminster is called with the org.eclipse.graphiti.site feature
(note feature id is used here, not the project name - here thi sis not
relevant since they are the same, but they might differ), and the
site.p2 call. As a default (is option controlled) the content of the
passed in feature is used for the content of the site, so the feature
project (org.eclipse.graphiti.site) itself is not part of the update
site.</p>

<h4>Code Coverage</h4>
<p>It is quite easy to enable code coverage analysis as part of the
test run. This section explains how to set it up.</p>
<p>First make sure that the
org.eclipse.buckminster.emma.headless.feature.feature.group feature is
installed as part of the Buckminster install. See the
install.buckminster target in Graphiti's build.xml. To enable code
coverage analysis in Buckminster you have to use the 'emma' command
instead of the 'junit' command when calling Buckminster and pass an
extra xml parameter with the output location of the code coverage
analysis.</p>