Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Teneo/Teneo Build Setup"

(Build folder content)
(Build folder content)
Line 88: Line 88:
 
* teneo_build.xml: the ant file which executes the build step by calling Buckminster to create the workspace and target platform, download the development projects and dependencies, build the plugins and create the update site with the resulting features/plugins.
 
* teneo_build.xml: the ant file which executes the build step by calling Buckminster to create the workspace and target platform, download the development projects and dependencies, build the plugins and create the update site with the resulting features/plugins.
  
The content of the cquery file:
+
The mspec file is the starting point for Buckminster. The teneo_build.xml calls Buckminster with this file passed as a parameter. Its content:
 +
<source lang="xml">
 +
<mspec:mspec xmlns:mspec="http://www.eclipse.org/buckminster/MetaData-1.0" materializer="p2" name="teneo.mspec" url="teneo.cquery" conflictResolution="UPDATE">
 +
  <mspec:property key="target.os" value="*"/>
 +
  <mspec:property key="target.ws" value="*"/>
 +
  <mspec:property key="target.arch" value="*"/>
 +
  <mspec:mspecNode
 +
filter="(buckminster.source=true)"
 +
materializer="workspace"/>
 +
</mspec:mspec>
 +
</source>
 +
This file specifies several things:
 +
* which query file to use, the cquery file is explained below, it defines which root feature project to download as a starting point for the build.
 +
* how to materialize: as a default materialize to the target platform (materializer="p2"), only if source is "true" materialize to the workspace
 +
 
 +
As noted above the mspec file defines which cquery file to use, the content of the cquery file:
 
<source lang="xml">
 
<source lang="xml">
 
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="teneo.rmap">
 
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="teneo.rmap">
Line 95: Line 110:
 
</cq:componentQuery>
 
</cq:componentQuery>
 
</source>
 
</source>
This defines the root feature project which should be downloaded. Buckminster needs a root (feature) project to use as a starting point. This is defined by the rootRequest tag. The advisorNode tag is used to tell Buckminster to ignore dependencies (defined in feature.xml files) which have a pattern starting with org.eclipse.emf.teneo and ending with source. This is needed because the SDK feature projects contain these dependencies. However, these *.source plugins/features do not exist in cvs but are created by Buckminster/Eclipse during the build. So they should be ignored when downloading dependencies. See [http://www.eclipse.org/forums/index.php?t=tree&th=162136& here] for a thread on the Buckminster newsgroup on this.
+
* The cquery file defines which rmap file to use, the rmap file specifies the download locations (update sites) and cvs locations to download dependencies.  
 +
* Buckminster needs a root (feature) project to use as a starting point. This is defined by the rootRequest tag.  
 +
* The advisorNode tag is used to tell Buckminster to ignore dependencies (defined in feature.xml files) which have a pattern starting with org.eclipse.emf.teneo and ending with source. This is needed because the SDK feature projects contain these dependencies. However, these *.source plugins/features do not exist in cvs but are created by Buckminster/Eclipse during the build. So they should be ignored when downloading dependencies. See [http://www.eclipse.org/forums/index.php?t=tree&th=162136& here] for a thread on the Buckminster newsgroup on this.
  
 
=== Test folder content ===
 
=== Test folder content ===

Revision as of 17:56, 26 February 2010

This page describes the Teneo build setup. Teneo uses Buckminster and Hudson for its continuous build infrastructure.

The Teneo Hudson job which uses the build setup as it is described in this wiki page, is available here.

Try it

With the current build setup it is quite easy to do a Teneo build. To download Teneo and do a build (building, testing, generating p2 site) you need to do the following steps:

cvs -d dev.eclipse.org:/cvsroot/modeling co -d org.eclipse.emf.teneo.releng.buckminster \
org.eclipse.emf/org.eclipse.emf.teneo/releng/org.eclipse.emf.teneo.releng.buckminster
cd org.eclipse.emf.teneo.releng.buckminster
ant

Note the above assumes that ant is installed and that JAVA_HOME is set to Java 5.0 or higher.

After about 20 minutes you will see a build folder with a number of subfolders:

  • build: contains the targetplatform (the tp subfolder) for building the main plugins and the build workspace itself (the workspace subfolder)
  • test: contains the targetplatform for testing (in the tp subfolder), the workspace subfolder contains the workspace used for the test projects.
  • output: contains temporary output from the build steps.
  • result: contains the generation result (p2 site) and the test results.
  • tools: contains buckminster and the p2 director applications. These are downloaded as part of the build.

The build workspace (build/workspace) and test workspace (test/workspace) can be opened in Eclipse as workspaces.

Main builds steps (build, generate, test)

The build process is driven by an ant script which calls buckminster for the main build steps. The ant scripts goes through the following steps:

  1. download the p2 director and using the director, download buckminster.
  2. build the main deliverables (using buckminster):
    1. download the development projects from CVS and use them to create a workspace with a target platform
    2. download/retrieve the dependencies and set them in the targetplatform
    3. build the software (compile etc.)
    4. build the update site
    5. create downloadable zips
  3. build the test projects (using buckminster):
    1. download the development test projects from CVS and use them to create a workspace with a target platform
    2. download/retrieve the dependencies and set them in the targetplatform. This includes the artifacts generated in the previous step.
    3. build/compile the test plugins
  4. run the test cases and convert the test results to a format which can be read by hudson

The ant script also take care of copying the output (p2 site and features/plugins) and creating zip files etc.

The following sections describe the necessary files for the build: the ant xml files and the buckminster files.

When reading on it is good to take into account some of the following aspects:

  • An important part of the work done by Buckminster is that Buckminster creates a workspace with target platform for you. The workspace is the same or similar to what you see in your development project. The builds and test runs are done using the created workspace/target platform.
  • Buckminster uses the dependency definitions in the MANIFEST.MF (for a plugin project) and the feature.xml (for a feature project) to download related projects or binaries. The build and buckminster files often only refer to so-called root (feature) projects. The dependencies of the root projects (often a feature project) then automatically drive the download of all other relevant projects and binaries into the workspace and target platform.
  • When downloading dependencies, Buckminster determines which dependency to place in the workspace (as a development project) and which of the dependencies to place in the target platform. The choice where to place what is defined in the buckminster *.rmap and *.mspec file using an attribute: source="true", which signals a dependency to be a source (so should be placed in the workspace).

Structure of build projects/features

The build infrastructure for Teneo is centered around a releng project containing the ant build xml file and the main Buckminster files. Then a number of other feature projects are used to define which plugins are part of the build and which are part of the tests etc.

The releng and feature projects can be found in this location in cvs:

  • dev.eclipse.org
  • /cvsroot/modeling
  • org.eclipse.emf/org.eclipse.emf.teneo/releng and org.eclipse.emf/org.eclipse.emf.teneo/features

The org.eclipse.emf.teneo.releng.buckminster project contains the ant files, and the buckminster files for the build and test step. The content of this project and the buckminster setup is described in more detail below. This project is the most important one to look at when trying to understand how Teneo's build is setup.

The feature projects are used to structure the output of the build in the correct way, i.e. which features/plugins are made available in an update site, which features/plugins are involved in testing etc.

A summary of these feature projects:

  • org.eclipse.emf.teneo.site-feature: defines the update site and the main features it contains (Teneo Hibernate runtime and SDK and Teneo EclipseLink runtime and SDK). This project contains a buckminster.spex file to define extra dependencies (this is described in more detail below). The included features:
    • org.eclipse.emf.teneo.hibernate-feature: the runtime feature, this is a standard PDE feature project (the plugins are defined in the feature.xml).
    • org.eclipse.emf.teneo.hibernate.sdk-feature: the SDK feature includes the runtime feature and source feature (see below for more information on building SDK features).
    • org.eclipse.emf.teneo.eclipselink-feature: the EclipseLink runtime feature, this is a standard PDE feature project (the plugins are defined in the feature.xml).
    • org.eclipse.emf.teneo.eclipselink.sdk-feature: the SDK feature includes the runtime feature and source feature (see below for more information on building SDK features).
  • org.eclipse.emf.teneo.tests-feature: contains the test plugins and their dependencies. The main test plugins contain test launch configurations (see below running test cases).

Main build project: org.eclipse.emf.teneo.releng.buckminster

After downloading the main build project from cvs you can see that it consists of several files in the root and two subfolders: build and test. Here is a summary for the meaning of different files:

  • ant_common.xml: contains macros which are used both in test and build phases, the main macros are for calling buckminster in an easy way. This ant file is imported by other build files.
  • buckminster.properties: defines several properties related to how to sign plugins and where to download the director and buckminster applications from.
  • build.xml: the main build file which drives the build and test phases. Its content is described in more detail below.

The Teneo build consists of two steps: 1) build the core plugins and use these as part of the target platform when 2) building and running the test plugins. These two steps are visible as subfolders in the main build project, the build and test subfolders.

Build folder content

The build setup consists of 4 files:

  • teneo.cquery: this files defines the root feature project to retrieve from cvs, the content of the root feature project in the feature.xml (and an additional dependency definition file, a cspex file) specifies which development projects to download.
  • teneo.mspec: this file basically specifies that dependencies which are source (source="true") should be materialized in the workspace, while the other dependencies are materialized in the target platform.
  • teneo.rmap: defines where and how development projects and dependencies can be downloaded. This file for example lists cvs locations and several update sites.
  • teneo_build.xml: the ant file which executes the build step by calling Buckminster to create the workspace and target platform, download the development projects and dependencies, build the plugins and create the update site with the resulting features/plugins.

The mspec file is the starting point for Buckminster. The teneo_build.xml calls Buckminster with this file passed as a parameter. Its content:

<mspec:mspec xmlns:mspec="http://www.eclipse.org/buckminster/MetaData-1.0" materializer="p2" name="teneo.mspec" url="teneo.cquery" conflictResolution="UPDATE">
  <mspec:property key="target.os" value="*"/>
  <mspec:property key="target.ws" value="*"/>
  <mspec:property key="target.arch" value="*"/>
  <mspec:mspecNode
		filter="(buckminster.source=true)"
		materializer="workspace"/>
</mspec:mspec>

This file specifies several things:

  • which query file to use, the cquery file is explained below, it defines which root feature project to download as a starting point for the build.
  • how to materialize: as a default materialize to the target platform (materializer="p2"), only if source is "true" materialize to the workspace

As noted above the mspec file defines which cquery file to use, the content of the cquery file:

<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="teneo.rmap">
    <cq:rootRequest name="org.eclipse.emf.teneo.site" componentType="eclipse.feature"/>
    <cq:advisorNode namePattern="org\.eclipse\.emf\.teneo.*\.source" skipComponent="true"/>
</cq:componentQuery>
  • The cquery file defines which rmap file to use, the rmap file specifies the download locations (update sites) and cvs locations to download dependencies.
  • Buckminster needs a root (feature) project to use as a starting point. This is defined by the rootRequest tag.
  • The advisorNode tag is used to tell Buckminster to ignore dependencies (defined in feature.xml files) which have a pattern starting with org.eclipse.emf.teneo and ending with source. This is needed because the SDK feature projects contain these dependencies. However, these *.source plugins/features do not exist in cvs but are created by Buckminster/Eclipse during the build. So they should be ignored when downloading dependencies. See here for a thread on the Buckminster newsgroup on this.

Test folder content

The test folder content is very similar to the build folder content:

  • teneo.cquery: this files defines the root feature project to retrieve from cvs, the content of the root feature project in the feature.xml (and an additional dependency definition file, a cspex file) specifies which development projects to download.
  • teneo.mspec: this file basically specifies that dependencies which are source (source="true") should be materialized in the workspace, while the other dependencies are materialized in the target platform.
  • teneo.rmap: defines where and how development projects and dependencies can be downloaded. This file for example lists cvs locations and several update sites. One of the update sites to download dependencies from is the update site created by the build step.
  • teneo_build.xml: the ant file which executes the build step by calling Buckminster to create the workspace and target platform, download the development projects and dependencies, build the plugins and create the update site with the resulting features/plugins.
  • convert-test-results.xsl: a xslt file which converts the test results created by Buckminster/Eclipse to a format which can be read correctly by Hudson.

buckminster.spex in PDE Feature projects

Hudson Configuration

Testing

  • Launch setup
  • xslt script execution to process test results

Code Coverage

  • change junit to emma in buckminster call
  • specify xml parameter
  • set hudson configuration, note value is the xml file not a pattern
  • code coverage works on all the projects in the workspace, so for the current setup of Teneo it makes less sense because Teneo separates the build and test steps in separate workspaces

Solving some warnings

  • [java] WARN: CVS file org.eclipse.emf/org.eclipse.emf.cdo/hibernate/org.eclipse.emf.cdo.tests.hibernate/src/org/eclipse/emf/cdo/tests/hibernate/external_model1_4.persistence.xml either contains invalid line endings on the server (CR/LF instead of just LF) or is a binary file that is not marked as -kb.
  • [java] Warning: file /opt/users/hudsonbuild/.hudson/jobs/emf-teneo-nightly/workspace/build/build/workspace/plugins/org.eclipse.emf.teneo.annotations/META-INF/MANIFEST.MF,line 9: The JRE container on the classpath is not a perfect match to the 'J2SE-1.5' execution environment

[append code snippet from build file]

Links

For the Teneo setup some scripts have been re-used from the Buckminster build itself. The Buckminster releng/build project can be found here in svn.

I good tutorial on using Buckminster and Hudson to build RCP can be found here.

Back to the top