Difference between revisions of "JWT Automatic Build"

From Eclipsepedia

Jump to: navigation, search
(Creating the base with from repositories (recommended))
(JWT build minimal Eclipse base)
(7 intermediate revisions by 2 users not shown)
Line 61: Line 61:
 
P2 is the recommended way to install feature since it checks dependencies and avoids to forget some bundles that are necessary at runtime for testing.
 
P2 is the recommended way to install feature since it checks dependencies and avoids to forget some bundles that are necessary at runtime for testing.
  
* Download and expand a platform that is OK with you build host architecture => ''~/downloads/eclipse/downloads/[buildId]/eclipse-platform-[version]-linux-gtk-ppc.tar.gz''
+
* Download a platform that is OK with you build host architecture => ''~/downloads/eclipse/downloads/[buildId]/eclipse-SDK-[version]-linux-gtk-ppc.tar.gz'' from [http://download.eclipse.org/eclipse/downloads/ http://download.eclipse.org/eclipse/downloads/]
 +
* Transfer to the /base directory on build.eclipse.org (e.g. with SCP) and expand
 
* Go into the new eclipse folder, then type
 
* Go into the new eclipse folder, then type
**  ''export LAUNCHER="java -jar $(ls plugins/org.eclipse.equinox.launcher_*.jar) -application org.eclipse.equinox.p2.director.app.application"''
+
**  ''export LAUNCHER="java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.equinox.p2.director"''
** ''$LAUNCHER -metadataRepository http://download.eclipse.org/eclipse/updates/3.4 -artifactRepository http://download.eclipse.org/eclipse/updates/3.4 -installIU org.eclipse.pde.feature.group --version 3.4''
+
** ''$LAUNCHER -metadataRepository http://download.eclipse.org/releases/galileo -artifactRepository http://download.eclipse.org/releases/galileo -installIU org.eclipse.pde.feature.group -installIU org.eclipse.jdt.feature.group -installIU org.eclipse.emf.feature.group -installIU org.eclipse.gef -installIU org.eclipse.m2m.atl.feature.group'' (dependency list to be updated following JWT deps)
** ''$LAUNCHER -metadataRepository http://download.eclipse.org/eclipse/updates/3.4 -artifactRepository http://download.eclipse.org/eclipse/updates/3.4 -installIU org.eclipse.jdt.feature.group --version 3.4''
+
** ''$LAUNCHER -metadataRepository http://download.eclipse.org/modeling/emf/updates/releases/ -artifactRepository http://download.eclipse.org/modeling/emf/updates/releases/ -installIU org.eclipse.emf.feature.group --version 2.4.1''
+
** ''$LAUNCHER -metadataRepository http://download.eclipse.org/tools/gef/updates/releases/ -artifactRepository http://download.eclipse.org/tools/gef/updates/releases/ -installIU org.eclipse.gef --version 3.4.1''
+
** ''$LAUNCHER -metadataRepository http://download.eclipse.org/modeling/m2m/updates/releases/ -artifactRepository http://download.eclipse.org/modeling/m2m/updates/releases/ -installIU org.eclipse.m2m.atl.feature.group --version 2.0.2''
+
  
 
In case you want to install a feature from a repository that is not p2-ready, you can use
 
In case you want to install a feature from a repository that is not p2-ready, you can use
* ''java -jar $(ls plugins/org.eclipse.equinox.launcher_*.jar) -application org.eclipse.update.core.standaloneUpdate -command install -featureId [FEATURE_ID] -from [SITE_URL] -version [VERSION]''
+
* ''java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.update.core.standaloneUpdate -command install -featureId [FEATURE_ID] -from [SITE_URL] -version [VERSION]''
  
 
==== Creating the base using drops (not fair) ====
 
==== Creating the base using drops (not fair) ====
Line 133: Line 130:
 
==== Start a build ====
 
==== Start a build ====
  
Anyone who has access to build.eclipse.org can now start a build of cvs HEAD by typing from JWT home folder ''./build.sh [buildType]'' where build type is I (default), S or R, respectively for integration, stable or release build.
+
Anyone who has access to build.eclipse.org can now start a build of cvs HEAD by browsing to JWT home folder (''/shared/technology/jwt''), going into the ''builder'' folder and typing ''./build.sh [buildType]'' where build type is I (default), S or R, respectively for integration, stable or release build.
  
 
This script will start the whole build process and update the update-site for the specified build type.
 
This script will start the whole build process and update the update-site for the specified build type.
 +
 +
Follow your build on [https://build.eclipse.org/hudson/view/JWT/ Hudson].
  
 
=== Start a build on your local machine ===
 
=== Start a build on your local machine ===
Line 171: Line 170:
 
* Add those plugins CVS path to the builder map file (currently releng/builder/maps/jwt.map)
 
* Add those plugins CVS path to the builder map file (currently releng/builder/maps/jwt.map)
 
* If your new plugin depends on other plugins (eg EMFT or M2M projects) that are not included in the build base, add its dependencies to the build base
 
* If your new plugin depends on other plugins (eg EMFT or M2M projects) that are not included in the build base, add its dependencies to the build base
* Add the test plugins as dependencies to the top-level test plugin (currently jwt-we-tests)
+
* Add the test plugins as dependencies to the top-level test plugin (currently jwt-tests-plugin)
** Add them as dependencies to the plugin.xml
+
** Add them as dependencies to the MANIFEST.MF
 
** Add them into the AllTests testSuite of the plugin
 
** Add them into the AllTests testSuite of the plugin
** Add them to the top-level test feature (so that they will be fetched from CVS at build time)
+
** Add them to the top-level test feature (so that they will be fetched from CVS at build time, currently jwt-tests-feature, edit the feature.xml)
 
** Add those plugins CVS path to the builder map file (currently releng/maps/jwt.map)
 
** Add those plugins CVS path to the builder map file (currently releng/maps/jwt.map)

Revision as of 10:36, 17 December 2009

Contents

Automatic build of JWT Features and Plugins

Helpful resources

What we did

Before going further, it is necessary that you read and understand links pointed in the "Helpful Resources" section, since they explain how everything works. Next notes are just the steps we followed to get everything working, the don't explain how it works.

Our strategy was to copy ("monkey see, monkey do") an existing and working builder and to adapt it to our needs. The "development process" of our build system was a kind of "Error Driven Development", that infinitely runs the following build loop:

  • Try
  • Check the errors
  • Fix them

Get and adapt a builder

The builder is the directory that contains build informations (ant files and build.properties) and scripts to start the build.

  • We copied the ide4edu.releng project (see link in the "Helpful Resources" section)
  • We externalized all ide4edu specific contents (such as plugins names and paths) from ant script to properties in the build.properties file, so that all project-specific data can be found in the build.properties.
  • We adapted all ide4edu specific value in the build.properties to replace it by JWT specific values
  • We created the map file with JWT plugins and features CVS directories
  • We created a light and sufficient Eclipse base for building
  • We customized the ant scripts to add JWT-specific behavior (such as the fact to automatically choose an update-site according to the build type.

Handle automated testing

Another main point was to have the ability to run unit test on the just build JWT plugins, to avoid silent regressions. Our steps to do so were:

  • Copied an example tester (we used GEF and Platform ones as examples)
  • Same strategy as for the builder: externalize strings from the test.xml to the builder build.properties file, and then adapt values to JWT
  • Added the automated fetching of the test feature in the customTargets.xml
  • Added test plugins and features to the map file
  • In the master test plugin, copied a test.xml from another test plugin (not a test.xml from a tester)
  • In the tester's test.xml, adapt the creation of the runtime we'll use for testing
    • The test runtime is created dynamically as a copy of the build base
    • And we add the JWT plugins and JWT tests plugins in it (expanding zips into the test runtime directory)
  • Update the createUpdateSite.xml so that it copies test results into the update site

P2

We encountered a problem with p2 file generation: Indeed, setting the generate.p2.metadata to true made metadate generated also for the test feature, whereas we do not expect to deliver them.

Then the solution was to set generate.p2.metadata to false, and then to run the p2 generator on the generated update site. The p2.generate ant task does not seem to support the site attribute, thus we simply used an Ant exec task calling the eclipse p2 generator application as we would have done with command-line.

What can still be done

  • Try to use a build/continuous integration server
  • Try to have nightly build
  • Merge builder and tester: Move the test.xml from tester to builder, test and replace references to tester by builder.

What we have

JWT build minimal Eclipse base

In order for the build and automated testing to be working as fast as possible, it is necessary to maintain a build base (in /shared/technology/jwt/base on build.eclipse.org), that contains only the projects that are necessary to build and test JWT, and to ensure that these project are built for for the build server you are using.

Creating the base with from repositories (recommended)

P2 is the recommended way to install feature since it checks dependencies and avoids to forget some bundles that are necessary at runtime for testing.

  • Download a platform that is OK with you build host architecture => ~/downloads/eclipse/downloads/[buildId]/eclipse-SDK-[version]-linux-gtk-ppc.tar.gz from http://download.eclipse.org/eclipse/downloads/
  • Transfer to the /base directory on build.eclipse.org (e.g. with SCP) and expand
  • Go into the new eclipse folder, then type
    • export LAUNCHER="java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.equinox.p2.director"
    • $LAUNCHER -metadataRepository http://download.eclipse.org/releases/galileo -artifactRepository http://download.eclipse.org/releases/galileo -installIU org.eclipse.pde.feature.group -installIU org.eclipse.jdt.feature.group -installIU org.eclipse.emf.feature.group -installIU org.eclipse.gef -installIU org.eclipse.m2m.atl.feature.group (dependency list to be updated following JWT deps)

In case you want to install a feature from a repository that is not p2-ready, you can use

  • java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.update.core.standaloneUpdate -command install -featureId [FEATURE_ID] -from [SITE_URL] -version [VERSION]

Creating the base using drops (not fair)

To create a build base, you'll need to expand the following project in the directory you want to use ase the eclipse base:

  • Eclipse platform linux/ppc/gtk => ~/downloads/eclipse/downloads/[buildId]/eclipse-platform-[version]-linux-gtk-ppc.tar.gz
  • Eclipse PDE => ~/downloads/eclipse/downloads/[buildId]/eclipse-PDE-[version].zip
  • Eclipse JDT => ~/downloads/eclipse/downloads/[buildId]/eclipse-JDT-[version].zip
  • EMF Runtime => ~/downloads/modeling/emf/emf/downloads/drops/[version]/[buildId]/emf-runtime-[buildId].zip
  • GEF Runtime => ~/downloads/tools/gef/downloads/drops/[version]/[buildId]/GEF-runtime-[buildId].zip
  • ATL Runtime (for transformations) => ~/downloads/modeling/m2m/atl/downloads/drops/[version]/[buildId]/m2m-atl-runtime-[version].zip

Choosing the version of each project:

  • Currently, JWT build on Eclipse server uses the latest version that contains a Maintenance or Release build (ie buildId starting with R or M)
  • However, we could discuss about trying to build usng stable or milestone builds, to ensure the compatibility with the latest versions of projects. But this would require much more work until we succeed to automate the creation of the base with the latest drops...

Check an existing base

There are 2 ways to check that an eclipse installation matches your machine platform:

  • Try to run the eclipse launcher program (./eclipse). If eclipse starts, it is OK
  • From the eclipse directory, try command ls plugins/ | grep org.eclipse.swt, then you should get a list of plugins, and one of them specifies a os/ws/arch 3-uplet in its name. It must be match your server platform.

If you don't have the same eclipse target and server platforms, then you might be able to build, but testing will be impossible for you.

How to use it

Update sites organisation

First, as it is the case in most of "one feature" projects (such as GEF), JWT now has 3 update sites:

Building on build.eclipse.org

Requirements

Since testing requires the ability to start the eclipse UI, an X server must be started and the DISPLAY environment variable must be correctly set on the build server. See EMF advices about X configuration for more details.

On build.eclipse.org, you can use the Xvfb server started by root on localhost:1.0 (DISPLAY is automatically set to this value when running build.sh)

If you there is a problem with X server, you'll have the bahavior described above:

  • In the build trace, the Java Result: 13 tells you that eclipse could not start
java-test:
     [echo] Running org.eclipse.jwt.we.tests.AllTests. Result file: /shared/technology/jwt/tester/results//org.eclipse.jwt.we.tests.AllTests.xml.
     [java] Java Result: 13
  • In the result file, you see a desperately empty test result
<?xml version="1.0" encoding="UTF-8" ?>
<testsuites />
  • In the eclipse log (tester/results/org.eclipse.jwt.we.tests.AllTests.txt), you can see a message telling you that a problems occurs with display:
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
        at org.eclipse.swt.SWT.error(SWT.java:3803)

Start a build

Anyone who has access to build.eclipse.org can now start a build of cvs HEAD by browsing to JWT home folder (/shared/technology/jwt), going into the builder folder and typing ./build.sh [buildType] where build type is I (default), S or R, respectively for integration, stable or release build.

This script will start the whole build process and update the update-site for the specified build type.

Follow your build on Hudson.

Start a build on your local machine

If you want to build locally JWT, simply checkout the org.eclipse.jwt/releng CVS module and write a script that starts the build with your local configuration (please do not modify build.sh or build.properties, that are build.eclipse.org files). See build.mistria.openwide.sh for example.

Continuous integration

TODO Note that the PDE build process supports unit testing. I've read about several Eclipse projects that used a continuous integration tool (such as cruise, continuum or hudson) to monitor their builds and test results. I'll try to choose if one of these solutions is fine for JWT.

Releasing Policy

When releasing JWT, here are the steps to follow:

  1. Tag the feature and all plugins included into the release with tag R{releaseVersion} (for example R0.5.0)
  2. Build on build.eclipse.org the feature to release (./build.sh R). The update site is automatically updated.
  3. Increment the feature version number (See Version Numbering)
  4. Increment all plugins versions that will be modified in next release)
  5. Create a buzz about this release!

Adding new plugins to the JWT release

This section describes the steps to follow to add a new plugin to JWT release and get the build working with this add-ons. We'll use an example to facilitate the explanations.

Assume that jwt-we (plugin) now depends on a new plugin (jwt.we.conf.model) that was not part of the automated build before. In such case, some things needs to be updated to build the jwt-we automatically, else you'll get a "missing bundle" error telling you that the build cannot find the jwt.we.conf.model.

  • Ensure it conforms to Eclipse.org legal and technical requirements
    • Plugin version must endswith .qualifier (see Version Numbering)
    • Message strings must be externalized and messaging must be based on NLS (See Message Bundle Conversion Tool and related docs)
    • More generally, it must conform with Eclipse requirements (See list on Galileo Simultaneous Release)
    • Ensure it conforms to legal Eclipse requirements
      • Provider must be Eclipse.org
      • Plugin must include license and about files
      • License headers are necessary on all source files
      • More generally, see [1]
  • Add those plugins to the top-level feature (currently releng/jwt-feature) with version 0.0.0 (automatically chosen at build time)
  • Add those plugins CVS path to the builder map file (currently releng/builder/maps/jwt.map)
  • If your new plugin depends on other plugins (eg EMFT or M2M projects) that are not included in the build base, add its dependencies to the build base
  • Add the test plugins as dependencies to the top-level test plugin (currently jwt-tests-plugin)
    • Add them as dependencies to the MANIFEST.MF
    • Add them into the AllTests testSuite of the plugin
    • Add them to the top-level test feature (so that they will be fetched from CVS at build time, currently jwt-tests-feature, edit the feature.xml)
    • Add those plugins CVS path to the builder map file (currently releng/maps/jwt.map)