Jump to: navigation, search

E4/Builds

Builds

In e4 (like in the other projects under the Eclipse top-level project), we produce regular builds. One important kind of build is called integration build - usually run once a week to produce a version that is good enough to for purposes of integrating with other teams' work, and for self-hosting. Integration builds do not just take whatever is in HEAD at the time of running the build - given the amount of changes produced by the committers, this is not predictable enough. Instead, each team needs to make a build submission, which basically amounts to tagging their projects at a known good state with a tag that then gets put into the team's map file. The map files are located in so-called releng projects. These projects are checked out from HEAD by the build system, followed by checking out the referenced projects using the specified tags. This means that if a team forgets to submit to the integration build, the build will just check out whatever was submitted last time.

To make a build submission, one team member would first check out (or update) all the team's projects, and run their tests locally. Then, to tag all changed projects, and updating the map file to contain those tags, they would use the releng tools (see below for installation instructions). By using the releng tools instead of tagging and updating the map file manually, the process is made less error-prone, and as an additional benefit, change log information will be generated for you at the same time. The change log is produced by searching for bugzilla ids in commit comments, and doing a lookup in the Bugzilla system to retrieve the bugs' descriptions. Sending the change log information to the mailing list makes everyone aware of what the team has worked on, and can help later if you find regressions and would like to be able to map back to what has been changed and why.

Plug-in Versioning

Our current release, e4 0.9, has all of our plugins marked as (Incubation) and with a version 0.9.0.qualifier.

Now that HEAD is open for our 1.0 release we are following the standard Version Numbering guidelines. For example, for existing bundles:

  • if you contribute bug fixes for a bundle, please increment the number to 0.9.1.qualifier
  • if you change the public API or public classes (to add new API or to refactor API for the 1.0 release), please increment the number to 0.10.0.qualifier

For new bundles, please mark them as 0.10.0.qualifier

When we graduate bundles so they are ready for Eclipse 4.0 we will vet the API and change their version number to 1.0.0.qualifier

Milestone Builds

Milestone Delivery Date
M1 10/09/2009
M2 11/20/2009
M3 01/08/2010
M4 02/19/2010
M5 04/02/2010
M6 05/21/2010

Integration Builds

E4 Integration Builds
Date Time Frequency
Thursdays 21:00 EST Weekly

e4 Downloads Page

Submitting For the Build

In General:

Checking in code: The checkin comment should at a minimum include "bug #", and preferably be of the form "bug # summary"

For the build:

Tool: Teams should use the "Eclipse Releng Tools" to update their releng project for each build. The releng tools can be installed from the update sites already included in 3.5.

Install-Releng-Tools.png

Tags: tags should be of the form v<date>, so v20081120 or v20081120-1300

Procedure:

  1. check out your team releng project from e4/releng
  2. use the psf files to check out your core plugins, tests, and examples
  3. Run Team>Release from the context menu:
    1. Select your team's releng project, like org.eclipse.e4.ui.releng.
      Release-1.png
    2. Select the projects you want to release from your map file.
      Release-2.png
    3. You can review the changed projects in this pane. Make sure 'Generate Build Notes' is selected.
      Release-3.png
    4. Copy out the build note changes so you can send an email to e4-dev@eclipse.org, with a subject that starts with [releng]. You don not have to select a build notes file to continue.
      Release-4.png
    5. Enter the release tag for you build. This would normally be the build time and date with a v in front, v20090101-1930
      Release-5.png
    6. You can review the changes to your map file here, or simply continue.
      Release-6.png
    7. Commit the changes so they will be picked up by the I-build


Versioning Plugins: Teams should try and follow the standard versioning rules posted at Version Numbering. If you need to copy over an existing eclipse project to make changes, please increment the minor number by 100. i.e. 3.5.0 becomes 3.105.0. When you need to break API, then update the major number. i.e. 3.105.0 becomes 4.0.0

Build Infrastructure

We run our builds on build.eclipse.org (a ppc linux machine) with a local userid, e4Build.

Build Scripts

The main build project is in e4/releng/org.eclipse.e4.builder. It contains the masterBuild.sh script that can be modified to run a test build or a real build (tagged and uploaded to download.eclipse.org). It also contains the PDE build directory, e4/releng/org.eclipse.e4.builder/builder/general, that builds our master feature. Our PDE build directory also contains modified code/XML to support using p2 to run our automated tests.


Machine Setup

There are a number of directories that need to be available to run a build.

  • /opt/public/common: the java installs can be appropriate for your architecture.
    apache-ant-1.7.0/         ibm-java2-ppc-50@      ibm-java-jdk-ppc-60@
    

flex_sdk_3.2.0.3794_mpl/ ibm-java2-ppc-50-old@ ibm-java-ppc-60@ ibm-java2-142@ ibm-java2-ppc64-50@

  • /shared/eclipse/e4/flex_sdk_3.2.0.3794_mpl
  • /shared/eclipse/e4/build/e4

To run the tests, we need a virtual X server on :8:

Xvfb :8 -screen 0 1280x1024x24 -auth auth.cfg
metacity --display=:8.0 --replace --sm-disable 

Where auth.cfg contains at least:

localhost

Build Requirements

The build is currently controlled from the masterBuild.sh script. There are a number of dependencies that make this a linux only build, and the machine has to be set up the same way as build.eclipse.org. This section will capture the requirements/process so that all of the needed steps are executed. See bug releng - Run a build from any platform for any discussions about this section.

Setup

In masterBuild.sh

We have a set of variables that are needed in order to make the system go.

Variable Comments
writableBuildRoot

The base directory that contains all build related files except the JREs.

projRelengBranch

The branch to check out the releng project (this contains subprojects that contain all our maps). Default is HEAD.

projRoot

The CVS root used to check out the releng project. For anonymous test builds, it is :pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse

arch

Build architecture, currently x86 or ppc.

supportDir

The directory that contains the eclipse install used to build (by default we use the eclipse basebuilder), and usually org.eclipse.e4.builder directory that we launch the build against.

builderDir

Points to the org.eclipse.e4.builder so that it can be in a different place than the supportDir.

builddate

Build date in the format YYYYMMDD, ex: 20090625

buildtime

Build time in the 24 hour clock format HHMM, ex: 1930

tagMaps

Controls if we tag the map files after we check out our releng project.

publishDir

Controls if we should upload the finished build. Currently this is the form of an SCP destination directory.

basebuilderBranch

The tag that determines which base builder we use for the build.

eclipseIBuild

The ID of the eclipse build we use as a target platform. With the changes to use a zipped repo for the eclipse SDK, this ID is only used to generate URLs on the download page. It should potentially be used to collect the correct p2 repo for the eclipse/equinox SDK.

javaHome

The java that is used to launch JVMs from the masterBuild.sh script.

buildTimestamp

The date and time together: 20090625-1930

buildDir

The directory to build in, by default $writableBuildRoot/build/e4/downloads/drops/4.0.0.

targetDir

A directory that contains "target platforms" to work against. It contains a zip download cache, any eclipse target platforms, an untransformed repo directory (a directory with zipped repos to use in the build), and the runnable repo generated by the build.

buildDirectory

The directory for the current build, $buildDir/I$buildTimestamp by default.

testDir

The directory used for running the automated tests.

buildResults

The directory for the build results. This will contain the index.html, compile logs, p2 repo, platform zips, etc. Defaults to $buildDirectory/I$buildTimestamp. So for example, .../4.0.0/I20090625-1930/I20090625-1930.

relengBaseBuilderDir

The root directory for the eclipse install that will run the build. By default we use the eclipse basebuilder.

writableBuildRoot


In general/build.properties

We also code/re-generate a number of properties in the PDE builder.

Variable Comments
buildType

Set to I, but not currently used to do more than create the matching build results directory and correct build "tag".

eclipseBaseURL

The URL of the eclipse SDK base used for this build. This was used to download targets and automated test frameworks, but is now only used to generate links in the download page.

emfBaseURL

Now only used to generate links in the download page.

gefBaseURL

Now only used to generate links in the download page.

wstBaseURL

Now only used to generate links in the download page.

JAVA60_HOME

A base java 6 install, set to /opt/public/common/ibm-java-jdk-ppc-60 to match build.eclipse.org.

JAVA50_64_HOME

A base java 5 64-bit install, set to /opt/public/common/ibm-java2-ppc64-50 to match build.eclipse.org.

JAVA50_HOME

A base java 5 install, set to /opt/public/common/ibm-java2-ppc-50 to match build.eclipse.org.

JAVA14_HOME

A base java 1.4.2 install, set to /opt/public/common/ibm-java2-142 to match build.eclipse.org.

JavaSE-1.6

Definition of the jars that make up the JSE 1.6 BREE.

J2SE-1.5

Definition of the jars that make up the J2SE 1.5 BREE.

J2SE-1.4

Definition of the jars that make up the J2SE 1.4 BREE.

flex.sdk

A property to the base directory for a flex SDK, needed to build the actionScript supporting tools.

repoBaseLocation

This property points to the directory of zipped repos to be used by the built. It is processed through the repo2runnable ant task.

transformedRepoLocation

The runnable repo produced by repo2runnable. Defaults to ${repoBaseLocation}-trans

Manual Setup

A lot of the directory structures must be set up before you can run a build. Perhaps the directory structure needs to be flattened so that it can be more easily created.

The basebuilder (eclipse install used to run the build) and org.eclipse.e4.builder project have to be on disk somewhere. A build updates them to the latest required versions and links to them using symlinks so that they can be used to run the build, drive the build, and provide the testing instructions.

The runnable repo is used for the build as well as to produce the platform zips and install supporting bundles for the automated tests. The untransformed repo location must be created, and populated with repos downloaded by hand. Currently, the eclipse SDK repo must be hand rolled as well. It would be nice to have the runnable repo populated by downloading the needed zipped repos.

The other possibility is to use the repo2runnable task directly and not through repoBaseLocation. Then multiple repo locations can be specified either as local files or URLs (to zipped repos or p2 repos).

There must be at least 3 JREs/JDKs installed on a machine used to build, 1.4.2, 1.5, and 1.6. A bundles BREE instructs PDE build which set of libraries to use during compile.

Steps

In masterBuild.sh

The steps we go through to run a build are controlled by the masterBuild.sh script.

Step Description
realBuildProperties

Set all of the build properties so that the build will be tagged and copied over to download.eclipse.org when complete.

testBuildProperties

Set the build properties to run a test build. Complete with tests, this is not tagged or copied to download.eclipse.org. Only one of testBuildProperties or realBuildProperties should be set.

commonProperties

Just common properties that are set for both types of builds.

updateBaseBuilder

Update the basebuilder to the correct version. Not done in certain types of test builds, like when using the basebuilder checked out into your workspace.

updateE4Builder

Get the latest of the templates and general builder. This is not done in certain types of test builds, when testing changes that are not checked in.

updateBaseBuilderInfo

Collect info on the basebuilder to run the build. This is done in such a way that the launcher jar can be used to launch other eclipse applications (like the publisher or director) as part of the PDE build.

buildMasterFeature

This runs the PDE build. It runs repo2runnable, builds all of the features and plugins in e4 and generates the e4 p2 repository, and generates platform zips.

copyCompileLogs

Script that generates an html page that links to any compile logs generated during the compile.

generateRepoHtml

Generates a default index for the repo URL ... to stop 404 complaints when people click on it in their browser.

generateSwtZip

Generates the 2 SWT action script projects, that can be imported into the workspace for use with the ActionScript tools. Checks out 2 projects, fiddles with some of the text files, and then zips them up into the results directory.

runTheTests

Copy the modified test.xml/runtests (linux) scripts to the testing directory along with one of the generated platform zips. Run the automated tests on the virtual X machine, Xvfb.

CVS

The resources, ui, and swt teams now have plugins in their team area in the e4 project. Our project is under :pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse


Resources

The resources CVS structure is:

  • e4
    • org.eclipse.e4.resources
      • bundles
      • doc
      • examples
      • features
      • tests
    • releng
      • org.eclipse.e4.resources.releng

The org.eclipse.e4.resources.releng contains the map files for building as well as Team Project Sets that should be used for checking out the projects to be worked on.

Resources currently has 4 features. 2 feature patches for the plugins (one for rcp, one for platform), a feature that contains the 2 patches, and a test feature for the tests.

Setup/Restrictions

Resources can currently update a 3.5 SDK I build and run.

UI

The UI CVS structure is:

  • e4
    • org.eclipse.e4.ui
      • bundles
      • doc
      • examples
      • features
      • tests
    • releng
      • org.eclipse.e4.ui.releng

The org.eclipse.e4.ui.releng contains the map files for building as well as Team Project Sets that should be used for checking out the projects to be worked on.

UI currently has one main feature for the modelled workbench and one feature for the demos. The build master feature, org.eclipse.e4.master, is also in UI for now.

Setup/Restrictions

UI currently requires EMF 2.4.1 or 2.5.0 in installed on 3.5 in order to run. They can be installed from the update site included with the I builds.

SWT

The SWT CVS structure is:

  • e4
    • org.eclipse.e4.swt
      • bundles
      • doc
      • examples
      • features
      • tests
    • releng
      • org.eclipse.e4.swt.releng

The org.eclipse.e4.swt.releng contains the map files for building as well as Team Project Sets that should be used for checking out the projects to be worked on.

SWT has 2 features, one for the ActionScript tools and another for the ActionScript tests.

Setup/Restrictions

To update to the ActionScript tools you must first download and install the Open Source Flex SDK.

Setup Flex environment:

  1. Download and install the Adobe Open Source Flex SDK (available from http://opensource.adobe.com). NOTE: The path where you install the Flex SDK must contain no spaces. This is due to a bug in FCSH.
  2. update your eclipse.ini file and add another line at the end, -Dflex.sdk=<your path to the installed sdk>

Then you can update to the ActionScript tools. To work on the tools themselves, you must also set the FLEX_SDK classpath variable in Window>Preferences>Java>Build Path>Classpath Variables to point to the sdk location.


To work on org.eclipse.swt, org.eclipse.swt.e4.jcl, org.eclipse.swt.e4.examples, and org.eclipse.swt.examples you need to create a Linked Resource variable, WORKSPACE, that points to your current workspace root, from Window>Preferences>General>Workspace>Linked Resources. Also if .classpath_flex exists it must be copied to the .classpath entry.

XWT

The XWT CVS structure is:

  • e4
    • org.eclipse.e4.xwt
      • bundles
      • doc
      • examples
      • features
      • tests
    • releng
      • org.eclipse.e4.xwt.releng

The org.eclipse.e4.xwt.releng contains the map files for building as well as Team Project Sets that should be used for checking out the projects to be worked on.

XWT has 6 features for different purposes:

Feature Description
org.eclipse.e4.xwt.feature The core of XWT
org.eclipse.e4.tools.feature Tools of development
org.eclipse.e4.emf.feature XWT with EMF support
org.eclipse.e4.css.feature XWT with CSS support
org.eclipse.e4.workbench.feature XWT with e4 Modeled workbench integration  
org.eclipse.e4.tests.feature The Auto tests of XWT

 

Setup/Restrictions

XWT Tools currently requires the following components on eclipse 3.4 or + in order to run:

  • EMF runtime 2.4.0 or +
  • GEF SDK 3.4.0 or +
  • WST SDK 3.0 or +