Jump to: navigation, search

Orion/Releng Setup Details

This page outlines the details of the Orion automated build.

General Overview

The Orion Releng build is a relatively basic PDE/Build Product Build using p2. The releng setup is contained in the releng/org.eclipse.orion.releng project in the orion server git repository.

The build runs as the "e4Build" user on build.eclipse.org

Starting the build

A scheduled build is launched via cron job. The crontab entries look something like:

0 02 * * 2         /opt/buildhomes/e4Build/bootstrap.orion.sh -I
0 20 * * 0,1,2,3,4 /opt/buildhomes/e4Build/bootstrap.orion.sh -N

The first is an integration build that runs at 2:00 am on Tuesday morning. The second is a nightly build that runs at 8:00pm Sunday through Thursday.

bootstrap.orion.sh is a copy of org.eclipse.orion.releng/builder/scripts/bootstrap.sh that has been modified to define an environment variable for running the hudson tests.

bootstrap.sh

The bootstrap script performs the following steps:

  1. Change directory to /shared/eclipse/e4/orion
  2. Get the master shell script and copy it to /shared/eclipse/e4/orion/masterBuild.sh
  3. define the hudson security token environment variable
  4. execute the masterBuild.sh shell script and pipe the output to /shared/eclipse/e4/orion/logs/current.log

masterBuild.sh

The master shell script performs the following steps:

  1. Tag the git repositories and update the map files
  2. Get the org.eclipse.orion.releng project from git (includes the map file changes from step #1)
  3. Get the org.eclipse.releng.basebuilder project from cvs
  4. Set JDK specific classpaths to compile against (J2SE-1.4, J2SE-1.5, JavaSE-1.6)
  5. Launch the Product Build
  6. Build and launch the tests
  7. Publish the results

Important Locations & Variable

  • writableBuildRoot = /shared/eclipse/e4/orion , this is the root of everything (shell scripts)
  • supportDir = /shared/eclipse/e4/build/e4 , where the releng.basebuilder is kept (shell scripts)
  • buildDirectory = /shared/eclipse/e4/orion/<build-id> (eg I20110601-2200), the working directory for each build (ant)
  • base = /shared/eclipse/e4/orion, same as writableBuildRoot, used in the ant scripts
  • ${base}/jsdoc-toolkit, jsdoc-toolkit for generating js docs
  • ${base}/requirejs, require js for optimizing javascript and css
  • ${base}/target, contains p2 repositories to hold the results, also the transformedRepos used during the build
  • ${base}/tests, contains the test results
  • ${base}/tests/JsTestDriver.jar, jstestdriver for running javascript tests

Orion Product Build

The build is a product build based on the org.eclipse.orion.releng/builder/orion.product file. This file specifies:

  • The list of features to include in the product
  • Configuration start levels and autostart=true on plugins that need it
  • <property/> elements specifying properties to go into the config.ini
  • Launcher and vm arguments to go into the orion.ini file. Including some mac specific arguments.

PDE/Build uses the .product file to run the build. It will call into the customTargets.xml file at specific times during the build.

customTargets.xml

Targets are invoked in the following order:

getMapFiles

This target is called by PDE/Build to get the map files that specify the locations of all the code to retrieve from CVS and Git. The target copies the map files out of the org.eclipse.orion.releng project that was retrieved by masterBuild.sh

preProcessRepos

In this target, we mirror the launchers (org.eclipse.equinox.executable) from the eclipse platform repository (http://download.eclipse.org/eclipse/updates/3.7-I-builds) into a local ${repoBaseLocation}(=${base}/target/repoBase)

PDE/Build will then run a "transformRepos" step which will run p2.repo2runnable on ${repoBaseLocation}. The results of this will be under ${transformedRepoLocation} (=${base}/target/transformedRepos}

postFetch

After processing the repositories, PDE/Build will then fetch all the source files as specified in the map files. We take things from a combination of Git, CVS and p2 repositories. Once this is finished, the postFetch target is called.

postFetch does the following thins:

  • For nightly N builds, we fetch cvs=HEAD and git=origin/master, we update the versions to be used for the plugin qualifiers to be the build label.
  • Run jsdoc-toolkit to generate the javascript docs into org.eclipse.orion.doc.isv/jsdoc
  • Run requirejs optimization

postBuild

The build then compiles everything, generates p2 metadata and platform specific archives. Once this is done, the postBuild step is called. This step mirrors the resulting metadata to a location which will be made publicly available. It also copies an index.html and the logs files for the download page.

Details for jsdocs

jsdoctoolkit is installed at /shared/eclipse/e4/orion/jsdoc-toolkit (which is a softlink to /shared/common/jsdoc-toolkit). It is invoked from the <jsdocs> target in customTargets.xml. The command line used is

 app/run.js -s -v -a -r=3 -t=templates/jsdoc -d <org.eclipse.orion.doc.isv>/jsdoc  <inputs>

The <inputs> are currently

 org.eclipse.orion.client.core/static/js
 org.eclipse.orion.client.editor/web/js
 org.eclipse.orion.client.git/static/js
 org.eclipse.orion.client.users.ui/static/profile/js

Details for requirejs optimization

requirejs is installed at /shared/eclipse/e4/orion/requirejs (which is a softlink to /shared/common/requirejs-0.24.0). It is invoked from the <requirejs> target in customTargets.xml.

Before calling requirejs, we first create a folder ${buildDirectory}/optimization and

  • extract org.dojotoolkit_*.jar into optimization/org.dojotoolkit
  • copy org.eclipse.orion.client.core/static into optimization/
  • copy org.eclipse.orion.client.core/static/dojox/ into optimization/org.dojotoolkit/dojox

We then invoke requirejs (using the <optimize> macro) on the following to generate a corresponding "built-*.js" file.

 edit/edit.js
 navigate/table.js
 navigate/tree.js
 compare/compare.js
 search/search.js
 sites/sites.js
 sites/site.js

In each corresponding .html file, we replace "require(["<name>"]);" with "require(["built-<name>"]);" Once this is done, we invoke requirejs once more on the entire optimization/ folder to optimize all the *.css files, which we then copy back over to org.eclipse.orion.client.core.