Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Orion/Releng Setup Details
This page outlines the details of the Orion automated build.
Contents
General Overview
The Orion Releng build is a relatively basic PDE/Build Product Build using p2. The releng setup is contained in the releng/org.eclipse.orion.releng project in the orion server git repository.
The build runs as the "e4Build" user on build.eclipse.org
Starting the build
A scheduled build is launched via cron job. The crontab entries look something like:
0 02 * * 2 /opt/buildhomes/e4Build/bootstrap.orion.sh -I 0 20 * * 0,1,2,3,4 /opt/buildhomes/e4Build/bootstrap.orion.sh -N
The first is an integration build that runs at 2:00 am on Tuesday morning. The second is a nightly build that runs at 8:00pm Sunday through Thursday.
bootstrap.orion.sh is a copy of org.eclipse.orion.releng/builder/scripts/bootstrap.sh that has been modified to define an environment variable for running the hudson tests.
bootstrap.sh
The bootstrap script performs the following steps:
- Change directory to /shared/eclipse/e4/orion
- Get the master shell script and copy it to /shared/eclipse/e4/orion/masterBuild.sh
- define the hudson security token environment variable
- execute the masterBuild.sh shell script and pipe the output to /shared/eclipse/e4/orion/logs/current.log
masterBuild.sh
The master shell script performs the following steps:
- Tag the git repositories and update the map files
- Get the org.eclipse.orion.releng project from git (includes the map file changes from step #1)
- Get the org.eclipse.releng.basebuilder project from cvs
- Set JDK specific classpaths to compile against (J2SE-1.4, J2SE-1.5, JavaSE-1.6)
- Launch the Product Build
- Build and launch the tests
- Publish the results
Important Locations & Variable
- writableBuildRoot = /shared/eclipse/e4/orion , this is the root of everything (shell scripts)
- supportDir = /shared/eclipse/e4/build/e4 , where the releng.basebuilder is kept (shell scripts)
- buildDirectory = /shared/eclipse/e4/orion/<build-id> (eg I20110601-2200), the working directory for each build (ant)
- base = /shared/eclipse/e4/orion, same as writableBuildRoot, used in the ant scripts
- ${base}/jsdoc-toolkit, jsdoc-toolkit for generating js docs
- ${base}/requirejs, require js for optimizing javascript and css
- ${base}/target, contains p2 repositories to hold the results, also the transformedRepos used during the build
- ${base}/tests, contains the test results
- ${base}/tests/JsTestDriver.jar, jstestdriver for running javascript tests
Orion Product Build
The build is a product build based on the org.eclipse.orion.releng/builder/orion.product file. This file specifies:
- The list of features to include in the product
- Configuration start levels and autostart=true on plugins that need it
- <property/> elements specifying properties to go into the config.ini
- Launcher and vm arguments to go into the orion.ini file. Including some mac specific arguments.
PDE/Build uses the .product file to run the build. It will call into the customTargets.xml file at specific times during the build.
customTargets.xml
Targets are invoked in the following order:
getMapFiles
This target is called by PDE/Build to get the map files that specify the locations of all the code to retrieve from CVS and Git. The target copies the map files out of the org.eclipse.orion.releng project that was retrieved by masterBuild.sh
preProcessRepos
In this target, we mirror the launchers (org.eclipse.equinox.executable) from the eclipse platform repository (http://download.eclipse.org/eclipse/updates/3.7-I-builds) into a local ${repoBaseLocation}(=${base}/target/repoBase)
PDE/Build will then run a "transformRepos" step which will run p2.repo2runnable on ${repoBaseLocation}. The results of this will be under ${transformedRepoLocation} (=${base}/target/transformedRepos}
postFetch
After processing the repositories, PDE/Build will then fetch all the source files as specified in the map files. We take things from a combination of Git, CVS and p2 repositories. Once this is finished, the postFetch target is called.
postFetch does the following thins:
- For nightly N builds, we fetch cvs=HEAD and git=origin/master, we update the versions to be used for the plugin qualifiers to be the build label.
- Run jsdoc-toolkit to generate the javascript docs into org.eclipse.orion.doc.isv/jsdoc
- Run requirejs optimization
postBuild
The build then compiles everything, generates p2 metadata and platform specific archives. Once this is done, the postBuild step is called. This step mirrors the resulting metadata to a location which will be made publicly available. It also copies an index.html and the logs files for the download page.
Details for jsdocs
jsdoctoolkit is installed at /shared/eclipse/e4/orion/jsdoc-toolkit (which is a softlink to /shared/common/jsdoc-toolkit). It is invoked from the <jsdocs> target in customTargets.xml. The command line used is
app/run.js -s -v -a -r=3 -t=templates/jsdoc -d <org.eclipse.orion.doc.isv>/jsdoc <inputs>
The <inputs> are currently
org.eclipse.orion.client.core/static/js org.eclipse.orion.client.editor/web/js org.eclipse.orion.client.git/static/js org.eclipse.orion.client.users.ui/static/profile/js
Details for requirejs optimization
requirejs is installed at /shared/eclipse/e4/orion/requirejs (which is a softlink to /shared/common/requirejs-0.24.0). It is invoked from the <requirejs> target in customTargets.xml.
Before calling requirejs, we first create a folder ${buildDirectory}/optimization and
- extract org.dojotoolkit_*.jar into optimization/org.dojotoolkit
- copy org.eclipse.orion.client.core/static into optimization/
- copy org.eclipse.orion.client.core/static/dojox/ into optimization/org.dojotoolkit/dojox
We then invoke requirejs (using the <optimize> macro) on the following to generate a corresponding "built-*.js" file.
edit/edit.js navigate/table.js navigate/tree.js compare/compare.js search/search.js sites/sites.js sites/site.js
In each corresponding .html file, we replace "require(["<name>"]);" with "require(["built-<name>"]);" Once this is done, we invoke requirejs once more on the entire optimization/ folder to optimize all the *.css files, which we then copy back over to org.eclipse.orion.client.core.