Difference between revisions of "Orion/Releng Builds"

From Eclipsepedia

Jump to: navigation, search
(Adding a new client bundle to the JS build)
(Building stand-alone features)
Line 68: Line 68:
=== Building stand-alone features ===
=== Building stand-alone features ===
Right now stand-alone features (such as the <tt>built-editor.min.js</tt>, the standalone Orion editor) are still handled separately by the build script. You must add logic to <tt>orion.mini.xml</tt> that invokes <tt>r.js</tt>, passing the build file for your feature. Use the standalone editor build as a reference.
Orion pages and plugins are handled by a multi-module invocation of r.js that minifies all of them in one shot. However, stand-alone features need different build config options, which means r.js must be invoked separately for each of these features.
To build a stand-alone feature, you must add logic to the <tt>requirejs</tt> task in the Ant script <tt>orion.mini.xml</tt>. You must invoke <tt>r.js</tt>, passing the build file for your feature. Here is an example showing how a file used for the standalone Orion editor is minified:
<source lang="xml" enclose="div" line="line" highlight="6">
<!-- Clean widget output directory -->
<target name="widgetClean" depends="widget.check" if="widget.run">
    <delete dir="${widgetOutDirectory}" />
    <mkdir dir="${widgetOutDirectory}" />
    <copy file="${builder}/scripts/editor.build-js.js" tofile="${staging}/editor.build-js.js" overwrite="true"/>
    <!-- .. Additional copy tasks omitted -->
<target name="requirejs" depends="checkDirs, cleanStaging, cleanOptimized">
    <!-- much logic omitted -->
    <!-- build standalone editor (unless skipEditor is set) -->
    <antcall target="editorOptimize">
        <param name="staging" value="${staging}" />
        <param name="out" value="${widgetOutDirectory}" />
<target name="editorOptimize" unless="skipEditor">
            staging="${staging}" />
        <!-- Additional calls to r.js can be placed here -->
= PDE build =
= PDE build =

Revision as of 16:39, 27 March 2014

Orion currently has 3 kinds of builds:

  • The PDE build uses infrastructure from Eclipse platform releng. As of March 2014, the PDE build is deprecated, and will be removed shortly. See bug 414425 for info.
  • The Tycho build is based on Tycho and Maven. Compared to the PDE build, the Tycho build is easier for downstream products to consume, and easier to reproduce in other environments.
  • The Node build is a small script that runs on Node.js. It produces an Orion server that runs under Node.js.


How to contribute

This section applies to all builds.

Contributing changes to the build

Contributing to a build is easy: just commit your code to the master stream of the Orion repository and it will be included in the next integration build.

Setting up your workspace

  1. See Orion/Getting the source for instructions on how to set up your workspace to work with Orion source code.
  2. (If you want to try running the PDE build locally): Import 'org.eclipse.orion.releng' into your workspace. It is located in the org.eclipse.orion.server.git repository under the releng folder.

Contributing your changes

Once you have made code changes, you can contribute them to the Integration build as follows:

  • Commit your changes to the local repository and push them to the master
git commit -a -m "fix lots of bugs"                   #commits all changes
git push                                              #push local commits to the master


The Orion build performs minification of client pages and plugins. This section has instructions on how to add new content for minification in the build.

Adding a new module to the client build

To add a new page or plugin to the Orion client UI, you must update this file: releng/org.eclipse.orion.client.releng/builder/scripts/orion.build.js Find the property modules, and add an entry for your page/plugin.

Every entry needs a name property giving the module's web path, and a bundle property giving the source bundle that the minified module will be copied back into. For example, here is the Git Commit page's entry:

    { name: "git/git-commit",
      bundle: "${orionClient}/bundles/org.eclipse.orion.client.git"

^ Ant-style property replacement is supported in the bundle field, but ${orionClient} is the only property you should use there. The PDE, Tycho, and Node builds all share orion.build.js, so your new module will be correctly built everywhere.

How optimized modules are generated

For every module {name} listed in the modules section:

  • The build outputs an optimized file, named built-{name}.js, with AMD dependencies inlined and code minified.
  • The build also determines the caller. The caller is whatever HTML page or script is responsible for calling require(["{name}.js"]) or <script data-main="{name}" ... /> to actually load the module.
  • The caller is modified such that the optimized file built-{name}.js is loaded rather than the original {name}.js.

For a module {name}.js, the build by convention looks for a caller named {name}.html. If the caller is not named according to convention, it must be specified in the module entry using a caller property, so that the build can locate it. As of this writing, the caller must live in the same folder as the module.

For example, this is how the JS tools web worker (which is loaded by another script file named javascriptWorkerBootstrap.js, not an HTML file) is optimized:

    { name: "javascript/plugins/javascriptPluginWorker",
      bundle: "${orionClient}/bundles/org.eclipse.orion.client.javascript",
      caller: "javascriptWorkerBootstrap.js"

^ Here, the build will modify the caller javascriptWorkerBootstrap.js such that it loads the optimized module "built-javascriptPluginWorker.js" instead of "javascriptPluginWorker.js".

Adding a new client bundle to the JS build

  1. Add an entry to the bundles section of <tt>orion.build.js.
  2. If your bundle contains any JSDocs, add an entry to the jsdocs section of orion.build.js.
  3. For each of the bundle's modules you want to minify, add an entry to the modules section of orion.build.js. (see previous section).
  4. Add a pom.xml file in the root folder of your bundle.
  5. Add an entry to the <modules> section of the client repo's top-level pom.xml. This ensures that the Maven builder finds your bundle.

Building stand-alone features

Orion pages and plugins are handled by a multi-module invocation of r.js that minifies all of them in one shot. However, stand-alone features need different build config options, which means r.js must be invoked separately for each of these features.

To build a stand-alone feature, you must add logic to the requirejs task in the Ant script orion.mini.xml. You must invoke r.js, passing the build file for your feature. Here is an example showing how a file used for the standalone Orion editor is minified:

  1.  <!-- Clean widget output directory -->
  2.  <target name="widgetClean" depends="widget.check" if="widget.run">
  3.     <delete dir="${widgetOutDirectory}" />
  4.     <mkdir dir="${widgetOutDirectory}" />
  6.     <copy file="${builder}/scripts/editor.build-js.js" tofile="${staging}/editor.build-js.js" overwrite="true"/>
  7.     <!-- .. Additional copy tasks omitted -->
  8.  </target>
  10.  <target name="requirejs" depends="checkDirs, cleanStaging, cleanOptimized">
  11.      <!-- much logic omitted -->
  13.      <!-- build standalone editor (unless skipEditor is set) -->
  14.      <antcall target="editorOptimize">
  15.          <param name="staging" value="${staging}" />
  16.          <param name="out" value="${widgetOutDirectory}" />
  17.      </antcall>
  19.  <target name="editorOptimize" unless="skipEditor">
  20.      <sequential>
  21.          <widgetOptimizeFile
  22.             out="${out}/built-editor-amd.min.js"
  23.             buildfile="${staging}/editor.build-js.js"
  24.             optimize="closure"
  25.             staging="${staging}" />
  27.          <!-- Additional calls to r.js can be placed here -->
  28.      </sequential>
  29.  </target>

PDE build

The PDE builds follow many of the same practices used by the Eclipse Platform Project. The builds are based on PDE/Build and run automatically on build.eclipse.org via cron job.

There is only one kind of build. Integration builds run daily from tagged repository versions. Map files are used to specify where a project is located in the repository and what version of that project to use in the build. The map files are located in the org.eclipse.orion.releng project. The builder itself performs the tagging so all the latest changes in "master" stream are picked up by every build. No manual tagging by committers is required.

How the build works


The build is kicked off by a "bootstrap" script that is not under version control. The only purpose of this script is to fetch the real build script and invoke the build. The script is located at:


The build type is passed as an argument. -I means integration build (with tagging). -N is equivalent but does not perform any tagging (in both cases the contents of master are used to perform the build).

You can also pass an -email argument to cause the build notifications to only be sent to an individual. This is useful for test builds:

bootstrap.orion.sh -I -email bob_mackenzie@example.com

The bootstrap script is typically invoked by a cron entry in the e4Build user account. For example here is a cron entry to start a build every Wednesday at 8:55am:

55 08 * * Wed /opt/buildhomes/e4Build/bootstrap.orion.sh -I

Invoke "crontab -e" to edit the cron entries (using vi), or "crontab -l" to simply list them.


The bootstrap fetches and invokes the main build script, called masterBuild.sh. This script is under version control, so you need to push any changes to this script into the master branch for them to take effect. If you scroll to the very bottom of this script you can see the steps it performs:

tagRepositories # perform auto-tagging of the Orion git repositories
updateRelengProject # fetch latest from Orion releng project (our scripts and maps)
updateBaseBuilder # fetch the PDE builder to use in the build
setProperties # various initialization steps
runBuild # invoke the PDE build
runTests # invoke client and server tests
publish # send results email

build log

The results of the build, including any failures from any step, can found found in the build log at:


This is the first place to check when the build fails. It is typically very long so it is often helpful to use tail to start near the end. This exact same log gets copied into the download directory so it is available for reference for any build we produce. For example:


Repository cache

For performance reasons, PDE build maintains a cache of fetched prerequisite bundles. This lives at /shared/eclipse/e4/orion/target/transformedRepos on the build machine. The build will pick up the newest version of each required bundle from that location. If you ever need to force the build to go back to an *older* version of a bundle, this cache needs to be cleared. Simply delete the transformedRepos directory prior to the build.

Deploying builds to orion.eclipse.org or orionhub.org

Builds are deployed using the script deploy.sh. This script should be copied to your home directory and run from there. Your home directory will contain a symlink to the downloads directory, so you can perform a deploy directly from the downloads area. The script takes a single argument which is the location of the zip containing the download. For our servers we want the Linux 64-bit build. It is useful to log the output of this script so it can be reviewed later. Example:

./deploy.sh -archive downloads/orion/drops/I201201102230/eclipse-orion-I20120110-2230-linux.gtk.x86_64.zip >> deploy.log

This script simply copies the build onto the deployment server, and invokes an upgrade script on that server - upgrade.sh. The upgrade script shuts down the old server, moves it, unzips and configures the new build, and finally starts it.

The entire deploy/upgrade process takes about 5 seconds when it runs smoothly. Occasionally there will be a communication error copying the new build onto the target machine. In this case simply re-running the script usually succeeds.

Server configuration

The server configuration file on orion.eclipse.org and orionhub.org is found in ~admin/current/orion.conf. This file gets copied into the server by the deployment script. So, any change made to this configuration file will take effect only on the next deployment.

Tycho build

The Tycho builds use the Eclipse Tycho infrastructure, centered around Maven. Orion committers can managed Tycho builds using the Orion Hudson instance. The build comprises 2 jobs:

Builds Orion's client-side JavaScript code and performs JS minification.
Downstream project of orion-client. Builds Orion's Java server and runs the tests.

As of this writing, Tycho builds only run Orion's Java server tests; Orion's JavaScript tests are not yet supported.

Node build

The Node build produces a Node.js-based Orion server with minified client-side JavaScript code.

As of this writing, the Node build does not happen automatically. Members of the Orion dev team run the build manually - to prepare Orion for publishing to the npm repository. Builds are published on a regular basis, at least once per milestone.

Running the Node build

Run the shell script make-publish.sh from the modules/orionode/build/ directory:

$ cd modules/orionode/build
$ ./make-publish.sh /path/to/tempDir

The script takes a single argument, which is the output directory where the minified server will be generated. Everything in tempDir gets deleted, so do not publish to an important directory.