Jump to: navigation, search

STEM Releng

Revision as of 11:16, 1 August 2011 by Mattadav.us.ibm.com (Talk | contribs) (Adding a third-party (non-Eclipse) dependency from Eclipse Orbit)

STEM Release Engineering

This page details how to operate the STEM builder to create binary, platform-specific builds of STEM. This information is for advanced users and committers only.


The STEM build process is built on several foundational pieces of the Eclipse Platform. These pieces include:

  • Eclipse Products
  • Eclipse Features
  • PDE Build Process
  • P2 Provisioning

These fundamental aspects of the Eclipse Platform are orchestrated to build platform-specific binaries of STEM.

Adding New Code to the Builder

This section describes how to add new plug-ins, features, and third-party dependencies to the STEM Build process.

Important Note About Feature Cross-Dependencies

Plug-in dependencies across features are not as simple as they appear. For a feature (Feature A) containing plug-ins built from source (e.g. not binary), then any feature(s) (Feature B) it depends on must either be built completely first or the plug-ins in Feature B that Feature A depends on must also be included directly in Feature A.


  • STEM has Features org.eclipse.stem.core.feature and org.eclipse.stem.models.epidemiology.feature
  • Plug-in org.eclipse.stem.ui is in the core feature but depends on the org.eclipse.stem.diseasemodels plug-in, which is (logically) in the epidemiology feature
  • Plug-in org.eclipse.stem.diseasemodels depends on the org.eclipse.stem.core plug-in, which is in the core feature
  • The epidemiology feature cannot be built completely first because it depends on plug-ins in the core feature
  • The core feature cannot be built completely first because it depends on plug-ins in the epidemiology feature
  • Therefore, to solve this cross-dependency, the org.eclipse.stem.diseasemodels plug-in must be in both core and epidemiology features

Adding a new plug-in to an existing feature

STEM is built from Eclipse plug-ins. This section describes how to add one or more new plug-ins to the STEM build process to an existing feature. If you want to create a new feature, see the next section.

  • Determine which Feature(s) the plug-in(s) should be added to
    See note above about feature cross-dependencies to help determine which features you should add the plug-in to
    1. In Eclipse, expand the feature project the plug-in(s) should be added to and double-click feature.xml
    2. In the Feature Editor, select the Plug-ins tab
    3. On the Plug-ins and Fragments page, click the Add... button
    4. In the Plug-in Selection dialog, type the name of the plug-in you wish to add. When it shows up in the list, select it and click OK
      Repeat this step for all plug-ins you wish to add to this feature
    5. Save feature.xml (File-Save or Control-S)
    6. Commit feature.xml (Right-click Team-Commit)
  • Add the plug-in to SVN
    1. In Eclipse, select the plug-in project you wish to add.
    2. Right click on the project, highlight Team and select Share Project
    3. Follow the steps to add it to the repository.
      Note: The STEM Source Repository has a specific layout. Please make sure to place your new plug-in in an appropriate location. Plug-ins are grouped in folders with other plug-ins of similar functionality (such as disease models ). Remember this location, as it's needed for the build.map step below.
  • Add the plug-in to the build map
    1. In Eclipse, expand the org.eclipse.stem.releng project
    2. Expand the maps folder and open stem.map in the text editor
    3. Add the plug-in to the build map. The syntax is as follows:
      [PLUGIN_NAME] should be replaced with the plug-in name (Example: org.eclipse.stem.core)
      [REPOSITORY_PATH] should be replaced with the path in SVN where the plug-in was added, relative to trunk (Example: plug-ins added to /trunk/core would use core for the [REPOSITORY_PATH])
      Important Note: Plug-ins are grouped in the build map for project set generation. To ensure that a plug-in gets checked out, make sure it's grouped appropriately with similar plug-ins
    4. Save and commit build.map

Adding a new feature

If the new functionality you're adding is substantially different from existing functionality in STEM (such as a new type of modeling, like climate modeling ), then it may be appropriate to add it as a new feature. These steps will help you add a new Eclipse Feature to STEM's runtime and build process.

Important Note: These steps add a feature to the default build process (which, in turn, adds the feature to the main STEM distribution). If you do not wish for the feature to be part of the STEM distribution, only follow the first step below (Creating the Feature Project).

  • Create the Feature Project
    1. In Eclipse, select the File menu and click New
    2. In the New wizard, expand the Plug-in Development category and select Feature Project . Click Next
    3. On the New Feature page, populate the fields
      • Project Name should be fully qualified and end with .feature (Example: org.eclipse.stem.new.feature)
      • Feature ID should be the same as Project Name
      • Feature Name should be a short description of what functionality the feature provides
      • Feature Provider should be your organization's name (Example: Eclipse Foundation)
    4. Click Next
    5. On the Referenced Plug-ins and Fragments page, select the plug-ins to be part of this feature.
      Note: If the plug-ins aren't created yet, use the steps from the previous section to add them to the feature later
    6. Click Finish
    7. Add the new feature project to SVN
      Note: In the STEM Source repository, feature projects are placed in the /trunk/features folder.
  • Add the Feature to the STEM Product Feature (optional)
    1. In Eclipse, expand the org.eclipse.stem.product.feature project. Open feature.xml
    2. In the Feature Editor , select the Included Features tab
    3. On the Included Features page, click Add
    4. In the Feature Selection dialog, type in your Feature Project's name. When found, select it and click OK
    5. Save and commit feature.xml
  • Add the Feature to the build map
    1. In Eclipse, expand the org.eclipse.stem.releng project
    2. Expand the maps folder and open stem.map in the text editor
    3. Add the feature to the build map at the end of the rest of the features. The syntax is as follows:
      [FEATURE_NAME] should be replaced with the feature project name (Example: org.eclipse.stem.core.feature)
    4. Save and commit build.map

Add a new dependency

Adding a dependency from another Eclipse project

Adding a third-party (non-Eclipse) dependency from Eclipse Orbit

The easiest way to include a third-party (non-Eclipse) dependency is to select one from the Eclipse Orbit project. Orbit is a formal process by which non-Eclipse, third-party libraries are vetted by Eclipse Legal and approved for general use and redistribution by Eclipse projects. This significantly reduces the amount of work required for both committers and Eclipse Legal to get libraries approved for redistribution.

  1. Select the library that you wish to include from the latest Orbit release (See here for releases)
    Note: Keep note of the IPZilla CQ number from the release table
  2. Committers Only In the Eclipse Committer Portal, fill out the use a third-party library from Orbit form to request approval to use the library
  3. Add the library to the Target Platform
    1. In Eclipse, expand the org.eclipse.stem.releng.targetplatforms project
    2. Open the STEM-Eclipse36.target target platform
      • If the existing Orbit release repository is in the target platform , select the repository, click Edit , and choose the library from the list to include, and click Finish
      • If the existing Orbit release repository is NOT in the target platform , click Add , select Software Site , enter the Orbit Release Repository URL, select the library you wish to use, and click Finish
    3. Save the Target Platform
  4. Add the Library to the Build Map
    1. In Eclipse, expand the org.eclipse.stem.releng project
    2. Expand the maps folder and open stem.map with a text editor

Adding a non-Eclipse, third-party dependency not available in Orbit

Running the STEM Builder

The section describes how to setup and run the Eclipse PDE Builder to create a STEM binary.

Setting up your environment

  • Environment with a compatible shell (bash)
    • Windows users are recommended to run the builder from cygwin or a similar interpreter
  • Subversion command-line client (svn)
  • Java J2SE Development Kit (JDK) 6.0 (Oracle or IBM)
  • Eclipse Classic SDK with the following features installed:
    • Eclipse Modeling Framework (EMF)
    • Eclipse Business Intelligence and Reporting Tools (BIRT) Framework
    • Eclipse Zest Framework
    • Eclipse RCP Delta Pack (for Eclipse 3.7.0, see here)
    • Eclipse Releng Tools (For Eclipse 3.7.0, see here)

Checking out the source from SVN

The STEM Builder looks for projects based on the SVN Repository layout, not in the flat structure of an Eclipse workspace. Therefore, you should checkout the repository AS IS using the SVN command line client.

This svn command checks out the STEM source from the SVN trunk into the current working directory.

svn co http://dev.eclipse.org/svnroot/technology/org.eclipse.stem/trunk .

To checkout from a different branch, replace trunk with branches and include the branch name. Example that checks out the STEM_1_2_1 branch:

svn co http://dev.eclipse.org/svnroot/technology/org.eclipse.stem/branches/STEM_1_2_1 .

One advantage of checking out this way is you can quickly update the entire code base using:

svn update

Running the STEM Builder

The STEM builder is built on top of the Eclipse PDE build process. For STEM, it's three steps that are controlled by environment variables:

  1. Build the STEM Features
  2. Create and Package the STEM Product
  3. Run JUnit Tests

Each step can be enabled or disabled using appropriate environment switches

Environment Variables

The following are the environment variables that should be defined before invoking the STEM build script.

Important Note: The org.eclipse.stem.releng/bootstrapManual.sh script defines these environment variables.

Variable Name Required Example Description
SKIPFEATURE yes false Skip creating the STEM Features. In most cases, this should be false
SKIPPRODUCT yes false Skip creating and packaging the STEM Product. In most cases, this should be false
RUNTESTS yes true Run the JUnit tests after the features and product are created.
JAVA_HOME yes /usr/local/java Path to the root folder of your JDK. The java executable should be located at /bin/java relative to this path
ECLIPSE_HOME yes /usr/local/eclipse Path to the root folder of your Eclipse SDK installation. The eclipse plugins folder should be located at /plugins relative to this path
MAJOR_VERSION yes 1.2.1 The release train this build is member of. Should be in X.Y.Z form. Should not include other information, such as a qualifier.
BUILD_TYPE yes nightly The type of release. One of three options:
  • nightly - unstable builds created automatically
  • integration - weekly or semi-weekly that are mostly stable
  • stable - pre-release stable builds, such as milestones and release candidates
  • release - official, stable releases
VERSION_QUALIFIER no RC1 Version qualifier for stable builds only. Examples are M1, M2, RC1, RC2, etc.
PUBLISH_BUILD no false Whether a build should be prepared for publishing to Eclipse. Only valid for STEM project release engineering.
PUBLISH_DIR no /opt/stem/builds Path that builds should be published to. Only valid for STEM project release engineering.

Launching the Builder from Command Line

The builder scripts are located in the releng/org.eclipse.stem.releng folder.

The launcher script is bootstrapManual.sh. This file contains the environment variable definitions from above. Edit this file to so that the values are appropriate for your enironment.

  1. From a command line or terminal, change into the org.eclipse.stem.releng directory
    cd [STEM_CHECKOUT_PATH]/releng/org.eclipse.stem.releng
  2. Edit the bootstrapManual.sh file to configure the builder for your environment (see above for help)
  3. Invoke bootstrapManual.sh  :
    bash bootstrapManual.sh
  4. Wait while STEM builder compiles and packages the binaries
    Depending on your system speed and options specified, this step could take from 30 minutes to 1 hour to finish
  5. After it's finished, the zip files containing the binaries are located in the build folder:
    cd build/I.WeeklyBuild

Launching the Builder from Hudson/Jenkins

If you wish to setup STEM builds using Hudson or Jenkins, you can easily do so. In Hudson/Jenkins, you can configure a job to define the environment variables listed above and then invoke the releng/org.eclipse.stem.releng/bootstrapHudson.sh script. This will invoke the builder. See section on Continuous Integration below for more information.

Running the JUnit Tests

JUnit tests are run as part of the standard build process. Whether or not they are run is controlled by the RUNTESTS environment variable. To run JUnits as part of the build process, simply set RUNTESTS=true when invoking the builder script (bootstrapManual.sh).

Test results are aggregated and collected in the releng/org.eclipse.stem.releng/tests/ folder.

Important Note: Running JUnit tests requires a windowing environment to be available to the terminal invoking the build script (e.g. $DISPLAY must be defined). There are several options available for all-text environments, such as running Xvnc. Most users won't be affected by this.

Continuous Integration (CI) Automated Builds