Linux Tools Project/Eclipse Build/Inner Workings

Given an archive with Eclipse SDK sources Eclipse Build will extract the sources and remove all the pre-built OSGI jars, but collects all the OSGI metadata (contained in MANIFEST.MF files). Then Eclipse Build tries use jars provided by the system, verifying that versions match the OSGI information provided in the eclipse sources archive.

Preparation

Setting Build IDs, tags, and labels

The build IDs and tags determines what version of the various Eclipse components will be checked out. Labels specify the version string which is appended to the source tarball and appears in the About dialogue.

When you update the source tarball label you must change it in the following places:

• eclipse-build/buildSDKSource.sh
• eclipse-build/build.properties
• eclipse-build/pdebuild.properties
• eclipse-build/regenerateBootstrapFiles.sh

...more about tags and buildID...

Fetching Sources

When an SDK source tarball is not available, sources can be fetched using buildSDKSource.sh. This can be the case for example if a release has not yet been made of the version you wish to build.

To fetch the sources, you need the build ID. The build ID is the git (or CVS) tag which is used to create the particular version. For more details about the upstream Eclipse build cycle click here. Here you can find recent builds, the build IDs (tags) which were used to create them and and you can inspect the build results.

Example:

./buildSDKSource.sh I20110719-0800

Running that does the following:

• Checkout the base builder (org.eclipse.releng.basebuilder: a binary version of Eclipse used for running the build)
• Checkout the Eclipse builder (org.eclipse.releng.eclipsebuilder: the build driver)
• Use the above to checkout sources for Eclipse
• Extract some compressed sources
• Remove all prebuilt binaries.

Generating Pregenerated Build Scripts

These build scripts live eclipse-build-generatedScripts.tar.bz2 and are generated using an upstream eclipse. During the first stage of bootstraping these scripts are extracted to generatedScripts and are used to build a minimal eclipse which is in turn used to regenerate these build scripts and finish the build.

The version of the upstream prebuilt Eclipse you use should be as close as possible to the one you are trying to build, if not the exact same tag. To generate the scripts you do the following:

cd eclipse-build
ant copyDeps && ./regenerateBootstrapFiles.sh <path to downloaded eclipse binary>

Note that while normally we create symlinks in the plugins directory to the system jars (ant target symlinkDeps), when creating the pregenerated build scripts we copy the dependencies instead (ant target copyDeps). The reason for this is that the genereated build scripts will always follow the symlinks and contain paths which point to the final destination of the symlink. This makes the scripts system specific. By copying the deps instead the generated build files will contain references to the files in the plugins directory which during the actual build will be symlinks to the system jars.

Dependencies

Packages which Eclipse depends on are normally provided in an Eclipse download as JARs coming from the Orbit project. Eclipse-Build replaces those with JARs provided by the distribution. In eclipsebuild/*.properties files you will find a list of Eclipse dependencies and a search path indicating where they are found on various Linux distributions. If Eclipse-Build does not know about a needed dependency this is where you would add it. If eclipse-build knows about a dependency but cannot find it, you would edit the search path for that dependency in one of these .properties files.

Each entry in a .properties file must have a corresponding MANIFEST.MF file in eclipse-build/dependencyManifests/. These manifest files should be obtained from a JAR from the corresponding eclipse-build or from an Orbit download. The directory structure should be in this format:

eclipse-build/dependencyManifests/<name and version of the JAR>/META-INF/MANIFEST.MF

At a later stage eclipse-build will use these files to ensure that the system files which you have pointed it to match the Eclipse requirement.

Building Eclipse

This process is started when you run ant in the eclipse-build directory. This runs build.xml which does the following:

• extracts source tarballs
• applies eclipse-build patches
• creates symlinks to system JARs to satisfy Eclipse dependencies.
• starts the bootstrapping process by calling the bootstart target in pdebuild.xml

Applying Patches

Patches are applied to files belonging to plugins that are built as part of the Eclipse SDK or in running the SDK tests.

All patches to be applied are located within eclipse-build/patches, and have a corresponding "<patch .." directive in build.xml under the applyPatches or applyTestPatches task, depending on where it will be applied.

For example,

eclipse-build/patches/tests-BZ295666.patch, has a corresponding entry in build.xml under the task applyTestPatches:

<patch patchfile="${basedir}/patches/tests-BZ295666.patch" dir="${testsBuildDirectory}" strip="0" />

Searching And Symlinking System JARs

The following Java files are used to define ant tasks which perform the version verification and symlinking of the system JARs

task-src/org/eclipse/linuxtools/eclipsebuild/SymlinkInstalledOSGiJars.java
task-src/org/eclipse/linuxtools/eclipsebuild/SymlinkOSGiJars.java
task-src/org/eclipse/linuxtools/eclipsebuild/SymlinkNonOSGiJars.java

The search path used by the ant tasks above is specified in *.properties files. For example:

org.objectweb.asm_3.3.1.v201101071600.jar=/usr/share/java/objectweb-asm/asm-all.jar:/usr/share/java/asm3-all.jar

means that the MANIFEST.MF files of /usr/share/java/objectweb-asm/asm-all.jar and /usr/share/java/asm3-all.jar will be examined to see if they satisfied the requirement of org.objectweb.asm_3.3.1.v201101071600.jar. If a match if found a symlink to that JAR is created in the plugins directory.

Bootstrapping Eclipse

Bootstrapping is done in 3 steps:

1. use pre-generated build scripts to build org.eclipse.pde.build and all of its dependencies. At this step we have enough bootstrapped to generate build.xml files.
2. use the result from step one to generate build.xml files for build utilities like pde.apitools and others. Now we have something very close to what Eclipse releng calls a "basebuilder".
3. use the result from step two to cleanup, regeneate all the build.xml files, and build the whole SDK. These steps try to mimic the regular releng procedure where possible.

pdebuild.xml will first unzip pre-generated build scripts found in eclipse-build-generatedScripts.tar.bz2. It will then use those scripts to build Eclipse up to and including PDE Build. This PDE Build is then used to generate build.xml files and rebuild it self again to ensure that a full build is done (pre-generated ant scripts may not run all the pre/post ant targets if any exist). The result is then used to finish the build.