Jump to: navigation, search

Difference between revisions of "Gyrex/Contributor Guide/Builds"

m
m (Really Quick HowTo)
 
(9 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
Gyrex builds are produced on the Eclipse.org infrastructure. It's also possible to build locally. We use Maven and Tycho as our build technology which is also the standard build technology recommended by [[CBI]].
 
Gyrex builds are produced on the Eclipse.org infrastructure. It's also possible to build locally. We use Maven and Tycho as our build technology which is also the standard build technology recommended by [[CBI]].
 +
 +
== Really Quick HowTo ==
 +
 +
# create temp directory
 +
mkdir -p /tmp/gyrex
 +
cd /tmp/gyrex
 +
 +
# clone source into local directory
 +
git clone git://git.eclipse.org/gitroot/gyrex/gyrex-server.git .
 +
 +
# run Maven
 +
mvn clean verify -f releng/aggregator/pom.xml
  
 
== Prerequisites ==
 
== Prerequisites ==
Line 55: Line 67:
 
All Gyrex Git repositories must be cloned into ''the same parent directory''.
 
All Gyrex Git repositories must be cloned into ''the same parent directory''.
  
   git clone git://git.eclipse.org/gitroot/gyrex/gyrex-releng.git
+
   git clone git://git.eclipse.org/gitroot/gyrex/gyrex-server.git
  git clone git://git.eclipse.org/gitroot/gyrex/gyrex-platform.git
+
  git clone git://git.eclipse.org/gitroot/gyrex/gyrex-admin.git
+
  git clone git://git.eclipse.org/gitroot/gyrex/addons/gyrex-jersey-jaxrs.git
+
  git clone git://git.eclipse.org/gitroot/gyrex/addons/gyrex-search.git
+
  git clone git://git.eclipse.org/gitroot/gyrex/addons/gyrex-mongodb-persistence.git
+
  
  
 
== Running The Build ==
 
== Running The Build ==
From the releng root (directory <code>'gyrex-releng'</code>), run:
+
From the aggregator directory (directory <code>'releng/aggregator'</code>), run:
  
 
   mvn clean verify
 
   mvn clean verify
Line 85: Line 92:
 
== Build Process Insights ==
 
== Build Process Insights ==
  
== RelEng Project ==
+
=== RelEng Project ===
The root for all build related stuff is the Gyrex RelEng (Release Engineering) project. It contains:
+
The root for all build related stuff is the Gyrex RelEng (Release Engineering) tree. It contains:
* [http://git.eclipse.org/c/gyrex/gyrex-releng.git/tree/org.eclipse.gyrex.releng/builder/ build scripts],
+
* [http://git.eclipse.org/c/gyrex/gyrex-server.git/tree/releng/targetplatform/ target platform definitions],
* [http://git.eclipse.org/c/gyrex/gyrex-releng.git/tree/org.eclipse.gyrex.releng/maps/ map files],
+
* [http://git.eclipse.org/c/gyrex/gyrex-server.git/tree/releng/products/ product configurations] and
* [http://git.eclipse.org/c/gyrex/gyrex-releng.git/tree/org.eclipse.gyrex.releng/targets/ target platform definitions],
+
* [http://git.eclipse.org/c/gyrex/gyrex-server.git/tree/releng/features/ features].
* [http://git.eclipse.org/c/gyrex/gyrex-releng.git/tree/org.eclipse.gyrex.releng/products/ product configurations] and
+
* [http://git.eclipse.org/c/gyrex/gyrex-releng.git/tree/org.eclipse.gyrex.releng/features/ features].
+
 
<pre>
 
<pre>
git://git.eclipse.org/gitroot/gyrex/gyrex-releng.git
+
git clone git://git.eclipse.org/gitroot/gyrex/gyrex-server.git  
 +
cd gyrex-server/releng
 
</pre>
 
</pre>
  
Line 102: Line 108:
 
The Product Configuration generates the various downloadable artifacts (eg. Gyrex Server ZIP). It references Eclipse Features which define collection of bundles that are meant to be build/deployed together. There are two types of features - regular Gyrex features and dependency-only features. Gyrex features collect Gyrex bundles. Dependency-only feature collect all dependencies. This makes it easy to customize Gyrex based products with different dependencies by using custom features.
 
The Product Configuration generates the various downloadable artifacts (eg. Gyrex Server ZIP). It references Eclipse Features which define collection of bundles that are meant to be build/deployed together. There are two types of features - regular Gyrex features and dependency-only features. Gyrex features collect Gyrex bundles. Dependency-only feature collect all dependencies. This makes it easy to customize Gyrex based products with different dependencies by using custom features.
  
=== Build Scripts and Map Files ===
 
The build scripts and map files are used to drive the automatic build and tagging process on Hudson. A cron script checks Git repositories regularly for updates. If new commits are found, a repository is tagged, the map file is updated with the new tag and a Hudson build will be started. PDE Build uses the map files to calculate a bundle qualifier.
 
  
 +
=== Headless Build Process ===
 +
The build process is entirely based on CBI. CBI is the common build infrastructure at Eclipse.org which is based on Maven and Tycho and has some additional Maven plug-ins for signing of bundles/binaries, etc.
  
 +
We use Hudson for CI. The build process on Hudson is as follows:
 +
* Checkout all Git repos into same directory
 +
* Invoke Maven using <code>releng/aggregator/pom.xml</code> which will
 +
** Build target platform from target definition
 +
** Build modules referenced in POM
 +
** Assemble Gyrex Server Product
 +
** Produce final p2 Repo (including signing of artifacts when executed on Eclipse.org infrastructure with Maven profile)
 +
** Publish to download.eclipse.org (when executed on Eclipse.org infrastructure with Maven profile)
  
 +
=== Hudson Build Jobs ===
 +
* [https://hudson.eclipse.org/gyrex/job/gyrex-latest/ gyrex-latest]
 +
The integration job is used for producing I, S and R builds. Care must be taken when producing S and R builds. The results are automatically published to download.eclipse.org (when running on Eclipse.org infrastructure).
  
== Headless Build Process ==
+
* [https://hudson.eclipse.org/gyrex/job/gyrex-maintenance/ gyrex-maintenance]
The build process is entirely based on PDE Build (which is based on Ant). The main build script includes targets for calling from a plain Ant as well as from an Eclipse Ant. The plain Ant target bootstraps a PDE Builder from Eclipse CVS and then launches the Eclipse target using Eclipse Ant. There is also a shared external tool launch configuration that can be used for building from within Eclipse.
+
The integration job is used for producing M and R builds of the maintenance branch.
 
+
The build process if as follows:
+
* Bootstrap PDE Builder
+
* Build target platform from target definition (using p2 mirror functionality)
+
* Fetch sources using map files
+
* Build org.eclipse.gyrex.features.master
+
* Assemble Gyrex Server Product
+
* Produce final p2 Repo (including signing of artifacts when executed on Eclipse.org infrastructure)
+
* Publish to download.eclipse.org (when executed on Eclipse.org infrastructure)
+
 
+
In order to use the headless build locally, you need to clone the Gyrex Releng repository. You also need to have a working Ant, Git and CVS executable on your system and in your path (so that the build can invoke it). The build script to invoke is <code>org.eclipse.gyrex.releng/builder/build.xml</code>. Please also create a local <code>build.local.properties</code> before in order to set some important, required paths.
+
 
+
=== runInEclipse ===
+
As part of the build process the Ant scripts bootstrap PDE Builder from Eclipse CVS repos. It then launches the PDE Builder with the same build script but this time calls the Ant target <code>runInEclipse</code>. Of course, this target can also be called directly from a running Eclipse. However, you must have Eclipse PDE, CVS as well as the Git Fetch Factory for PDE installed in your running Eclipse. The Git Fetch Factory can be installed from http://download.eclipse.org/egit/pde/updates-nightly/.
+
 
+
 
+
== Hudson Build Jobs ==
+
* [https://hudson.eclipse.org/hudson/view/Technology/job/gyrex-nightly/ gyrex-nightly]
+
The nightly job is used for CI compile check builds. It monitors the full source tree for changes and kicks of a build every once in a while when somebody checked something into source control. However, it always uses HEAD when performing checkouts. The build results should not be used.
+
 
+
* [https://hudson.eclipse.org/hudson/view/Technology/job/gyrex-integration/ gyrex-integration]
+
The integration job is used for producing I, S and R builds. Care must be taken when producing S and R builds. The results are automatically published to download.eclipse.org (when running on Eclipse.org infrastructure).
+

Latest revision as of 14:35, 28 January 2014

Gyrex builds are produced on the Eclipse.org infrastructure. It's also possible to build locally. We use Maven and Tycho as our build technology which is also the standard build technology recommended by CBI.

Really Quick HowTo

# create temp directory
mkdir -p /tmp/gyrex
cd /tmp/gyrex

# clone source into local directory
git clone git://git.eclipse.org/gitroot/gyrex/gyrex-server.git .

# run Maven
mvn clean verify -f releng/aggregator/pom.xml

Prerequisites

The build is modular with build-order dependencies between the modules. We do not use a single "aggregator" project that aggregates the various Git repositories. But the build process expects that repositories are cloned into the same parent directory in order to allow relative path reference in build scripts/descriptor files.

Oracle Java 7 or higher

  • Oracle Java 7 JDK installed
    Note: Open JDK might work too but hasn't been tested yet. Feedback is welcome!
  • Environment variable JAVA_HOME set to the JDK directory
  • Please ensure that the java executable is available in the PATH!

Maven 3

  • Download from http://maven.apache.org/download.html or use your Linux distribution provided version (whether it works depends on the distribution).
    Note: At the time of writing, Maven 3.0.4 has been used and tested.
  • Please ensure that the mvn executable is available in the PATH!

Git

  • Please ensure that a recent version of the git executable is on the PATH!
  • Depending on your operating system the setup might vary. A good setup guide is available at Set Up Git (GitHub).


Building on Windows

In general, the build should also work on Windows. However, please be aware that there might be some limitations which make it a difficult to produce good results. For example, the path length is limited. Thus, the code tree should not be cloned into an exciting deep hierarchy of folders.

Additionally, builds on Windows can't produce the correct permissions in produced archives on Linux. Therefore, good quality builds for multiple platforms should not be produced from a Windows system.


Build Environment Setup

The following instructions are required to prepare your build environment. Unless otherwise stated, they apply to Linux, Mac OS X, and Windows platforms.

Maven Settings

To avoid getting hit by bug 365727 and to make sure your local environment configuration does not interfere with CBI build, make sure build user account does not have a Maven settings.xml

your_user@desktop:~$ ls -l ~/.m2/settings.xml
ls: cannot access /home/your_user/.m2/settings.xml: No such file or directory

On Windows XP this is located in:

C:\Documents and Settings\your_user\.m2\settings.xml
File should not exist, if it does delete it.

On Windows 7 this is located in:

C:\Users\your_user\.m2\settings.xml
File should not exist, if it does delete it.

Maven Memory Consumption

It may be necessary to grant Maven more memory. In this case, 2GB of RAM should be enough.

 export MAVEN_OPTS=-Xmx2048m

If you're using Windows:

 SET MAVEN_OPTS=-Xmx2048m


Cloning Gyrex Source Tree

All Gyrex Git repositories must be cloned into the same parent directory.

 git clone git://git.eclipse.org/gitroot/gyrex/gyrex-server.git


Running The Build

From the aggregator directory (directory 'releng/aggregator'), run:

 mvn clean verify

Multiple Streams

If you build multiple streams on the same system, you'll want to add '-Dmaven.repo.local=/some/directory/somewhere' to the end of each 'mvn' command to avoid collisions (using a different local repo for each stream). Most casual developers won't be affected.

Alternatively, you can add the additional argument to the MAVEN_OPTS environment variable instead.

On Windows use '-Dmaven.repo.local=C:\path\to\somewhere'.

Building directly from Eclipse

Instead of using Maven and Tycho, it is also possible to simply use the PDE Product Export wizard that's available in Eclipse. It requires to have the target platform configured and all Gyrex bundles imported in your workspace. The export can be started via File -> Export... -> Product configuration menu.

Gyrex-product-export.png

Note, however, that this is not recommended for producing official, production build results (to be published on Eclipse.org). But it's a convenient way of testing a product setup.


Build Process Insights

RelEng Project

The root for all build related stuff is the Gyrex RelEng (Release Engineering) tree. It contains:

git clone git://git.eclipse.org/gitroot/gyrex/gyrex-server.git 
cd gyrex-server/releng

Target Definition

The Target Definition is used for dependency management. All dependencies are listed as p2 repositories hosted at Eclipse.

Product Configuration and Features

The Product Configuration generates the various downloadable artifacts (eg. Gyrex Server ZIP). It references Eclipse Features which define collection of bundles that are meant to be build/deployed together. There are two types of features - regular Gyrex features and dependency-only features. Gyrex features collect Gyrex bundles. Dependency-only feature collect all dependencies. This makes it easy to customize Gyrex based products with different dependencies by using custom features.


Headless Build Process

The build process is entirely based on CBI. CBI is the common build infrastructure at Eclipse.org which is based on Maven and Tycho and has some additional Maven plug-ins for signing of bundles/binaries, etc.

We use Hudson for CI. The build process on Hudson is as follows:

  • Checkout all Git repos into same directory
  • Invoke Maven using releng/aggregator/pom.xml which will
    • Build target platform from target definition
    • Build modules referenced in POM
    • Assemble Gyrex Server Product
    • Produce final p2 Repo (including signing of artifacts when executed on Eclipse.org infrastructure with Maven profile)
    • Publish to download.eclipse.org (when executed on Eclipse.org infrastructure with Maven profile)

Hudson Build Jobs

The integration job is used for producing I, S and R builds. Care must be taken when producing S and R builds. The results are automatically published to download.eclipse.org (when running on Eclipse.org infrastructure).

The integration job is used for producing M and R builds of the maintenance branch.