Skip to main content
Jump to: navigation, search

Virgo/Build

< Virgo
Revision as of 07:06, 17 May 2011 by Gnormington.vmware.com (Talk | contribs) (Releasing a Single Repository Project)



Hudson Builds

Hudson manages Virgo CI builds. Committers are notified of build failures caused by their pushes but anyone who wants to know about other build failures should subscribe to virgo-build (this list is not currently in use but can be enabled when needed).

Virgo Build

Virgo has its own build system known as "Virgo build" which is a collection of Ant/Ivy scripts.

Prerequisites

Make sure you have git, Java 6 or later, and ant 1.7.1 or later installed.

To use any of the Ruby scripts (e.g. update_dependencies.rb or ripplor.rb) then you must install Ruby, RubyGems and Rubygem choice:

  1. To install Ruby use the command sudo port install ruby (on Mac);
  2. To install RubyGems use the command sudo port install rb-rubygems (on Mac);
  3. To install Rubygem choice use the command sudo gem install choice (on Mac).

Building Individual Repositories

To build a given git repository, first clone it:

git clone <repository URL>

and then make sure that the Virgo build submodule is correctly initialised by changing directory to the cloned repository and issuing:

git submodule update --init

then change directory into the build-xxx directory and invoke ant with the appropriate targets, usually:

ant clean clean-integration test

This will do a fresh build and run all the unit and integration tests. It will only report "Build successful" if everything compiles cleanly and all the tests pass. The directory build-xxx/target/test-results contains the test results in html and xml formats. You will find the built bundles in the target/artifact directory of each individual project.

Running all tests

Sometimes it is useful to run all the tests even if some fail. You can do this by specifying the property -Dci.build=true on the ant command, for example:

ant -Dci.build=true clean clean-integration test

This will report "Build successful" regardless of test success or failure and you will need to look at the test results to see which tests failed.

Common targets

ant -p displays commonly used targets. Use ant clean-ivy to delete the ivy cache (or use git clean -dfx to get everything back to a reset state, including ditching any changes you haven't committed).

If you simply want to compile the runtime code, you can use ant clean clean-integration jar but that will not compile (or run) the tests.

ant precommit performs more extensive checks including code coverage and "findbugs". (You will need a licence for Clover to perform test coverage analysis.)

Exceptions to the above

Building the packaged web server is somewhat different:

ant clean clean-integration jar package smoke-test

You can also use the package target in the kernel to build a standalone kernel:

ant clean clean-integration test package

Building the documentation is different too:

ant doc

Example

For example, to build the test stubs component, issue the following commands:

>git clone git://git.eclipse.org/gitroot/virgo/org.eclipse.virgo.osgi-test-stubs.git 
>cd org.eclipse.virgo.osgi-test-stubs
>git submodule update --init
>cd build-osgi-stubs
>ant clean clean-integration test

After this has completed successfully, the project org.eclipse.virgo.teststubs.osgi contains the built binary and source jars in its target/artifacts directory.

Updating Dependencies

In general, dependencies are declared in files named ivy.xml.

Dependencies between Virgo components are a special case of this. The properties file build.versions in the root directory of each git repository defines the versions of the Virgo components that the build will download. Updating any of the versions in build.versions is error-prone as the same values occur in certain other files (which cannot use property substitution). To update a version in build.versions, run the "update dependency" ruby script, for example:

~/virgo/web-server/scripts/update-dependency.rb -v <variable> -n <new version> 

where <variable> is the relevant property name in build.versions and <new version> is the replacement version. The script will report the changes it makes and may issue warnings, so pay attention to its output.

Please ensure you are running the latest version of update-dependency script by updating web-server git repository.

Rippling

Since the Virgo components form a directed, acyclic graph, changing the graph to use a new version of a specific component involves not only updating that version throughout the graph, but also updating the versions of the components which have been changed. This is the tedious process known as rippling and so the ripplor ruby script was written to automate it. Essentially, Ripplor handles the most commonly updated components and flattens the dependency graph into a linear sequence. The sequence is (at the time of writing):

osgi-test-stubs
osgi-extensions
util
test
medic
artifact-repository
kernel
web
apps
documentation
web-server

So, for example, if you change the kernel and want to produce a package web server containing the updated kernel, you would run:

~/virgo/web-server/scripts/ripplor/ripplor.rb -r kernel

This would build and publish a new version of the kernel, update the dependency of the web layer on the newly built kernel, build and publish a new version of the web layer, update the dependency of the Virgo-supplied applications on the newly built web layer and kernel, and so on until it builds and publishes a new version of the packaged web-server. This is called a ripple from kernel.

Ripplor (after 13May2010) allows a ripple to be executed across a list of repositories some in git.eclipse.org and some in git.springsource.org, though all of them are in git.eclipse.org at the time of writing.

The definitive list of repositories and the order of a ripple is stored in the ripplor.rb ruby script source.

Please ensure you are running the latest version of ripplor script by updating web-server git repository.

When should Ripplor be used?

In general, Ripplor is used any time a change needs to be propagated up the graph of dependent repositories. There are two main cases that cause this to happen.

  1. An external dependency needs to be upgraded in all modules of Virgo
  2. A change has been made to code in a repository (e.g. the kernel) which affects its externally visible behaviour. The code above (e.g. the web layer, Virgo-supplied applications, etc.) needs to be updated to see the changes.

Usage

Usage: ripplor.rb [-rvmu]

Required arguments:
    -r, --repo=REPO                  The name of the starting repo

Optional arguments:
    -v, --version=VARIABLE:VERSION[,...]
                                     Versions to substitute during the ripple
    -m, --map=REPO-MAP               The property file containing a mapping from a repository name to a location
                                     (defaults to ~/repository.map)
    -u, --remote-user=REMOTE-USER    User id to use for remote repository access
        --dry-run                    Show what would happen but do not actually do anything
 Option  Description
 -r The name of the repository you want to start the ripple at. A typical repository name is util or kernel.
 -v When updating a dependency use this flag to describe both the variable to update and the new version to update to. Can be a comma separated list if multiple dependencies need updating at the same time.
 -m The location of the repository map file (see below). By default, this file is found in ~/repository.map, but in cases where you have more than one file (one for simple ripples, one for complex ripples that happen in your working copy) you may need to pass in this value.
 -u The login name used for ssh access to the git.eclipse.org server. By default, this is the local login name.

Repository Map

By default, Ripplor checks out each repository working copy to the working directory that it was run from. If you are prepared to run Ripplor in a dedicated working directory, this is fine. However, if you run a ripple from within your development shell environment it can pollute your filesystem. To account for this, Ripplor takes a repository map file. This file tells Ripplor where to checkout a repository working copy to. The file itself is a simple mapping from a repository name to a filesystem location.

osgi-extensions = ~/dev/ripplor/osgi-extensions
util = ~/dev/ripplor/util
...

Tip: In the repository map file, ~ will be expanded to the current user’s home directory.

Broken Build

Ripplor was smart enough to restart a failed ripple.

[If a ripple fails (due to test or compile failure), simply fix any problems in the working copy that Ripplor is using and restart the ripple at the same location you originally used. Ripplor will bring in any new commits since you last started the ripple and continue up the stack as done previously. This behaviour is very useful if you are rippling a breaking API change and need to fix code as the ripple moves upwards.]

Unfortunately this behaviour is broken, since the version updates (accumulated during the ripple and propagated up) are not correctly restored. Also, when committing the changes (to all the repositories) at the end, after a failure in the middle, not all of the rippled repositories will be committed at the end. The standard solution at the moment is to push the repositories that succeeded by hand and start another ripple, with version updates that are likely to be lost inserted by hand on the ripplor command line. This is not perfect and a repeat of the ripple from the lowest point is safer.

A full ripple of all of the repositories takes an hour and a half or more, even on a high-end Mac.

Examples

Upgrading the version of Spring used in util and upward:

./ripplor.rb -r util -v "org.springframework:3.1.0.RELEASE" -m ~/ripple.repository.map

Rippling a breaking API change to kernel

./ripplor.rb -r kernel -m ~/workingcopy.repository.map

Releasing a Single Repository Project

Simply follow these steps to produce a release of a product that is within a single Git repository. The aim is to have a tag of the right name pointing to the codebase at release time and to leave master or the current development branch version at the next version ready for development to continue. Only committers can produce releases.

1. Create a new branch and make sure everything is up to date and clean

      git pull
      git clean -f -d
      git submodule update --init
      git checkout -b release 

2. Update the build.properties file to have the right version, have a release.type of 'release' (or 'milestone' if appropriate), and have a build.stamp with the appropriate version qualifier. For example:

      version=1.1.0
      release.type=release
      build.stamp=RELEASE

3. Do the build, commit the changes and then tag it with the version and release type appropriate. In order for publishing to eclipse to work you must have your ssh public key registered with the virgoBuild account.

      cd build-xxx
      ant clean clean-integration clean-ivy test package publish publish-eclipse -propertyfile ~/s3.properties -Declipse.committerId=gnormington
      cd ..
      git add .
      git commit -m 'Build properties for the release 1.1.0.RELEASE'
      git tag 1.1.0.RELEASE 

4. After the tag go back to master and delete the release branch. The branch will still be available from the tag.

      git checkout master
      git branch -D release

5. Back on master the build.properties file needs to be revved up to the following release for development to continue. Ensure the release type is still integration and that the build.stamp property is not present. Set the version to the following version of the product.

      version=1.2.0
      release.type=integration

6. Finally add, commit and push it all back up to origin. Ensure the new tag is also pushed up.

      git add .
      git commit
      git push origin
      git push --tags

Once this is done the usual steps to make the release available on download servers and to make announcements should be followed. For Eclipse, a committer with a full shell should sftp in to download1.eclipse.org and beam the file up. More information available here.

Releasing a Multi Repository Project

This is for any release of the full Virgo Server. It makes use of the Releasolor ruby script. This is very similar to Ripplor but it also performs the steps above for a single repository to all the repositories it visits. Only committers can produce releases.

The usage instructions are:

   Usage: releaselor.rb [-vqtnbmurghjk]
   Required arguments:
       -v, --version=VERSION            The version number of the release
       -q, --build-stamp=BUILD-STAMP    The build stamp for the release, e.g. M01, RELEASE
       -t, --release-type=RELEASE-TYPE  The release type for the release, e.g. milestone, release
       -n, --new-version=NEW-VERSION    The new version number to be used after the release
   Optional arguments:
       -b, --virgo-build-version=VIRGO-BUILD-VERSION
                                        The version to update Virgo Build to
       -m, --map=REPOSITORY-MAP         The property file containing a mapping from a repository name to a location
                                        (defaults to ~/repository.map)
       -u, --remote-user=REMOTE-USER    User id to use for remote repository access
                                        (defaults to local login id)
       -r, --product-release=PRODUCT    The product to be released, kernel, virgo, web-server, full-product
                                        (defaults to releasing all the virgo repos)
       -g, --gemini-version=GEMINI-BUILD-VERSION
                                        When producing a full-product build the Gemini version must be given
       -h, --gemini-build-stamp=GEMINI-BUILD-STAMP
                                        The build stamp for the Gemini Web release, e.g. M01, RELEASE
       -j, --gemini-release-type=GEMINI-RELEASE-TYPE
                                        The release type for the Gemini Web release, e.g. milestone, release
       -k, --gemini-new-version=GEMINI-NEW-VERSION
                                        The new Gemini Web version number to be used after the release
           --dry-run                    Show what would happen but do not actually do anything


An example invocation for a release is:

       ./releaselor.rb -v 2.0.1 -s RELEASE -t release -n 2.0.2 -u cfrost -r kernel

This will release the Kernel only at 2.0.1 and leave the repos versioned 2.0.2 for work to continue after the release. Repos other than those needed for the kernel will not be modified.

An example invocation for a milestone including Gemini Web is:

       ./releaselor.rb -v 3.0.0 -q M01 -t milestone -n 3.0.0 -g 1.2.0 -h M02 -j milestone -k 1.2.0 -u cfrost -r full-product

An example invocation for a release not including Gemini Web is:

       ./releaselor.rb -v 3.0.0 -q RELEASE -t release -n 3.1.0 -u cfrost -r virgo


Note.png
Publishing the Gemini Zips
If you do a 'full-product' release, Gemini Web will also be released. There will be a manual step to publish the Gemini zips to download.eclipse.org as they need to go in a different directory than the rest of Virgo. The Zips will be built as normal and will be available from the 'build-web-container' directory.


Note.png
Updating Plan versions
When doing a full release there are some plan files that need manually editing.
   org.eclipse.virgokernel.userregion.springdm (Kernel repo in the build-kernel/repository/ext directory)
   org.eclipse.virgo.apps.admin.plan (Servlet/Apps repo in the src of the admin project)
   org.eclipse.virgo.web (In the web repo in the src of the web project)
In addition, the abstract integration test in the web.test project and AdminConsoleAppTests in the system-verification-tests repository (still to be contributed at the time of writing) also need to be updated to use the new version for the tests to pass. Ideally the script that does version updating should cover this but in doing so it would no longer be generic. Ideas welcome.

Back to the top