Jump to: navigation, search

Difference between revisions of "Virgo/Build"

(Hudson Builds)
m (Hot-fixing third party dependencies)
(20 intermediate revisions by 2 users not shown)
Line 46: Line 46:
  
 
3. Execute the following targets:
 
3. Execute the following targets:
<pre>ant clean clean-integration clean-ivy test test-e2e publish-package-build</pre>  
+
<pre>ant clean clean-integration clean-ivy test test-e2e publish-build</pre>  
 
This publishes a RELEASE version of the ''virgo-build-tools'' assembly to build.eclipse.org.  
 
This publishes a RELEASE version of the ''virgo-build-tools'' assembly to build.eclipse.org.  
  
Line 55: Line 55:
 
<pre>git commit -m 'Build properties for 1.2.0 integration'
 
<pre>git commit -m 'Build properties for 1.2.0 integration'
 
git push origin</pre>  
 
git push origin</pre>  
At this point all repositories consuming the ''virgo-build-tools'' assembly can be updated to the newly released version.  
+
At this point all repositories consuming the ''virgo-build-tools'' assembly can be updated to the newly released version.
  
 
=== Virgo-build-tools prerequisites  ===
 
=== Virgo-build-tools prerequisites  ===
Line 77: Line 77:
 
| 1.3.0
 
| 1.3.0
 
| 2.22 or higher
 
| 2.22 or higher
 +
|-
 +
| 1.4.0
 +
| 2.32 or higher
 +
|-
 +
| 1.4.1
 +
| 2.32 or higher
 
|}
 
|}
  
Line 215: Line 221:
 
   &lt;param name="bundles" value="${bundles.dir}"/&gt;
 
   &lt;param name="bundles" value="${bundles.dir}"/&gt;
 
   &lt;param name="features" value="${features.dir}"/&gt;
 
   &lt;param name="features" value="${features.dir}"/&gt;
&lt;/antcall&gt;</pre> <pre>@Since 2.15
+
&lt;/antcall&gt;</pre> <pre>@Since 2.31
 
&lt;antcall target="p2.publish-product"&gt;
 
&lt;antcall target="p2.publish-product"&gt;
 
   &lt;param name="repository" value="${repository.dir}"/&gt;
 
   &lt;param name="repository" value="${repository.dir}"/&gt;
 
   &lt;param name="product.file.location" value="${p2.products.location}/example.product"/&gt;
 
   &lt;param name="product.file.location" value="${p2.products.location}/example.product"/&gt;
 +
  &lt;param name="javaprofile.location" value="${javaprofile.location}/java-server.profile"/&gt;
 
&lt;/antcall&gt;</pre> <pre>@Since 2.15
 
&lt;/antcall&gt;</pre> <pre>@Since 2.15
 
&lt;antcall target="p2.publish-category"&gt;
 
&lt;antcall target="p2.publish-category"&gt;
Line 285: Line 292:
 
'''Important:'''  
 
'''Important:'''  
  
'''- The conversion from p2 to Ivy currently ignores any dependencies between bundles. Respecting these dependencies is currently under development but until then the mirrored and converted bundles and their dependencies need to be referenced explicitly.'''
+
'''- <span style="color:#ff0000">The conversion from p2 to Ivy currently ignores any dependencies between bundles. Respecting these dependencies is currently under development but until then the mirrored and converted bundles and their dependencies need to be referenced explicitly.</span>'''
  
 
- If the specified mirror.source contains more than one version of an IU all of them will be mirrored and converted which may lead to problems with the versions propagation. Use source repositories with single version, e.g. the release train repositories. Orbit usually contains more than one version, and often we are not using the latest one available so mirroring Orbit involves a manual step of choosing which version to use and deleting the others from the resulting build.versions file.
 
- If the specified mirror.source contains more than one version of an IU all of them will be mirrored and converted which may lead to problems with the versions propagation. Use source repositories with single version, e.g. the release train repositories. Orbit usually contains more than one version, and often we are not using the latest one available so mirroring Orbit involves a manual step of choosing which version to use and deleting the others from the resulting build.versions file.
  
 
- Keep in mind the time to mirror a large number of artifacts depends on your network and how well you can connect to Eclpse servers. Usually it takes some time, so go treat yourself with a cup of coffee :-)
 
- Keep in mind the time to mirror a large number of artifacts depends on your network and how well you can connect to Eclpse servers. Usually it takes some time, so go treat yourself with a cup of coffee :-)
 +
 +
=== Hot-fixing third party dependencies ===
 +
Sometimes we need to hot-fix an already used third party dependency until the upstream project produces an official release of this dependency with the desired fix.
 +
In order to do that first we must request the "modified" keyword added to the corresponding CQ.
 +
Then we make the fix, produce a jar or patch the original with the fixed class.
 +
 +
When you patch a jar make sure that it's new version qualifier ends with '''''-virgo-<index>''''', where ''<index>'' is the modifications count. For instance the first modification of logback would be ''ch.qos.logback-1.0.0.timestamp-virgo-1''.
 +
 +
If you produce a new jar through a build make sure to modify its filename and manifest to match the current version used in the build and the add the qualifier extension to it.
 +
 +
To make this hot-fixed version available for virgo-build follow these steps:
 +
 +
1. Clone the eclipse-mirror repo
 +
 +
2. Place the fixed jar into '''''eclipse-mirror-root/hotfix/plugins'''''
 +
 +
3. Place the fixed sources into '''''eclipse-mirror-root/hotfix/source/<project-name>'''''
 +
 +
4. Place a README file in '''''eclipse-mirror-root/hotfix/source/<project-name>''''' that points to the original source repository with the right tag
 +
 +
5. Go into build-eclipse-mirror and execute
 +
<pre>ant clean update-hotfixed-artifacts</pre>
 +
 +
6. Update the ''build.versions'' file with the new versions from the ''hotfixed.build.versions'' file
 +
 +
That's it. Now you can refer to the hot-fixed artifacts from Ivy xmls.
  
 
== Rippling  ==
 
== Rippling  ==
  
 
Since the Virgo components form a directed, acyclic graph, changing the graph to use a new version of a specific component involves not only updating that version throughout the graph, but also updating the versions of the components which have been changed. This is the tedious process known as ''rippling'' and so the <code>'''ripplor'''</code> ruby script was written to automate it. Essentially, Ripplor handles the most commonly updated components and flattens the dependency graph into a linear sequence. The sequence is (at the time of writing):  
 
Since the Virgo components form a directed, acyclic graph, changing the graph to use a new version of a specific component involves not only updating that version throughout the graph, but also updating the versions of the components which have been changed. This is the tedious process known as ''rippling'' and so the <code>'''ripplor'''</code> ruby script was written to automate it. Essentially, Ripplor handles the most commonly updated components and flattens the dependency graph into a linear sequence. The sequence is (at the time of writing):  
<pre>osgi-test-stubs
+
<pre>eclipse-mirror
osgi-extensions
+
 
util
 
util
 
test
 
test
 
medic
 
medic
 +
nano
 
artifact-repository
 
artifact-repository
 
kernel
 
kernel
Line 306: Line 339:
 
apps
 
apps
 
documentation
 
documentation
web-server
+
packaging
jetty-server
+
 
</pre>  
 
</pre>  
 
So, for example, if you change the kernel and want to produce a package web server containing the updated kernel, you would run:  
 
So, for example, if you change the kernel and want to produce a package web server containing the updated kernel, you would run:  
Line 380: Line 412:
 
[If a ripple fails (due to test or compile failure), simply fix any problems in the working copy that Ripplor is using and restart the ripple at the same location you originally used. Ripplor will bring in any new commits since you last started the ripple and continue up the stack as done previously. This behaviour is very useful if you are rippling a breaking API change and need to fix code as the ripple moves upwards.]  
 
[If a ripple fails (due to test or compile failure), simply fix any problems in the working copy that Ripplor is using and restart the ripple at the same location you originally used. Ripplor will bring in any new commits since you last started the ripple and continue up the stack as done previously. This behaviour is very useful if you are rippling a breaking API change and need to fix code as the ripple moves upwards.]  
  
Unfortunately this behaviour is broken, since the version updates (accumulated during the ripple and propagated up) are not correctly restored. Also, when committing the changes (to all the repositories) at the end, after a failure in the middle, not all of the rippled repositories will be committed at the end.  
+
==== How to rebuild a broken rippled repository ====
 +
Use the normally executed for that repository targets plus one additional -D parameter:
 +
<pre>
 +
ant -Dvirgo.deps.location=integration-repo clean clean-integration test
 +
</pre>
 +
The important part here is to add the -Dvirgo.deps.location parameter. Its default value is "ivy-cache". It is automatically set by the ruby scripts. It is also not needed during a single repository build.
 +
Its purpose is to point the integration tests to the right location of the Virgo artifacts. During a ripple - the integration repo, during a single repo build - the ivy-cache.
 +
 
 +
Unfortunately this behaviour is broken, since the version updates (accumulated during the ripple and propagated up) are not correctly restored. Also, when committing the changes (to all the repositories) at the end, after a failure in the middle, not all of the rippled repositories will be committed at the end.
  
 
=== Re-Building  ===
 
=== Re-Building  ===
Line 409: Line 449:
 
Rippling a breaking API change to kernel  
 
Rippling a breaking API change to kernel  
 
<pre>./ripplor.rb -r kernel -m ~/workingcopy.repository.map
 
<pre>./ripplor.rb -r kernel -m ~/workingcopy.repository.map
</pre>  
+
</pre>
 +
 
 
== Releasing a Single Repository Project  ==
 
== Releasing a Single Repository Project  ==
  
Line 438: Line 479:
 
Note: if you need to publish to p2, add the target publish-updatesite-download to the above list.
 
Note: if you need to publish to p2, add the target publish-updatesite-download to the above list.
  
Note: if you are building bundlor, until [https://bugs.eclipse.org/bugs/show_bug.cgi?id=384139 bug 384139] is fixed, remember to update the version in org.eclipse.virgo.bundlor.maven/src/main/resources/META-INF/maven/plugin.xml follow these [https://bugs.eclipse.org/bugs/show_bug.cgi?id=379061#c3 instructions] for updating the composite p2 repository. Also, until [https://bugs.eclipse.org/bugs/show_bug.cgi?id=384138 bug 384138] is fixed the update site needs manually relocating.
+
Note: if you are building bundlor, until [https://bugs.eclipse.org/bugs/show_bug.cgi?id=384139 bug 384139] is fixed, remember to update the version in org.eclipse.virgo.bundlor.maven/src/main/resources/META-INF/maven/plugin.xml and follow these [https://bugs.eclipse.org/bugs/show_bug.cgi?id=379061#c3 instructions] for updating the composite p2 repository. Also, until [https://bugs.eclipse.org/bugs/show_bug.cgi?id=384138 bug 384138] is fixed the update site needs manually relocating.
  
 
4. After the tag go back to master and delete the release branch. The branch will still be available from the tag.  
 
4. After the tag go back to master and delete the release branch. The branch will still be available from the tag.  
Line 457: Line 498:
 
     git push --tags
 
     git push --tags
  
<span style="color:#ff0000">Once this is done the usual steps to '''make the release available on download servers''' and to make announcements should be followed. For Eclipse, a committer with a full shell should sftp in to download1.eclipse.org and beam the file up.</span> More information available [http://wiki.eclipse.org/IT_Infrastructure_Doc#Put_files_on_the_download_server.3F here].
+
<span style="color:#ff0000">Once this is done the usual steps to '''make the release available on download servers''' and to make announcements should be followed. For Eclipse, a committer with a full shell should sftp in to build.eclipse.org and beam the file up.</span> More information available [http://wiki.eclipse.org/IT_Infrastructure_Doc#Put_files_on_the_download_server.3F here].
  
 
== Releasing a Multi Repository Project  ==
 
== Releasing a Multi Repository Project  ==
Line 536: Line 577:
 
Virgo Tooling for the Eclipse IDE is built using the same toolchain as the rest of the Virgo project. The job can be found here: https://hudson.eclipse.org/hudson/job/virgo.ide.snapshot/.
 
Virgo Tooling for the Eclipse IDE is built using the same toolchain as the rest of the Virgo project. The job can be found here: https://hudson.eclipse.org/hudson/job/virgo.ide.snapshot/.
  
'''''How to build from the command line?'''''
+
'''''[How to build from the command line?]'''''
  
 
== Building in Eclipse ==
 
== Building in Eclipse ==
Line 545: Line 586:
 
#Clone the git repos at http://git.eclipse.org/c/virgo/org.eclipse.virgo.ide.git/  
 
#Clone the git repos at http://git.eclipse.org/c/virgo/org.eclipse.virgo.ide.git/  
 
#Import all of the projects within the repos.
 
#Import all of the projects within the repos.
#Add Virgo IDE dependencies to your target platform. The best way to do that is to add the composite update site at http://download.eclipse.org/virgo/snapshot/tooling to your target platform and select the following items ''only''. You'll need to uncheck "Group Items by Category" to see all of these.
+
#Add Virgo IDE dependencies to your target platform '''''[What if I haven't got a target platform? How do I create one? Does it need any special contents or configuration?]'''''. The best way to do that is to add the composite update site at http://download.eclipse.org/virgo/snapshot/tooling to your target platform and select the following items ''only''. You'll need to uncheck "Group Items by Category" to see all of these.
 
##All Libra Features
 
##All Libra Features
 
##Eclipse Virgo Bundlor
 
##Eclipse Virgo Bundlor

Revision as of 03:20, 23 November 2012


Hudson Builds

Hudson manages Virgo CI builds. Committers are notified of build failures caused by their pushes but anyone who wants to know about other build failures should subscribe to virgo-build (this list is not currently in use but can be enabled when needed).

Virgo Runtime Build

Virgo has its own build system known as "Virgo build" which is a collection of Ant/Ivy scripts.

Prerequisites

Make sure you have git, Java 6 or later, and ant 1.8.2 or later installed.

To use any of the Ruby scripts (e.g. update_dependencies.rb or ripplor.rb) then you must install Ruby, RubyGems and Rubygem choice:

  1. To install Ruby use the command sudo port install ruby (on Mac);
  2. To install RubyGems use the command sudo port install rb-rubygems (on Mac);
  3. To install Rubygem choice use the command sudo gem install choice (on Mac).

Building Virgo Build

Building virgo-build-tools

Virgo Build, as of version 2.15, introduces p2 tools assembly and common targets and properties to make its consumption easy from the git repositories in the build chain. The p2 tools assembly and validation happens as part of a normal build of Virgo Build. In future other tools may be added, which is why the tools assembly carries the generic virgo-build-tools name.

Virgo Build isn't included in the normal build chain, so whenever there are changes to virgo-build-tools a manual rebuild and publish has to be executed, like so:

ant clean clean-integration test test-e2e

These targets assemble and validate virgo-build-tools. The target test-e2e has a dependency to package, so "package" need not be specified. test-e2e stands for end-to-end test execution: it is all performed in ant, because there is a small risk of stumbling into cyclic dependencies if we have a reference from virgo-build to the test framework. Another benefit of testing in ant is that this target exercises the exact same p2-related Ant targets other repositories will call to publish, install and uninstall.

Releasing virgo-build-tools

Virgo Build is not part of the normal release process. It is executed on demand and manually. We don't need to release new version along with the other repositories if there are no changes to virgo-build-tools.

Releasing the tools is really simple and similar to "Releasing a single repository project" described in this page. All you have to do is:

1. Tag the latest commit:

git tag VBT-1.1.0

Use the syntax VBT-x.y.z for the tags.

2. Update the following properties in Virgo Build's build.properties:

version=1.1.0
release.type=release
build.stamp=RELEASE

When making a release the version stays the same - it is incremented after releasing. What's important is to update the other properties.

3. Execute the following targets:

ant clean clean-integration clean-ivy test test-e2e publish-build

This publishes a RELEASE version of the virgo-build-tools assembly to build.eclipse.org.

4. Edit "build.properties" to increment the version, update release.type properties, and remove the build.stamp property:

version=1.2.0
release.type=integration

5. Commit the updated version and push:

git commit -m 'Build properties for 1.2.0 integration'
git push origin

At this point all repositories consuming the virgo-build-tools assembly can be updated to the newly released version.

Virgo-build-tools prerequisites

VBT released version requires
1.0.0 2.15 or higher
1.1.0 2.15 or higher
1.2.0 2.19 or higher
1.2.1 2.19 or higher
1.3.0 2.22 or higher
1.4.0 2.32 or higher
1.4.1 2.32 or higher

Building Individual Repositories

To build a given git repository, first clone it:

git clone <repository URL>

and then make sure that the Virgo build submodule is correctly initialised by changing directory to the cloned repository and issuing:

git submodule update --init

then change directory into the build-xxx directory and invoke ant with the appropriate targets, usually:

ant clean clean-integration test

This will do a fresh build and run all the unit and integration tests. It will only report "Build successful" if everything compiles cleanly and all the tests pass. The directory build-xxx/target/test-results contains the test results in html and xml formats. You will find the built bundles in the target/artifact directory of each individual project.

Running all tests

Sometimes it is useful to run all the tests even if some fail. You can do this by specifying the property -Dci.build=true on the ant command, for example:

ant -Dci.build=true clean clean-integration test

This will report "Build successful" regardless of test success or failure and you will need to look at the test results to see which tests failed.

Common targets

ant -p displays commonly used targets. Use ant clean-ivy to delete the ivy cache (or use git clean -dfx to get everything back to a reset state, including ditching any changes you haven't committed).

If you simply want to compile the runtime code, you can use ant clean clean-integration jar but that will not compile (or run) the tests.

ant precommit performs more extensive checks including code coverage and "findbugs". (You will need a licence for Clover to perform test coverage analysis.)

Exceptions to the above

Web Server

Building the packaged web server is somewhat different:

ant clean clean-integration jar package smoke-test

Stand-alone Runtime

You can also use the package target in the nano and kernel repositories to build a standalone runtime:

ant clean clean-integration test package

Documentation

For the web and pdf and other docs do:

ant doc

See also #Virgo Tooling Builds.

Samples

Samples just need packaging (some of them, such as Greenpages, produce a package that requires the user to build them using Maven):

ant clean clean-integration package

Publishing Development Builds

If you want to publish development build you may use the publish-ivy target (except for nano - see below):

ant clean clean-integration test package publish-ivy -Declipse.committerId=<committer id>

To use this build you need to update the dependencies of the referring project(s) with the new version you just published.

If you want to publish a development build of nano, use:

ant clean clean-integration test package publish-ivy publish-updatesite-build -Declipse.committerId=<committer id>

Example

For example, to build the test stubs component, issue the following commands:

>git clone git://git.eclipse.org/gitroot/virgo/org.eclipse.virgo.osgi-test-stubs.git 
>cd org.eclipse.virgo.osgi-test-stubs
>git submodule update --init
>cd build-osgi-stubs
>ant clean clean-integration test

After this has completed successfully, the project org.eclipse.virgo.teststubs.osgi contains the built binary and source jars in its target/artifacts directory.

Configuring p2 Publishing for Individual Repositories

Since version 2.15 Virgo Build has the capability to assemble build-tools that other repositories can use. So far these tools include only various p2 Eclipse applications.

To understand more about the virgo-build synergy with p2 and how we organized our products and packaging repositories look here.

In order to consume these tools there is a prerequisite. You have to include two references in the project's build.versions file. One for the virgo-build-tools and one for the org.eclipse.equinox.launcher used to launch the tools.

org.eclipse.virgo.build.tools=1.2.1.RELEASE

The next step is to fetch virgo-build-tools. You just need to call the predefined target for that. Here's how

<antcall target="fetch-unzipped-virgo-build-tools"/>

This call fetches and unzips the virgo-build-tools of the specified version in the project's "${target.dir}/virgo-build-tools-${org.eclipse.virgo.build.tools}" directory.

Another way to achieve the same results is to add this ant target as a dependency to a target of your choice in which you plan to use any of the virgo-build-tools common targets. Here's how

<target name="p2.publish" depends="package, fetch-unzipped-virgo-build-tools" description="Publishes bundles, features and product">

Here's a list of all available p2 tools targets that can be consumed

 p2.generate-inf                  Generates p2.inf-s for all feature directories inside a wrapping 'features' directory.
 p2.install-iu                    Installs one or more IUs to a target p2 installation's profile.
 p2.install-virgo-product         Installs a Virgo product to a desired destination. Default profile(roaming) is VIRGOProfile, the environment cofigurations are ANY.
 p2.mirror-artifacts              Mirrors an artifact set of rootIUs from a local or remote p2 repository to a local destination. The artifacts mirror app must be ran after the metadata one.
 p2.mirror-metadata               Mirrors a metadata set of rootIUs from a local or remote p2 repository to a local destination. The metadata mirror app must be ran first.
 p2.publish-binary                Publishes a single zipped artifact with unzip instructions to a p2 repository.
 p2.publish-binary-chmod          Publishes a single zipped artifact with proper unzip and permissions instructions to a p2 repository.
 p2.publish-bundles-features      Publishes bundles and features located in separate folders in the source location to a p2 repository.
 p2.publish-bundles-features-alt  Alternative form. Publishes bundles and features passed as separate sources.
 p2.publish-category              Categorizes features published in a p2 repository.
 p2.publish-product               Publishes a product to a p2 repository. The publishing uses ANY environment configurations.
 p2.uninstall-iu                  Uninstalls one or more IUs to a target p2 installation's profile.
 convert-p2-to-ivy                Prepares p2 repository artifacts for Ivy publishing

And an example of how to use each in the project's build.xml file

@Since 2.22
<antcall target="convert-p2-to-ivy">
  <param name="tools.dir" value="${package.output.dir}"/>
  <param name="source.plugins.dir" value="${mirrored.repository.dir}/plugins"/>
  <param name="destination" value="${converted.repository.dir}"/>
  <param name="offset.to.virgo.build" value="../../../..virgo-build"/>
  <param name="updated.build.versions.location" value="${converted.repository.dir}/build.versions"/>
</antcall>
@Since 2.22
<antcall target="p2.mirror-metadata">
  <param name="tools.dir" value="${package.output.dir}"/>
  <param name="source" value="file:${repository.dir}"/>
  <param name="destination" value="${mirrored.repository.dir}"/>
  <param name="roots" value="testBundle"/>
</antcall>
@Since 2.22
<antcall target="p2.mirror-artifacts">
  <param name="tools.dir" value="${package.output.dir}"/>
  <param name="source" value="file:${repository.dir}"/>
  <param name="destination" value="${mirrored.repository.dir}"/>
  <param name="roots" value="testBundle"/>
</antcall>
@Since 2.19
<antcall target="p2.generate-inf">
  <param name="source" value="${target.dir}/assembly/features"/>
</antcall>
@Since 2.15
<antcall target="p2.publish-binary">
  <param name="repository" value="${repository.dir}"/>
  <param name="source" value="${resources.location}"/>
</antcall>
@Since 2.15

<antcall target="p2.publish-binary-chmod">

 <param name="repository" value="${repository.dir}"/>
 <param name="source" value="${resources.location}"/>
 <param name="chmod.args" value="exampleFile.sh@/exampleLocation#755"/>
</antcall>

chmod.args accepts values with file@location#permission syntax. Where location is the path where the file is located relative to the root of the zip. For example if you want to give permissions to a file in the zip's root, location is must be /. Another example is /directory/example.

@Since 2.15
<antcall target="p2.publish-bundles-features">
  <param name="repository" value="${repository.dir}"/>
  <param name="source" value="${target.dir}/assembly"/>
</antcall>
@Since 2.16

<antcall target="p2.publish-bundles-features-alt">

 <param name="repository" value="${repository.dir}"/>
 <param name="bundles" value="${bundles.dir}"/>
 <param name="features" value="${features.dir}"/>
</antcall>
@Since 2.31

<antcall target="p2.publish-product">

 <param name="repository" value="${repository.dir}"/>
 <param name="product.file.location" value="${p2.products.location}/example.product"/>
 <param name="javaprofile.location" value="${javaprofile.location}/java-server.profile"/>
</antcall>
@Since 2.15

<antcall target="p2.publish-category">

 <param name="repository" value="${repository.dir}"/>
 <param name="category.file.location" value="${p2.category.dir}/category.xml"/>
</antcall>
@Since 2.15

<antcall target="p2.install-virgo-product">

 <param name="repository" value="${repository.dir}"/>
 <param name="destination" value="${package.output.dir}"/>
 <param name="product.iu" value="example.product"/>
</antcall>
@Since 2.15

<antcall target="p2.uninstall-iu">

 <param name="repository" value="${repository.dir}"/>
 <param name="destination" value="${destination.dir}"/>
 <param name="iu" value="exampleIU"/>
</antcall>
@Since 2.15

<antcall target="p2.install-iu">

 <param name="repository" value="${repository.dir}"/>
 <param name="destination" value="${destination.dir}"/>
 <param name="iu" value="exampleIU"/>
</antcall>

A real-world example is the build.xml of the Nano repository found here.

Updating Dependencies

In general, dependencies are declared in files named ivy.xml.

Dependencies between Virgo components are a special case of this. The properties file build.versions in the root directory of each git repository defines the versions of the Virgo components that the build will download. Updating any of the versions in build.versions is error-prone as the same values occur in certain other files (which cannot use property substitution). To update a version in build.versions, run the "update dependency" ruby script in the root folder of the git repository, for example:

~/virgo/web-server/scripts/update-dependency.rb -v <variable> -n <new version> 

where <variable> is the relevant property name in build.versions and <new version> is the replacement version. The script will report the changes it makes and may issue warnings, so pay attention to its output.

Please ensure you are running the latest version of update-dependency script by updating web-server git repository.

Updating Eclipse dependencies

Virgo can consume artifacts from both Eclipse p2 repositories and Orbit. The Juno contribution is even required to consume the same artifacts that are present in Orbit and the other Juno-contributed p2 repositories. As of version 2.22 virgo-build is capable of doing this by introducing new ant targets and a new git repository called "eclipse-mirror". Building this repository mirrors the artifacts from a specified p2 repository, adds Ivy metadata and publishes them to the Ivy repositories at build.eclipse.org. From then on it's business as usual for virgo-build.

Virgo's Eclipse dependencies are updated manually, on-demand, without this being an automatic step of a ripple or a release process.

Here's how to do it:

1. The build.properties file in the eclipse-mirror repository contains two important properties

mirror.source - the URI to the p2 repository from which we will mirror
mirror.ius - a coma-separated list of symbolic names that we want to mirror locally

Prior triggering the update process these properties must be set with the desired values. Note the source bundles need to be described in the IU list as well if they are needed.

Here's an example of both properties:

mirror.ius=\
   org.eclipse.osgi,\
   org.eclipse.osgi.source
mirror.source=http://download.eclipse.org/releases/juno

Local locations are accepted as well:

mirror.source=file:/<location-to-local-p2-repo>


2. Trigger ant build of the eclipse-mirror repository like this:

ant clean update-eclipse-artifacts

Even if some versions are already published the process will overwrite them, so it won't fail. As a result of this operation a new mirrored.build.versions file is generated in the repository root that contains the newly mirrored versions. In order to add them for propagation during a ripple copy the desired versions in the build.versions file of the eclipse-mirror repository.

All Eclipse artifacts are published with the distinctive Ivy organisation: "org.eclipse.virgo.mirrored"

3. Trigger a ripple or release to let the scripts propagate the updated versions upstream.

Important:

- The conversion from p2 to Ivy currently ignores any dependencies between bundles. Respecting these dependencies is currently under development but until then the mirrored and converted bundles and their dependencies need to be referenced explicitly.

- If the specified mirror.source contains more than one version of an IU all of them will be mirrored and converted which may lead to problems with the versions propagation. Use source repositories with single version, e.g. the release train repositories. Orbit usually contains more than one version, and often we are not using the latest one available so mirroring Orbit involves a manual step of choosing which version to use and deleting the others from the resulting build.versions file.

- Keep in mind the time to mirror a large number of artifacts depends on your network and how well you can connect to Eclpse servers. Usually it takes some time, so go treat yourself with a cup of coffee :-)

Hot-fixing third party dependencies

Sometimes we need to hot-fix an already used third party dependency until the upstream project produces an official release of this dependency with the desired fix. In order to do that first we must request the "modified" keyword added to the corresponding CQ. Then we make the fix, produce a jar or patch the original with the fixed class.

When you patch a jar make sure that it's new version qualifier ends with -virgo-<index>, where <index> is the modifications count. For instance the first modification of logback would be ch.qos.logback-1.0.0.timestamp-virgo-1.

If you produce a new jar through a build make sure to modify its filename and manifest to match the current version used in the build and the add the qualifier extension to it.

To make this hot-fixed version available for virgo-build follow these steps:

1. Clone the eclipse-mirror repo

2. Place the fixed jar into eclipse-mirror-root/hotfix/plugins

3. Place the fixed sources into eclipse-mirror-root/hotfix/source/<project-name>

4. Place a README file in eclipse-mirror-root/hotfix/source/<project-name> that points to the original source repository with the right tag

5. Go into build-eclipse-mirror and execute

ant clean update-hotfixed-artifacts

6. Update the build.versions file with the new versions from the hotfixed.build.versions file

That's it. Now you can refer to the hot-fixed artifacts from Ivy xmls.

Rippling

Since the Virgo components form a directed, acyclic graph, changing the graph to use a new version of a specific component involves not only updating that version throughout the graph, but also updating the versions of the components which have been changed. This is the tedious process known as rippling and so the ripplor ruby script was written to automate it. Essentially, Ripplor handles the most commonly updated components and flattens the dependency graph into a linear sequence. The sequence is (at the time of writing):

eclipse-mirror
util
test
medic
nano
artifact-repository
kernel
kernel-tools
web
snaps
apps
documentation
packaging

So, for example, if you change the kernel and want to produce a package web server containing the updated kernel, you would run:

~/virgo/web-server/scripts/ripplor/ripplor.rb -r kernel

This would build and publish a new version of the kernel, update the dependency of the web layer on the newly built kernel, build and publish a new version of the web layer, update the dependency of the Virgo-supplied applications on the newly built web layer and kernel, and so on until it builds and publishes a new version of the packaged web-server. This is called a ripple from kernel.

Ripplor (after 13May2010) allows a ripple to be executed across a list of repositories some in git.eclipse.org and some in git.springsource.org, though all of them are in git.eclipse.org at the time of writing.

The definitive list of repositories and the order of a ripple is stored in the ripplor.rb ruby script source.

Please ensure you are running the latest version of ripplor script by updating web-server git repository.

When should Ripplor be used?

In general, Ripplor is used any time a change needs to be propagated up the graph of dependent repositories. There are two main cases that cause this to happen.

  1. An external dependency needs to be upgraded in all modules of Virgo
  2. A change has been made to code in a repository (e.g. the kernel) which affects its externally visible behaviour. The code above (e.g. the web layer, Virgo-supplied applications, etc.) needs to be updated to see the changes.

Usage

Usage: ripplor.rb [-rbvmuc]

Required arguments:
    -r, --repo=REPO                  The name of the starting repo

Optional arguments:
    -b, --virgo-build-version=VIRGO-BUILD-VERSION
                                     The version to update Virgo Build to
    -v, --version=VARIABLE:VERSION[,...]
                                     Versions to substitute during the ripple
    -m, --map=REPO-MAP               The property file containing a mapping from a repository name to a location
                                     (defaults to ~/repository.map)
    -u, --remote-user=REMOTE-USER    User id to use for remote repository access
    -c, --branch=BRANCH-NAME         The branch to be rippled
                                     (defaults to master)
        --dry-run                    Show what would happen but do not actually do anything
 Option  Description
 -r The name of the repository you want to start the ripple at. A typical repository name is util or kernel.
 -v When updating a dependency use this flag to describe both the variable to update and the new version to update to. Can be a comma separated list if multiple dependencies need updating at the same time.
 -m The location of the repository map file (see below). By default, this file is found in ~/repository.map, but in cases where you have more than one file (one for simple ripples, one for complex ripples that happen in your working copy) you may need to pass in this value.
 -u The login name used for ssh access to the git.eclipse.org server. By default, this is the local login name.
 -c The branch to be rippled. By default, this is master.

Repository Map

By default, Ripplor checks out each repository working copy to the working directory that it was run from. If you are prepared to run Ripplor in a dedicated working directory, this is fine. However, if you run a ripple from within your development shell environment it can pollute your filesystem. To account for this, Ripplor takes a repository map file. This file tells Ripplor where to checkout a repository working copy to. The file itself is a simple mapping from a repository name to a filesystem location.
osgi-extensions = ~/dev/ripplor/osgi-extensions
util = ~/dev/ripplor/util
...
Tip: In the repository map file, ~ will be expanded to the current user’s home directory.

Broken Build

Ripplor was smart enough to restart a failed ripple.

[If a ripple fails (due to test or compile failure), simply fix any problems in the working copy that Ripplor is using and restart the ripple at the same location you originally used. Ripplor will bring in any new commits since you last started the ripple and continue up the stack as done previously. This behaviour is very useful if you are rippling a breaking API change and need to fix code as the ripple moves upwards.]

How to rebuild a broken rippled repository

Use the normally executed for that repository targets plus one additional -D parameter:

ant -Dvirgo.deps.location=integration-repo clean clean-integration test

The important part here is to add the -Dvirgo.deps.location parameter. Its default value is "ivy-cache". It is automatically set by the ruby scripts. It is also not needed during a single repository build. Its purpose is to point the integration tests to the right location of the Virgo artifacts. During a ripple - the integration repo, during a single repo build - the ivy-cache.

Unfortunately this behaviour is broken, since the version updates (accumulated during the ripple and propagated up) are not correctly restored. Also, when committing the changes (to all the repositories) at the end, after a failure in the middle, not all of the rippled repositories will be committed at the end.

Re-Building

To re-start ripplor and fix a broken build you can use two strategies:

Restart from lowest

The standard solution at the moment is to push the repositories that succeeded by hand and start another ripple, with version updates that are likely to be lost inserted by hand on the ripplor command line. This is not perfect and a repeat of the ripple from the lowest point is safer.

A full ripple of all of the repositories takes an hour and a half or more, even on a high-end Mac.

Safe/Lucky points

Ripplor can be restarted from repository that contains all the version updates so far (this may be called a safe or lucky point). The repositories that contain all version updates:

  • medic
  • kernel
  • web
  • web-server

For example if the build broke at snaps you can re-start from the last successfully built safe repository. In our example this is the web repo.

Examples

Upgrading the version of Spring used in util and upward:

./ripplor.rb -r util -v "org.springframework:3.1.0.RELEASE" -m ~/ripple.repository.map

Rippling a breaking API change to kernel

./ripplor.rb -r kernel -m ~/workingcopy.repository.map

Releasing a Single Repository Project

Simply follow these steps to produce a release of a product that is within a single Git repository. The aim is to have a tag of the right name pointing to the codebase at release time and to leave master or the current development branch version at the next version ready for development to continue. Only committers can produce releases.

1. Create a new branch and make sure everything is up to date and clean

      git pull
   git clean -f -d
   git submodule update --init
   git checkout -b release 

2. Update the build.properties file to have the right version, have a release.type of 'release' (or 'milestone' if appropriate), and have a build.stamp with the appropriate version qualifier. For example:

      version=1.1.0
   release.type=release
   build.stamp=RELEASE

3. Do the build, commit the changes and then tag it with the version and release type appropriate. In order for publishing to eclipse to work you must have your ssh public key registered with the virgoBuild account.

      cd build-xxx
   ant clean clean-integration clean-ivy test package publish-ivy publish-maven -Declipse.committerId=gnormington
   cd ..
   git add .
   git commit -m 'Build properties for the release 1.1.0.RELEASE'
   git tag 1.1.0.RELEASE 

Note: if you need to publish to p2, add the target publish-updatesite-download to the above list.

Note: if you are building bundlor, until bug 384139 is fixed, remember to update the version in org.eclipse.virgo.bundlor.maven/src/main/resources/META-INF/maven/plugin.xml and follow these instructions for updating the composite p2 repository. Also, until bug 384138 is fixed the update site needs manually relocating.

4. After the tag go back to master and delete the release branch. The branch will still be available from the tag.

      git checkout master
   git branch -D release

5. Back on master the build.properties file needs to be revved up to the following release for development to continue. Ensure the release type is still integration and that the build.stamp property is not present. Set the version to the following version of the product.

      version=1.2.0
   release.type=integration

6. Finally add, commit and push it all back up to origin. Ensure the new tag is also pushed up.

      git add .
   git commit
   git push origin
   git push --tags

Once this is done the usual steps to make the release available on download servers and to make announcements should be followed. For Eclipse, a committer with a full shell should sftp in to build.eclipse.org and beam the file up. More information available here.

Releasing a Multi Repository Project

This is for any release of the full Virgo Server. It makes use of the Releasolor ruby script. This is very similar to Ripplor but it also performs the steps above for a single repository to all the repositories it visits. Only committers can produce releases.

The usage instructions are:

   Usage: releaselor.rb [-vqtnbmurcghjkd]
Required arguments:
  -v, --version=VERSION            The version number of the release
  -q, --build-stamp=BUILD-STAMP    The build stamp for the release, e.g. M01, RELEASE
  -t, --release-type=RELEASE-TYPE  The release type for the release, e.g. milestone, release
  -n, --new-version=NEW-VERSION    The new version number to be used after the release

Optional arguments:

   -b, --virgo-build-version=VIRGO-BUILD-VERSION
                                   The version to update Virgo Build to
  -m, --map=REPOSITORY-MAP         The property file containing a mapping from a repository name to a location
                                   (defaults to ~/repository.map)
  -u, --remote-user=REMOTE-USER    User id to use for remote repository access
                                   (defaults to local login id)
  -r, --product-release=PRODUCT    The product to be released, kernel, web-server, virgo, full-product
                                   (defaults to releasing all the virgo repos)
  -c, --branch=BRANCH-NAME         The branch to be released
                                   (defaults to master)
  -g, --gemini-version=GEMINI-BUILD-VERSION
                                   When producing a full-product build the Gemini version must be given
  -h, --gemini-build-stamp=GEMINI-BUILD-STAMP
                                   The build stamp for the Gemini Web release, e.g. M01, RELEASE
  -j, --gemini-release-type=GEMINI-RELEASE-TYPE
                                   The release type for the Gemini Web release, e.g. milestone, release
  -k, --gemini-new-version=GEMINI-NEW-VERSION
                                   The new Gemini Web version number to be used after the release
  -d, --gemini-branch=BRANCH-NAME  The Gemini Web branch to be released
                                   (defaults to master)
      --dry-run                    Show what would happen but do not actually do anything


An example invocation for a release is:

       ./releaselor.rb -v 2.0.1 -s RELEASE -t release -n 2.0.2 -u cfrost -r kernel

This will release the Kernel only at 2.0.1 and leave the repos versioned 2.0.2 for work to continue after the release. Repos other than those needed for the kernel will not be modified.

For 3.0 release including Gemini Web at 2.0.1 you may use:

       ./releaselor.rb -v 3.0.0 -q RELEASE -t release -n 3.0.1 -c 3.0.x -g 2.0.0 -h RELEASE -j release -k 2.0.1 -d 2.0.x -r full-product -u cfrost

An example invocations for a milestone including Gemini Web is:

       ./releaselor.rb -v 3.0.0 -q M01 -t milestone -n 3.0.0 -g 1.2.0 -h M02 -j milestone -k 1.2.0 -u cfrost -r full-product

An example invocation for a release not including Gemini Web is:

       ./releaselor.rb -v 3.0.0 -q RELEASE -t release -n 3.1.0 -u cfrost -r virgo

Note.png
Publishing the Gemini Zips
If you do a 'full-product' release, Gemini Web will also be released. There will be a manual step to publish the Gemini zips to download.eclipse.org as they need to go in a different directory than the rest of Virgo. The Zips will be built as normal and will be available from the 'build-web-container' directory.


Note.png
Updating Plan versions
When doing a full release there are some plan files that need manually editing.
   org.eclipse.virgokernel.userregion.springdm (Kernel repo in the build-kernel/repository/ext directory)
   org.eclipse.virgo.apps.admin.plan (Apps repo in the org.eclipse.virgo.apps.admin/src directory)
   org.eclipse.virgo.web (Web repo in the org.eclipse.virgo.web.tomcat/src directory)
In addition, the abstract integration test in the web.test project and AdminConsoleAppTests in the system-verification-tests repository (still to be contributed at the time of writing) also need to be updated to use the new version for the tests to pass. Ideally the script that does version updating should cover this but in doing so it would no longer be generic. Ideas welcome.

Virgo Tooling Builds

Hudson Builds

Virgo Tooling for the Eclipse IDE is built using the same toolchain as the rest of the Virgo project. The job can be found here: https://hudson.eclipse.org/hudson/job/virgo.ide.snapshot/.

[How to build from the command line?]

Building in Eclipse

If you want to explore, modify or contribute to the Virgo Tools, follow these steps to setup your workspace:

  1. In most (but not all) cases, you'll want to have JEE and WTP features installed.
  2. Clone the git repos at http://git.eclipse.org/c/virgo/org.eclipse.virgo.ide.git/
  3. Import all of the projects within the repos.
  4. Add Virgo IDE dependencies to your target platform [What if I haven't got a target platform? How do I create one? Does it need any special contents or configuration?]. The best way to do that is to add the composite update site at http://download.eclipse.org/virgo/snapshot/tooling to your target platform and select the following items only. You'll need to uncheck "Group Items by Category" to see all of these.
    1. All Libra Features
    2. Eclipse Virgo Bundlor
    3. Eclipse Virgo Kernel
    4. Eclipse Virgo Medic
    5. Eclipse Virgo Utils

To launch the Virgo tools, just

  1. Add all features but *..tests to an Eclipse runtime.
  2. Add any WTP or JEE features appropriate for your target environment.
  3. Be sure to select "Add Required Plugins".

Building Documentation

The Eclipse documentation builds from the same documentation source as the rest of the Virgo build, all of which is maintained in the documentation repos. Currently, Eclipse doc builds are invoked manually. See this bug for more information.

  1. If you do not already have it, clone the documentation repos at: http://git.eclipse.org/c/virgo/org.eclipse.virgo.documentation.git/.
  2. The build should be in a parallel directory to your tooling repos. Alternatively, you can set the virgo.docs.dir property to the location of the documentation repos.
  3. Optional: Update the Wiki Docs, see below.
  4. From org.eclipse.virgo.ide.doc, execute "Build Virgo Docs" in External Tools. (This launches the build.xml using the workspace JRE.)

Updating Wiki Text Documentation

We are also re-providing this Wiki documentation as part of the Eclipse Virgo feature. Unfortunately, we aren't able to update those documents automatically, as that would require committing changes into git, and resolving ordering issues there. Currently, the process works like this:

  1. Execute ant build.xml in wiki-guide. This:
    1. Causes the Wiki Text to be built for use in the Virgo IDE Documentation.
    2. Builds docbook xml for use in the tooling-guide, and copies the relevant XML to that guide.
  2. Execute the IDE build as described above.
  3. The Virgo documentation will be updated with the tooling guide additions during the next scheduled Virgo docs build.

Editing Media Wiki Documents

Note that there is now mediawiki support for the tooling documentation. This means that you can use mediawiki to write documentation, and we can easily crowd-source the latest documentation from the Virgo wiki pages. You can mix and match mediawiki documentation with ordinary docbook documentation. Just be sure not to edit mediawiki produced docs in the wikixml directory directly as they will be overwritten ! To build the wiki docs, just call:

ant wiki-to-docbook
  • Each wikitext document is a chapter.
  • To add a new chapter to the tooling docs, edit src/virgo-tooling-guide and add the chapter to the xi:include entries with the path ../wikixml/your_chapter.xml.

(This is not a build time task, it's simply an aid for creating the entire documentation set.)