Difference between revisions of "Virgo/Committers/Process Advice"

From Eclipsepedia

Jump to: navigation, search
m (Product inheritance)
(Understanding virgo-build's synergy with p2)
Line 52: Line 52:
The content from '''''publish_resources''''' is not ready for publishing that's why it is moved to ${target.dir} and prepared in the build.xmls for the publishing step.
The content from '''''publish_resources''''' is not ready for publishing that's why it is moved to ${target.dir} and prepared in the build.xmls for the publishing step.
===== Download packaged p2 repo =====
'''In this step a zipped p2 repo is downloaded from build.eclipse.org. As the build chain progresses this repository is appended with new artifacts and re-uploaded back to build.eclipse.org. The final packaging repository in the chain (Jetty) uploads an unzipped p2 repository to download.eclipse.org. This final repository contains all Virgo products.'''
===== Preparing zipped resources for publishing =====
===== Preparing zipped resources for publishing =====

Revision as of 09:29, 31 January 2012


Development advice for committers

These sections are incomplete. Please either supply the required details or request more information if you require it.

ssh keys; why and how to get them

Set up ssh keys so that you are not prompted for a password at many and/or inconvenient points.

  • git.eclipse.org - instructions here
  • build.eclipse.org - same procedure as for git.eclipse.org

To request a full shell, either raise a bugzilla or email webmaster@eclipse.org. Probably best to email and ask for a full shell on both the above servers.

how to subscribe to virgo-inbox

You need to do this so you are notified of new bugs.

  • Open bugzilla in a browser.
  • Click on Preferences.
  • Click on the Email Preferences tab.
  • Add virgo-inbox@eclipse.org to the list of users to watch. gemini.web-inbox@eclipse.org and equinox.framework-inbox@eclipse.org may also be of interest.

how to fix a bug

All our enhancements and bugs (in fact, every change that happens to the product or the code base) start life with a bugzilla. We generically refer to them all as bugs. Bugzilla entries should record any information associated with the change.

To make it easier to share the work it is helpful to follow some simple guidelines:

  • starting work on a bug
    • assign it to yourself while you are working on it: change the "Assigned to" field to contain your email address and change the status to ASSIGNED.
    • if you make a code change for the bug, ensure the commit log message begins with "bug nnnnnn: "
  • giving up on a bug
    • if you want to pass the bug to someone else to work on, please change the "Assigned to" field.
    • if you want to put the bug back in the backlog, please change the "Assigned to" field to "virgo-inbox@eclipse.org" and change the status to NEW.
  • closing a bug
    • If you want to remember to make sure the raiser validates the fix, change the status to RESOLVED and let the raiser change it to CLOSED.
    • if the fix is straightforward and you don't want to involve the raiser, change the status to CLOSED.

what is a ripple and when to do one

See Rippling.

Understanding virgo-build's synergy with p2

As of 3.5.0 virgo-build produces its own build-tools that leverage p2 technology. P2 is now actively involved in the packaging process of virgo-build and the build-tools are used in every packaging repository (excluding snaps). There are a number of steps that are performed over a packaging repository during builds.

An overview is shown below:


The assembly output zips from every Virgo server packaging repository are in fact p2 initial installations from the published Virgo update site and then zipped into the assembly file. The provisioning itself installs only the kernel region content in the OSGi framework. Common for every packaging repository is the publish_resources directory. It contains p2 metadata that will be used to publish the Virgo update site. The publish_resources directory contains features and product files, sometimes categories too.

The content from publish_resources is not ready for publishing that's why it is moved to ${target.dir} and prepared in the build.xmls for the publishing step.

Download packaged p2 repo

In this step a zipped p2 repo is downloaded from build.eclipse.org. As the build chain progresses this repository is appended with new artifacts and re-uploaded back to build.eclipse.org. The final packaging repository in the chain (Jetty) uploads an unzipped p2 repository to download.eclipse.org. This final repository contains all Virgo products.

Preparing zipped resources for publishing

In this step the resource content(configuration files, scripts, bundles, etc.) is assembled in a directory. The user region content is installed as a resource that is unzipped as part of the installation - this is how the /repository directory is populated.

Prepare p2 repo plugins and features for publishing

Here the features from publish_resources are copied to ${target.dir} substituting their version placeholders along the way.

Understanding features

Features mainly consist of a feature.xml - that is the only mandatory part. A feature.xml file contains references to other bundles or plugins. It is important to understand that a feature ideally should contain only the bundles that form its semantic core, not their dependencies too. For example in order to have DS working we need three bundles, but their feature only includes the single DS bundle itself. The feature dependencies still need to be published in the Virgo update site so a feature install operation can drag them over to the targeted runtime. There are other accompanying files to a feature.xml such as properties and license. These are not mandatory but I believe are standard practice within Eclipse.

It is highly recommended to refer bundles with exact versions from their grouping feature.

Optional dependencies and fragments in features

From 3.8 optional dependencies are not installed automatically in p2. This means if you have an optional dependency you want to add you need to include it explicitly in your feature.

OSGi fragments are not automatically installed by p2, therefore if you want to include a fragment in you installation that isn't required by anyone you need to explicitly include it in your feature.xml file.

Understanding products

Products are the highest level p2 grouping artifact that fully describes a runtime. Conceptually a major difference between products and features is their completeness. Products should be transitively complete in order to ensure successful initial provisioning. On the other hand features are designed to be installed on top of existing runtimes usually materialized from products. Products are defined by a single yourname.product file. They contains more complete information of a product such as required features and runtime configurations. It can also directly require bundles(plugins) but this requires additional configuration and isn't used in any of our current products. The configurations can contain system properties that are written in the config.ini file during install and can also contain autostart and start-level information about bundles.

Product inheritance

Virgo's products are organized in such a way that they are inherited to reuse the definition of their content. The inheritance itself is realized by a p2.inf instruction added to a product saying this product requires another one. Here's an example containing the nano.product and the instructions for it to require the base.product IU.

You'll notice there is more than one product per packaging repository. The reason for this lies in the specific for a packaging repository resources. It isn't a good idea to propagate them to higher packaging repositories so there is one product without the specific resources that higher packaging repositories inherit and one product that includes the specific resources that is used to produce the assembly zip of this packaging repository.

Here's a list of products that are reused by the packaging repositories:

Packaging Repo inherits
nano n/a (first repo in chain)
kernel base.product from nano
web/jetty kernel-base.product from kernel

Understanding p2.inf files

A p2.inf file contributes more information to the p2 publisher helping us achieve a more precisely configured result. This instruction file can modify almost anything about a published InstallableUnit (a.k.a IU) and is quite powerful. Examples are modify an IU's requirements, what it provides, add action instructions to the IU that will be executed on various phases of the provisioning process such as unzip on the install phase or change the permissions on the configure phase. More details on it can be found here. For features and products the accompanying p2.inf file must be placed in the same directory as the feature.xml and the yourname.product file.

In our scenarios we use p2.inf files only for the product inheritance because they are simple enough to handle them manually. The features' more complex p2.inf files that contain autostart instructions are generated by the p2.generate-inf tool.

In principle you should be extremely careful when handling p2.inf files. I would recommend to stick to the auto-generation tool (ant target: p2.generate-inf, described here - it is sufficient for the Virgo use cases.

Managing bundle startup

You might have noticed that bundle startup seems to be manageable both from the product file and a p2.inf file. We utilise both approaches. Normally an Eclipse product contains the autostart information of all bundles with the features only describing the included bundles. When new features are provisioned Eclipse autostarts all new bundles and their depependencies. This approach firstly has the downfall of having performance overhead and secondly it may lead to activating bundles that really must have been just RESOLVED and that could cause inconsistencies and clashes if activated in an inappropriate environment. Another downfall of this approach is that you can't take a feature and install it anywhere else and expect it to be properly starting only its necessary parts.

The chosen approach for Virgo would be to take the startup responsibilities down to the features. This has the benefits of

  • explicitly defined active parts of your feature so it can work properly, starting only what you need
  • reusing the feature ensures consistent runtime state on any environment

So the guideline here is to define in the feature.xml which bundles described in the feature should be started with the default start level. If you need any custom start level configuration define that in your product file. For example the medic bundles need to be started on start level 3 and this is described in nano's base.product

Auto-starting bundles

In order to autostart a bundle in a feature just include the autostart="true" attribute in the bundle's description like it is done here. This attribute is recognised by the p2.generate-inf ant target included in virgo-build 2.19 and virgo-build-tools-1.2.0 onward so make sure you execute this target over the features directory of your prepared for publishing features location. This location should be already configured for the packaging repositories and normally is in ${target.dir}/assembly/features. This will configure your bundle to autostart with the default start level.

In case you need to autostart bundles that are dependencies of your features add autostart configurations for them in the product file where they will be included. Use the product file for unusual start level settings too like it is done here.

Understanding publishing

In a nutshell the publishing process generates p2 metadata(content and artifacts.jar) for the Virgo artifacts. The input for publishing is a directory that contains a /plugins, /features and optionally /binaries. The /binaries directory contains the zipped resources that are published by the virgo-build-tools. All features are published in a jar form in the /features directory and all bundles must be places in the /plugins directory.

How to add new bundles (to /plugins)

To add new bundles to plugins you need to either create a new feature for them or include them in already existing feature depending on their semantics.

  • To add a new feature just copy an existing one and edit its feature.xml and feature.properties files. Remember that when describing plugins in a feature it is important to specify an exact version - 0.0.0 will complicate matters on the long run. An example is this feature where the nano's current build version is used correctly for plugin version.
  • Take your time determining which bundles of your feature you'd like to autostart, as described above in the Auto-starting bundles section.
  • Your next step is to require this feature from a product file.
  • Lastly you need to make sure to download via Ivy these bundle dependecies into the /plugins directory in your prepared for publishing location.
  • Normally the packaging repositories use ${target.dir}/assembly as their prepared publishing location.

If everything is well your feature will be installed as part of the product provisioning and your bundles will be downloaded as part of the provisioning in the /plugins folder and will be configured with the desired autostart instructions.

How to publish new zipped resources

Zipped resources are published in such a way that their content is unzipped in the root directory of your provisioning destination. If many zipped resources contain the same file it isn't determined in which order they will be unzipped.

Here's an example of preparing zipped resources for publishing. And here's an example of the actual publishing using the p2.publish-binary and p2.publish-binary-chmod targets.

Creating a release (milestone or higher)

The hardest and riskiest thing to do is run successfully the releaselor script as described here and here.

What remains if everything goes well is to update the online documentation (User guide, etc.) and the relevant web pages of the Virgo project.

Both are manually checked in to CVS.

CVS settings are as follows:

Repository: :extssh:<commiter_id>@dev.eclipse.org:/cvsroot/org.eclipse

Path: HEAD/www/virgo/documentation and HEAD/www/virgo/download

Here's a list of simple steps after the CVS access is configured:

1. Delete the current documentation from HEAD/www/virgo/documentation/version/docs/*

2. Upload on the same location the new documentation that was produced by the releaselor script in its working directory

3. Move the index.php file from HEAD/www/virgo/documentation/old_version to the new version folder you just created for the new release and change the links inside it to match the new version.

4. Create new release notes for the new version and upload them in HEAD/www/virgo/download/release-notes. Follow the naming standard of the existing release notes there.

5. Change index.php or milestones.php in HEAD/www/virgo/download depending on what release you are producing to include the new version on the download page.

Creating Tomcat binaries as bundles

Tomcat binaries do not provide OSGi metadata, so we use bundlor in order to make them bundles.

Tomcat binaries and sources can be obtained from the maven repository: http://repo2.maven.org/maven2/org/apache/tomcat/

For Tomcat 7.0.12 we use the following 9 jar files :