Jump to: navigation, search

Difference between revisions of "Virgo/Committers/Process Advice"

m
m
Line 44: Line 44:
 
As of 3.5.0 virgo-build produces its own build-tools that leverage p2 technology. P2 is now actively involved in the packaging process of virgo-build and the build-tools are used in every packaging repository (excluding snaps).
 
As of 3.5.0 virgo-build produces its own build-tools that leverage p2 technology. P2 is now actively involved in the packaging process of virgo-build and the build-tools are used in every packaging repository (excluding snaps).
 
There are a number of steps that are performed over a packaging repository during builds.
 
There are a number of steps that are performed over a packaging repository during builds.
 +
 
An overview is shown below:
 
An overview is shown below:
<place pic here>
+
[[Image:Packaging-repo.jpg|center|700px]]
  
 
Note that the assembly outcome zips from every Virgo server packaging repository are in fact p2 initial installations from the published Virgo update site and then zipped into the assembly file. The provisioning itself installs only the kernel region content in the OSGi framework. The user region content is installed as a resource that is unzipped as part of the installation - this is how the ''/repository'' directory is populated.
 
Note that the assembly outcome zips from every Virgo server packaging repository are in fact p2 initial installations from the published Virgo update site and then zipped into the assembly file. The provisioning itself installs only the kernel region content in the OSGi framework. The user region content is installed as a resource that is unzipped as part of the installation - this is how the ''/repository'' directory is populated.

Revision as of 03:58, 25 January 2012

Development advice for committers

These sections are incomplete. Please either supply the required details or request more information if you require it.

ssh keys; why and how to get them

Set up ssh keys so that you are not prompted for a password at many and/or inconvenient points.

  • git.eclipse.org - instructions here
  • build.eclipse.org - same procedure as for git.eclipse.org

To request a full shell, either raise a bugzilla or email webmaster@eclipse.org. Probably best to email and ask for a full shell on both the above servers.

how to subscribe to virgo-inbox

You need to do this so you are notified of new bugs.

  • Open bugzilla in a browser.
  • Click on Preferences.
  • Click on the Email Preferences tab.
  • Add virgo-inbox@eclipse.org to the list of users to watch. gemini.web-inbox@eclipse.org and equinox.framework-inbox@eclipse.org may also be of interest.

how to fix a bug

All our enhancements and bugs (in fact, every change that happens to the product or the code base) start life with a bugzilla. We generically refer to them all as bugs. Bugzilla entries should record any information associated with the change.

To make it easier to share the work it is helpful to follow some simple guidelines:

  • starting work on a bug
    • assign it to yourself while you are working on it: change the "Assigned to" field to contain your email address and change the status to ASSIGNED.
    • if you make a code change for the bug, ensure the commit log message begins with "bug nnnnnn: "
  • giving up on a bug
    • if you want to pass the bug to someone else to work on, please change the "Assigned to" field.
    • if you want to put the bug back in the backlog, please change the "Assigned to" field to "virgo-inbox@eclipse.org" and change the status to NEW.
  • closing a bug
    • If you want to remember to make sure the raiser validates the fix, change the status to RESOLVED and let the raiser change it to CLOSED.
    • if the fix is straightforward and you don't want to involve the raiser, change the status to CLOSED.

what is a ripple and when to do one

(don't)

Understanding virgo-build's synergy with p2

As of 3.5.0 virgo-build produces its own build-tools that leverage p2 technology. P2 is now actively involved in the packaging process of virgo-build and the build-tools are used in every packaging repository (excluding snaps). There are a number of steps that are performed over a packaging repository during builds.

An overview is shown below:

Packaging-repo.jpg

Note that the assembly outcome zips from every Virgo server packaging repository are in fact p2 initial installations from the published Virgo update site and then zipped into the assembly file. The provisioning itself installs only the kernel region content in the OSGi framework. The user region content is installed as a resource that is unzipped as part of the installation - this is how the /repository directory is populated. Common for every packaging repository is the publish_resources directory. It contains p2 metadata that will be used to publish the Virgo update site. The publish_resources directory contains features and product files, sometimes categories too.

Understanding features

Features mainly consist of a feature.xml - that is the only mandatory part. A feature.xml file contains references to other bundles or plugins. It is important to understand that a feature ideally should contain only the bundles that form its semantic core, not their dependencies too. For example in order to have DS working we need three bundles, but their feature only includes the single DS bundle itself. The feature dependencies still need to be published in the Virgo update site so a feature install operation can drag them over to the targeted runtime. There are other accompanying files to a feature.xml such as properties and license. These are not mandatory but I believe are standard practice within Eclipse.

Understanding products

Products are defined by a single yourname.product file. It contains more complete information of a product such as required features and runtime configurations. It can also directly require bundles(plugins) but cannot mix the two modes. Either features or bundles are required, not both. The configurations can contain system properties that are written in the config.ini file during install and can also contain autostart and start-level information about bundles.

Understanding p2.inf files

A p2.inf file contributes more information to the p2 publisher helping us achieve a more precisely configured result. This instruction file can modify almost anything about a published InstallableUnit (a.k.a IU) and is quite powerful. Examples are modify an IU's requirements, what it provides, add action instructions to the IU that will be executed on various phases of the provisioning process such ass unzip on the install phase or change the permissions on the configure phase. More details on it can be found here. For features and products the accompanying p2.inf file must be placed in the same directory as the feature.xml and the yourname.product file.

In principle you should be extremely careful when handling p2.inf files. I would recommend to stick to the auto-generation tool - it is sufficient for the Virgo use cases.

Managing bundle startup

You might have noticed that bundle startup seems to be manageable both from the product file and a p2.inf file. We utilise both approaches. Normally an Eclipse product contains the autostart information of all bundles with the features only describing the included bundles. When new features are provisioned Eclipse autostarts all new bundles and their depependencies. This approach firstly has the downfall of having performance overhead and secondly it may lead to activating bundles that really must have been just RESOLVED and that could cause inconsistencies and clashes if activated in an inappropriate environment. Another downfall of this approach is that you can't take a feature and install it anywhere else and expect it to be properly starting only its necessary parts.

The right approach for Virgo would be to take the startup responsibilities down to the features. This has the benefits of being able to define exactly which are the active parts of your feature so it can work properly, so starting only what you need. Another benefit is that installing this same feature to another system that doesn't start every incoming bundle too will result in a proper startup.

So the guideline here is to define in the feature what part of its semantic core must be started with the default start level. In the product file we will define startup instructions for bundles that are not included in any feature but features depend on them. Also the product file is the place for any non-default start level configurations. For example the medic bundles need to be started on start level 3 and this is described in nano's base.product

Understanding publishing

The input for publishing is a directory that contains a /plugins, /features and optionally /binaries. The /binaries directory contains the zipped resources that are published by the virgo-build-tools. All features are published in a jar form in the /features directory and all bundles must be places in the /plugins directory. As a result the p2 publisher produces content and artifacts repositories - optionally compressed - in the publishing directory.

Product inheritance

Virgo's products are organized in such a way that they are inherited to reuse the definition of their content. The inheritance itself is realized by a p2.inf instruction added to a product saying this product requires another one. Here's an example containing the nano.product and the instructions for it to require the base.product IU.

Here's a list of products that are reused by the packaging repositories:

Packaging Repo inherits
nano n/a (first repo in chain)
kernel base.product from nano
web/jetty kernel-base.product from kernel

Auto-starting bundles

In order to autostart a bundle in a feature just include the autostart="true" attribute in the bundle's description like it is done here. This attribute is recognised by the p2.generate-inf ant target included in virgo-build 2.19 and virgo-build-tools-1.2.0 onward so make sure you execute this target over the features directory of your prepared for publishing example. This should be already configured for the packaging repositories. This will configure your bundle to autostart with the default start level.

In case you need to autostart bundles that are dependencies of your features add autostart configurations for them in the product file where they will be included. Use the product file for unusual start level settings too like it is done here.

How to add new bundles (to /plugins)

To add new bundles to plugins you need to either create a new feature for them or include them in already existing feature depending on their semantics.

  • To add a new feature just copy an existing one and edit its feature.xml and feature.properties files.
  • Your next step is to require this feature from a product file.
  • Lastly you need to make sure to download via Ivy these bundle dependecies into the /plugins directory in your prepared for publishing location.
  • Normally the packaging repositories use ${target.dir}/assembly as their prepared publishing location.

If everything is well your feature will be installed as part of the product provisioning and your bundles will be downloaded as part of the provisioning in the /plugins folder and will be configured with the desired autostart instructions.

How to publish new zipped resources

Zipped resources are published in such a way that their content is unzipped in the root directory of your provisioning destination. Also note that if many zipped resources contain the same file it isn't determined in which order they will be unzipped.

Here's an example of preparing zipped resources for publishing. And here's an example of the actual publishing using the p2.publish-binary and p2.publish-binary-chmod targets.

Creating a release (milestone or higher)

The hardest and riskiest thing to do is run successfully the releaselor script as described here and here.

What remains if everything goes well is to update the online documentation (User guide, etc.) and the relevant web pages of the Virgo project.

Both are manually checked in to CVS.


CVS settings are as follows:

Repository: :extssh:<commiter_id>@dev.eclipse.org:/cvsroot/org.eclipse

Path: HEAD/www/virgo/documentation and HEAD/www/virgo/download


Here's a list of simple steps after the CVS access is configured:

1. Delete the current documentation from HEAD/www/virgo/documentation/version/docs/*

2. Upload on the same location the new documentation that was produced by the releaselor script in its working directory

3. Move the index.php file from HEAD/www/virgo/documentation/old_version to the new version folder you just created for the new release and change the links inside it to match the new version.

4. Create new release notes for the new version and upload them in HEAD/www/virgo/download/release-notes. Follow the naming standard of the existing release notes there.

5. Change index.php or milestones.php in HEAD/www/virgo/download depending on what release you are producing to include the new version on the download page.

Creating Tomcat binaries as bundles

Tomcat binaries do not provide OSGi metadata, so we use bundlor in order to make them bundles.

Tomcat binaries and sources can be obtained from the maven repository: http://repo2.maven.org/maven2/org/apache/tomcat/

For Tomcat 7.0.12 we use the following 9 jar files :