Jump to: navigation, search

Buckminster Aggregator User Guide

Introduction

The Aggregator is based on and part of the Buckminster project. Buckminster provides a versatile and adaptable framework supporting build, assembly and deployment processes. It supports a rich set of usecases. One of those - the aggregation of repositories - is the focus of the Buckminster Aggregator tool.

The Aggregator combines p2 repositories from various sources into a new aggregated p2 repository. It can also be configured to produce an aggregated repository that can be used as a p2 and Maven repository. There appears to be an increasing need for delivering aggregated repositories. The reasons vary from licensing issues to organisational requirements:

  1. Owners of a p2 repo for a given project may not be in position to host all required or recommended components due to licensing issues - Buckminster's SVN support can serve as an example here, as it requires components available through the main Eclipse p2 repo as well as third-party components. Hence users have to visit several repos for a complete install.
  2. Projects want to provide convenient access to their products - Installation instructions requiring the user to visit several repos for a complete install are not uncommon. An aggregated repo for all those locations provides a convenient one-stop strategy. The aggregation may support mirroring all consumed p2 repos or simply providing an indirection via a composite repo.
  3. Organisations or teams want control over internally used components - It may be necessary to have gated access to relevant p2 repos and do an organisational "healthcheck" of those before internal distribution. Furthermore, internally used aggregated repos can provide a common basis for all organisational users.

The Aggregator tool is focused on supporting those specific requirements, rather than the complete set of usecases that Buckminster covers. Furthermore, it:

  • is based on EMF models and as such is part of the transition of the Buckminster models to EMF-based models
  • provides a simple user interface that does not delve into build specifics and should be accessible by a larger audience
  • and the aggregation can be run headlessly once an aggregation definition has been created

Installation

Start by installing a fresh Eclipse 3.5 SDK from http://www.eclipse.org/downloads/ (The latest version used at the time of writing was http://download.eclipse.org/eclipse/downloads/drops/R-3.5.1-200909170800/index.php)

Aggregator standalone

Aggregator with Buckminster

The Aggregator is not part of the official Galileo release. If you install the Aggregator from the repo mentioned above, into a workbench with Buckminster already installed, you may run into dependency conflicts. Instead update your Buckminster or install it from the following p2 repo: http://download.eclipse.org/tools/buckminster/updates-3.5

Getting started with standard examples

In the following we provide two simple examples that are easy to replicate and should highlight some of the features of the Aggregator. The first example deals with the creation of two variations of a p2 repo. The second shows the Aggregator's Maven support.

Aggregating a p2 repo

The first sample aggregation is build around Buckminster and its support for Subversive. The objective of this aggregated repo is to:

  • provide a "one-stop shop" experience
  • conveniently pull in third-party components that are not hosted at Eclipse
  • provide this repo as an indirection mechanism if required

Mirroring contributions

The example aggregation can be downloaded and opened in an appropriately set up workbench: Media:buckminster_galileo_m.build

Buckminster provides support for Subversive. Apart from the Subversive components at Eclipse (http://www.eclipse.org/subversive/), a complete installation will also require Subversive SVN connectors provided by Polarion (http://community.polarion.com/projects/subversive/download/eclipse/2.0/update-site/). We want to create a repo that combines all these components and makes them accessible from one location and for several pre-defined configurations.

Buckminster Aggregator - Sample 1

This example already includes some of the more advanced aggregation features. Stepping through the model view from the top node the following components can be identified:

  1. The top node Aggregator groups all model definitions regarding Configurations and Contributions. Looking at the properties section at the bottom it is shown
    • the top node has been provided with a mandatory Label with the value "Galileo + Buckminster for Subversive"; this will also be the label that would be used when accessing the aggregated repo via the p2 update manager
    • the Build Root identifies the location to which the artifacts of the aggregated repo will be deployed
  2. The aggregation is defined for three configurations (i.e. os=win32,ws=win32,arch=x86; etc)
    • any number of configurations can be defined
    • during the aggregation process all dependencies of the contributed components will be verified for all provided configurations, unless exceptions are defined (see below)
  3. The first Contribution to the aggregation is labeled "Galileo 3.5".
    • this contribution represents the simplest example of a contribution
    • one Mapped Repository is defined for this contribution (it could be multiple); all that is needed is a user-defined label and the URL of the repository that should be included
    • the result of this definition is that the complete Galileo p2 repo will be included in the aggregated repo
  4. The second Contribution is labeled "Subversive SVN connectors" and deals with the inclusion of bundles provided by Polarion. This contribution includes binary configuration-specific artifacts which are only available for win32. If a simple contribution would be defined the aggregation would fail for all non-win32 configurations, and hence the aggregation would fail as a whole.
    • this requires a definition of Valid Configurations Rules that state exceptions
    • the rules defined for the the three components in question essentially state that the verification process for those components should only be performed for win32-based configurations
  5. The third Contribution is labeled "Buckminster (latest)". It shows another advanced feature - an Exclusion Rule.
    • the objective of the sample repo is to provide convenient setup of Buckminster with Subversive support. Since Buckminster's Subclipse and Subversive support are mutually exclusive, the features relevant for Subclipse can be excluded from the aggregated repo
    • this is done using an Exclusion Rule defined for each Installable Unit that should be excluded
  6. At the bottom of the model editor view a list of all included repos is displayed.
    • this list allows browsing the contents of all repos
    • this part of the model is not editable

The aggregation can be run by right-clicking any node in the model and selecting Build Repository. This example was setup to use a mirroring approach for all contributed repos. Hence, the complete contents of all included can be found in the aggregated repos target location specified under Build Root.

INSERT OVERVIEW OF RESULTING STRUCTURE AND ARTIFACTS ... TBD !!!

Check the next section for a slightly different approach.


Providing a repo indirection

Buckminster Aggregator - mirroring disabled

Mirroring all repo artifacts of your aggregated contributions may neither seem prudent nor necessary. This can be avoided by changing one property for the defined contributions. Each Mapped Repository has property called Mirror Artifacts which can be set to false in order to prevent copying all artifacts of the contributed repo to the aggregated repo.

The following Media:buckminster_galileo_i.build is a variation of the first example with the Mirror Artifacts property set to false for all contributed repos. Running this aggregation will result in a composite repository that provides an indirection to the contributed repos.

INSERT OVERVIEW OF RESULTING STRUCTURE AND ARTIFACTS ... TBD !!!




Creating a Maven-conformant p2 repo

Buckminster Aggregator - Maven result

A powerful feature of the Aggregator is the ability to create aggregated repos that can be consumed as Maven repos, that is providing the structure and artifacts required by Maven. Those repos can in fact be consumed both as p2 and Maven repos. This flexibility is provided due to p2's separation of meta-data about dependencies and the actual location of the referenced artifacts.

In order to create a Maven-conformant aggregate repo all that is required is to set the property Maven Result property of the Aggregator to true. The aggregation deployed to the Build Root location will be a Maven repo.

The sample Media:buckminster_galileo_maven.build is a variation of the previous aggregations configured to produce a Maven repo.

INSERT OVERVIEW OF RESULTING STRUCTURE AND ARTIFACTS ... TBD !!!

Headless support

You will need a headless installation of Buckminster with the Aggregator feature installed.

Running from the command line

Just type:
buckminster aggregate <options>

For a detailed listing of the available options consult the next section.

Command line options

Option Value Default Description
--buildModel <path to build model> This value is required Appoints the aggregation definition that drives the execution
--action VERIFY

BUILD

CLEAN

CLEAN_BUILD

BUILD Specifies the type of the execution.
  • VERIFY - verifies model validity and resolves dependencies; no artifacts are copied or created
  • BUILD - performs the aggregation and creates the aggregated repository in the target location
  • CLEAN - cleans all traces of previous aggregations in the specified target location
  • CLEAN_BUILD - performs a CLEAN followed by a BUILD
--buildId <string> build-<timestamp in the format yyyyMMddHHmm> Assigns a build identifier to the aggregation. The identifier is used to identify the build in notification emails. Defaults to: build-<timestamp> where <timestamp> is formatted according as yyyyMMddHHmm, i.e. build-200911031527
--buildRoot <path to directory> buildRoot declared in the aggregation model Controls the output. Defaults to the build root defined in the aggregation definition.
--production N/A N/A Indicates that the build is running in real production. That means that no mock emails will be sent. Instead, the contacts listed for each contribution will get emails when things go wrong.
--emailFrom <email> Address of build master Becomes the sender of the emails sent from the aggregator.
--mockEmailTo <email>  ? Becomes the receiver of the mock-emails sent from the aggregator.
--mockEmailCC <email>  ? Becomes the CC receiver of the mock-emails sent from the aggregator.
--logURL <url> N/A The URL that will be pasted into the emails. Should normally point to the a public URL for output log for the aggregator so that the receiver can browse the log for details on failures.
--subjectPrefix <string>  ? The prefix to use for the subject when sending emails. Defaults to the label defined in the aggregation definition. The subject is formatted as: "[<subjectPrefix>] Failed for build <buildId>"
--smtpHost <host name> localhost The SMTP host to talk to when sending emails. Defaults to "localhost".
--smtpPort <port number> 25 The SMTP port number to use when talking to the SMTP host. Default is 25.

Hudson integration

TBD

Aggregator model components and specific actions

This section provides an in-depth description and reference of the Aggregator model, listing all model components, properties and available actions.

When referring to ... TBD (conventions)

Global actions

The following aggregator-specific actions are available via the context menu that can be invoked on any node in the Aggregator model editor:

  • Clean Repository - cleans all traces of previous aggregations in the specified target location (Build Root)
  • Verify Repository - verifies model validity and resolves dependencies; no artifacts are copied or created
  • Build Repository - performs the aggregation and creates the aggregated repository in the target location (Build Root)
  • Clean then Build Repository - performs a Clean followed by a Build

Aggregator

The root node of any aggregation model is the Aggregator node. It specifies a number of global properties including the Build Root (the target location of the aggregated repository) as well as the repo structure (maven-conformant or classic p2 setup). There are several child components some of which can be reference in other parts of the model: Configuration, Contribution, Contact, Custom Category, Validation Repository, Maven Mapping.

Property Value(s) Default Value Comment
Buildmaster <Contact>? - Specifies an optional build master that will be notified of progress and problems if sending of mail is enabled. This is a reference to any Contact that has been added to this Aggregator model.
Build Root <urn> - This is a required property specifying the target location of the aggregated repository.
Description <string> - An optional description of the aggregated repository that would be visible if accessing the aggregated repository via the p2 update manager.
Label <string> - A required label that will be used for the aggregated repository. This would be visible as the repo label if accessing the aggregated repository via the p2 update manager.
Maven Mappings <Maven Mapping>* - References Maven Mappings added as children to the Aggregator model root node. See Maven Mapping for details.
Maven Result true

false

false Controls the output structure of the aggregated repo. If true, the aggregated repo will be Maven-conformant. Both the structure and meta-data of the aggregated repository will follow the conventions required by Maven.

NOTE that due to the flexibility of p2 (separation of meta-data about dependencies and location of artifacts) the aggregated repo will also function as a valid p2 repository.

Packed Strategy Copy

Verify

Unpack as Sibling

Unpack

Skip

Copy This property controls how packed artifacts found in contributed repositories are handled when building the Aggregation:
  • Copy - if the source contains packed artifacts, copy and store them verbatim. No further action
  • Verify - same as copy but unpack the artifact and then discard the result
  • Unpack as Sibling - same as copy but unpack the artifact and store the result as a sibling
  • Unpack - use the packed artifact for data transfer but store it in unpacked form
  • Skip - do not consider packed artifacts. This option will require unpacked artifacts in the source
Sendmail false

true

false Controls whether or not emails are sent when errors are detected during the aggregation process. A value of false disables sending of emails. This includes mock emails.
Type S

I

N

M

C

R

S Indicates the Aggregation type. This is an annotation merely for the benefit of the build master. It is not visible in the resulting repo.
  • S - stable
  • I - integration
  • N - nightly
  • M - milestone
  • C - continuous
  • R - release

Configuration

An Aggregation may have one or more Configuration definitions. The aggregated repo will be verified for all added configurations. If dependencies for any of the given configurations fails the aggregation as a whole fails. It is however possible to specify exceptions for individual Contributions.

A Configuration is a combination of the following properties:

Property Value(s) Default Value Comment
Architecture X86

PPC

X86_64

- Specifies the architecture for which this configuration should be verified.
Operating System Win32

Linux

MacOSX

- Specifies the operating system for which this configuration should be verified.
Window System Win32

GTK

Carbon

cocoa

- Specifies the windowing system for which this configuration should be verified.

Contribution

Contributions are the key element of any aggregation. Contributions specify which repositories, or parts thereof (category, feature, product, IU), and according to which constraints they should be included in the aggregated repository. A contribution definition may consist of several Mapped Repository and Maven Mapping components.

Property Value(s) Default Value Comment
Contacts <Contact>* - Zero or more references to Contacts defined in the given Aggregator model. The referenced contacts will be notified if aggregation errors related to the contribution as a whole occur.
Description <string> - Optional description of the contribution. This serves documentation purposes and will not end up in the aggregated repository.
Enabled true

false

true Disables (false) or enables (true) the contribution for the aggregation process. Note that this may lead to missing dependencies and hence verification errors.
Label <string> - Mandatory label for this contribution.
Maven Mappings <Maven Mapping>* - See Maven Mapping for details ...


Mapped Repository

A Contribution may define several Mapped Repositories defining the actual content of the contribution. The Aggregator provides a fine-grained control over the contribution from each Mapped Repository through references to Products, Bundles, Features, Categorys, Exclusion Rules, Valid Configuration Rules.

Property Value(s) Default Value Comment
Category Prefix <string> - A prefix added to this repositories' label when displayed in the p2 update manager. In the absence of custom categories this allows a useful grouping of repositories in an aggregated repository.
Description <string> - Description of the Mapped Repository. For documentation purposes only.
Enabled true

false

true Controls whether a Mapped Repository is considered as part of the Contribution. Setting this property to false excludes the repository from the verification and aggregation process.
Location <URL> This specifies the location of the repository that should be added to the enclosing Contribution and is a required property.
Mirror Artifacts true

false

true Controls whether the contents (artifacts) of the specified repository will be copied to the target location of the aggregated repository.


Product

Defining Product components allows to add individual Eclipse products to the aggregation (when defined as part of the repository) rather than the complete contents of a given Mapped Repository.

Property Value(s) Default Value Comment
Enabled true

false

true Controls whether a Product is considered as part of the Contribution. Setting this property to false excludes the product from the verification and aggregation process.
Installable Unit <product IU> - A reference to the product to be included in the aggregation.
Valid Configurations <Configuration>* - References to zero or more configurations for which this product has to be verified. If no references are given the product has to be verified for all Configurations defined for the aggregation.


Category

Defining Category components allows to add individual categories provided by the contributed repository rather than the complete contents of a given Mapped Repository.

Property Value(s) Default Value Comment
Enabled true

false

true Controls whether a repository Category is considered as part of the Contribution. Setting this property to false excludes the category from the verification and aggregation process.
Installable Unit <IU> - A reference to the category to be included in the aggregation.
Label Override <string> - New category name used instead of the default name.
Valid Configurations <Configuration>* - References to zero or more configurations for which those category's contents have to be verified. If no references are given the category has to be verified for all Configurations defined for the aggregation.


Bundle

Defining Bundle components allows to add individual Eclipse bundles to the aggregation rather than the complete contents of a given Mapped Repository.

Property Value(s) Default Value Comment
Enabled true

false

true Controls whether a referenced bundle is considered as part of the Contribution. Setting this property to false excludes the bundle from the verification and aggregation process.
Installable Unit <IU> - A reference to the bundle to be included in the aggregation.
Valid Configurations <Configuration>* - References to zero or more configurations for which the referenced bundle has to be verified. If no references are given the bundle has to be verified for all Configurations defined for the aggregation.


Feature

Defining Feature components allows to add individual Eclipse features to the aggregation (when defined as part of the repository) rather than the complete contents of a given Mapped Repository. Furthermore, this component provides the means to group features implicitly into Custom Categorys.

Property Value(s) Default Value Comment
Categories <Custom Category>* - Optionally references the Custom Categorys into which the feature should be placed upon aggregation. The relationship to the custom category is bi-directional so adding the feature to a custom category will update this property automatically in the Custom Category definition, and vice versa.
Enabled true

false

true Controls whether a referenced feature is considered as part of the Contribution. Setting this property to false excludes the feature from the verification and aggregation process.
Installable Unit <IU> - A reference to the feature to be included in the aggregation.
Valid Configurations <Configuration>* - References to zero or more configurations for which the referenced feature has to be verified. If no references are given the feature has to be verified for all Configurations defined for the aggregation.


Exclusion Rule

Exclusion Rules are another tool providing control over the aggregated repository. An exclusion rule may reference any bundle, feature or product to exclude. The excluded IU will not be considered in the aggregation and verification process. Each exclusion rule can only reference one IU.

Property Value(s) Default Value Comment
Description <string> - Description for documentation purposes.
Installable Unit <IU> - Reference to any IU (product, feature, bundle) in the enclosing Mapped Repository that should be excluded from the aggregation.


Valid Configuration Rule

By default all contributed contents of a Mapped Repository will have to be verified for all Configurations defined for the aggregation. A Valid Configuration Rule allows constrains this and defines and exception. For an IU (product, feature, bundle) referenced by this rule only chosen subset of configuration will be verified and aggregated.

Property Value(s) Default Value Comment
Description <string> - Description for documentation purposes.
Installable Unit <IU> - Reference to any IU (product, feature, bundle) in the enclosing Mapped Repository for which only the referenced configurations should be verified.
Valid Configurations <Configuration>* - References to one or more configurations for which the referenced IU has to be verified. This implicitly excludes verification and aggregation for all other Configurations defined as part of the aggregation model.


Maven Mapping (Contribution)

See Maven Mapping for a detailed description and list of properties.

Contact

Defines a resuseable contact which can be referenced in other parts of the model and may be used to send notifications about the aggregation process.

Property Value(s) Default Value Comment
Email <email> - The email to be used when notifying the contact.
Name <string> - Full name of the contact. May be displayed as label when referenced in other parts of the model.

Custom Category

Property Value(s) Default Value Comment
Description - - Comment...
Features - - Comment...
Identifier - - Comment...
Label - - Comment...

Validation Repository

Property Value(s) Default Value Comment
Enabled - - Comment...
Location - - Comment...
Metadata Repository - - Comment...

Maven Mapping

The Aggregator supports the creation of Maven-conformant repositories. Those require a structure and naming conventions that may have to be achieved by a transformation of the Bundle-SymbolicName (BSN) when working with Eclipse bundles found at contributed repositories. Custom transformations are supported by the definition of one or more Maven Mappings which can be defined at the Aggregator and the Contribution level.

This only applies when the Maven Result property of the Aggregator model is set to true. In that case all defined Maven Mappings are applied in the order in which they appear in the model starting from the most specific to the most generic. That means for each artifact that a Contribution adds to the aggregated repository:

  1. first Maven Mappings defined as children of a Contribution are applied in the order in which they appear as children of the parent Contribution node
  2. second Maven Mappings defined as children of the Aggregator model are applied in the order in which they appear as children of the parent Aggregator node
  3. finally the default Maven Mapping is applied

The most generic mapping is a default pattern that is applied whenever a Maven result should be created. It does not need to be added explicitly to the model. A mapping is specified by a regular expression (applied to each BSN) and two replacements (one for groupId, another one for artifactId) which refer to the resulting Maven repo structure. The default pattern is:

^([^.]+(?:\.[^.]+(?:\.[^.]+)?)?)(?:\.[^.]+)*$

The default maven mapping defines as replacements $1 for groupId and $0 for artifactId. Hence, when applying the mapping to a BSN up to 3 segments (with dots as segment delimiters) are considered as the group, and the whole BSN is considered as the artifact name. If this is a applied to something like org.eclipse.buckminster.aggregator the groupId would be org.eclipse.buckminster and the artifactId org.eclipse.buckminster.aggregator. The resulting Maven repo would have the folder structure:

org
   |
   eclipse
      |
      buckminster 
         |
         org.eclipse.buckminster.aggregator
            |
            <folder name after version>
               |
               org.eclipse.buckminster.aggregator-<version>.jar
               ...


Property Value(s) Default Value Comment
Name Pattern regex - A regular expression which is applied to the Bundle-SymbolicName (BSN). Pattern groups may be referenced in the replacement properties.
Group Id <string> (may contain pattern group references (i.e. $1, ...)) - Group Id applied in a maven mapping structure (groupId/artifactId). The Group Id replacement may contain pattern group references (i.e. $1, ...).
Artifact Id <string> (may contain pattern group references (i.e. $1, ...)) - Artifact Id applied in a maven mapping structure (groupId/artifactId). The Artifact Id replacement may contain pattern group references (i.e. $1, ...).