Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "CBI/aggregator/manual"
(→Exclusion Rule) |
(added "The *.b3aggr model files shown be below can be created via File > New > b3 > Repository Aggregation.") |
||
(71 intermediate revisions by 10 users not shown) | |||
Line 15: | Line 15: | ||
* the Aggregator is based on EMF models | * the Aggregator is based on EMF models | ||
* headless execution of aggregation definitions (once they have been created) is possible using a command line packaging of the Aggregator | * headless execution of aggregation definitions (once they have been created) is possible using a command line packaging of the Aggregator | ||
+ | |||
+ | =Functional Overview= | ||
+ | The b3 Aggregator performs aggregation and validation of repositories. The input to the aggregator engine (that tells it what to do) is a ''b3aggr'' EMF model. Such a model is most conveniently created by using the b3 Aggregator editor. This editor provides both editing and interactive execution of aggregation commands. The editor is based on a standard EMF "tree and properties view" style editor where nodes are added and removed to from a tree, and the details of nodes are edited in a separate properties view. Once a b3aggr model has been created it is possible to use the command line / headless aggregator to perform aggregation (and other related commands). (Note that since the b3aggr is "just an EMF model", it can be produced via EMF APIs, transformation tools, etc., and thus support advanced use cases). | ||
+ | |||
+ | The model mainly consists of ''Contributions'' - specifications of what to include from different repositories, and ''Validation Repositories'' - repositories that are used when validating, but which are not included in the produced aggregation (i.e., they are not copied). ''Contributions'' and ''Validation Repositories'' are grouped into ''Validation Sets''. Everything in a ''Validation Set'' will be validated as one unit, i.e. it must be possible to install everything in a ''Validation Set'' together. The model also contains specification of various processing rules (exclusions, transformation of names, etc.), and specification of ''Contacts'' - individuals/mailing-lists to inform when processing fails. | ||
+ | |||
+ | Here are some of the important features supported by the b3 Aggregator: | ||
+ | * '''p2''' and '''maven2''' support — the aggregator can aggregate from and to both p2 and maven2 repositories. | ||
+ | * '''Maven2 name mapping support''' — names in the p2 domain are automatically mapped to maven2 names using built-in rules. Custom rules are also supported. | ||
+ | * '''Mirroring''' — artifacts from repositories are mirrored/downloaded/copied to a single location. | ||
+ | * '''Selective mirroring''' — an aggregation can produce an aggregate consisting of a mix of references to repositories and mirrored repositories. | ||
+ | * '''Cherry picking''' — it is possible to pick individual items when the entire content of a repository is not wanted. Detailed picking is supported as well as picking transitive closures like a product, or a category to get everything it contains/requires. | ||
+ | * '''Pruning''' — it is possible to specify mirroring based on version ranges. This can be used to reduce the size of the produced result when historical versions are not needed in the aggregated result. | ||
+ | * '''Categorization''' — categorization of installable units is important to the consumers of the aggregated repository. Categories are often choosen by repository publishers in a fashion that makes sense when looking at a particular repository in isolation, but when they are combined with others it can be very difficult for the user to understand what they relate to. An important task for the constructor of an aggregation is to be able to organize the aggregated material in an easily consumable fashion. The b3 aggregator has support for category prefixing, category renaming, addition of custom categories, as well as adding and removing features in categories. | ||
+ | * '''Validation''' — the b3 aggregator validates the aggregated '''Validation Sets''' to ensure that everything in them is installable at the same time. | ||
+ | * '''Blame Email''' — when issues are found during validation, the aggregator supports sending emails describing the issue. This is very useful when aggregating the result of many different projects. Advanced features include specifying contacts for parts of the aggregation, which is useful in large multi-layer project structures where issues may be related to the combination of a group of projects rather than one individual project - someone responsible for the aggregation itself should be informed about these cross-project issues. The aggregator supports detailed control over email generation, including handling of mock emails when testing aggregation scripts. | ||
=Installation= | =Installation= | ||
− | Start by installing a fresh Eclipse | + | Start by installing a fresh Eclipse 4.3 SDK from http://download.eclipse.org/eclipse/downloads |
The b3 aggregator can either be integrated in your Eclipse SDK or it can be installed as a standalone headless product (i.e. pure command line, without any graphical UI). | The b3 aggregator can either be integrated in your Eclipse SDK or it can be installed as a standalone headless product (i.e. pure command line, without any graphical UI). | ||
Line 23: | Line 39: | ||
The instructions below show the current URLs (when this document was written). Always check the latest information on the [http://eclipse.org/modeling/emft/b3/download/ b3 download page] before installing. | The instructions below show the current URLs (when this document was written). Always check the latest information on the [http://eclipse.org/modeling/emft/b3/download/ b3 download page] before installing. | ||
− | ==Eclipse SDK installation== | + | == Eclipse SDK installation == |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | | | + | {| |
− | | | + | |- |
− | + | | colspan="2" | | |
− | + | *Start your Eclipse installation and open the '''''Install New Software wizard'''''. You'll find in under the top menu ''Help'' | |
− | [[Image: | + | [[Image:B3 Install New Software.png|thumb|center|580x492px|Install New Software wizard]] |
+ | *Click the ''Add...'' button and enter the URL '''''<nowiki>http://download.eclipse.org/modeling/emft/b3/updates-4.3/</nowiki>''''' in the ''Location'' field. Or alternatively, if you feel adventurous and want to use the latest build, '''''<nowiki>https://hudson.eclipse.org/hudson/job/emft-b3-build/lastSuccessfulBuild/artifact/b3.p2.repository/</nowiki>'''''. | ||
+ | *Select the '''''b3 Aggregator Editor''''' and click ''Next'' twice. | ||
+ | [[Image:B3 Install Aggregator.png|thumb|center|580x492px|b3 Aggregator Editor selection]] | ||
+ | *Accept the Eclipse Public License and click ''Finish'' | ||
+ | *Restart the IDE once the installation is finished. | ||
+ | |- | ||
|} | |} | ||
Line 47: | Line 61: | ||
#'''Install b3''' with the following command: | #'''Install b3''' with the following command: | ||
#*<code>director -r <HEADLESS_REPO> -d <INSTALL_DIR> -p b3 -i org.eclipse.b3.cli.product -i org.eclipse.b3.aggregator.engine.feature.feature.group</code> where | #*<code>director -r <HEADLESS_REPO> -d <INSTALL_DIR> -p b3 -i org.eclipse.b3.cli.product -i org.eclipse.b3.aggregator.engine.feature.feature.group</code> where | ||
− | #**'''''-r <HEADLESS_REPO>''''' is the headless p2 update site: Current version is: http://download.eclipse.org/modeling/emft/b3/headless- | + | #**'''''-r <HEADLESS_REPO>''''' is the headless p2 update site: Current stable version is: '''''<nowiki>http://download.eclipse.org/modeling/emft/b3/headless-4.3</nowiki>''''', and our latest build can be found at '''''<nowiki>https://hudson.eclipse.org/hudson/job/emft-b3-build/lastSuccessfulBuild/artifact/b3.headless.p2.repository</nowiki>''''' |
#**'''''-d <INSTALL_DIR>''''' is the chosen install location of the headless b3 | #**'''''-d <INSTALL_DIR>''''' is the chosen install location of the headless b3 | ||
#**'''''-p b3''''' is the name of the p2 profile | #**'''''-p b3''''' is the name of the p2 profile | ||
− | #**'''''-i org.eclipse.cli.product''''' is the name of the headless b3 base | + | #**'''''-i org.eclipse.b3.cli.product''''' is the name of the headless b3 base |
#**'''''-i org.eclipse.b3.aggregator.engine.feature.feature.group''''' is the name of the aggregator feature | #**'''''-i org.eclipse.b3.aggregator.engine.feature.feature.group''''' is the name of the aggregator feature | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
=Getting started with standard examples= | =Getting started with standard examples= | ||
In the following sections we provide two simple examples that are easy to replicate and should highlight the most important features of the Aggregator. | In the following sections we provide two simple examples that are easy to replicate and should highlight the most important features of the Aggregator. | ||
The first example deals with the creation of two variations of a p2 repo. The second shows the Aggregator's Maven support. | The first example deals with the creation of two variations of a p2 repo. The second shows the Aggregator's Maven support. | ||
+ | |||
+ | The *.b3aggr model files shown be below can be created via File > New > b3 > Repository Aggregation. | ||
==Aggregating a p2 repo== | ==Aggregating a p2 repo== | ||
Line 71: | Line 79: | ||
*provide this repo as an indirection mechanism if required | *provide this repo as an indirection mechanism if required | ||
− | ===Mirroring contributions=== | + | === Mirroring contributions === |
− | + | ||
− | + | (This example aggregation can be downloaded via the b3 project SVN and opened in an appropriately set up workbench: [http://http://git.eclipse.org/c/b3/b3.git/tree/org.eclipse.b3.aggregator.samples/buckminster-indigo.b3aggr]). | |
− | + | ||
− | + | The background is that Buckminster provides support for Subclipse. In addition to all components hosted at Eclipse, a complete installation will also require Subclipse components from Tigris.org (http://subclipse.tigris.org/update_1.6.x). We want to create a repo that combines these components and makes them accessible from one location. We want to make several platform configurations available. | |
− | This example already includes some of the more advanced aggregation features. Stepping through the model view from the top node the following components can be identified: | + | [[Image:B3 aggregator sample 1.png|thumb|center|800x750px|b3 Aggregator - Sample 1]] |
− | #The top node ''''' | + | |
− | #*the top node has been provided with a mandatory ''Label'' with the value " | + | This example already includes some of the more advanced aggregation features. Stepping through the model view from the top node the following components can be identified: |
− | #*the ''Build Root'' identifies the location to which the artifacts of the aggregated repo will be deployed | + | |
− | #The aggregation is defined for three configurations (i.e. os=win32, ws=win32, arch=x86; etc) | + | #The top node '''''Aggregation''''' groups all model definitions regarding '''''ValidationSet'''''s and '''''Contribution'''''s. Looking at the properties section at the bottom we see that: |
+ | #*the top node has been provided with a mandatory ''Label'' with the value "Indigo + Buckminster for Subclipse"; this is also the label that is shown to users when accessing the aggregated repo via the p2 update manager | ||
+ | #*the ''Build Root'' identifies the location to which the artifacts of the aggregated repo will be deployed | ||
+ | #The aggregation is defined for three configurations (i.e. os=win32, ws=win32, arch=x86; etc) | ||
#*any number of configurations can be defined | #*any number of configurations can be defined | ||
− | #*during the aggregation process all dependencies of the ''contributed'' components will be verified for all provided configurations, unless exceptions are defined (see below) | + | #*during the aggregation process all dependencies of the ''contributed'' components will be verified for all provided configurations, unless exceptions are defined (see below) |
− | # | + | #We have one ValidationSet labeled "main". A ValidationSet constitutes everything that will be validated as one unit by the p2 planner. |
− | + | #The ''main'' ValidationSet contains three different contributions. | |
− | # | + | #The first '''''Contribution''''' to the aggregation is labeled "Indigo". This contribution includes binary configuration-specific artifacts which are only available for linux. If a simple contribution would be defined the aggregation would fail for all non-linux configurations, and hence the aggregation would fail as a whole. |
− | + | #*this requires a definition of '''''Valid Configurations Rule'''''s that state exceptions | |
− | #The | + | #*the rules defined for the the three components in question essentially state that the verification process for those components should only be performed for linux-based configurations |
− | #*this requires a definition of '''''Valid Configurations Rule'''''s that state exceptions | + | #*one '''''Mapped Repository''''' is defined for this contribution (it can have multiple); all that is needed is a user-defined label and the URL of the repository that should be included |
− | #*the rules defined for the the three components in question essentially state that the verification process for those components should only be performed for | + | #*the result of this definition is that all categories, products, and features from Indigo p2 repo will be included in the aggregated repo. |
− | #The third '''''Contribution''''' is labeled "Buckminster (latest)". It shows another advanced feature - an '''''Exclusion Rule'''''. | + | #The second '''''Contribution''''' is labeled "Subclipse" and deals with the inclusion of bundles provided from the Subclipse project. #*this contribution represents the simplest example of a contribution |
− | #*remember that the objective of the sample repo is to provide convenient setup of Buckminster with | + | #The third '''''Contribution''''' is labeled "Buckminster (latest)". It shows another advanced feature - an '''''Exclusion Rule'''''. |
− | #*this is done using an '''''Exclusion Rule''''' defined for each Installable Unit that should be excluded | + | #*remember that the objective of the sample repo is to provide convenient setup of Buckminster with Subclipse support, and since Buckminster's Subclipse and Subversive support are mutually exclusive, we want to exclude the features for Subversive from the aggregated repo to make it easier for the user. |
+ | #*this is done using an '''''Exclusion Rule''''' defined for each Installable Unit that should be excluded | ||
#A list of all included repos is displayed at the bottom of the model editor view | #A list of all included repos is displayed at the bottom of the model editor view | ||
− | #*this list allows browsing the contents of all repos | + | #*this list allows browsing the contents of all repos |
#*this part of the model is not editable | #*this part of the model is not editable | ||
− | The aggregation can be run by right-clicking any node in the model and selecting '''''Build | + | The aggregation can be run by right-clicking any node in the model and selecting '''''Build Aggregation'''''. This example was setup to use a mirroring approach for all contributed repos. Hence, the complete contents of all included can be found in the aggregated repos target location specified under '''''Build Root'''''. |
− | This example was setup to use a mirroring approach for all contributed repos. Hence, the complete contents of all included can be found in the aggregated repos target location specified under '''''Build Root'''''. | + | |
Check the next section for a slightly different approach. | Check the next section for a slightly different approach. | ||
Line 112: | Line 121: | ||
Each '''''Mapped Repository''''' has a boolean property called ''Mirror Artifacts'' which can be set to <code>false</code> in order to prevent copying all artifacts of the contributed repo to the aggregated repo. | Each '''''Mapped Repository''''' has a boolean property called ''Mirror Artifacts'' which can be set to <code>false</code> in order to prevent copying all artifacts of the contributed repo to the aggregated repo. | ||
− | The following [http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.b3/trunk/org.eclipse.b3.aggregator.samples/buckminster_helios_redirect.b3aggr | + | The following [http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.b3/trunk/org.eclipse.b3.aggregator.samples/buckminster_helios_redirect.b3aggr buckminster_indigo_redirect.b3aggr] is a variation of the first example with the ''Mirror Artifacts'' property set to <code>false</code> for all contributed repos. Running this aggregation will result in a composite repository that provides indirection to the contributed repos. |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
==Creating a Maven-conformant p2 repo== | ==Creating a Maven-conformant p2 repo== | ||
Line 127: | Line 132: | ||
The sample [http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.b3/trunk/org.eclipse.b3.aggregator.samples/buckminster_helios_maven.b3aggr buckminster_helios_maven.b3aggr] is a variation of the previous aggregations configured to produce a Maven repo. | The sample [http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.b3/trunk/org.eclipse.b3.aggregator.samples/buckminster_helios_maven.b3aggr buckminster_helios_maven.b3aggr] is a variation of the previous aggregations configured to produce a Maven repo. | ||
− | |||
− | |||
− | |||
− | |||
=Headless support= | =Headless support= | ||
Line 137: | Line 138: | ||
==Running from the command line== | ==Running from the command line== | ||
Just type:<pre>b3 aggregate <options></pre> | Just type:<pre>b3 aggregate <options></pre> | ||
+ | |||
+ | Note: if you install the aggregator into an Eclipse SDK or Platform (rather than a pure headless installation), you can run from the command line, by using | ||
+ | <pre>eclipse -application org.eclipse.b3.cli.headless aggregate <options></pre> | ||
For a detailed listing of the available options consult the next section. | For a detailed listing of the available options consult the next section. | ||
Line 156: | Line 160: | ||
|- valign="top" | |- valign="top" | ||
| '''--action''' | | '''--action''' | ||
− | | VERIFY | + | | VALIDATE (VERIFY in 0.1) |
BUILD | BUILD | ||
Line 166: | Line 170: | ||
| BUILD | | BUILD | ||
| Specifies the type of the execution. | | Specifies the type of the execution. | ||
− | *''' | + | *'''VALIDATE''' - verifies model validity and resolves dependencies; no artifacts are copied or created |
*'''BUILD''' - performs the aggregation and creates the aggregated repository in the target location | *'''BUILD''' - performs the aggregation and creates the aggregated repository in the target location | ||
*'''CLEAN''' - cleans all traces of previous aggregations in the specified target location | *'''CLEAN''' - cleans all traces of previous aggregations in the specified target location | ||
Line 182: | Line 186: | ||
| ''buildRoot'' declared in the aggregation model | | ''buildRoot'' declared in the aggregation model | ||
| Controls the output. Defaults to the build root defined in the aggregation definition. The value must be a valid path to a directory (absolute or relative to current folder). If the desired directory structure does not exist then it is created. | | Controls the output. Defaults to the build root defined in the aggregation definition. The value must be a valid path to a directory (absolute or relative to current folder). If the desired directory structure does not exist then it is created. | ||
+ | |||
+ | |- valign="top" | ||
+ | | '''--agentLocation''' | ||
+ | | <path to directory> | ||
+ | | '''<buildRoot>'''/p2 | ||
+ | | Controls the location of the p2 agent. When specified as outside of the build root, the agent will not be cleaned out by the aggregator and thus retain its cache. | ||
|- valign="top" | |- valign="top" | ||
Line 242: | Line 252: | ||
| INFO | | INFO | ||
| Controls the verbosity of the eclipse log trace output. Defaults to global b3 settings. | | Controls the verbosity of the eclipse log trace output. Defaults to global b3 settings. | ||
+ | |||
+ | |- valign="top" | ||
+ | | '''--stacktrace''' | ||
+ | | --- | ||
+ | | ''no value'' | ||
+ | | Display stack trace on uncaught error. | ||
|- valign="top" | |- valign="top" | ||
Line 283: | Line 299: | ||
A comma separated list of contributions with repositories that will be referenced directly (through a composite repository) rather than mirrored into the final repository (even if the repository is set to mirror artifacts by default). | A comma separated list of contributions with repositories that will be referenced directly (through a composite repository) rather than mirrored into the final repository (even if the repository is set to mirror artifacts by default). | ||
+ | ''Note: this option is mutually exclusive with option --mavenResult (see below)'' | ||
+ | |||
+ | |- valign="top" | ||
+ | | '''--validationContributions''' | ||
+ | | <comma separated list> | ||
+ | | ''no value'' | ||
+ | | ''Deprecated (Use only with on-the-fly transformation from deprecated build models)'' | ||
+ | A comma separated list of contributions with repositories that will be used for aggregation validation only rather than mirrored or referenced into the final repository. | ||
+ | |||
+ | |- valign="top" | ||
+ | | '''--mavenResult''' | ||
+ | | --- | ||
+ | | ''no value'' | ||
+ | | ''Deprecated (Use only with on-the-fly transformation from deprecated build models)'' | ||
+ | Tells the aggregator to generate a hybrid repository that is compatible with p2 and maven2. | ||
+ | ''Note: this option is mutually exclusive with option --trustedContributions (see above)'' | ||
+ | |||
+ | |- valign="top" | ||
+ | | '''--mirrorReferences''' | ||
+ | | --- | ||
+ | | ''no value'' | ||
+ | | Mirror meta-data repository references from the contributed repositories. Note the current recommendation is to not have update sites in feature definitions, but instead in "products", that is, to allow adopters or consumers to specify where to get updates from and what "extras" to offer. See {{bug|358415}}. But, this option can still useful for some situations. | ||
+ | |||
+ | |- valign="top" | ||
+ | | '''--referenceIncludePattern''' | ||
+ | | <regexp> | ||
+ | | ''no value'' | ||
+ | | Include only those references that matches the given regular expression pattern. | ||
+ | |||
+ | |- valign="top" | ||
+ | | '''--referenceExcludePattern''' | ||
+ | | <regexp> | ||
+ | | ''no value'' | ||
+ | | Exclude all references that matches the given regular expression pattern. An exclusion takes precedence over an inclusion in case both patterns match a reference. | ||
|} | |} | ||
Line 288: | Line 338: | ||
Currently there is no support for Hudson. | Currently there is no support for Hudson. | ||
− | = | + | =Aggregation model components and specific actions= |
− | This section provides an in-depth description and reference of the | + | This section provides an in-depth description and reference of the Aggregation model, listing all model components, properties and available actions. |
==Global actions== | ==Global actions== | ||
− | The following aggregator-specific actions are available via the context menu that can be invoked on any node in the | + | The following aggregator-specific actions are available via the context menu that can be invoked on any node in the Aggregation model editor: |
− | *'''Clean | + | *'''Clean Aggregation''' - cleans all traces of previous aggregations in the specified target location (''Build Root'') |
− | *''' | + | *'''Validate Aggregation''' - verifies model validity and resolves dependencies; no artifacts are copied or created |
− | *'''Build | + | *'''Build Aggregation''' - performs the aggregation and creates the aggregated repository in the target location (''Build Root'') |
− | *'''Clean then Build | + | *'''Clean then Build Aggregation''' - performs a ''Clean'' followed by a ''Build'' |
− | == | + | ==Aggregation== |
− | The root node of any aggregation model is the | + | The root node of any aggregation model is the Aggregation node. It specifies a number of global properties including the ''Build Root'' (the target location of the aggregated repository) as well as the repo structure (maven-conformant or classic p2 setup). |
− | There are several child components some of which can be reference in other parts of the model: [[#Configuration|Configuration]], [[# | + | There are several child components some of which can be reference in other parts of the model: [[#Configuration|Configuration]], [[#Contact|Contact]],[[#Validation Set|Validation Set]], [[#Custom Category|Custom Category]], [[#Maven Mapping|Maven Mapping]]. |
{| {{Greytable}} | {| {{Greytable}} | ||
Line 313: | Line 363: | ||
| <[[#Contact|Contact]]>? | | <[[#Contact|Contact]]>? | ||
| - | | - | ||
− | | Specifies an optional build master that will be notified of progress and problems if sending of mail is enabled. This is a reference to any [[#Contact|Contact]] that has been added to this | + | | Specifies an optional build master that will be notified of progress and problems if sending of mail is enabled. This is a reference to any [[#Contact|Contact]] that has been added to this Aggregation model. |
|- valign="top" | |- valign="top" | ||
Line 337: | Line 387: | ||
| <[[#Maven Mapping|Maven Mapping]]>* | | <[[#Maven Mapping|Maven Mapping]]>* | ||
| - | | - | ||
− | | References [[#Maven Mapping|Maven Mapping]]s added as children to the | + | | References [[#Maven Mapping|Maven Mapping]]s added as children to the Aggregation model root node. See [[#Maven Mapping|Maven Mapping]] for details. |
|- valign="top" | |- valign="top" | ||
Line 417: | Line 467: | ||
'''IA64''' | '''IA64''' | ||
− | | | + | |
+ | '''IA64_32''' | ||
+ | |||
+ | '''Sparc''' | ||
+ | |||
+ | '''PPC64''' | ||
+ | |||
+ | '''S390''' | ||
+ | |||
+ | '''S390x''' | ||
+ | |'''X86''' | ||
|Specifies the architecture for which this configuration should be verified. | |Specifies the architecture for which this configuration should be verified. | ||
Line 434: | Line 494: | ||
'''MacOSX''' | '''MacOSX''' | ||
− | | | + | |
+ | '''AIX''' | ||
+ | |||
+ | '''HPUX''' | ||
+ | |||
+ | '''Solaris''' | ||
+ | |||
+ | '''QNX''' | ||
+ | |'''Win32''' | ||
|Specifies the operating system for which this configuration should be verified. | |Specifies the operating system for which this configuration should be verified. | ||
Line 445: | Line 513: | ||
'''Carbon''' | '''Carbon''' | ||
− | ''' | + | '''Cocoa''' |
− | | | + | |
+ | '''Motif''' | ||
+ | |||
+ | '''Photon''' | ||
+ | |'''Win32''' | ||
|Specifies the windowing system for which this configuration should be verified. | |Specifies the windowing system for which this configuration should be verified. | ||
+ | |} | ||
+ | |||
+ | ===Validation Set=== | ||
+ | The aggregator uses the p2 planner tool to determine what needs to be copied. The validation set determines the scope of one such planning session per valid configuration. This vouches for that everything that is contained within a validation set will be installable into the same target. Sometimes it is desirable to have more than one version of a feature in a repository even though multiple versions cannot be installed. This can be done using one validation set for each version. | ||
+ | |||
+ | A validation set can extend other validation set and thereby provide a convenient way for sharing common things that multiple validation sets will make use of. An extended validation set will not be validated by its own. It will only be validated as part of the sets that extend it. | ||
+ | |||
+ | A validation set can have two types of children: [[#Contribution|Contribution]] and [[#Validation Repository|Validation Repository]]. | ||
+ | |||
+ | Versions prior to 0.2.0 did not have the validation set node. Older b3aggr files will therefore be converted and given one validation set called ''main'' | ||
+ | |||
+ | {| {{Greytable}} | ||
+ | |- | ||
+ | !Property | ||
+ | !Value(s) | ||
+ | !Default Value | ||
+ | !Comment | ||
+ | |||
+ | |- valign="top" | ||
+ | |Description | ||
+ | | <string> | ||
+ | | - | ||
+ | | An optional description of the validation set. For documentation purposes only. | ||
+ | |||
+ | |- valign="top" | ||
+ | |Enabled | ||
+ | | '''true''' | ||
+ | '''false''' | ||
+ | | true | ||
+ | | Disables (false) or enables (true) the validation set to be considered in the aggregation process. Note that this may lead to missing dependencies and hence verification errors. | ||
+ | |||
+ | |- valign="top" | ||
+ | |||
+ | |Extends | ||
+ | | '''<[[#Validation Set|Validation Set]]>'''* | ||
+ | | - | ||
+ | | Zero or more references to [[#Validation Set|Validation Set]]s defined in the given Aggregation model that this validation set extends. | ||
+ | |||
+ | |- valign="top" | ||
+ | |Label | ||
+ | | <string> | ||
+ | | - | ||
+ | | Mandatory label for this validation set. | ||
|} | |} | ||
Line 466: | Line 581: | ||
| '''<[[#Contact|Contact]]>'''* | | '''<[[#Contact|Contact]]>'''* | ||
| - | | - | ||
− | | Zero or more references to [[#Contact|Contact]]s defined in the given | + | | Zero or more references to [[#Contact|Contact]]s defined in the given Aggregation model. The referenced contacts will be notified if aggregation errors related to the contribution as a whole occur. |
|- valign="top" | |- valign="top" | ||
Line 615: | Line 730: | ||
|- valign="top" | |- valign="top" | ||
|Name | |Name | ||
− | | < | + | | <category IU id> |
| - | | - | ||
− | | The id of the category to be included in the aggregation. A drop-down of available names is provided. | + | | The id of the category to be included in the aggregation. A drop-down of available names is provided. |
|- valign="top" | |- valign="top" | ||
Line 664: | Line 779: | ||
|- valign="top" | |- valign="top" | ||
|Name | |Name | ||
− | | < | + | | <bundle IU id> |
| - | | - | ||
− | | The id of the bundle to be included in the aggregation. A drop-down of available names is provided. | + | | The id of the bundle to be included in the aggregation. A drop-down of available names is provided. |
|- valign="top" | |- valign="top" | ||
Line 715: | Line 830: | ||
|- valign="top" | |- valign="top" | ||
|Name | |Name | ||
− | | < | + | | <feature IU id> |
| - | | - | ||
− | | The id of the feature to be included in the aggregation. A drop-down of available names is provided. | + | | The id of the feature to be included in the aggregation. A drop-down of available names is provided. |
|- valign="top" | |- valign="top" | ||
Line 751: | Line 866: | ||
|- valign="top" | |- valign="top" | ||
|Name | |Name | ||
− | | < | + | | <IU id> |
| - | | - | ||
− | | The id of the installable unit to be excluded from the aggregation. A drop-down of available names is provided. | + | | The id of the installable unit to be excluded from the aggregation. A drop-down of available names is provided. |
|- valign="top" | |- valign="top" | ||
Line 764: | Line 879: | ||
=====Valid Configuration Rule===== | =====Valid Configuration Rule===== | ||
− | By default all contributed contents of a [[#Mapped Repository|Mapped Repository]] will | + | By default all contributed contents of a [[#Mapped Repository|Mapped Repository]] will be verified for all [[#Configuration|Configuration]]s defined for the aggregation. A [[#Valid_Configuration_Rule|Valid Configuration Rule]] provides more control over validation. When using a Valid Configuration Rule, the referenced IUs (product, feature, or bundle) will only be verified and aggregated for the configurations specified in the rule. The rule only applies if the whole repository is mapped (i.e. when no explicit features, products, bundles or categories are mapped, regardless if they are enabled or disabled). |
{| {{Greytable}} | {| {{Greytable}} | ||
Line 781: | Line 896: | ||
|- valign="top" | |- valign="top" | ||
|Name | |Name | ||
− | | < | + | | <IU id> |
| - | | - | ||
| The id of the IU to be included in the verification process. A drop-down of available names is provided. | | The id of the IU to be included in the verification process. A drop-down of available names is provided. | ||
Line 789: | Line 904: | ||
| <[[#Configuration|Configuration]]>* | | <[[#Configuration|Configuration]]>* | ||
| - | | - | ||
− | | References to one or more configurations for which the referenced IU | + | | References to one or more configurations for which the referenced IU should be verified. This implicitly excludes verification and aggregation for all other [[#Configuration|Configuration]]s defined as part of the aggregation model. |
|- valign="top" | |- valign="top" | ||
Line 803: | Line 918: | ||
===Contact=== | ===Contact=== | ||
− | Defines a resuseable contact which can be referenced in other parts of the model and may be used to send notifications about the aggregation process. | + | Defines a resuseable contact element which can be referenced in other parts of the model and may be used to send notifications about the aggregation process. |
{| {{Greytable}} | {| {{Greytable}} | ||
Line 828: | Line 943: | ||
===Custom Category=== | ===Custom Category=== | ||
A Custom Category provides a grouping mechanism for features in the aggregated repository. A custom category can be referenced by [[#Feature|Feature]]s defined for inclusion from a [[#Mapped Repository|Mapped Repository]]. | A Custom Category provides a grouping mechanism for features in the aggregated repository. A custom category can be referenced by [[#Feature|Feature]]s defined for inclusion from a [[#Mapped Repository|Mapped Repository]]. | ||
− | The relationship to | + | The relationship to between Custom Category and a [[#Feature|Feature]] is bi-directional. Thus, adding the feature to a custom category will update this property automatically in the [[#Feature|Feature]] definition, and vice versa. |
{| {{Greytable}} | {| {{Greytable}} | ||
Line 864: | Line 979: | ||
===Validation Repository=== | ===Validation Repository=== | ||
− | A Validation Repository | + | A Validation Repository is used to define that a repository should be used when validating dependencies for the aggregation but whose contents should not be included in the aggregated repository. |
− | It supports the cases where the objective is to create a repository that is not self sufficient, but rather a complement to something else. | + | It supports the cases where the objective is to create a repository that is not self sufficient, but rather a complement to something else (typical use case is an aggregation of everything produced by an organization with validation against the main Eclipse repository). |
{| {{Greytable}} | {| {{Greytable}} | ||
Line 896: | Line 1,011: | ||
===Maven Mapping=== | ===Maven Mapping=== | ||
− | The Aggregator supports the creation of Maven-conformant repositories. | + | The Aggregator supports the creation of Maven-conformant repositories. A Maven repository requires a structure and use of naming conventions that may have to be achieved by a transformation of the Bundle-SymbolicName (BSN) when working with Eclipse bundles. There is a default translation from BSN naming standard to Maven naming. If that is not satisfactory, |
+ | custom transformations are supported by the definition of one or more Maven Mappings which can be defined at the [[#Aggregator|Aggregator]] and the [[#Contribution|Contribution]] level. | ||
This only applies when the ''Maven Result'' property of the [[#Aggregator|Aggregator]] model is set to true. In that case all defined Maven Mappings are applied in the order in which they appear in the model starting from the most specific to the most generic. That means for each artifact that a [[#Contribution|Contribution]] adds to the aggregated repository: | This only applies when the ''Maven Result'' property of the [[#Aggregator|Aggregator]] model is set to true. In that case all defined Maven Mappings are applied in the order in which they appear in the model starting from the most specific to the most generic. That means for each artifact that a [[#Contribution|Contribution]] adds to the aggregated repository: | ||
Line 903: | Line 1,019: | ||
#finally the default Maven Mapping is applied | #finally the default Maven Mapping is applied | ||
− | The most generic mapping is a default pattern that is applied whenever a Maven | + | The most generic mapping is a default pattern that is applied whenever a Maven is created. It does not need to be added explicitly to the model. |
− | A mapping is specified | + | A mapping is specified using a regular expression that is applied to each BSN. The regular expression must specify two replacements; one for the maven groupId, and one for the maven artifactId. The group and artifact ids have an effect on the resulting Maven repo structure. The default pattern is: |
<pre> | <pre> | ||
Line 910: | Line 1,026: | ||
</pre> | </pre> | ||
− | The default maven mapping | + | The default maven mapping use the replacement <code>'''$1'''</code> for groupId and <code>'''$0'''</code> for artifactId. Hence, when applying the default maven mapping regular expression to a BSN, up to 3 segments (with dots as segment delimiters) are taken as the group id, and the entire BSN is taken as the artifact name. If this is a applied to something like <code>org.eclipse.b3.aggregator</code> the groupId would be <code>org.eclipse.b3</code> and the artifactId <code>org.eclipse.b3.aggregator</code>. The resulting Maven repo will have the folder structure: |
<pre> | <pre> | ||
Line 955: | Line 1,071: | ||
|} | |} | ||
+ | |||
+ | =Legacy format support= | ||
+ | As time passes, the aggregator model will undergo changes. All such changes will result in a new XML schema describing the aggregator model. The aggregator will always detect the schema in use by a model and, if needed, transform it into the current schema. | ||
+ | |||
+ | The transformation is fully automated when you run in headless mode but you will see a message like ''The build model file is obsolete, using the default transformation'' indicating that a transformation took place. The new model only exists in memory while the aggregation runs and is never stored on disk. | ||
+ | |||
+ | When you are opening an old model in the IDE, a transformation wizard will be presented where you are given a chance to specify a new file name for the transformed model. Please note that if you are using detached contributions (contributions that live in files of their own), and you want to keep it that way, then you should not change the name. If the name is changed, the aggregator has no choice but to create a new file that embeds all the contributions. This is because a detached contribution is referencing the aggregation file by name. | ||
+ | |||
+ | The behavior of the transformation wizard is: | ||
+ | * If the name is unchanged, also retain the detached contributions | ||
+ | * If a new name is given, embed all contributions in the new file | ||
+ | |||
+ | If you want a new name and still retain the detached contributions then you have two options | ||
+ | |||
+ | Either: | ||
+ | # Do the transformation using the same name | ||
+ | # Manually rename the main b3aggr file | ||
+ | # Manually search/replace and change the name in all your contributions | ||
+ | |||
+ | Or: | ||
+ | # Do the transformation using a different name | ||
+ | # Remove all your detached contributions | ||
+ | # Detach all contributions again | ||
=What else should be documented= | =What else should be documented= | ||
Line 963: | Line 1,102: | ||
# 'available versions' tree | # 'available versions' tree | ||
# context menu actions (mapping IUs from repository view, searching for IUs from 'available version' tree...) | # context menu actions (mapping IUs from repository view, searching for IUs from 'available version' tree...) | ||
− | |||
==Questions== | ==Questions== | ||
* How is blame-mail handled when running in the UI? Are the options "production" etc. available then? | * How is blame-mail handled when running in the UI? Are the options "production" etc. available then? | ||
+ | ''When launched from IDE, only --buildModel and --action options are set (all other options have default values). Perhaps it's worth adding a launching dialog which would enable setting all options that are available in headless mode.'' | ||
* How do you know what the "log output url" is? How can it be set? | * How do you know what the "log output url" is? How can it be set? | ||
+ | ''You can, for example, redirect a copy of the console output to a publicly available resource (a text file served by a http server). The public URL should be passed in the --logURL option so that a link to it would appear in the informational email.'' | ||
* Why is there a section that says "there is no hudson support" - is hudson support needed/required in any way?? | * Why is there a section that says "there is no hudson support" - is hudson support needed/required in any way?? | ||
+ | ''The Hudson Support article was copied from the old buckminster documentation. It can be removed.'' | ||
* Some configurations seems to be missing - or is the list open ended? (Sparc, Motif, etc..) - I assume user can put anything there? | * Some configurations seems to be missing - or is the list open ended? (Sparc, Motif, etc..) - I assume user can put anything there? | ||
+ | ''Only listed configurations are supported (they are a part of the model). Adding an option means changing the model and creation of a new aggregator build.'' | ||
* It seems that it is possible to load maven repositories. I suppose the result of aggregation is a valid p2 repository (or a combination of p2/maven)?? | * It seems that it is possible to load maven repositories. I suppose the result of aggregation is a valid p2 repository (or a combination of p2/maven)?? | ||
+ | ''Yes, it is possible to load p2 and maven2 repositories by default (if someone adds a custom loader then he can load any type of repository). The output is always p2 compatible, optionally combined with maven2 (with no extensibility - at least now). However, if a maven2 repo is loaded and the result is also maven2 compatible, it is not identical to the original (not all attributes loaded from maven are stored in the p2 model).'' | ||
[[Category:Eclipse b3]] | [[Category:Eclipse b3]] |
Revision as of 13:39, 28 March 2014
Contents
- 1 Introduction
- 2 Functional Overview
- 3 Installation
- 4 Getting started with standard examples
- 5 Headless support
- 6 Aggregation model components and specific actions
- 7 Legacy format support
- 8 What else should be documented
Introduction
The Aggregator is based on and part of the Eclipse b3 project. b3 provides a versatile and adaptable framework supporting build, assembly and deployment processes. It supports a rich set of use cases. One of those - the aggregation of repositories - is the focus of the b3 Aggregator tool.
The Aggregator combines repositories from various sources into a new aggregated p2 repository. It can also be configured to produce a hybrid p2/Maven2 repository. There are many situations where using aggregated repositories is a good solution. The reasons vary from licensing issues to organizational requirements:
- Owners of a p2 repo for a given project may not be in position to host all required or recommended components due to licensing issues - Buckminster's SVN support can serve as an example here, as it requires components available through the main Eclipse p2 repo as well as third-party components. Hence users have to visit several repos for a complete install.
- Projects want to provide convenient access to their products - Installation instructions requiring the user to visit several repos for a complete install are not uncommon. An aggregated repo for all those locations provides a convenient one-stop strategy. The aggregation may support mirroring all consumed p2 repos or simply providing an indirection via a composite repo.
- Organizations or teams want control over internally used components - It may be necessary to have gated access to relevant p2 repos and do an organizational "healthcheck" of those before internal distribution. Furthermore, internally used aggregated repos can provide a common basis for all organizational users.
- Increase repository availability - by aggregating and mirroring what is used from multiple update sites into an internally controlled server (or several).
- Distributed Development Support - an overall product repository is produced by aggregating contributions from multiple teams.
The Aggregator tool is focused on supporting these specific requirements, and it plays an important role in the full scope of the b3 project. The Aggregator is however used in scenarios outside of the traditional "build domain" and this has been reflected in the user interface which does not delve into the details of "building" and should therefore be easy to use by non build experts.
Furthermore, it is worth noting that:
- the Aggregator is based on EMF models
- headless execution of aggregation definitions (once they have been created) is possible using a command line packaging of the Aggregator
Functional Overview
The b3 Aggregator performs aggregation and validation of repositories. The input to the aggregator engine (that tells it what to do) is a b3aggr EMF model. Such a model is most conveniently created by using the b3 Aggregator editor. This editor provides both editing and interactive execution of aggregation commands. The editor is based on a standard EMF "tree and properties view" style editor where nodes are added and removed to from a tree, and the details of nodes are edited in a separate properties view. Once a b3aggr model has been created it is possible to use the command line / headless aggregator to perform aggregation (and other related commands). (Note that since the b3aggr is "just an EMF model", it can be produced via EMF APIs, transformation tools, etc., and thus support advanced use cases).
The model mainly consists of Contributions - specifications of what to include from different repositories, and Validation Repositories - repositories that are used when validating, but which are not included in the produced aggregation (i.e., they are not copied). Contributions and Validation Repositories are grouped into Validation Sets. Everything in a Validation Set will be validated as one unit, i.e. it must be possible to install everything in a Validation Set together. The model also contains specification of various processing rules (exclusions, transformation of names, etc.), and specification of Contacts - individuals/mailing-lists to inform when processing fails.
Here are some of the important features supported by the b3 Aggregator:
- p2 and maven2 support — the aggregator can aggregate from and to both p2 and maven2 repositories.
- Maven2 name mapping support — names in the p2 domain are automatically mapped to maven2 names using built-in rules. Custom rules are also supported.
- Mirroring — artifacts from repositories are mirrored/downloaded/copied to a single location.
- Selective mirroring — an aggregation can produce an aggregate consisting of a mix of references to repositories and mirrored repositories.
- Cherry picking — it is possible to pick individual items when the entire content of a repository is not wanted. Detailed picking is supported as well as picking transitive closures like a product, or a category to get everything it contains/requires.
- Pruning — it is possible to specify mirroring based on version ranges. This can be used to reduce the size of the produced result when historical versions are not needed in the aggregated result.
- Categorization — categorization of installable units is important to the consumers of the aggregated repository. Categories are often choosen by repository publishers in a fashion that makes sense when looking at a particular repository in isolation, but when they are combined with others it can be very difficult for the user to understand what they relate to. An important task for the constructor of an aggregation is to be able to organize the aggregated material in an easily consumable fashion. The b3 aggregator has support for category prefixing, category renaming, addition of custom categories, as well as adding and removing features in categories.
- Validation — the b3 aggregator validates the aggregated Validation Sets to ensure that everything in them is installable at the same time.
- Blame Email — when issues are found during validation, the aggregator supports sending emails describing the issue. This is very useful when aggregating the result of many different projects. Advanced features include specifying contacts for parts of the aggregation, which is useful in large multi-layer project structures where issues may be related to the combination of a group of projects rather than one individual project - someone responsible for the aggregation itself should be informed about these cross-project issues. The aggregator supports detailed control over email generation, including handling of mock emails when testing aggregation scripts.
Installation
Start by installing a fresh Eclipse 4.3 SDK from http://download.eclipse.org/eclipse/downloads
The b3 aggregator can either be integrated in your Eclipse SDK or it can be installed as a standalone headless product (i.e. pure command line, without any graphical UI).
The instructions below show the current URLs (when this document was written). Always check the latest information on the b3 download page before installing.
Eclipse SDK installation
|
Headless installation
Installation of the headless version of the Aggregator is similar to a typical headless b3 installation. The following steps focus on the installation of the headless Aggregator feature.
- Start by Downloading the (headless) director which can be found here.
- Next unpack the director to a location of your choice. You may want to set the
PATH
to include the install location and make the director accessible for additional use. - Install b3 with the following command:
director -r <HEADLESS_REPO> -d <INSTALL_DIR> -p b3 -i org.eclipse.b3.cli.product -i org.eclipse.b3.aggregator.engine.feature.feature.group
where- -r <HEADLESS_REPO> is the headless p2 update site: Current stable version is: http://download.eclipse.org/modeling/emft/b3/headless-4.3, and our latest build can be found at https://hudson.eclipse.org/hudson/job/emft-b3-build/lastSuccessfulBuild/artifact/b3.headless.p2.repository
- -d <INSTALL_DIR> is the chosen install location of the headless b3
- -p b3 is the name of the p2 profile
- -i org.eclipse.b3.cli.product is the name of the headless b3 base
- -i org.eclipse.b3.aggregator.engine.feature.feature.group is the name of the aggregator feature
Getting started with standard examples
In the following sections we provide two simple examples that are easy to replicate and should highlight the most important features of the Aggregator. The first example deals with the creation of two variations of a p2 repo. The second shows the Aggregator's Maven support.
The *.b3aggr model files shown be below can be created via File > New > b3 > Repository Aggregation.
Aggregating a p2 repo
The first sample aggregation is build around Buckminster and its support for Subversive. The objective of this aggregated repo is to:
- provide a "one-stop shop" experience
- conveniently pull in third-party components that are not hosted at Eclipse
- provide this repo as an indirection mechanism if required
Mirroring contributions
(This example aggregation can be downloaded via the b3 project SVN and opened in an appropriately set up workbench: [1]).
The background is that Buckminster provides support for Subclipse. In addition to all components hosted at Eclipse, a complete installation will also require Subclipse components from Tigris.org (http://subclipse.tigris.org/update_1.6.x). We want to create a repo that combines these components and makes them accessible from one location. We want to make several platform configurations available.
This example already includes some of the more advanced aggregation features. Stepping through the model view from the top node the following components can be identified:
- The top node Aggregation groups all model definitions regarding ValidationSets and Contributions. Looking at the properties section at the bottom we see that:
- the top node has been provided with a mandatory Label with the value "Indigo + Buckminster for Subclipse"; this is also the label that is shown to users when accessing the aggregated repo via the p2 update manager
- the Build Root identifies the location to which the artifacts of the aggregated repo will be deployed
- The aggregation is defined for three configurations (i.e. os=win32, ws=win32, arch=x86; etc)
- any number of configurations can be defined
- during the aggregation process all dependencies of the contributed components will be verified for all provided configurations, unless exceptions are defined (see below)
- We have one ValidationSet labeled "main". A ValidationSet constitutes everything that will be validated as one unit by the p2 planner.
- The main ValidationSet contains three different contributions.
- The first Contribution to the aggregation is labeled "Indigo". This contribution includes binary configuration-specific artifacts which are only available for linux. If a simple contribution would be defined the aggregation would fail for all non-linux configurations, and hence the aggregation would fail as a whole.
- this requires a definition of Valid Configurations Rules that state exceptions
- the rules defined for the the three components in question essentially state that the verification process for those components should only be performed for linux-based configurations
- one Mapped Repository is defined for this contribution (it can have multiple); all that is needed is a user-defined label and the URL of the repository that should be included
- the result of this definition is that all categories, products, and features from Indigo p2 repo will be included in the aggregated repo.
- The second Contribution is labeled "Subclipse" and deals with the inclusion of bundles provided from the Subclipse project. #*this contribution represents the simplest example of a contribution
- The third Contribution is labeled "Buckminster (latest)". It shows another advanced feature - an Exclusion Rule.
- remember that the objective of the sample repo is to provide convenient setup of Buckminster with Subclipse support, and since Buckminster's Subclipse and Subversive support are mutually exclusive, we want to exclude the features for Subversive from the aggregated repo to make it easier for the user.
- this is done using an Exclusion Rule defined for each Installable Unit that should be excluded
- A list of all included repos is displayed at the bottom of the model editor view
- this list allows browsing the contents of all repos
- this part of the model is not editable
The aggregation can be run by right-clicking any node in the model and selecting Build Aggregation. This example was setup to use a mirroring approach for all contributed repos. Hence, the complete contents of all included can be found in the aggregated repos target location specified under Build Root.
Check the next section for a slightly different approach.
Providing a repo indirection
Mirroring all repo artifacts of your aggregated contributions is a very valuable and important feature when performing aggregation, but there are also many cases where this is not necessary. It is possible to turn off artifact mirroring/copying by changing one property for a defined contribution.
Each Mapped Repository has a boolean property called Mirror Artifacts which can be set to The following buckminster_indigo_redirect.b3aggr is a variation of the first example with the Mirror Artifacts property set to Creating a Maven-conformant p2 repo
|