Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "CBI/aggregator/manual"
(added "The *.b3aggr model files shown be below can be created via File > New > b3 > Repository Aggregation.") |
m (Update to 4.7) |
||
(25 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
=Introduction= | =Introduction= | ||
− | + | Historically, the CBI Aggregator is based on the "b3 Aggregator" from the now defunct "Eclipse b3 Project". | |
+ | |||
+ | The purpose of the CBI Aggregator is to aggregate or combine several p2 repositories. The aggregator does a number of things above and beyond simply "p2 mirroring" the different repositories: first and foremost, it makes sure that all the bundles could be "installed together" (that is, all the constraints are met and none are contradictory); it also ensures any "pack.gz" files are valid (i.e. can be unpacked into valid jars); and it also offers custom categories since the aggregated repository in most cases would not want to use the same categories as each individual repository uses. More details on its functionality follows. | ||
+ | |||
+ | The Aggregator combines repositories from various sources into a new aggregated p2 repository. It can also be configured to produce a hybrid p2/Maven2 repository. (This feature is currently labeled "experimental" in the CBI aggregator, since it is not yet production quality). | ||
− | |||
There are many situations where using aggregated repositories is a good solution. The reasons vary from licensing issues to organizational requirements: | There are many situations where using aggregated repositories is a good solution. The reasons vary from licensing issues to organizational requirements: | ||
− | #'''Owners of a p2 repo for a given project may not be in position to host all required or recommended components due to licensing issues''' - | + | #'''Owners of a p2 repo for a given project may not be in position to host all required or recommended components due to licensing issues''' - SVN support one example: as it requires components available through the main Eclipse p2 repo as well as third-party components. Hence users would normally have to visit several repos for a complete install, but by using the aggregator an institution could create a custom repository that has everything needed. |
#'''Projects want to provide convenient access to their products''' - Installation instructions requiring the user to visit several repos for a complete install are not uncommon. An aggregated repo for all those locations provides a convenient one-stop strategy. The aggregation may support mirroring all consumed p2 repos or simply providing an indirection via a composite repo. | #'''Projects want to provide convenient access to their products''' - Installation instructions requiring the user to visit several repos for a complete install are not uncommon. An aggregated repo for all those locations provides a convenient one-stop strategy. The aggregation may support mirroring all consumed p2 repos or simply providing an indirection via a composite repo. | ||
#'''Organizations or teams want control over internally used components''' - It may be necessary to have gated access to relevant p2 repos and do an organizational "healthcheck" of those before internal distribution. Furthermore, internally used aggregated repos can provide a common basis for all organizational users. | #'''Organizations or teams want control over internally used components''' - It may be necessary to have gated access to relevant p2 repos and do an organizational "healthcheck" of those before internal distribution. Furthermore, internally used aggregated repos can provide a common basis for all organizational users. | ||
Line 10: | Line 13: | ||
#'''Distributed Development Support''' - an overall product repository is produced by aggregating contributions from multiple teams. | #'''Distributed Development Support''' - an overall product repository is produced by aggregating contributions from multiple teams. | ||
− | The Aggregator tool is focused on supporting these specific requirements | + | The Aggregator tool is focused on supporting these specific requirements. The Aggregator is used in scenarios outside of the traditional "build domain" and this has been reflected in the user interface which does not delve into the details of "building" and should therefore be easy to use by non build experts. |
Furthermore, it is worth noting that: | Furthermore, it is worth noting that: | ||
Line 17: | Line 20: | ||
=Functional Overview= | =Functional Overview= | ||
− | The | + | The CBI Aggregator performs aggregation and validation of repositories. The input to the aggregator engine (that tells it what to do) is a ''aggr'' EMF model. Such a model is most conveniently created by using the CBI Aggregator editor. This editor provides both editing and interactive execution of aggregation commands. The editor is based on a standard EMF "tree and properties view" style editor where nodes are added and removed to from a tree, and the details of nodes are edited in a separate properties view. Once a aggr model has been created it is possible to use the command line / headless aggregator to perform aggregation (and other related commands). (Note that since the aggr is "just an EMF model", it can be produced via EMF APIs, transformation tools, etc., and thus support advanced use cases). |
The model mainly consists of ''Contributions'' - specifications of what to include from different repositories, and ''Validation Repositories'' - repositories that are used when validating, but which are not included in the produced aggregation (i.e., they are not copied). ''Contributions'' and ''Validation Repositories'' are grouped into ''Validation Sets''. Everything in a ''Validation Set'' will be validated as one unit, i.e. it must be possible to install everything in a ''Validation Set'' together. The model also contains specification of various processing rules (exclusions, transformation of names, etc.), and specification of ''Contacts'' - individuals/mailing-lists to inform when processing fails. | The model mainly consists of ''Contributions'' - specifications of what to include from different repositories, and ''Validation Repositories'' - repositories that are used when validating, but which are not included in the produced aggregation (i.e., they are not copied). ''Contributions'' and ''Validation Repositories'' are grouped into ''Validation Sets''. Everything in a ''Validation Set'' will be validated as one unit, i.e. it must be possible to install everything in a ''Validation Set'' together. The model also contains specification of various processing rules (exclusions, transformation of names, etc.), and specification of ''Contacts'' - individuals/mailing-lists to inform when processing fails. | ||
− | Here are some of the important features supported by the | + | Here are some of the important features supported by the CBI Aggregator: |
* '''p2''' and '''maven2''' support — the aggregator can aggregate from and to both p2 and maven2 repositories. | * '''p2''' and '''maven2''' support — the aggregator can aggregate from and to both p2 and maven2 repositories. | ||
* '''Maven2 name mapping support''' — names in the p2 domain are automatically mapped to maven2 names using built-in rules. Custom rules are also supported. | * '''Maven2 name mapping support''' — names in the p2 domain are automatically mapped to maven2 names using built-in rules. Custom rules are also supported. | ||
Line 28: | Line 31: | ||
* '''Cherry picking''' — it is possible to pick individual items when the entire content of a repository is not wanted. Detailed picking is supported as well as picking transitive closures like a product, or a category to get everything it contains/requires. | * '''Cherry picking''' — it is possible to pick individual items when the entire content of a repository is not wanted. Detailed picking is supported as well as picking transitive closures like a product, or a category to get everything it contains/requires. | ||
* '''Pruning''' — it is possible to specify mirroring based on version ranges. This can be used to reduce the size of the produced result when historical versions are not needed in the aggregated result. | * '''Pruning''' — it is possible to specify mirroring based on version ranges. This can be used to reduce the size of the produced result when historical versions are not needed in the aggregated result. | ||
− | * '''Categorization''' — categorization of installable units is important to the consumers of the aggregated repository. Categories are often choosen by repository publishers in a fashion that makes sense when looking at a particular repository in isolation, but when they are combined with others it can be very difficult for the user to understand what they relate to. An important task for the constructor of an aggregation is to be able to organize the aggregated material in an easily consumable fashion. The | + | * '''Categorization''' — categorization of installable units is important to the consumers of the aggregated repository. Categories are often choosen by repository publishers in a fashion that makes sense when looking at a particular repository in isolation, but when they are combined with others it can be very difficult for the user to understand what they relate to. An important task for the constructor of an aggregation is to be able to organize the aggregated material in an easily consumable fashion. The CBI Aggregator has support for category prefixing, category renaming, addition of custom categories, as well as adding and removing features in categories. |
− | * '''Validation''' — the | + | * '''Validation''' — the CBI Aggregator validates the aggregated '''Validation Sets''' to ensure that everything in them is installable at the same time. |
* '''Blame Email''' — when issues are found during validation, the aggregator supports sending emails describing the issue. This is very useful when aggregating the result of many different projects. Advanced features include specifying contacts for parts of the aggregation, which is useful in large multi-layer project structures where issues may be related to the combination of a group of projects rather than one individual project - someone responsible for the aggregation itself should be informed about these cross-project issues. The aggregator supports detailed control over email generation, including handling of mock emails when testing aggregation scripts. | * '''Blame Email''' — when issues are found during validation, the aggregator supports sending emails describing the issue. This is very useful when aggregating the result of many different projects. Advanced features include specifying contacts for parts of the aggregation, which is useful in large multi-layer project structures where issues may be related to the combination of a group of projects rather than one individual project - someone responsible for the aggregation itself should be informed about these cross-project issues. The aggregator supports detailed control over email generation, including handling of mock emails when testing aggregation scripts. | ||
=Installation= | =Installation= | ||
− | + | It is best to start by downloading a fresh release of the [http://download.eclipse.org/eclipse/downloads Eclipse SDK]. The latest version of the aggregator, from its own [https://www.eclipse.org/cbi/downloads#p2repoAggregator software repository], typically goes with the most recently released version of the Eclipse SDK or the Eclipse Platform. The aggregator builds do not typically keep up with Milestones. It might work, but "user beware". | |
− | The | + | The CBI aggregator can either be integrated with your Eclipse SDK or it can be installed as a standalone headless product (i.e. pure command line, without any graphical UI). |
− | The instructions below show the | + | The instructions below show the URLs at the time this section was written (that is, not necessarily the current URLs). |
+ | |||
+ | Always check the for the latest aggregator software repository on the [https://www.eclipse.org/cbi/downloads#p2repoAggregator CBI download] page. | ||
== Eclipse SDK installation == | == Eclipse SDK installation == | ||
Line 46: | Line 51: | ||
*Start your Eclipse installation and open the '''''Install New Software wizard'''''. You'll find in under the top menu ''Help'' | *Start your Eclipse installation and open the '''''Install New Software wizard'''''. You'll find in under the top menu ''Help'' | ||
[[Image:B3 Install New Software.png|thumb|center|580x492px|Install New Software wizard]] | [[Image:B3 Install New Software.png|thumb|center|580x492px|Install New Software wizard]] | ||
− | *Click the ''Add...'' button and enter the URL '''''<nowiki>http://download.eclipse.org/ | + | *Click the ''Add...'' button and enter the URL '''''<nowiki>http://download.eclipse.org/cbi/updates/aggregator/ide/4.7/</nowiki>''''' in the ''Location'' field. |
− | *Select the ''''' | + | *Select the '''''Eclipse CBI Aggregator Editor''''' and click ''Next'' twice. |
− | [[Image:B3 Install Aggregator.png|thumb|center|580x492px| | + | [[Image:B3 Install Aggregator.png|thumb|center|580x492px|CBI Aggregator Editor selection]] |
*Accept the Eclipse Public License and click ''Finish'' | *Accept the Eclipse Public License and click ''Finish'' | ||
*Restart the IDE once the installation is finished. | *Restart the IDE once the installation is finished. | ||
Line 55: | Line 60: | ||
==Headless installation== | ==Headless installation== | ||
− | Installation of the headless version of the Aggregator is similar to | + | Installation of the headless version of the Aggregator is similar to any typical headless installation using the p2Director. The following steps focus on the installation of the headless Aggregator feature. |
− | + | #'''Install CBI Aggregator''' with the following command: | |
− | + | #*<code><p2_DIRECTOR> -r <HEADLESS_REPO> -d <INSTALL_DIR> -p CBIProfile -i org.eclipse.cbi.p2repo.cli.product</code> | |
− | #'''Install | + | where |
− | #*<code> | + | #**'''''<p2_DIRECTOR>''''' is whatever method you use to invoke the [http://help.eclipse.org/neon/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fp2_director.html p2Director application]. |
− | #**'''''-r <HEADLESS_REPO>''''' is the headless p2 update site: Current stable version is: '''''<nowiki>http://download.eclipse.org/ | + | #**'''''-r <HEADLESS_REPO>''''' is the headless p2 update site: Current stable version is: '''''<nowiki>http://download.eclipse.org/cbi/updates/aggregator/headless/4.7/</nowiki>''''' |
− | #**'''''-d <INSTALL_DIR>''''' is the chosen install location of the headless | + | #**'''''-d <INSTALL_DIR>''''' is the chosen install location of the headless aggregator |
− | #**'''''-p | + | #**'''''-p CBIProfile''''' is the name of the p2 profile |
− | #**'''''-i org.eclipse. | + | #**'''''-i org.eclipse.cbi.p2repo.cli.product''''' is the name of the headless CBI Aggregator |
− | + | ||
=Getting started with standard examples= | =Getting started with standard examples= | ||
Line 71: | Line 75: | ||
The first example deals with the creation of two variations of a p2 repo. The second shows the Aggregator's Maven support. | The first example deals with the creation of two variations of a p2 repo. The second shows the Aggregator's Maven support. | ||
− | The *. | + | The *.aggr model files shown be below can be created via File > New > CBI Aggregator > Repository Aggregation. |
==Aggregating a p2 repo== | ==Aggregating a p2 repo== | ||
Line 81: | Line 85: | ||
=== Mirroring contributions === | === Mirroring contributions === | ||
− | (This example aggregation can be downloaded via the | + | [NOTE: this section is out of date and has yet to be updated for the CBI aggregator] |
+ | |||
+ | (This example aggregation can be downloaded via the CBI project Git and opened in an appropriately set up workbench: [http://git.eclipse.org/c/b3/b3.git/tree/org.eclipse.cbi.p2repo.aggregator.samples/buckminster-indigo.aggr]). | ||
The background is that Buckminster provides support for Subclipse. In addition to all components hosted at Eclipse, a complete installation will also require Subclipse components from Tigris.org (http://subclipse.tigris.org/update_1.6.x). We want to create a repo that combines these components and makes them accessible from one location. We want to make several platform configurations available. | The background is that Buckminster provides support for Subclipse. In addition to all components hosted at Eclipse, a complete installation will also require Subclipse components from Tigris.org (http://subclipse.tigris.org/update_1.6.x). We want to create a repo that combines these components and makes them accessible from one location. We want to make several platform configurations available. | ||
− | [[Image:B3 aggregator sample 1.png|thumb|center|800x750px| | + | [[Image:B3 aggregator sample 1.png|thumb|center|800x750px|CBI Aggregator - Sample 1]] |
This example already includes some of the more advanced aggregation features. Stepping through the model view from the top node the following components can be identified: | This example already includes some of the more advanced aggregation features. Stepping through the model view from the top node the following components can be identified: | ||
Line 121: | Line 127: | ||
Each '''''Mapped Repository''''' has a boolean property called ''Mirror Artifacts'' which can be set to <code>false</code> in order to prevent copying all artifacts of the contributed repo to the aggregated repo. | Each '''''Mapped Repository''''' has a boolean property called ''Mirror Artifacts'' which can be set to <code>false</code> in order to prevent copying all artifacts of the contributed repo to the aggregated repo. | ||
− | The following [http:// | + | The following [http://git.eclipse.org/c/b3/b3.git/tree/org.eclipse.cbi.p2repo.aggregator.samples/buckminster_helios_redirect.aggr buckminster_indigo_redirect.aggr] is a variation of the first example with the ''Mirror Artifacts'' property set to <code>false</code> for all contributed repos. Running this aggregation will result in a composite repository that provides indirection to the contributed repos. |
==Creating a Maven-conformant p2 repo== | ==Creating a Maven-conformant p2 repo== | ||
Line 129: | Line 135: | ||
A powerful feature of the Aggregator is the ability to create aggregated repos that can be consumed as Maven repos, (i.e. providing the directory/file structure and artifacts required by Maven). An aggregated repository that supports Maven can be consumed both as a p2 and a Maven repo (at the same time). This flexibility is possible thanks to p2's separation of dependency meta-data and the actual location of the referenced artifacts. | A powerful feature of the Aggregator is the ability to create aggregated repos that can be consumed as Maven repos, (i.e. providing the directory/file structure and artifacts required by Maven). An aggregated repository that supports Maven can be consumed both as a p2 and a Maven repo (at the same time). This flexibility is possible thanks to p2's separation of dependency meta-data and the actual location of the referenced artifacts. | ||
− | In order to create a Maven-conformant aggregate repo all that is required is to set the property '''''Maven Result''''' property of the ''''' | + | In order to create a Maven-conformant aggregate repo all that is required is to set the property '''''Maven Result''''' property of the '''''Aggregation''''' to <code>true</code>. The aggregation deployed to the '''''Build Root''''' location will be a Maven repo. In this repo all <code>.source</code> artifacts will be co-located with their main artifact, sharing the same artifactId but with qualifier <code>-sources</code>, as expected by Maven. |
+ | |||
+ | Additionally, the property '''''Version Format''''' of the '''''Aggregation''''' allows to select between three strategies for version numbers: | ||
+ | :;Normal | ||
+ | ::Versions numbers are kept unmodified | ||
+ | :;Strict Maven | ||
+ | ::Use a '-' (dash) to separate three-part versions from any qualifier, to make the resulting versions parsable by Maven (so 3.12.2.v20161117-1814 will be converted to 3.12.2'''<font color=red>-</font>'''v20161117-1814). | ||
+ | ::''In previous versions this strategy was selected via property ''Strict Maven Versions'' which is now deprecated in favor of ''Version Format. | ||
+ | :;Maven Release | ||
+ | ::Cut off any qualifier, i.e., produce three-part version numbers to tell Maven that all artifacts are "releases". | ||
+ | If more fine grained control over version formats is needed, each [[#Maven Mapping|Maven Mapping]] can be fine tuned with a pair of '''Version Pattern''' and '''Version Template''', which will only be applied to artifacts matching the corresponding Maven Mapping. In this case, the existing version will be matched against the pattern, and if it matches, the version will be replaced by applying the corresponding template, where $1, $2 ... represent match groups from the pattern. As an example consider the case where the micro digit of a version should be omitted if it is equal to '0', so "4.12.0.v201504281640" will be translated to "4.12": | ||
+ | * Version Pattern: ([^.]+)\.([^.]+)\.0(?:\..*)? | ||
+ | * Version Template: $1.$2 | ||
+ | If no version pattern is defined for any Maven Mapping, or if a defined version pattern does not match, the aggregator will fall back to the global '''Version Format''' strategy. | ||
− | + | A sample aggr file, [https://git.eclipse.org/c/cbi/org.eclipse.cbi.p2repo.aggregator.git/plain/org.eclipse.cbi.p2repo.aggregator/samples/SDK4Mvn.aggr SDK4Mvn.aggr], shows an easy-to-use example of how to create a Maven-conformant p2 repository (i.e. one that works with Maven or p2). The sample has a short [https://git.eclipse.org/c/cbi/org.eclipse.cbi.p2repo.aggregator.git/plain/org.eclipse.cbi.p2repo.aggregator/samples/SDK4Mvn.description.txt description file] that explains some of the details. | |
=Headless support= | =Headless support= | ||
− | You will need a [[#Headless installation|headless installation of | + | You will need a [[#Headless installation|headless installation of CBI]] with the Aggregator feature installed. |
==Running from the command line== | ==Running from the command line== | ||
− | Just type:<pre> | + | Just type:<pre>cbiAggr aggregate <options></pre> |
Note: if you install the aggregator into an Eclipse SDK or Platform (rather than a pure headless installation), you can run from the command line, by using | Note: if you install the aggregator into an Eclipse SDK or Platform (rather than a pure headless installation), you can run from the command line, by using | ||
− | <pre>eclipse -application org.eclipse. | + | <pre>eclipse -application org.eclipse.cbi.p2repo.cli.headless aggregate <options></pre> |
For a detailed listing of the available options consult the next section. | For a detailed listing of the available options consult the next section. | ||
Line 239: | Line 258: | ||
ERROR | ERROR | ||
| INFO | | INFO | ||
− | | Controls the verbosity of the console trace output. Defaults to global | + | | Controls the verbosity of the console trace output. Defaults to global settings. |
|- valign="top" | |- valign="top" | ||
Line 251: | Line 270: | ||
ERROR | ERROR | ||
| INFO | | INFO | ||
− | | Controls the verbosity of the eclipse log trace output. Defaults to global | + | | Controls the verbosity of the eclipse log trace output. Defaults to global settings. |
|- valign="top" | |- valign="top" | ||
Line 530: | Line 549: | ||
A validation set can have two types of children: [[#Contribution|Contribution]] and [[#Validation Repository|Validation Repository]]. | A validation set can have two types of children: [[#Contribution|Contribution]] and [[#Validation Repository|Validation Repository]]. | ||
− | Versions prior to 0.2.0 did not have the validation set node. Older | + | Versions prior to 0.2.0 did not have the validation set node. Older aggr files will therefore be converted and given one validation set called ''main'' |
{| {{Greytable}} | {| {{Greytable}} | ||
Line 1,026: | Line 1,045: | ||
</pre> | </pre> | ||
− | The default maven mapping use the replacement <code>'''$1'''</code> for groupId and <code>'''$0'''</code> for artifactId. Hence, when applying the default maven mapping regular expression to a BSN, up to 3 segments (with dots as segment delimiters) are taken as the group id, and the entire BSN is taken as the artifact name. If this is a applied to something like <code>org.eclipse. | + | The default maven mapping use the replacement <code>'''$1'''</code> for groupId and <code>'''$0'''</code> for artifactId. Hence, when applying the default maven mapping regular expression to a BSN, up to 3 segments (with dots as segment delimiters) are taken as the group id, and the entire BSN is taken as the artifact name. If this is a applied to something like <code>org.eclipse.cbi.p2repo.aggregator</code> the groupId would be <code>org.eclipse.cbi.p2repo</code> and the artifactId <code>org.eclipse.cbi.p2repo.aggregator</code>. The resulting Maven repo will have the folder structure: |
<pre> | <pre> | ||
Line 1,033: | Line 1,052: | ||
eclipse | eclipse | ||
| | | | ||
− | + | cbi | |
| | | | ||
− | org.eclipse. | + | org.eclipse.cbi.p2repo.aggregator |
| | | | ||
<folder name after version> | <folder name after version> | ||
| | | | ||
− | org.eclipse. | + | org.eclipse.cbi.p2repo.aggregator-<version>.jar |
... | ... | ||
</pre> | </pre> | ||
Line 1,069: | Line 1,088: | ||
| - | | - | ||
| Artifact Id applied in a maven mapping structure (groupId/artifactId). The Artifact Id replacement may contain pattern group references (i.e. $1, ...). | | Artifact Id applied in a maven mapping structure (groupId/artifactId). The Artifact Id replacement may contain pattern group references (i.e. $1, ...). | ||
+ | |||
+ | |- valign="top" | ||
+ | | Version Pattern | ||
+ | | regex | ||
+ | | - | ||
+ | | An optional regular expression which is applied to the Version. Pattern groups may be referenced in the Version Template property. | ||
+ | |||
+ | |- valign="top" | ||
+ | | Version Template | ||
+ | | <string> (may contain pattern group references (i.e. $1, ...)) | ||
+ | | - | ||
+ | | If Version Pattern matched the given Version, then this Version Template is used to construct the version as seen in Maven, useful in particular to shorten versions ending in ".0" to 2-part versions etc.. The Version Template may contain pattern group references (i.e. $1, ...).<br> | ||
+ | If no Version Pattern is given or if the Version Pattern doesn't match, then the global strategy from Version Format is applied. | ||
|} | |} | ||
=Legacy format support= | =Legacy format support= | ||
+ | |||
+ | [Note: this section is not completely accurate, with the current version of the CBI aggregator. It is certainly desired to offer this level of support, but that is not currently the case.] | ||
+ | |||
As time passes, the aggregator model will undergo changes. All such changes will result in a new XML schema describing the aggregator model. The aggregator will always detect the schema in use by a model and, if needed, transform it into the current schema. | As time passes, the aggregator model will undergo changes. All such changes will result in a new XML schema describing the aggregator model. The aggregator will always detect the schema in use by a model and, if needed, transform it into the current schema. | ||
Line 1,087: | Line 1,122: | ||
Either: | Either: | ||
# Do the transformation using the same name | # Do the transformation using the same name | ||
− | # Manually rename the main | + | # Manually rename the main aggr file |
# Manually search/replace and change the name in all your contributions | # Manually search/replace and change the name in all your contributions | ||
Line 1,115: | Line 1,150: | ||
''Yes, it is possible to load p2 and maven2 repositories by default (if someone adds a custom loader then he can load any type of repository). The output is always p2 compatible, optionally combined with maven2 (with no extensibility - at least now). However, if a maven2 repo is loaded and the result is also maven2 compatible, it is not identical to the original (not all attributes loaded from maven are stored in the p2 model).'' | ''Yes, it is possible to load p2 and maven2 repositories by default (if someone adds a custom loader then he can load any type of repository). The output is always p2 compatible, optionally combined with maven2 (with no extensibility - at least now). However, if a maven2 repo is loaded and the result is also maven2 compatible, it is not identical to the original (not all attributes loaded from maven are stored in the p2 model).'' | ||
− | [[Category: | + | [[Category:CBI| ]] |
Revision as of 14:20, 27 July 2017
Contents
- 1 Introduction
- 2 Functional Overview
- 3 Installation
- 4 Getting started with standard examples
- 5 Headless support
- 6 Aggregation model components and specific actions
- 7 Legacy format support
- 8 What else should be documented
Introduction
Historically, the CBI Aggregator is based on the "b3 Aggregator" from the now defunct "Eclipse b3 Project".
The purpose of the CBI Aggregator is to aggregate or combine several p2 repositories. The aggregator does a number of things above and beyond simply "p2 mirroring" the different repositories: first and foremost, it makes sure that all the bundles could be "installed together" (that is, all the constraints are met and none are contradictory); it also ensures any "pack.gz" files are valid (i.e. can be unpacked into valid jars); and it also offers custom categories since the aggregated repository in most cases would not want to use the same categories as each individual repository uses. More details on its functionality follows.
The Aggregator combines repositories from various sources into a new aggregated p2 repository. It can also be configured to produce a hybrid p2/Maven2 repository. (This feature is currently labeled "experimental" in the CBI aggregator, since it is not yet production quality).
There are many situations where using aggregated repositories is a good solution. The reasons vary from licensing issues to organizational requirements:
- Owners of a p2 repo for a given project may not be in position to host all required or recommended components due to licensing issues - SVN support one example: as it requires components available through the main Eclipse p2 repo as well as third-party components. Hence users would normally have to visit several repos for a complete install, but by using the aggregator an institution could create a custom repository that has everything needed.
- Projects want to provide convenient access to their products - Installation instructions requiring the user to visit several repos for a complete install are not uncommon. An aggregated repo for all those locations provides a convenient one-stop strategy. The aggregation may support mirroring all consumed p2 repos or simply providing an indirection via a composite repo.
- Organizations or teams want control over internally used components - It may be necessary to have gated access to relevant p2 repos and do an organizational "healthcheck" of those before internal distribution. Furthermore, internally used aggregated repos can provide a common basis for all organizational users.
- Increase repository availability - by aggregating and mirroring what is used from multiple update sites into an internally controlled server (or several).
- Distributed Development Support - an overall product repository is produced by aggregating contributions from multiple teams.
The Aggregator tool is focused on supporting these specific requirements. The Aggregator is used in scenarios outside of the traditional "build domain" and this has been reflected in the user interface which does not delve into the details of "building" and should therefore be easy to use by non build experts.
Furthermore, it is worth noting that:
- the Aggregator is based on EMF models
- headless execution of aggregation definitions (once they have been created) is possible using a command line packaging of the Aggregator
Functional Overview
The CBI Aggregator performs aggregation and validation of repositories. The input to the aggregator engine (that tells it what to do) is a aggr EMF model. Such a model is most conveniently created by using the CBI Aggregator editor. This editor provides both editing and interactive execution of aggregation commands. The editor is based on a standard EMF "tree and properties view" style editor where nodes are added and removed to from a tree, and the details of nodes are edited in a separate properties view. Once a aggr model has been created it is possible to use the command line / headless aggregator to perform aggregation (and other related commands). (Note that since the aggr is "just an EMF model", it can be produced via EMF APIs, transformation tools, etc., and thus support advanced use cases).
The model mainly consists of Contributions - specifications of what to include from different repositories, and Validation Repositories - repositories that are used when validating, but which are not included in the produced aggregation (i.e., they are not copied). Contributions and Validation Repositories are grouped into Validation Sets. Everything in a Validation Set will be validated as one unit, i.e. it must be possible to install everything in a Validation Set together. The model also contains specification of various processing rules (exclusions, transformation of names, etc.), and specification of Contacts - individuals/mailing-lists to inform when processing fails.
Here are some of the important features supported by the CBI Aggregator:
- p2 and maven2 support — the aggregator can aggregate from and to both p2 and maven2 repositories.
- Maven2 name mapping support — names in the p2 domain are automatically mapped to maven2 names using built-in rules. Custom rules are also supported.
- Mirroring — artifacts from repositories are mirrored/downloaded/copied to a single location.
- Selective mirroring — an aggregation can produce an aggregate consisting of a mix of references to repositories and mirrored repositories.
- Cherry picking — it is possible to pick individual items when the entire content of a repository is not wanted. Detailed picking is supported as well as picking transitive closures like a product, or a category to get everything it contains/requires.
- Pruning — it is possible to specify mirroring based on version ranges. This can be used to reduce the size of the produced result when historical versions are not needed in the aggregated result.
- Categorization — categorization of installable units is important to the consumers of the aggregated repository. Categories are often choosen by repository publishers in a fashion that makes sense when looking at a particular repository in isolation, but when they are combined with others it can be very difficult for the user to understand what they relate to. An important task for the constructor of an aggregation is to be able to organize the aggregated material in an easily consumable fashion. The CBI Aggregator has support for category prefixing, category renaming, addition of custom categories, as well as adding and removing features in categories.
- Validation — the CBI Aggregator validates the aggregated Validation Sets to ensure that everything in them is installable at the same time.
- Blame Email — when issues are found during validation, the aggregator supports sending emails describing the issue. This is very useful when aggregating the result of many different projects. Advanced features include specifying contacts for parts of the aggregation, which is useful in large multi-layer project structures where issues may be related to the combination of a group of projects rather than one individual project - someone responsible for the aggregation itself should be informed about these cross-project issues. The aggregator supports detailed control over email generation, including handling of mock emails when testing aggregation scripts.
Installation
It is best to start by downloading a fresh release of the Eclipse SDK. The latest version of the aggregator, from its own software repository, typically goes with the most recently released version of the Eclipse SDK or the Eclipse Platform. The aggregator builds do not typically keep up with Milestones. It might work, but "user beware".
The CBI aggregator can either be integrated with your Eclipse SDK or it can be installed as a standalone headless product (i.e. pure command line, without any graphical UI).
The instructions below show the URLs at the time this section was written (that is, not necessarily the current URLs).
Always check the for the latest aggregator software repository on the CBI download page.
Eclipse SDK installation
|
Headless installation
Installation of the headless version of the Aggregator is similar to any typical headless installation using the p2Director. The following steps focus on the installation of the headless Aggregator feature.
- Install CBI Aggregator with the following command:
<p2_DIRECTOR> -r <HEADLESS_REPO> -d <INSTALL_DIR> -p CBIProfile -i org.eclipse.cbi.p2repo.cli.product
where
- <p2_DIRECTOR> is whatever method you use to invoke the p2Director application.
- -r <HEADLESS_REPO> is the headless p2 update site: Current stable version is: http://download.eclipse.org/cbi/updates/aggregator/headless/4.7/
- -d <INSTALL_DIR> is the chosen install location of the headless aggregator
- -p CBIProfile is the name of the p2 profile
- -i org.eclipse.cbi.p2repo.cli.product is the name of the headless CBI Aggregator
Getting started with standard examples
In the following sections we provide two simple examples that are easy to replicate and should highlight the most important features of the Aggregator. The first example deals with the creation of two variations of a p2 repo. The second shows the Aggregator's Maven support.
The *.aggr model files shown be below can be created via File > New > CBI Aggregator > Repository Aggregation.
Aggregating a p2 repo
The first sample aggregation is build around Buckminster and its support for Subversive. The objective of this aggregated repo is to:
- provide a "one-stop shop" experience
- conveniently pull in third-party components that are not hosted at Eclipse
- provide this repo as an indirection mechanism if required
Mirroring contributions
[NOTE: this section is out of date and has yet to be updated for the CBI aggregator]
(This example aggregation can be downloaded via the CBI project Git and opened in an appropriately set up workbench: [1]).
The background is that Buckminster provides support for Subclipse. In addition to all components hosted at Eclipse, a complete installation will also require Subclipse components from Tigris.org (http://subclipse.tigris.org/update_1.6.x). We want to create a repo that combines these components and makes them accessible from one location. We want to make several platform configurations available.
This example already includes some of the more advanced aggregation features. Stepping through the model view from the top node the following components can be identified:
- The top node Aggregation groups all model definitions regarding ValidationSets and Contributions. Looking at the properties section at the bottom we see that:
- the top node has been provided with a mandatory Label with the value "Indigo + Buckminster for Subclipse"; this is also the label that is shown to users when accessing the aggregated repo via the p2 update manager
- the Build Root identifies the location to which the artifacts of the aggregated repo will be deployed
- The aggregation is defined for three configurations (i.e. os=win32, ws=win32, arch=x86; etc)
- any number of configurations can be defined
- during the aggregation process all dependencies of the contributed components will be verified for all provided configurations, unless exceptions are defined (see below)
- We have one ValidationSet labeled "main". A ValidationSet constitutes everything that will be validated as one unit by the p2 planner.
- The main ValidationSet contains three different contributions.
- The first Contribution to the aggregation is labeled "Indigo". This contribution includes binary configuration-specific artifacts which are only available for linux. If a simple contribution would be defined the aggregation would fail for all non-linux configurations, and hence the aggregation would fail as a whole.
- this requires a definition of Valid Configurations Rules that state exceptions
- the rules defined for the the three components in question essentially state that the verification process for those components should only be performed for linux-based configurations
- one Mapped Repository is defined for this contribution (it can have multiple); all that is needed is a user-defined label and the URL of the repository that should be included
- the result of this definition is that all categories, products, and features from Indigo p2 repo will be included in the aggregated repo.
- The second Contribution is labeled "Subclipse" and deals with the inclusion of bundles provided from the Subclipse project. #*this contribution represents the simplest example of a contribution
- The third Contribution is labeled "Buckminster (latest)". It shows another advanced feature - an Exclusion Rule.
- remember that the objective of the sample repo is to provide convenient setup of Buckminster with Subclipse support, and since Buckminster's Subclipse and Subversive support are mutually exclusive, we want to exclude the features for Subversive from the aggregated repo to make it easier for the user.
- this is done using an Exclusion Rule defined for each Installable Unit that should be excluded
- A list of all included repos is displayed at the bottom of the model editor view
- this list allows browsing the contents of all repos
- this part of the model is not editable
The aggregation can be run by right-clicking any node in the model and selecting Build Aggregation. This example was setup to use a mirroring approach for all contributed repos. Hence, the complete contents of all included can be found in the aggregated repos target location specified under Build Root.
Check the next section for a slightly different approach.
Providing a repo indirection
Mirroring all repo artifacts of your aggregated contributions is a very valuable and important feature when performing aggregation, but there are also many cases where this is not necessary. It is possible to turn off artifact mirroring/copying by changing one property for a defined contribution.
Each Mapped Repository has a boolean property called Mirror Artifacts which can be set to The following buckminster_indigo_redirect.aggr is a variation of the first example with the Mirror Artifacts property set to Creating a Maven-conformant p2 repo
|