Jump to: navigation, search

Difference between revisions of "Orbit/Adding Bundles to Orbit"

(Contributing back to OSS communities)
m
 
(21 intermediate revisions by 5 users not shown)
Line 6: Line 6:
 
# Getting your project's PMC to grant their approval to use the 3rd-party library in their project.  This ensures that you should even be attempting an Orbit contribution, and that some Eclipse project actually intends to re-use the code that will be contributed to Orbit.  In fact, it's probably a good idea to solicit informal approval of your intention from your PMC before initiating the CQ process at all.
 
# Getting your project's PMC to grant their approval to use the 3rd-party library in their project.  This ensures that you should even be attempting an Orbit contribution, and that some Eclipse project actually intends to re-use the code that will be contributed to Orbit.  In fact, it's probably a good idea to solicit informal approval of your intention from your PMC before initiating the CQ process at all.
 
# If your project is eligible for the [[Development_Resources/HOWTO/Parallel_IP_Process|Parallel IP Process]], then you may as well begin working on your bundle.
 
# If your project is eligible for the [[Development_Resources/HOWTO/Parallel_IP_Process|Parallel IP Process]], then you may as well begin working on your bundle.
# When the Eclipse foundation has approved your 3rd party library for use in your project, raising a "piggy-back" (PB) CQ for the Orbit project to re-use the library. '''Orbit, like any other project, requires permission from the Eclipse Foundation to include any 3rd-party code in its repository and builds.'''
+
# When the Eclipse foundation has approved your 3rd party library for use in your project, raising a "piggy-back" (PB) CQ for the Orbit project to re-use the library. In the MyFoundation portal, [view] the tools.orbit project and click [add] an approved third-party library to Orbit. '''Orbit, like any other project, requires permission from the Eclipse Foundation to include any 3rd-party code in its repository and builds.'''
  
 
This is '''not a definitive guide''' to the legal process for 3rd-party content.  For that, see the wealth of information in the [[Development Resources]].
 
This is '''not a definitive guide''' to the legal process for 3rd-party content.  For that, see the wealth of information in the [[Development Resources]].
  
== Setting up CVS for the new bundle==
+
== Checklist ==
 +
 
 +
The [[Orbit Bundle Checklist]] provides a concise checklist of the following steps.
 +
 
 +
== Setting up GIT for the new bundle==
 
=== Creating a new project for an Orbit bundle ===  
 
=== Creating a new project for an Orbit bundle ===  
When adding an entirely new library to Orbit, you have to create a project in CVS to house the content.  Follow these steps to set this up the first time.
+
When adding an entirely new library to Orbit, you have to create a module in GIT to house the content.  Follow these steps to set this up the first time.
# In your workspace, create a project with the desired name (see [[Bundle Naming]]). For this example we'll use com.example.foo.
+
# First decide on the desired name (see [[Bundle Naming]]) and make sure to know the groupId, artifactId, and version for the maven artifact that will serve as a base for the bundle. For this example we'll use com.example.foo.
# In the project, create a readme.txt file that alerts users to the fact that the real content of the project is maintained in CVS branches. Something like <pre>Work in this project happens in various branches according to the &#10;version of the library being maintained. For example, if you are &#10;looking for javax.servlet version 2.3, check out the v2_3 branch &#10;of the javax.servlet project.</pre>
+
# Make sure you have the [http://git.eclipse.org/c/gerrit/orbit/orbit-recipes.git/ orbit-recipes] project cloned
# Share the project into the Orbit repository using the '''Team > Share Project...''' wizard. There select '''CVS''' then the "dev.eclipse.org/cvsroot/tools" repository.  In the following page, choose '''Use specified module name''' and enter "org.eclipse.orbit/com.example.foo" (remember to fill-in your bundle name). Finish off the wizard steps.
+
  $ git clone ssh://USERNAME@git.eclipse.org:29418/orbit/orbit-recipes
# Immediately create a new branch of the newly created project following the steps in [[#newBranch|Creating a new branch]].
+
# Choose a folder in orbit-recipes git repository under which to create the recipe and then create it there using :
 +
$ mvn ebr:create-recipe -DgroupId=com.example -DartifactId=foo -Dversion=1.0.0 -DbundleSymbolicName=com.example.foo
  
=== Creating a new branch of an existing Orbit project ===
+
== Library not available on Maven Central ==
Since all real content in Orbit is maintained in branches, HEAD never contains any real content (just the readme.txt file described above).  Follow these steps to create a new branch.
+
The new orbit build process, which makes use of CBI automates a large amount of the work required to create a bundle from a regular jar library. However, it depends heavily on the library being available from a maven repository. The advantage to this approach is it takes away the burden of uploading the library onto the committers while also opening up the possibility of having contributors propose changes. Many libraries meet this requirement and can be found on Maven Central, but for those that are not there, or cannot be deployed there is an alternative.
# Check out HEAD of the desired project, or '''Replace with...''' HEAD if HEAD of the project is not currently in the workspace.
+
# Create a new branch using the '''Team > Branch...''' wizard.  You will be asked to enter a branch name. The typical pattern for that is "v<version number>" where the version number is that which was assigned by the original library producers with the '.' (period) characters replaced with '_' (underscore) characters.  So a library whose version is 2.3 would go in the v2_3 branch. Ensure that the "Work with this branch" option is selected and and finish off the wizard.
+
# Delete the readme.txt file
+
# Proceed to add the desired content and eventually commit the changes into the branch.
+
  
== Moving an existing bundle to Orbit ==
+
Much like the old process of manually uploading the jar (exploded) into CVS, the Orbit project provides a more convenient way to upload the library as well as its sources to our own (Nexus) maven repository where approved artifacts can be hosted. This is provided through a HIPP (Hudson Instance Per-Project) at https://hudson.eclipse.org/orbit/job/upload-approved-artifacts/ . Committers would merely trigger a build by providing the library, its sources, the GAV (groupId, artifactId, version) that will identify it, and the build would deploy the artifact(s).
Taking a library that is already bundled in another Eclipse project and adding it to Orbit is quite straightforward.
+
# Create a [[#newProject|new project]] or [[#newBranch|new branch]] of an existing project to hold the bundle's content.
+
# Check out the existing version of the library into a project using a temporary name.  You need to do this if the new and old projects have the same name. Use the '''Team > Check Out As...''' wizard from the repository navigator.  Pick any name you like for the new project.  For example, com.example.foo.old helps keep things clear.  This project will only be in your workspace for a few minutes.
+
# Once the original project is checked out, copy and paste all content from the old project into the project you created at the beginning of these steps and delete the old project.  '''Note that you should use the Resource Navigator to ensure you get all the content.'''
+
# Commit the changes to the destination project to create an exact copy of the original bundle project in a branch in the Orbit repository.
+
  
 
== Adding a library for the first time ==
 
== Adding a library for the first time ==
Use this approach when you have a library (i.e., JAR) that you want to convert into a bundle.
+
The page titled, [[Adding Bundles To Orbit In 5 Minutes]] contains all the necessary information for adding a library to Orbit for the first time. The process is most straightforward when the library exists on Maven Central repository.
# Create a [[#newProject|new project]] or [[#newBranch|new branch]] of an existing project to hold the bundle's content.
+
# Start the '''New > Project > Plugin Development > Plug-in from existing JAR''' wizard to effectively import the library into a new project. 
+
# Enter the new bundle name etc.
+
# Ensure that the '''Analyze library contents and add dependencies''' checkbox is selected.
+
# Ensure that the '''Unzip the JAR into the project''' checkbox is selected. 
+
# Click '''Finish'''
+
 
+
== Adding a library that is already a bundle ==
+
See the com.ibm.icu*,  com.jcraft.jsch*, and org.sat4j* for existing examples.
+
 
+
Occasionally it may be desirable to add a bundle as a pre-built bundle, without exploding the jar and repackaging the .class files. This might be the case, for example, if the third party bundle was provided by the third party as a signed bundle. But, remember, that to qualify as a "pre-built" bundle, the bundle must have all the characteristics required of Orbit bundle, such as an acceptable about.html file, 4-part versioning, localization files, etc.
+
 
+
Libraries that are already bundles should be marked "prebuilt=true" in the map files. Also, the map entry must use the full paired-value form of entry, instead of the short-hand method, such as explicitly using 'CVS', 'tag', 'cvsroot', and 'path' for example:
+
 
+
plugin@com.ibm.icu,4.2.1=CVS,tag=v4_2_1_v20100413,cvsRoot=:pserver:anonymous@dev.eclipse.org:/cvsroot/tools,path=org.eclipse.orbit/com.ibm.icu/bin,prebuilt=true
+
  
Another, very important, thing to do for pre-built bundles is to update the build scripts to exclude those bundles from pre-conditioning for pack200. Hopefully they already where, from the third party. But, if we condition them, that would mean they would be different bits than the pre-existing versions, which is not desirable. Plus, in other cases, such as if the bundle was already signed, then trying to conditioning it will invalidate the signature. This 'pack exclude list' is currently "hard coded" in the buildutilities.xml file. As of this writing, 3/7/2009, the current pattern of files to exclude from conditioning is
+
== Moving an existing bundle to Orbit ==
"plugins/com.ibm.icu*,plugins/com.jcraft.jsch*,plugins/org.sat4j*"
+
Adding an existing bundle to Orbit is currently no too different from adding a library for the first time (See [[Adding Bundles To Orbit In 5 Minutes]]). The main difference is that with an existing bundle, certain OSGi metadata may need to be preserved while others may need to be reproduced to be as identical to the original metadata.
  
== Adding source for Orbit bundles ==
+
== <s> Adding source for Orbit bundles </s> ==
See [[Orbit Bundle Checklist#Create a Source Bundle]]
+
This is done automatically by the orbit-recipes build process.
  
 
== Project setup ==
 
== Project setup ==
 
No matter how you created your bundle in Orbit, the following outlines how the project itself should be configured.  Remember, it is often simpler to use the Resource Navigator rather than the Project Explorer since you really need to see all the files in your project and you aren't really compiling or building anything.
 
No matter how you created your bundle in Orbit, the following outlines how the project itself should be configured.  Remember, it is often simpler to use the Resource Navigator rather than the Project Explorer since you really need to see all the files in your project and you aren't really compiling or building anything.
  
* In general Orbit projects have the original JAR exploded into the root of the project.  This results in a directory structure such as <project root>org/apache/commons/logging/foo.class.
 
* Using the '''Project properties > Java Build Path > Source''' tab remove all source folders.  Check the '''Libraries''' tab to ensure that that project itself is listed as a library (since the class files are directly in the root of the project)
 
 
* In the MANIFEST.MF, ensure that the '''Bundle-Classpath:''' header is either missing or has only '.' as a value.
 
* In the MANIFEST.MF, ensure that the '''Bundle-Classpath:''' header is either missing or has only '.' as a value.
 
* For most cases there is no need to have '''Eclipse-Autostart: true''' or such headers since this library is generic and will not have an OSGi bundle activator.
 
* For most cases there is no need to have '''Eclipse-Autostart: true''' or such headers since this library is generic and will not have an OSGi bundle activator.
Line 67: Line 46:
 
* Where possible use '''Import-Package''' instead of '''Require-Bundle'''.  This reduces sensitivity to different bundlings of the same library.
 
* Where possible use '''Import-Package''' instead of '''Require-Bundle'''.  This reduces sensitivity to different bundlings of the same library.
 
* The Orbit community has agreed that '''Bundle-Version''' numbers should be the original library version number followed by .qualifier in the fourth segment.  In the event that the original number is already four segments, that version number should be used and then followed by "_qualifier".  ''Note that this versioning scheme differs from the normal Eclipse plug-in [[Version Numbering]] if the external library does not follow the Eclipse rules.''
 
* The Orbit community has agreed that '''Bundle-Version''' numbers should be the original library version number followed by .qualifier in the fourth segment.  In the event that the original number is already four segments, that version number should be used and then followed by "_qualifier".  ''Note that this versioning scheme differs from the normal Eclipse plug-in [[Version Numbering]] if the external library does not follow the Eclipse rules.''
* It is suggested that a 3 part version number be included in the project name, in the ''.project'' file. This is to enable those that use the Eclipse IDE preference (under Team, CVS) to "Use .project project name instead of module name on check out". This makes it easier to check out multiple versions of a bundle (project) from its various branches. For example,  
+
* It is suggested that a 3 part version number be included in the project name, in the ''.project'' file. This is to enable those that use the Eclipse IDE preference to "Use .project project name instead of module name on check out". This makes it easier to check out multiple versions of a bundle (project) from its various branches. For example,  
<pre><nowiki>
+
 
 
  <projectDescription>
 
  <projectDescription>
<name>javax.xml 1.3.4</name>
+
  <name>javax.xml_1.3.4</name>
<comment></comment>
+
  <comment></comment>
<projects>
+
  <projects>
</projects>
+
  ...
    .
+
  ...
    .
+
  </projects>
    .
+
</projectDescription>
 +
 
 +
== The osgi.bnd file ==
 +
 
 +
The osgi.bnd file is used to generate the MANIFEST.MF file for the bundle so it is very important the contents of this file are correct. This can be done by building the bundle locally to ensure the final manifest generated is correct. The following is a list of items to consider when finalizing the osgi.bnd file.
 +
 
 +
=== Import-Package should have an optional catch-all ===
 +
 
 +
All Import-package statements within the osgi.bnd file should end in a '*;resolution:=optional' statement that serves as a fallback in the event that any packages are missed.
 +
 
 +
Import-Package: \
 +
  org.foo.a.*;version="${range;[===,=+);${package-version}}", \
 +
  org.bar.b.*;version="${range;[===,=+);${package-version}}", \
 +
  *;resolution:=optional
 +
 
 +
It has been argued that things could be made easier by simply having 'Import-Package: *' and have all dependencies be automated. This is because contributors should be aware of every dependency that their package uses, so stating them explicitly should be required. This approach still allows grouping of various packages under a similar namespace using wild-card matches.
 +
 
 +
=== Emit No Import-Package ===
 +
 
 +
This can be done very simply by defining :
 +
Import-Package: !*
 +
 
 +
=== Permit Require-Bundle ===
 +
 
 +
By default, the Require-Bundle header is removed from the final generated manifest by the ebr-maven-plugin, and usage of this header in Orbit manifests is discouraged. If there is a good reason for why it needs to be used one can permit its use by overriding the property in the bundle's corresponding pom.xml file :
 +
 
 +
<properties>
 +
  <!-- Permit the use of Require-Bundle (in osgi.bnd file) -->
 +
  <recipe.removeadditionalheaders></recipe.removeadditionalheaders>
 +
</properties>
 +
 
 +
Now, the 'Require-Bundle' header may be defined in the osgi.bnd file.
 +
 
 +
=== <s> build.properties entries and .classpath file </s> ===
 +
This is not necessary as the process of creating the bundles is handled automatically.
  
</nowiki></pre>
 
=== build.properties entries and .classpath file ===
 
 
A special note about the <code>build.properties</code> file:  
 
A special note about the <code>build.properties</code> file:  
  
Line 104: Line 115:
 
But, the feature.xml file specifies  
 
But, the feature.xml file specifies  
  
     <plugin
+
     <plugin id="org.junit" version="4.1.0.qualifier" />
        id="org.junit"
+
        download-size="0"
+
        install-size="0"
+
        version="4.1.0.qualifier" />
+
  
And, the (multiversion) map file specifies
+
The resulting jar file is named as follows.  
 
+
    plugin@org.junit,4.1.0=v200701261101,:pserver:anonymous....
+
 
+
The resulting zip file is named as follows (zip, in this case, since it is not a jarred bundle).  
+
  
     org.junit_4.1.0.01_200704231307.zip
+
     org.junit_4.1.0.01_200704231307.jar
  
 
And, the resulting bundle version matches the file name, for example,  
 
And, the resulting bundle version matches the file name, for example,  
 
      
 
      
     Bundle-Version: 4.1.0.01_200704231307  
+
     Bundle-Version: 4.1.0.01_200704231307
  
Note, the final qualifier in this case does not "match" the cvs version tag, as the
 
usual case, but, is "translated" into current date/time to append to the
 
provided 4th segment prefix.
 
  
 
==== Avoid qualifiers that change with date and time of each build ====  
 
==== Avoid qualifiers that change with date and time of each build ====  
  
Note: to avoid the "current date and time" substitution for qualifier version -- which would cause a version increment each build, even though contents not changed, use the exact same form in the manifest, and in the feature.xml file. (See {{bug|355586}}.) For example, use 10.5.1.1_qualifier in the manifest.mf file, such as,  
+
Note: to avoid the "current date and time" substitution for qualifier version -- which would cause a version increment each build, even though contents not changed, use the exact same form in the feature.xml file. (See {{bug|355586}}.) For example, use 10.5.1.1_qualifier in the feature.xml file, such as,  
  
   Bundle-Version: 10.5.1.1_qualifier
+
   <plugin id="org.apache.derby" version="10.5.1.1_qualifier" />
  
and in feature.xml, such as
+
This will then cause the last time of modification to be substituted for 'qualifier', resulting in versions such as 10.5.1.1_v201108232300. This may look different from usual, but will remain constant from build to build.
 
+
  <plugin
+
        id="org.apache.derby"
+
        download-size="4859"
+
        install-size="4860"
+
        version="10.5.1.1_qualifier"
+
        unpack="false"/>
+
 
+
Then tag the version like usual, such as v201108232300 and use that tag in map file like usual, such as
+
 
+
  plugin@org.apache.derby,10.5.1=CVS,tag=v201108232300 ...
+
 
+
This will then cause the cvs tag to be substituted for 'qualifier', resulting in versions such as 10.5.1.1_v201108232300. This may look different from usual, but will remain constant from build to build.
+
  
 
=== Special Detailed Notes on the RC case ===  
 
=== Special Detailed Notes on the RC case ===  
Line 181: Line 168:
 
== Packaging licensing information ==
 
== Packaging licensing information ==
 
When packaging a third party library we have to ensure that the licensing information is present and correctly used.
 
When packaging a third party library we have to ensure that the licensing information is present and correctly used.
# Create an '''about_files''' directory and add a copy of the licensing text.
+
Generally, the '''about_files''', '''about.html''', '''build.properties''', and the '''ip_log.xml''' files will be automatically generated and correctly defined. They should still be checked to ensure the correct license has been detected and in the case of the '''ip_log.xml''' file, some additional fields will need to be filled in manually such as the corresponding ATO (Add To Orbit) CQ.
# Create an '''about.html''' file at the root of the third party bundle based on the '''template''' [http://nowhere here]... ''(eventually)''.
+
# Open your project's '''build.properties''' and switch to the PDE Build tab.
+
# Select and check the '''about_files''' directory and '''about.html''' under both your '''Binary Build''' and '''Source Build''' and then save your changes.
+
  
 
== National Language Considerations ==
 
== National Language Considerations ==
Line 196: Line 180:
 
* Be sure to "externalize" all the string in your manifest.mf files, and plugin.xml files, if you have any. There is usually at least two: the provider's name and the bundle's name.
 
* Be sure to "externalize" all the string in your manifest.mf files, and plugin.xml files, if you have any. There is usually at least two: the provider's name and the bundle's name.
  
== What to use as Provider Name? ==
+
== <s> What to use as Provider Name? </s> ==
  
Use "Eclipse Orbit".  
+
This is automatically generated by the orbit-recipes build but if for some reason this isn't being generated, Use "Eclipse Orbit".  
  
 
This is for the <code>Bundle-Vendor:</code> header in manifest.mf files. This should be externalized so what's in the manifest.mf file is a key, usually something like  
 
This is for the <code>Bundle-Vendor:</code> header in manifest.mf files. This should be externalized so what's in the manifest.mf file is a key, usually something like  
Line 211: Line 195:
  
 
There is one exception. We have some bundles that are pre-built and we just store them in our repository. There, the provider name is left as it is, of course, since it is pre-built. But anything we build/assemble should be "Eclipse Orbit".
 
There is one exception. We have some bundles that are pre-built and we just store them in our repository. There, the provider name is left as it is, of course, since it is pre-built. But anything we build/assemble should be "Eclipse Orbit".
 +
 +
== Avoid packing (pack200) nested jars ==
 +
 +
Unless there is an important reason to, avoid "packing" jars nested inside Orbit bundles. The reason is that changes in Java 7 make these packed children impossible to extract so tools such as p2 then has to "fall back" and use the plain jar version of the bundle anyway ... all adding up to a slight performance impact. See {{bug|361628}} for more details.
 +
 +
Mechanically, the way to accomplish this is to include a file named 'eclipse.inf' in the META-INF folder that has contents of
 +
 +
jarprocessor.exclude.children.pack=true
 +
 +
Or, if you want to excluded nested jars from packing and from signing, you can use
 +
 +
jarprocessor.exclude.children=true
 +
 +
Whether or not to exclude children from signing depends on how the jar is anticipated to be used. In some cases, consumers might expect it to be signed, in other cases, they might expect it not to be signed. For use in the Eclipse IDE, it does not particularly matter, the default Equinox settings will not 'verify' nested jars, though there are some server runtimes that do. Most bundles in Orbit use <code>jarprocessor.exclude.children=true</code>.
 +
 +
Occasionally, source-bundles also have "nested jars", so those would also need the eclipse.inf file put in their META-INF directory.
  
 
== Remember to retain important aspect of original jar manifest ==
 
== Remember to retain important aspect of original jar manifest ==
Line 217: Line 217:
  
 
In some cases, this means there may be aspects in the original jar's manifest that should be carried over to the manifest in the bundle.  
 
In some cases, this means there may be aspects in the original jar's manifest that should be carried over to the manifest in the bundle.  
 +
 +
=== Examples of things important to retain ===
  
 
One example, if the original jar's manifest contains a ''Main-Class'' directive, then the bundle's manifest should contain the same ''Main-Class'' directives. One case of this is the Rhino bundle (org.mozilla.javascript). It originally contained  
 
One example, if the original jar's manifest contains a ''Main-Class'' directive, then the bundle's manifest should contain the same ''Main-Class'' directives. One case of this is the Rhino bundle (org.mozilla.javascript). It originally contained  
 
   Main-Class: org.mozilla.javascript.tools.shell.Main
 
   Main-Class: org.mozilla.javascript.tools.shell.Main
 
which prints the version and starts up an interpreter "shell". It's best to maintain that sort of functionality whenever possible.
 
which prints the version and starts up an interpreter "shell". It's best to maintain that sort of functionality whenever possible.
 +
 +
=== Examples of things not to retain ===
 +
 +
An example of what not to retain was raised in {{bug|432032}}. Some jars have ''Class-Path:'' directives in their MANIFEST.MF files. This is the "non-OSGi" way of expressing a dependency on another jar (not to be confused with OSGi's ''Bundle-ClassPath:'' which is a different concept). In the cases I've seen, these 'Class-Path' directives become meaningless in OSGi bundles, either because the path or the jar referenced in that 'Class-Path' no longer exists, or it exists in a different form, such as with with version information added to the name. In some cases it becomes part of what a bundle "requires", which, naturally, is expressed in the "OSGi way". Having the 'Class-Path' there has no effect on the bundle when run with OSGi and none of our build systems pay attention to it, but as mentioned in {{bug|432032}} there are some build systems that will flag any "non-existing file" or "invalid path" as an error, interfering with the build. Hence, it's a good idea to remove any lines that have a 'Class-Path' directive. In most cases, though, this directive is probably useful in helping you "convert" the information when you create the OSGi bundle, that is, that you can add a proper ''Require-Bundle:'' to express the same relationship in the OSGi way.
  
 
== Export-Package guidelines ==  
 
== Export-Package guidelines ==  
Line 232: Line 238:
 
== Adding your bundle to the feature.xml ==
 
== Adding your bundle to the feature.xml ==
  
There is a feature in Orbit that is used solely to drive the build. To modify it, checkout the org.eclipse.orbit.build.feature.set1 project. You'd add your bundle to these feature like any other.  For example,  
+
There is a feature in Orbit that is used solely to drive the build. To modify it, checkout the [http://git.eclipse.org/c/gerrit/orbit/orbit-recipes.git/ orbit-recipes] project. It will be located under releng/aggregationfeature/feature.xml. You'd add your bundle to this feature like any other.  For example,  
 
+
    <plugin
+
        id="javax.servlet"
+
        download-size="0"
+
        install-size="0"
+
        version="2.3.0.qualifier"
+
        unpack="false"/>
+
 
+
If your bundle should be unzipped when it is installed, you need to omit the unpack="false" attribute, as usual. For example,
+
 
+
  <plugin
+
        id="org.junit"
+
        download-size="0"
+
        install-size="0"
+
        version="4.8.1.qualifier"/>
+
  
'''Important:''' If you have added a new bundle to the feature.xml file, then the feature project must be tagged and its map file entry must be updated. Read the instructions in the next section to learn how to update the map file.
+
    <plugin id="javax.servlet" version="2.3.0.qualifier" />
  
 
== Keeping track of legal information ==
 
== Keeping track of legal information ==
Line 255: Line 246:
  
 
== Adding your bundle to the Orbit Build ==
 
== Adding your bundle to the Orbit Build ==
See [[Orbit_Builds#Orbit_Builds_for_Orbit_Committers | Orbit Builds for Orbit Committers]].
+
See [[Adding_Bundles_To_Orbit_In_5_Minutes#Add_To_Build | Adding To Build]].
  
 
[[Category: Orbit]]
 
[[Category: Orbit]]

Latest revision as of 15:13, 22 February 2017

Before You Do Anything

Project Approvals and the CQ Process

Don't waste your time working up an OSGi bundling of a third-party library (or at least setting up a CVS project for a 3rd-party OSGi bundle) before you know that you will be able to use it. So, first start the legal approval process with the Eclipse Foundation's legal department by

  1. Raising a Contribution Questionnaire (CQ) requesting use of the 3rd-party code in your project (note: this is not the Orbit project, but the project in which you want to use the code). Do this in the MyFoundation portal. Raise one CQ for each JAR in the original 3rd-party software distribution that you intend to use.
  2. Getting your project's PMC to grant their approval to use the 3rd-party library in their project. This ensures that you should even be attempting an Orbit contribution, and that some Eclipse project actually intends to re-use the code that will be contributed to Orbit. In fact, it's probably a good idea to solicit informal approval of your intention from your PMC before initiating the CQ process at all.
  3. If your project is eligible for the Parallel IP Process, then you may as well begin working on your bundle.
  4. When the Eclipse foundation has approved your 3rd party library for use in your project, raising a "piggy-back" (PB) CQ for the Orbit project to re-use the library. In the MyFoundation portal, [view] the tools.orbit project and click [add] an approved third-party library to Orbit. Orbit, like any other project, requires permission from the Eclipse Foundation to include any 3rd-party code in its repository and builds.

This is not a definitive guide to the legal process for 3rd-party content. For that, see the wealth of information in the Development Resources.

Checklist

The Orbit Bundle Checklist provides a concise checklist of the following steps.

Setting up GIT for the new bundle

Creating a new project for an Orbit bundle

When adding an entirely new library to Orbit, you have to create a module in GIT to house the content. Follow these steps to set this up the first time.

  1. First decide on the desired name (see Bundle Naming) and make sure to know the groupId, artifactId, and version for the maven artifact that will serve as a base for the bundle. For this example we'll use com.example.foo.
  2. Make sure you have the orbit-recipes project cloned
$ git clone ssh://USERNAME@git.eclipse.org:29418/orbit/orbit-recipes
  1. Choose a folder in orbit-recipes git repository under which to create the recipe and then create it there using :
$ mvn ebr:create-recipe -DgroupId=com.example -DartifactId=foo -Dversion=1.0.0 -DbundleSymbolicName=com.example.foo

Library not available on Maven Central

The new orbit build process, which makes use of CBI automates a large amount of the work required to create a bundle from a regular jar library. However, it depends heavily on the library being available from a maven repository. The advantage to this approach is it takes away the burden of uploading the library onto the committers while also opening up the possibility of having contributors propose changes. Many libraries meet this requirement and can be found on Maven Central, but for those that are not there, or cannot be deployed there is an alternative.

Much like the old process of manually uploading the jar (exploded) into CVS, the Orbit project provides a more convenient way to upload the library as well as its sources to our own (Nexus) maven repository where approved artifacts can be hosted. This is provided through a HIPP (Hudson Instance Per-Project) at https://hudson.eclipse.org/orbit/job/upload-approved-artifacts/ . Committers would merely trigger a build by providing the library, its sources, the GAV (groupId, artifactId, version) that will identify it, and the build would deploy the artifact(s).

Adding a library for the first time

The page titled, Adding Bundles To Orbit In 5 Minutes contains all the necessary information for adding a library to Orbit for the first time. The process is most straightforward when the library exists on Maven Central repository.

Moving an existing bundle to Orbit

Adding an existing bundle to Orbit is currently no too different from adding a library for the first time (See Adding Bundles To Orbit In 5 Minutes). The main difference is that with an existing bundle, certain OSGi metadata may need to be preserved while others may need to be reproduced to be as identical to the original metadata.

Adding source for Orbit bundles

This is done automatically by the orbit-recipes build process.

Project setup

No matter how you created your bundle in Orbit, the following outlines how the project itself should be configured. Remember, it is often simpler to use the Resource Navigator rather than the Project Explorer since you really need to see all the files in your project and you aren't really compiling or building anything.

  • In the MANIFEST.MF, ensure that the Bundle-Classpath: header is either missing or has only '.' as a value.
  • For most cases there is no need to have Eclipse-Autostart: true or such headers since this library is generic and will not have an OSGi bundle activator.
  • Similarly, for most bundles there is no need to set the singleton:=true on the Bundle-SymbolicName header.
  • Of course, you should set the Bundle-RequiredExecutionEnvironment header to the absolute minimum JRE required by the library. That can often be difficult to figure out. Look for some tools in PDE to help with this...
  • Where possible use Import-Package instead of Require-Bundle. This reduces sensitivity to different bundlings of the same library.
  • The Orbit community has agreed that Bundle-Version numbers should be the original library version number followed by .qualifier in the fourth segment. In the event that the original number is already four segments, that version number should be used and then followed by "_qualifier". Note that this versioning scheme differs from the normal Eclipse plug-in Version Numbering if the external library does not follow the Eclipse rules.
  • It is suggested that a 3 part version number be included in the project name, in the .project file. This is to enable those that use the Eclipse IDE preference to "Use .project project name instead of module name on check out". This makes it easier to check out multiple versions of a bundle (project) from its various branches. For example,
<projectDescription>
  <name>javax.xml_1.3.4</name>
  <comment></comment>
  <projects>
  ...
  ...
  </projects>
</projectDescription>

The osgi.bnd file

The osgi.bnd file is used to generate the MANIFEST.MF file for the bundle so it is very important the contents of this file are correct. This can be done by building the bundle locally to ensure the final manifest generated is correct. The following is a list of items to consider when finalizing the osgi.bnd file.

Import-Package should have an optional catch-all

All Import-package statements within the osgi.bnd file should end in a '*;resolution:=optional' statement that serves as a fallback in the event that any packages are missed.

Import-Package: \
 org.foo.a.*;version="${range;[===,=+);${package-version}}", \
 org.bar.b.*;version="${range;[===,=+);${package-version}}", \
 *;resolution:=optional

It has been argued that things could be made easier by simply having 'Import-Package: *' and have all dependencies be automated. This is because contributors should be aware of every dependency that their package uses, so stating them explicitly should be required. This approach still allows grouping of various packages under a similar namespace using wild-card matches.

Emit No Import-Package

This can be done very simply by defining :

Import-Package: !*

Permit Require-Bundle

By default, the Require-Bundle header is removed from the final generated manifest by the ebr-maven-plugin, and usage of this header in Orbit manifests is discouraged. If there is a good reason for why it needs to be used one can permit its use by overriding the property in the bundle's corresponding pom.xml file :

<properties>
  <recipe.removeadditionalheaders></recipe.removeadditionalheaders>
</properties>

Now, the 'Require-Bundle' header may be defined in the osgi.bnd file.

build.properties entries and .classpath file

This is not necessary as the process of creating the bundles is handled automatically.

A special note about the build.properties file:

For your binary bundle (not your source bundle), in most cases it will have an entry like:

output..=.

This tells PDE that when someone has the project loaded into their workspace and other projects depend on it, then PDE should look at the project root for class files to compile against.

Note that it is important not to have a source..=. entry in your build.properties file as that will cause the bundle to be compiled and this can be a problem with the nested source bundles.

A special note about the .classpath file:

Be sure the class folders are listed in the project's build path properties and that they are marked exported. This only effects the .classpath file and effects how PDE and JDT "find" the classes if it happens to be checked out in a persons workspace while they are coding other plugins.

Special Detailed Notes on the four segment case

As documented above, in the event that the original package number is already four segments, that version number should be used and then followed by "_qualifier". These leads to a few differences in exactly how the final qualifier is computed and referred to in features and map files. The following example are taken from bug 152588.

For example, for the junit version 4.1.0.01, the manifest.mf file specifies

   Bundle-Version: 4.1.0.01_qualifier

But, the feature.xml file specifies

   <plugin id="org.junit" version="4.1.0.qualifier" />

The resulting jar file is named as follows.

    org.junit_4.1.0.01_200704231307.jar

And, the resulting bundle version matches the file name, for example,

    Bundle-Version: 4.1.0.01_200704231307


Avoid qualifiers that change with date and time of each build

Note: to avoid the "current date and time" substitution for qualifier version -- which would cause a version increment each build, even though contents not changed, use the exact same form in the feature.xml file. (See bug 355586.) For example, use 10.5.1.1_qualifier in the feature.xml file, such as,

  <plugin id="org.apache.derby" version="10.5.1.1_qualifier" />

This will then cause the last time of modification to be substituted for 'qualifier', resulting in versions such as 10.5.1.1_v201108232300. This may look different from usual, but will remain constant from build to build.

Special Detailed Notes on the RC case

Sometimes bundles need to be added that aren't full releases, but 'release candidates.' These release candidate plug-ins should share the same branch as their full release cousins.

For example, let's say we have a plug-in ch.ethz.iks.slp 1.0.0 RC2, the manifest.mf file specifies

   Bundle-Version: 1.0.0.RC2_qualifier

The branch the plug-in would be created in is v1_0_0 even though it's an RC2 release. Once 1.0.0 came out or say RC3, the branch would be updated with the info code. This essentially limits us to one "1.0.0" release at a time, but it's an acceptable workaround for the time being.

The feature.xml file specifies

   <plugin
       id="ch.ethz.iks.slp"
       download-size="0"
       install-size="0"
       version="1.0.0.qualifier" />

And, the (multiversion) map file specifies

    plugin@ch.ethz.iks.slp,1.0.0=v200701261101,:pserver:anonymous....

The resulting zip file is named as follows (zip, in this case, since it is not a jarred bundle).

    ch.ethz.iks.slp_1.0.0.RC2_200704231307.zip

And, the resulting bundle version matches the file name, for example,

    Bundle-Version: 1.0.0.RC2_200704231307

Note: It's incredibly important that the final release is lexicographically greater than the release candidate so it can be updated properly. For example, ch.ethz.iks.slp_1.0.0.RC2_v200801291200 is > than ch.ethz.iks.slp.1.0.0_v200801291200 because "R" is considered greater than "v"

Packaging licensing information

When packaging a third party library we have to ensure that the licensing information is present and correctly used. Generally, the about_files, about.html, build.properties, and the ip_log.xml files will be automatically generated and correctly defined. They should still be checked to ensure the correct license has been detected and in the case of the ip_log.xml file, some additional fields will need to be filled in manually such as the corresponding ATO (Add To Orbit) CQ.

National Language Considerations

The following are some tips to help make sure any text in the bundle is not "garbled" during network or storage transfers and it is capable of being properly translated to languages other than English.

  • As I think is true for all projects in an Eclipse repository, it is recommend you set the project's default encoding preference to ISO-8859-1. This project encoding applies to files that do not otherwise have any ability to specify their encoding (such as ".txt" or ".java" files). This is a good "single byte" encoding in general, and I think the CVS Repository Server assumes ISO-8859-1 as the default. Having it set as a project setting is good, so that, for example, if someone on windows checks it out and makes changes, and someone on Linux checks it out and makes changes, the default encoding for that project will be the same, no matter what the default encoding are on those different OS platforms.
  • Be sure to "externalize" all the string in your manifest.mf files, and plugin.xml files, if you have any. There is usually at least two: the provider's name and the bundle's name.

What to use as Provider Name?

This is automatically generated by the orbit-recipes build but if for some reason this isn't being generated, Use "Eclipse Orbit".

This is for the Bundle-Vendor: header in manifest.mf files. This should be externalized so what's in the manifest.mf file is a key, usually something like

Bundle-Vendor: %Bundle-Vendor.0

and then in the plugin.properties, or bundle.properties file is there you would have

Bundle-Vendor.0 = Eclipse Orbit

Historically, it used to be required to use "Eclipse.org". Beginning in Galileo, however, the recommendation (requirement, actually) is to use 'Eclipse' followed by the project name, avoiding acronyms and abbreviations etc, acknowledging that these are for end-users to read. The appropriate "project name" differs from one top level project to another. For example, in WTP, all sub-projects use the top level project name ("Eclipse Web Tools Platform"). But in Tools, since the sub-projects are all fairly different from one another, the Tools PMC decided to use "Eclipse" followed by sub-project name. So, that's "Eclipse Orbit" for us.

There is one exception. We have some bundles that are pre-built and we just store them in our repository. There, the provider name is left as it is, of course, since it is pre-built. But anything we build/assemble should be "Eclipse Orbit".

Avoid packing (pack200) nested jars

Unless there is an important reason to, avoid "packing" jars nested inside Orbit bundles. The reason is that changes in Java 7 make these packed children impossible to extract so tools such as p2 then has to "fall back" and use the plain jar version of the bundle anyway ... all adding up to a slight performance impact. See bug 361628 for more details.

Mechanically, the way to accomplish this is to include a file named 'eclipse.inf' in the META-INF folder that has contents of

jarprocessor.exclude.children.pack=true

Or, if you want to excluded nested jars from packing and from signing, you can use

jarprocessor.exclude.children=true

Whether or not to exclude children from signing depends on how the jar is anticipated to be used. In some cases, consumers might expect it to be signed, in other cases, they might expect it not to be signed. For use in the Eclipse IDE, it does not particularly matter, the default Equinox settings will not 'verify' nested jars, though there are some server runtimes that do. Most bundles in Orbit use jarprocessor.exclude.children=true.

Occasionally, source-bundles also have "nested jars", so those would also need the eclipse.inf file put in their META-INF directory.

Remember to retain important aspect of original jar manifest

Part of the goal of creating Orbit bundles, is that when ever possible, the Orbit bundle can continue to be used as a jar, just as it would have been originally.

In some cases, this means there may be aspects in the original jar's manifest that should be carried over to the manifest in the bundle.

Examples of things important to retain

One example, if the original jar's manifest contains a Main-Class directive, then the bundle's manifest should contain the same Main-Class directives. One case of this is the Rhino bundle (org.mozilla.javascript). It originally contained

 Main-Class: org.mozilla.javascript.tools.shell.Main

which prints the version and starts up an interpreter "shell". It's best to maintain that sort of functionality whenever possible.

Examples of things not to retain

An example of what not to retain was raised in bug 432032. Some jars have Class-Path: directives in their MANIFEST.MF files. This is the "non-OSGi" way of expressing a dependency on another jar (not to be confused with OSGi's Bundle-ClassPath: which is a different concept). In the cases I've seen, these 'Class-Path' directives become meaningless in OSGi bundles, either because the path or the jar referenced in that 'Class-Path' no longer exists, or it exists in a different form, such as with with version information added to the name. In some cases it becomes part of what a bundle "requires", which, naturally, is expressed in the "OSGi way". Having the 'Class-Path' there has no effect on the bundle when run with OSGi and none of our build systems pay attention to it, but as mentioned in bug 432032 there are some build systems that will flag any "non-existing file" or "invalid path" as an error, interfering with the build. Hence, it's a good idea to remove any lines that have a 'Class-Path' directive. In most cases, though, this directive is probably useful in helping you "convert" the information when you create the OSGi bundle, that is, that you can add a proper Require-Bundle: to express the same relationship in the OSGi way.

Export-Package guidelines

Normally, all packages in a bundle should be listed in the Export-Package declaration in MANIFEST.MF.

But, if a bundle has packages that are not normally part of its namespace, those should not be blindly exported, unless it is known that they are supposed to be. That is, the first assumption should be the bundle simply uses those packages internally, and are not part of its API to expose to the rest of the world. See bug 344560 for a case where exporting "foreign" packages can cause problems.

Also, normally, all exported packages should list the version they are exporting. Keep in mind that if a package in an interface, defined by a specification, then the version of the exported package should be the version of the specification, not the bundle it is in. For example, we use "Export-Package: javax.xml;version="1.3", even though the package is in the javax.xml bundle versioned at 1.3.4.

Adding your bundle to the feature.xml

There is a feature in Orbit that is used solely to drive the build. To modify it, checkout the orbit-recipes project. It will be located under releng/aggregationfeature/feature.xml. You'd add your bundle to this feature like any other. For example,

   <plugin id="javax.servlet" version="2.3.0.qualifier" />

Keeping track of legal information

Add a file in org.eclipse.orbit.releng/ip_logs that contains IP log information. See Orbit IP Log for more information.

Adding your bundle to the Orbit Build

See Adding To Build.

Final steps (a.k.a being a good OSS citizen)

It's a good practice to contribute back OSGi packaging - many upstream are not aware of OSGi and will appreciate patches. Having OSGi manifests upstream has a couple of advantages:

  • shared burden of maintenance
  • easier adoption (3rd party software can use newer versions before it is located in Orbit and drive Eclipse migration to latest-and-greatest versions).
  • spreading the word about OSGi