Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Orbit/Adding Bundles to Orbit"

m (Fixed a typo and slightly streamlined the text.)
(49 intermediate revisions by 15 users not shown)
Line 1: Line 1:
== Moving an existing bundle to Orbit ==
+
== Before You Do Anything ==
Taking a library that is already bundled in another Eclipse project and adding it to Orbit is quite straightforward.
+
=== Project Approvals and the CQ Process ===
 +
Don't waste your time working up an OSGi bundling of a third-party library (or at least setting up a CVS project for a 3rd-party OSGi bundle) before you know that you will be able to use it.  So, first start the legal approval process with the Eclipse Foundation's legal department by
 +
 
 +
# Raising a Contribution Questionnaire (CQ) requesting use of the 3rd-party code in your project (note: this is not the Orbit project, but the project in which you want to use the code).  Do this in the [http://portal.eclipse.org MyFoundation portal].  Raise one CQ for each JAR in the original 3rd-party software distribution that you intend to use.
 +
# Getting your project's PMC to grant their approval to use the 3rd-party library in their project.  This ensures that you should even be attempting an Orbit contribution, and that some Eclipse project actually intends to re-use the code that will be contributed to Orbit.  In fact, it's probably a good idea to solicit informal approval of your intention from your PMC before initiating the CQ process at all.
 +
# If your project is eligible for the [[Development_Resources/HOWTO/Parallel_IP_Process|Parallel IP Process]], then you may as well begin working on your bundle.
 +
# When the Eclipse foundation has approved your 3rd party library for use in your project, raising a "piggy-back" (PB) CQ for the Orbit project to re-use the library.  '''Orbit, like any other project, requires permission from the Eclipse Foundation to include any 3rd-party code in its repository and builds.'''
 +
 
 +
This is '''not a definitive guide''' to the legal process for 3rd-party content.  For that, see the wealth of information in the [[Development Resources]].
 +
 
 +
== Checklist ==
 +
 
 +
The [[Orbit Bundle Checklist]] provides a concise checklist of the following steps.
 +
 
 +
== Setting up CVS for the new bundle==
 +
=== Creating a new project for an Orbit bundle ===
 +
When adding an entirely new library to Orbit, you have to create a project in CVS to house the content.  Follow these steps to set this up the first time.
 
# In your workspace, create a project with the desired name (see [[Bundle Naming]]).  For this example we'll use com.example.foo.
 
# In your workspace, create a project with the desired name (see [[Bundle Naming]]).  For this example we'll use com.example.foo.
# In the project, create a readme.txt file that alerts users to the fact that the real content of the project is maintained in CVS branches.  Something like <pre><nowiki>Work in this project happens in various branches according to the  
+
# In the project, create a readme.txt file that alerts users to the fact that the real content of the project is maintained in CVS branches.  Something like <pre>Work in this project happens in various branches according to the &#10;version of the library being maintained.  For example, if you are &#10;looking for javax.servlet version 2.3, check out the v2_3 branch &#10;of the javax.servlet project.</pre>
version of the library being maintained.  For example, if you are  
+
looking for javax.servlet version 2.3, check out the v2_3 branch  
+
of the javax.servlet project.</nowiki></pre>
+
 
# Share the project into the Orbit repository using the '''Team > Share Project...''' wizard.  There select '''CVS''' then the "dev.eclipse.org/cvsroot/tools" repository.  In the following page, choose '''Use specified module name''' and enter "org.eclipse.orbit/com.example.foo" (remember to fill-in your bundle name).  Finish off the wizard steps.
 
# Share the project into the Orbit repository using the '''Team > Share Project...''' wizard.  There select '''CVS''' then the "dev.eclipse.org/cvsroot/tools" repository.  In the following page, choose '''Use specified module name''' and enter "org.eclipse.orbit/com.example.foo" (remember to fill-in your bundle name).  Finish off the wizard steps.
# Since all real content in Orbit is maintained in branches, immediately create a branch using the '''Team > Branch...''' wizard.  You will be asked to enter a branch name. The typical pattern for that is "v<version number>" where the version number is that which was assigned by the original library producers with the '.' (period) characters replaced with '_' (underscore) characters.  So a library whose version is 2.3 would go in the v2_3 branch.  Ensure that the "Work with this branch" option is selected and and finish off the wizard.
+
# Immediately create a new branch of the newly created project following the steps in [[#newBranch|Creating a new branch]].
 +
 
 +
=== Creating a new branch of an existing Orbit project ===
 +
Since all real content in Orbit is maintained in branches, HEAD never contains any real content (just the readme.txt file described above).  Follow these steps to create a new branch.
 +
# Check out HEAD of the desired project, or '''Replace with...''' HEAD if HEAD of the project is not currently in the workspace.
 +
# Create a new branch using the '''Team > Branch...''' wizard.  You will be asked to enter a branch name. The typical pattern for that is "v<version number>" where the version number is that which was assigned by the original library producers with the '.' (period) characters replaced with '_' (underscore) characters.  So a library whose version is 2.3 would go in the v2_3 branch.  Ensure that the "Work with this branch" option is selected and and finish off the wizard.
 
# Delete the readme.txt file
 
# Delete the readme.txt file
# Check out the existing version of the library using '''Team > Check Out As...''' wizard from the repository navigator.  Pick any name you like for the new project.  For example, com.example.foo.old helps keep things clear.  This project will only be in your workspace for a few minutes
+
# Proceed to add the desired content and eventually commit the changes into the branch.
# Once the original project is checked out, copy and paste all content from the old project into the project you created at the beginning of these steps and delete the old project.
+
 
# Commit the changes to the new project (e.g., com.example.foo) to create an exact copy of the original bundle project in a branch in the Orbit repository.
+
== Moving an existing bundle to Orbit ==
 +
Taking a library that is already bundled in another Eclipse project and adding it to Orbit is quite straightforward.
 +
# Create a [[#newProject|new project]] or [[#newBranch|new branch]] of an existing project to hold the bundle's content.
 +
# Check out the existing version of the library into a project using a temporary name.  You need to do this if the new and old projects have the same name.  Use the '''Team > Check Out As...''' wizard from the repository navigator.  Pick any name you like for the new project.  For example, com.example.foo.old helps keep things clear.  This project will only be in your workspace for a few minutes.
 +
# Once the original project is checked out, copy and paste all content from the old project into the project you created at the beginning of these steps and delete the old project. '''Note that you should use the Resource Navigator to ensure you get all the content.'''
 +
# Commit the changes to the destination project to create an exact copy of the original bundle project in a branch in the Orbit repository.
  
 
== Adding a library for the first time ==
 
== Adding a library for the first time ==
TBD
+
Use this approach when you have a library (i.e., JAR) that you want to convert into a bundle.
 +
# Create a [[#newProject|new project]] or [[#newBranch|new branch]] of an existing project to hold the bundle's content.
 +
# Start the '''New > Project > Plugin Development > Plug-in from existing JAR''' wizard to effectively import the library into a new project. 
 +
# Enter the new bundle name etc.
 +
# Ensure that the '''Analyze library contents and add dependencies''' checkbox is selected.
 +
# Ensure that the '''Unzip the JAR into the project''' checkbox is selected. 
 +
# Click '''Finish'''
  
 
== Adding a library that is already a bundle ==
 
== Adding a library that is already a bundle ==
TBD
+
See the com.ibm.icu*,  com.jcraft.jsch*, and org.sat4j* for existing examples.
 +
 
 +
Occasionally it may be desirable to add a bundle as a pre-built bundle, without exploding the jar and repackaging the .class files. This might be the case, for example, if the third party bundle was provided by the third party as a signed bundle. But, remember, that to qualify as a "pre-built" bundle, the bundle must have all the characteristics required of Orbit bundle, such as an acceptable about.html file, 4-part versioning, localization files, etc.
 +
 
 +
Libraries that are already bundles should be marked "prebuilt=true" in the map files. Also, the map entry must use the full paired-value form of entry, instead of the short-hand method, such as explicitly using 'CVS', 'tag', 'cvsroot', and 'path' for example:
 +
 
 +
plugin@com.ibm.icu,4.2.1=CVS,tag=v4_2_1_v20100413,cvsRoot=:pserver:anonymous@dev.eclipse.org:/cvsroot/tools,path=org.eclipse.orbit/com.ibm.icu/bin,prebuilt=true
 +
 
 +
Another, very important, thing to do for pre-built bundles is to update the build scripts to exclude those bundles from pre-conditioning for pack200. Hopefully they already where, from the third party. But, if we condition them, that would mean they would be different bits than the pre-existing versions, which is not desirable. Plus, in other cases, such as if the bundle was already signed, then trying to conditioning it will invalidate the signature. This 'pack exclude list' is currently "hard coded" in the buildutilities.xml file. As of this writing, 3/7/2009, the current pattern of files to exclude from conditioning is
 +
"plugins/com.ibm.icu*,plugins/com.jcraft.jsch*,plugins/org.sat4j*"
 +
 
 +
== Adding source for Orbit bundles ==
 +
See [[Orbit Bundle Checklist#Create a Source Bundle]]
 +
 
 +
== Project setup ==
 +
No matter how you created your bundle in Orbit, the following outlines how the project itself should be configured.  Remember, it is often simpler to use the Resource Navigator rather than the Project Explorer since you really need to see all the files in your project and you aren't really compiling or building anything.
 +
 
 +
* In general Orbit projects have the original JAR exploded into the root of the project.  This results in a directory structure such as <project root>org/apache/commons/logging/foo.class.
 +
* Using the '''Project properties > Java Build Path > Source''' tab remove all source folders.  Check the '''Libraries''' tab to ensure that that project itself is listed as a library (since the class files are directly in the root of the project)
 +
* In the MANIFEST.MF, ensure that the '''Bundle-Classpath:''' header is either missing or has only '.' as a value.
 +
* For most cases there is no need to have '''Eclipse-Autostart: true''' or such headers since this library is generic and will not have an OSGi bundle activator.
 +
* Similarly, for most bundles there is no need to set the '''singleton:=true''' on the '''Bundle-SymbolicName''' header.
 +
* Of course, you should set the '''Bundle-RequiredExecutionEnvironment''' header to the absolute minimum JRE required by the library.  That can often be difficult to figure out.  Look for some tools in PDE to help with this...
 +
* Where possible use '''Import-Package''' instead of '''Require-Bundle'''.  This reduces sensitivity to different bundlings of the same library.
 +
* The Orbit community has agreed that '''Bundle-Version''' numbers should be the original library version number followed by .qualifier in the fourth segment.  In the event that the original number is already four segments, that version number should be used and then followed by "_qualifier".  ''Note that this versioning scheme differs from the normal Eclipse plug-in [[Version Numbering]] if the external library does not follow the Eclipse rules.''
 +
* It is suggested that a 3 part version number be included in the project name, in the ''.project'' file. This is to enable those that use the Eclipse IDE preference (under Team, CVS) to "Use .project project name instead of module name on check out". This makes it easier to check out multiple versions of a bundle (project) from its various branches. For example,
 +
<pre><nowiki>
 +
<projectDescription>
 +
<name>javax.xml 1.3.4</name>
 +
<comment></comment>
 +
<projects>
 +
</projects>
 +
    .
 +
    .
 +
    .
 +
 
 +
</nowiki></pre>
 +
=== build.properties entries and .classpath file ===
 +
A special note about the <code>build.properties</code> file:
 +
 
 +
For your binary bundle (not your source bundle), in most cases it will have an entry like:
 +
<pre>
 +
output..=.
 +
</pre>
 +
This tells PDE that when someone has the project loaded into their workspace and other projects depend on it, then PDE should look at the project root for class files to compile against.
 +
 
 +
Note that it is important '''not''' to have a <code>source..=.</code> entry in your build.properties file as that will cause the bundle to be compiled and this can be a problem with the nested source bundles.
 +
 
 +
A special note about the <code>.classpath</code> file:
 +
 
 +
Be sure the class folders are listed in the project's <code>build path</code> properties and that they are marked ''exported''. This only effects the .classpath file and effects how PDE and JDT "find" the classes if it happens to be checked out in a persons workspace while they are coding other plugins.
 +
 
 +
=== Special Detailed Notes on the four segment case ===
 +
 
 +
As documented above, in the event that the original package number is already four segments, that version number should be used and then followed by "_qualifier". These leads to a few differences in exactly how the final qualifier is computed and referred to in features and map files. The following example are taken from [https://bugs.eclipse.org/bugs/show_bug.cgi?id=152588 bug 152588].
 +
 
 +
For example, for the junit version 4.1.0.01, the manifest.mf file specifies
 +
 
 +
    Bundle-Version: 4.1.0.01_qualifier
 +
 
 +
But, the feature.xml file specifies
 +
 
 +
    <plugin
 +
        id="org.junit"
 +
        download-size="0"
 +
        install-size="0"
 +
        version="4.1.0.qualifier" />
 +
 
 +
And, the (multiversion) map file specifies
 +
 
 +
    plugin@org.junit,4.1.0=v200701261101,:pserver:anonymous....
 +
 
 +
The resulting zip file is named as follows (zip, in this case, since it is not a jarred bundle).
 +
 
 +
    org.junit_4.1.0.01_200704231307.zip
 +
 
 +
And, the resulting bundle version matches the file name, for example,
 +
   
 +
    Bundle-Version: 4.1.0.01_200704231307
 +
 
 +
Note, the final qualifier in this case does not "match" the cvs version tag, as the
 +
usual case, but, is "translated" into current date/time to append to the
 +
provided 4th segment prefix.
 +
 
 +
==== Avoid qualifiers that change with date and time of each build ====
 +
 
 +
Note: to avoid the "current date and time" substitution for qualifier version -- which would cause a version increment each build, even though contents not changed, use the exact same form in the manifest, and in the feature.xml file. (See {{bug|355586}}.) For example, use 10.5.1.1_qualifier in the manifest.mf file, such as,
 +
 
 +
  Bundle-Version: 10.5.1.1_qualifier
 +
 
 +
and in feature.xml, such as
 +
 
 +
  <plugin
 +
        id="org.apache.derby"
 +
        download-size="4859"
 +
        install-size="4860"
 +
        version="10.5.1.1_qualifier"
 +
        unpack="false"/>
 +
 
 +
Then tag the version like usual, such as v201108232300 and use that tag in map file like usual, such as
 +
 
 +
  plugin@org.apache.derby,10.5.1=CVS,tag=v201108232300 ...
 +
 
 +
This will then cause the cvs tag to be substituted for 'qualifier', resulting in versions such as 10.5.1.1_v201108232300. This may look different from usual, but will remain constant from build to build.
 +
 
 +
=== Special Detailed Notes on the RC case ===
 +
 
 +
Sometimes bundles need to be added that aren't full releases, but 'release candidates.' These release candidate plug-ins should share the same branch as their full release cousins.
 +
 
 +
For example, let's say we have a plug-in ch.ethz.iks.slp 1.0.0 RC2, the manifest.mf file specifies
 +
 
 +
    Bundle-Version: 1.0.0.RC2_qualifier
 +
 
 +
The branch the plug-in would be created in is v1_0_0 even though it's an RC2 release. Once 1.0.0 came out or say RC3, the branch would be updated with the info code. This essentially limits us to one "1.0.0" release at a time, but it's an acceptable workaround for the time being.
 +
 
 +
The feature.xml file specifies
 +
 
 +
    <plugin
 +
        id="ch.ethz.iks.slp"
 +
        download-size="0"
 +
        install-size="0"
 +
        version="1.0.0.qualifier" />
 +
 
 +
And, the (multiversion) map file specifies
 +
 
 +
    plugin@ch.ethz.iks.slp,1.0.0=v200701261101,:pserver:anonymous....
 +
 
 +
The resulting zip file is named as follows (zip, in this case, since it is not a jarred bundle).
 +
 
 +
    ch.ethz.iks.slp_1.0.0.RC2_200704231307.zip
 +
 
 +
And, the resulting bundle version matches the file name, for example,
 +
   
 +
    Bundle-Version: 1.0.0.RC2_200704231307
 +
 
 +
Note: It's incredibly important that the final release is lexicographically greater than the release candidate so it can be updated properly. For example, ch.ethz.iks.slp_1.0.0.RC2_v200801291200 is > than ch.ethz.iks.slp.1.0.0_v200801291200 because "R" is considered greater than "v"
 +
 
 +
== Packaging licensing information ==
 +
When packaging a third party library we have to ensure that the licensing information is present and correctly used.
 +
# Create an '''about_files''' directory and add a copy of the licensing text.
 +
# Create an '''about.html''' file at the root of the third party bundle based on the '''template''' [http://nowhere here]... ''(eventually)''.
 +
# Open your project's '''build.properties''' and switch to the PDE Build tab.
 +
# Select and check the '''about_files''' directory and '''about.html''' under both your '''Binary Build''' and '''Source Build''' and then save your changes.
 +
 
 +
== National Language Considerations ==
 +
 
 +
The following are some tips to help make sure any text in the bundle
 +
is not "garbled" during network or storage transfers and it is capable of
 +
being properly translated to languages other than English.
 +
 
 +
* As I think is true for all projects in an Eclipse repository, it is recommend you set the project's default encoding preference to ISO-8859-1. This project encoding applies to files that do not otherwise have any ability to specify their encoding (such as ".txt" or ".java" files).  This is a good "single byte" encoding in general, and I think the CVS Repository Server assumes ISO-8859-1 as the default. Having it set as a project setting is good, so that, for example, if someone on windows checks it out and makes changes, and someone on Linux checks it out and makes changes, the default encoding for that project will be the same, no matter what the default encoding are on those different OS platforms.
 +
 
 +
* Be sure to "externalize" all the string in your manifest.mf files, and plugin.xml files, if you have any. There is usually at least two: the provider's name and the bundle's name.
 +
 
 +
== What to use as Provider Name? ==
 +
 
 +
Use "Eclipse Orbit".
 +
 
 +
This is for the <code>Bundle-Vendor:</code> header in manifest.mf files. This should be externalized so what's in the manifest.mf file is a key, usually something like
 +
 +
Bundle-Vendor: %Bundle-Vendor.0
 +
 
 +
and then in the plugin.properties, or bundle.properties file is there you would have
 +
 
 +
Bundle-Vendor.0 = Eclipse Orbit
 +
 
 +
Historically, it used to be required to use "Eclipse.org". Beginning in Galileo, however, the recommendation (requirement, actually) is to use 'Eclipse' followed by the project name, avoiding acronyms and abbreviations etc, acknowledging that these are for end-users to read. The appropriate "project name" differs from one top level project to another. For example, in WTP, all sub-projects use the top level project name ("Eclipse Web Tools Platform"). But in Tools, since the sub-projects are all fairly different from one another, the Tools PMC decided to use "Eclipse" followed by sub-project name. So, that's "Eclipse Orbit" for us.
 +
 
 +
There is one exception. We have some bundles that are pre-built and we just store them in our repository. There, the provider name is left as it is, of course, since it is pre-built. But anything we build/assemble should be "Eclipse Orbit".
 +
 
 +
== Avoid packing (pack200) nested jars ==
 +
 
 +
Unless there is an important reason to, avoid "packing" jars nested inside Orbit bundles. The reason is that changes in Java 7 make these packed children impossible to extract so tools such as p2 then has to "fall back" and use the plain jar version of the bundle anyway ... all adding up to a slight performance impact. See {{bug|361628}} for more details.
 +
 
 +
Mechanically, the way to accomplish this is to include a file named 'eclipse.inf' in the META-INF folder that has contents of
 +
 +
jarprocessor.exclude.children.pack=true
 +
 
 +
Or, if you want to excluded nested jars from packing and from signing, you can use
 +
 
 +
jarprocessor.exclude.children=true
 +
 
 +
Whether or not to exclude children from signing depends on how the jar is anticipated to be used. In some cases, consumers might expect it to be signed, in other cases, they might expect it not to be signed. For use in the Eclipse IDE, it does not particularly matter, the default Equinox settings will not 'verify' nested jars, though there are some server runtimes that do. Most bundles in Orbit use <code>jarprocessor.exclude.children=true</code>.
 +
 
 +
Occasionally, source-bundles also have "nested jars", so those would also need the eclipse.inf file put in their META-INF directory.
 +
 
 +
== Remember to retain important aspect of original jar manifest ==
 +
 
 +
Part of the goal of creating Orbit bundles, is that when ever possible, the Orbit bundle can continue to be used as a jar, just as it would have been originally.
 +
 
 +
In some cases, this means there may be aspects in the original jar's manifest that should be carried over to the manifest in the bundle.
 +
 
 +
=== Examples of things important to retain ===
 +
 
 +
One example, if the original jar's manifest contains a ''Main-Class'' directive, then the bundle's manifest should contain the same ''Main-Class'' directives. One case of this is the Rhino bundle (org.mozilla.javascript). It originally contained
 +
  Main-Class: org.mozilla.javascript.tools.shell.Main
 +
which prints the version and starts up an interpreter "shell". It's best to maintain that sort of functionality whenever possible.
 +
 
 +
=== Examples of things not to retain ===
 +
 
 +
An example of what not to retain was raised in {{bug|432032}}. Some jars have ''Class-Path:'' directives in their MANIFEST.MF files. This is the "non-OSGi" way of expressing a dependency on another jar (not to be confused with OSGi's ''Bundle-ClassPath:'' which is a different concept). In the cases I've seen, these 'Class-Path' directives become meaningless in OSGi bundles, either because the path or the jar referenced in that 'Class-Path' no longer exists, or it exists in a different form, such as with with version information added to the name. In some cases it becomes part of what a bundle "requires", which, naturally, is expressed in the "OSGi way". Having the 'Class-Path' there has no effect on the bundle when run with OSGi and none of our build systems pay attention to it, but as mentioned in {{bug|432032}} there are some build systems that will flag any "non-existing file" or "invalid path" as an error, interfering with the build. Hence, it's a good idea to remove any lines that have a 'Class-Path' directive. In most cases, though, this directive is probably useful in helping you "convert" the information when you create the OSGi bundle, that is, that you can add a proper ''Require-Bundle:'' to express the same relationship in the OSGi way.
 +
 
 +
== Export-Package guidelines ==
 +
 
 +
Normally, all packages in a bundle should be listed in the Export-Package declaration in MANIFEST.MF.
 +
 
 +
But, if a bundle has packages that are not normally part of its namespace, those should not be blindly exported, unless it is known that they are supposed to be. That is, the first assumption should be the bundle simply uses those packages internally, and are not part of its API to expose to the rest of the world. See {{bug|344560}} for a case where exporting "foreign" packages can cause problems.
 +
 
 +
Also, normally, all exported packages should list the version they are exporting. Keep in mind that if a package in an interface, defined by a specification, then the version of the exported package should be the version of the specification, not the bundle it is in. For example, we use <code>"Export-Package: javax.xml;version="1.3"</code>, even though the package is in the javax.xml bundle versioned at 1.3.4.
 +
 
 +
== Adding your bundle to the feature.xml ==
 +
 
 +
There is a feature in Orbit that is used solely to drive the build. To modify it, checkout the org.eclipse.orbit.build.feature.set1 project. You'd add your bundle to these feature like any other.  For example,
 +
 
 +
    <plugin
 +
        id="javax.servlet"
 +
        download-size="0"
 +
        install-size="0"
 +
        version="2.3.0.qualifier"
 +
        unpack="false"/>
 +
 
 +
If your bundle should be unzipped when it is installed, you need to omit the unpack="false" attribute, as usual. For example,
 +
 
 +
  <plugin
 +
        id="org.junit"
 +
        download-size="0"
 +
        install-size="0"
 +
        version="4.8.1.qualifier"/>
 +
 
 +
'''Important:''' If you have added a new bundle to the feature.xml file, then the feature project must be tagged and its map file entry must be updated. See [[Orbit Bundle Checklist#Update the Map File]]
 +
 
 +
== Keeping track of legal information ==
 +
Add a file in org.eclipse.orbit.releng/ip_logs that contains IP log information.  See [[Orbit/IP_Log | Orbit IP Log]] for more information.
 +
 
 +
== Adding your bundle to the Orbit Build ==
 +
See [[Orbit_Builds#Orbit_Builds_for_Orbit_Committers | Orbit Builds for Orbit Committers]].
  
 
[[Category: Orbit]]
 
[[Category: Orbit]]
 +
 +
== Final steps (a.k.a being a good OSS citizen) ==
 +
 +
It's a good practice to contribute back OSGi packaging - many upstream are not aware of OSGi and will appreciate patches.
 +
Having OSGi manifests upstream has a couple of advantages:
 +
* shared burden of maintenance
 +
* easier adoption (3rd party software can use newer versions before it is located in Orbit and drive Eclipse migration to latest-and-greatest versions).
 +
* spreading the word about OSGi

Revision as of 00:46, 8 April 2014

Before You Do Anything

Project Approvals and the CQ Process

Don't waste your time working up an OSGi bundling of a third-party library (or at least setting up a CVS project for a 3rd-party OSGi bundle) before you know that you will be able to use it. So, first start the legal approval process with the Eclipse Foundation's legal department by

  1. Raising a Contribution Questionnaire (CQ) requesting use of the 3rd-party code in your project (note: this is not the Orbit project, but the project in which you want to use the code). Do this in the MyFoundation portal. Raise one CQ for each JAR in the original 3rd-party software distribution that you intend to use.
  2. Getting your project's PMC to grant their approval to use the 3rd-party library in their project. This ensures that you should even be attempting an Orbit contribution, and that some Eclipse project actually intends to re-use the code that will be contributed to Orbit. In fact, it's probably a good idea to solicit informal approval of your intention from your PMC before initiating the CQ process at all.
  3. If your project is eligible for the Parallel IP Process, then you may as well begin working on your bundle.
  4. When the Eclipse foundation has approved your 3rd party library for use in your project, raising a "piggy-back" (PB) CQ for the Orbit project to re-use the library. Orbit, like any other project, requires permission from the Eclipse Foundation to include any 3rd-party code in its repository and builds.

This is not a definitive guide to the legal process for 3rd-party content. For that, see the wealth of information in the Development Resources.

Checklist

The Orbit Bundle Checklist provides a concise checklist of the following steps.

Setting up CVS for the new bundle

Creating a new project for an Orbit bundle

When adding an entirely new library to Orbit, you have to create a project in CVS to house the content. Follow these steps to set this up the first time.

  1. In your workspace, create a project with the desired name (see Bundle Naming). For this example we'll use com.example.foo.
  2. In the project, create a readme.txt file that alerts users to the fact that the real content of the project is maintained in CVS branches. Something like
    Work in this project happens in various branches according to the 
    version of the library being maintained.  For example, if you are 
    looking for javax.servlet version 2.3, check out the v2_3 branch 
    of the javax.servlet project.
  3. Share the project into the Orbit repository using the Team > Share Project... wizard. There select CVS then the "dev.eclipse.org/cvsroot/tools" repository. In the following page, choose Use specified module name and enter "org.eclipse.orbit/com.example.foo" (remember to fill-in your bundle name). Finish off the wizard steps.
  4. Immediately create a new branch of the newly created project following the steps in Creating a new branch.

Creating a new branch of an existing Orbit project

Since all real content in Orbit is maintained in branches, HEAD never contains any real content (just the readme.txt file described above). Follow these steps to create a new branch.

  1. Check out HEAD of the desired project, or Replace with... HEAD if HEAD of the project is not currently in the workspace.
  2. Create a new branch using the Team > Branch... wizard. You will be asked to enter a branch name. The typical pattern for that is "v<version number>" where the version number is that which was assigned by the original library producers with the '.' (period) characters replaced with '_' (underscore) characters. So a library whose version is 2.3 would go in the v2_3 branch. Ensure that the "Work with this branch" option is selected and and finish off the wizard.
  3. Delete the readme.txt file
  4. Proceed to add the desired content and eventually commit the changes into the branch.

Moving an existing bundle to Orbit

Taking a library that is already bundled in another Eclipse project and adding it to Orbit is quite straightforward.

  1. Create a new project or new branch of an existing project to hold the bundle's content.
  2. Check out the existing version of the library into a project using a temporary name. You need to do this if the new and old projects have the same name. Use the Team > Check Out As... wizard from the repository navigator. Pick any name you like for the new project. For example, com.example.foo.old helps keep things clear. This project will only be in your workspace for a few minutes.
  3. Once the original project is checked out, copy and paste all content from the old project into the project you created at the beginning of these steps and delete the old project. Note that you should use the Resource Navigator to ensure you get all the content.
  4. Commit the changes to the destination project to create an exact copy of the original bundle project in a branch in the Orbit repository.

Adding a library for the first time

Use this approach when you have a library (i.e., JAR) that you want to convert into a bundle.

  1. Create a new project or new branch of an existing project to hold the bundle's content.
  2. Start the New > Project > Plugin Development > Plug-in from existing JAR wizard to effectively import the library into a new project.
  3. Enter the new bundle name etc.
  4. Ensure that the Analyze library contents and add dependencies checkbox is selected.
  5. Ensure that the Unzip the JAR into the project checkbox is selected.
  6. Click Finish

Adding a library that is already a bundle

See the com.ibm.icu*, com.jcraft.jsch*, and org.sat4j* for existing examples.

Occasionally it may be desirable to add a bundle as a pre-built bundle, without exploding the jar and repackaging the .class files. This might be the case, for example, if the third party bundle was provided by the third party as a signed bundle. But, remember, that to qualify as a "pre-built" bundle, the bundle must have all the characteristics required of Orbit bundle, such as an acceptable about.html file, 4-part versioning, localization files, etc.

Libraries that are already bundles should be marked "prebuilt=true" in the map files. Also, the map entry must use the full paired-value form of entry, instead of the short-hand method, such as explicitly using 'CVS', 'tag', 'cvsroot', and 'path' for example:

plugin@com.ibm.icu,4.2.1=CVS,tag=v4_2_1_v20100413,cvsRoot=:pserver:anonymous@dev.eclipse.org:/cvsroot/tools,path=org.eclipse.orbit/com.ibm.icu/bin,prebuilt=true

Another, very important, thing to do for pre-built bundles is to update the build scripts to exclude those bundles from pre-conditioning for pack200. Hopefully they already where, from the third party. But, if we condition them, that would mean they would be different bits than the pre-existing versions, which is not desirable. Plus, in other cases, such as if the bundle was already signed, then trying to conditioning it will invalidate the signature. This 'pack exclude list' is currently "hard coded" in the buildutilities.xml file. As of this writing, 3/7/2009, the current pattern of files to exclude from conditioning is

"plugins/com.ibm.icu*,plugins/com.jcraft.jsch*,plugins/org.sat4j*"

Adding source for Orbit bundles

See Orbit Bundle Checklist#Create a Source Bundle

Project setup

No matter how you created your bundle in Orbit, the following outlines how the project itself should be configured. Remember, it is often simpler to use the Resource Navigator rather than the Project Explorer since you really need to see all the files in your project and you aren't really compiling or building anything.

  • In general Orbit projects have the original JAR exploded into the root of the project. This results in a directory structure such as <project root>org/apache/commons/logging/foo.class.
  • Using the Project properties > Java Build Path > Source tab remove all source folders. Check the Libraries tab to ensure that that project itself is listed as a library (since the class files are directly in the root of the project)
  • In the MANIFEST.MF, ensure that the Bundle-Classpath: header is either missing or has only '.' as a value.
  • For most cases there is no need to have Eclipse-Autostart: true or such headers since this library is generic and will not have an OSGi bundle activator.
  • Similarly, for most bundles there is no need to set the singleton:=true on the Bundle-SymbolicName header.
  • Of course, you should set the Bundle-RequiredExecutionEnvironment header to the absolute minimum JRE required by the library. That can often be difficult to figure out. Look for some tools in PDE to help with this...
  • Where possible use Import-Package instead of Require-Bundle. This reduces sensitivity to different bundlings of the same library.
  • The Orbit community has agreed that Bundle-Version numbers should be the original library version number followed by .qualifier in the fourth segment. In the event that the original number is already four segments, that version number should be used and then followed by "_qualifier". Note that this versioning scheme differs from the normal Eclipse plug-in Version Numbering if the external library does not follow the Eclipse rules.
  • It is suggested that a 3 part version number be included in the project name, in the .project file. This is to enable those that use the Eclipse IDE preference (under Team, CVS) to "Use .project project name instead of module name on check out". This makes it easier to check out multiple versions of a bundle (project) from its various branches. For example,
 <projectDescription>
	<name>javax.xml 1.3.4</name>
	<comment></comment>
	<projects>
	</projects>
    .
    .
    .

build.properties entries and .classpath file

A special note about the build.properties file:

For your binary bundle (not your source bundle), in most cases it will have an entry like:

output..=.

This tells PDE that when someone has the project loaded into their workspace and other projects depend on it, then PDE should look at the project root for class files to compile against.

Note that it is important not to have a source..=. entry in your build.properties file as that will cause the bundle to be compiled and this can be a problem with the nested source bundles.

A special note about the .classpath file:

Be sure the class folders are listed in the project's build path properties and that they are marked exported. This only effects the .classpath file and effects how PDE and JDT "find" the classes if it happens to be checked out in a persons workspace while they are coding other plugins.

Special Detailed Notes on the four segment case

As documented above, in the event that the original package number is already four segments, that version number should be used and then followed by "_qualifier". These leads to a few differences in exactly how the final qualifier is computed and referred to in features and map files. The following example are taken from bug 152588.

For example, for the junit version 4.1.0.01, the manifest.mf file specifies

   Bundle-Version: 4.1.0.01_qualifier

But, the feature.xml file specifies

   <plugin
       id="org.junit"
       download-size="0"
       install-size="0"
       version="4.1.0.qualifier" />

And, the (multiversion) map file specifies

    plugin@org.junit,4.1.0=v200701261101,:pserver:anonymous....

The resulting zip file is named as follows (zip, in this case, since it is not a jarred bundle).

    org.junit_4.1.0.01_200704231307.zip

And, the resulting bundle version matches the file name, for example,

    Bundle-Version: 4.1.0.01_200704231307 

Note, the final qualifier in this case does not "match" the cvs version tag, as the usual case, but, is "translated" into current date/time to append to the provided 4th segment prefix.

Avoid qualifiers that change with date and time of each build

Note: to avoid the "current date and time" substitution for qualifier version -- which would cause a version increment each build, even though contents not changed, use the exact same form in the manifest, and in the feature.xml file. (See bug 355586.) For example, use 10.5.1.1_qualifier in the manifest.mf file, such as,

  Bundle-Version: 10.5.1.1_qualifier

and in feature.xml, such as

  <plugin
        id="org.apache.derby"
        download-size="4859"
        install-size="4860"
        version="10.5.1.1_qualifier"
        unpack="false"/>

Then tag the version like usual, such as v201108232300 and use that tag in map file like usual, such as

  plugin@org.apache.derby,10.5.1=CVS,tag=v201108232300 ...

This will then cause the cvs tag to be substituted for 'qualifier', resulting in versions such as 10.5.1.1_v201108232300. This may look different from usual, but will remain constant from build to build.

Special Detailed Notes on the RC case

Sometimes bundles need to be added that aren't full releases, but 'release candidates.' These release candidate plug-ins should share the same branch as their full release cousins.

For example, let's say we have a plug-in ch.ethz.iks.slp 1.0.0 RC2, the manifest.mf file specifies

   Bundle-Version: 1.0.0.RC2_qualifier

The branch the plug-in would be created in is v1_0_0 even though it's an RC2 release. Once 1.0.0 came out or say RC3, the branch would be updated with the info code. This essentially limits us to one "1.0.0" release at a time, but it's an acceptable workaround for the time being.

The feature.xml file specifies

   <plugin
       id="ch.ethz.iks.slp"
       download-size="0"
       install-size="0"
       version="1.0.0.qualifier" />

And, the (multiversion) map file specifies

    plugin@ch.ethz.iks.slp,1.0.0=v200701261101,:pserver:anonymous....

The resulting zip file is named as follows (zip, in this case, since it is not a jarred bundle).

    ch.ethz.iks.slp_1.0.0.RC2_200704231307.zip

And, the resulting bundle version matches the file name, for example,

    Bundle-Version: 1.0.0.RC2_200704231307

Note: It's incredibly important that the final release is lexicographically greater than the release candidate so it can be updated properly. For example, ch.ethz.iks.slp_1.0.0.RC2_v200801291200 is > than ch.ethz.iks.slp.1.0.0_v200801291200 because "R" is considered greater than "v"

Packaging licensing information

When packaging a third party library we have to ensure that the licensing information is present and correctly used.

  1. Create an about_files directory and add a copy of the licensing text.
  2. Create an about.html file at the root of the third party bundle based on the template here... (eventually).
  3. Open your project's build.properties and switch to the PDE Build tab.
  4. Select and check the about_files directory and about.html under both your Binary Build and Source Build and then save your changes.

National Language Considerations

The following are some tips to help make sure any text in the bundle is not "garbled" during network or storage transfers and it is capable of being properly translated to languages other than English.

  • As I think is true for all projects in an Eclipse repository, it is recommend you set the project's default encoding preference to ISO-8859-1. This project encoding applies to files that do not otherwise have any ability to specify their encoding (such as ".txt" or ".java" files). This is a good "single byte" encoding in general, and I think the CVS Repository Server assumes ISO-8859-1 as the default. Having it set as a project setting is good, so that, for example, if someone on windows checks it out and makes changes, and someone on Linux checks it out and makes changes, the default encoding for that project will be the same, no matter what the default encoding are on those different OS platforms.
  • Be sure to "externalize" all the string in your manifest.mf files, and plugin.xml files, if you have any. There is usually at least two: the provider's name and the bundle's name.

What to use as Provider Name?

Use "Eclipse Orbit".

This is for the Bundle-Vendor: header in manifest.mf files. This should be externalized so what's in the manifest.mf file is a key, usually something like

Bundle-Vendor: %Bundle-Vendor.0

and then in the plugin.properties, or bundle.properties file is there you would have

Bundle-Vendor.0 = Eclipse Orbit

Historically, it used to be required to use "Eclipse.org". Beginning in Galileo, however, the recommendation (requirement, actually) is to use 'Eclipse' followed by the project name, avoiding acronyms and abbreviations etc, acknowledging that these are for end-users to read. The appropriate "project name" differs from one top level project to another. For example, in WTP, all sub-projects use the top level project name ("Eclipse Web Tools Platform"). But in Tools, since the sub-projects are all fairly different from one another, the Tools PMC decided to use "Eclipse" followed by sub-project name. So, that's "Eclipse Orbit" for us.

There is one exception. We have some bundles that are pre-built and we just store them in our repository. There, the provider name is left as it is, of course, since it is pre-built. But anything we build/assemble should be "Eclipse Orbit".

Avoid packing (pack200) nested jars

Unless there is an important reason to, avoid "packing" jars nested inside Orbit bundles. The reason is that changes in Java 7 make these packed children impossible to extract so tools such as p2 then has to "fall back" and use the plain jar version of the bundle anyway ... all adding up to a slight performance impact. See bug 361628 for more details.

Mechanically, the way to accomplish this is to include a file named 'eclipse.inf' in the META-INF folder that has contents of

jarprocessor.exclude.children.pack=true

Or, if you want to excluded nested jars from packing and from signing, you can use

jarprocessor.exclude.children=true

Whether or not to exclude children from signing depends on how the jar is anticipated to be used. In some cases, consumers might expect it to be signed, in other cases, they might expect it not to be signed. For use in the Eclipse IDE, it does not particularly matter, the default Equinox settings will not 'verify' nested jars, though there are some server runtimes that do. Most bundles in Orbit use jarprocessor.exclude.children=true.

Occasionally, source-bundles also have "nested jars", so those would also need the eclipse.inf file put in their META-INF directory.

Remember to retain important aspect of original jar manifest

Part of the goal of creating Orbit bundles, is that when ever possible, the Orbit bundle can continue to be used as a jar, just as it would have been originally.

In some cases, this means there may be aspects in the original jar's manifest that should be carried over to the manifest in the bundle.

Examples of things important to retain

One example, if the original jar's manifest contains a Main-Class directive, then the bundle's manifest should contain the same Main-Class directives. One case of this is the Rhino bundle (org.mozilla.javascript). It originally contained

 Main-Class: org.mozilla.javascript.tools.shell.Main

which prints the version and starts up an interpreter "shell". It's best to maintain that sort of functionality whenever possible.

Examples of things not to retain

An example of what not to retain was raised in bug 432032. Some jars have Class-Path: directives in their MANIFEST.MF files. This is the "non-OSGi" way of expressing a dependency on another jar (not to be confused with OSGi's Bundle-ClassPath: which is a different concept). In the cases I've seen, these 'Class-Path' directives become meaningless in OSGi bundles, either because the path or the jar referenced in that 'Class-Path' no longer exists, or it exists in a different form, such as with with version information added to the name. In some cases it becomes part of what a bundle "requires", which, naturally, is expressed in the "OSGi way". Having the 'Class-Path' there has no effect on the bundle when run with OSGi and none of our build systems pay attention to it, but as mentioned in bug 432032 there are some build systems that will flag any "non-existing file" or "invalid path" as an error, interfering with the build. Hence, it's a good idea to remove any lines that have a 'Class-Path' directive. In most cases, though, this directive is probably useful in helping you "convert" the information when you create the OSGi bundle, that is, that you can add a proper Require-Bundle: to express the same relationship in the OSGi way.

Export-Package guidelines

Normally, all packages in a bundle should be listed in the Export-Package declaration in MANIFEST.MF.

But, if a bundle has packages that are not normally part of its namespace, those should not be blindly exported, unless it is known that they are supposed to be. That is, the first assumption should be the bundle simply uses those packages internally, and are not part of its API to expose to the rest of the world. See bug 344560 for a case where exporting "foreign" packages can cause problems.

Also, normally, all exported packages should list the version they are exporting. Keep in mind that if a package in an interface, defined by a specification, then the version of the exported package should be the version of the specification, not the bundle it is in. For example, we use "Export-Package: javax.xml;version="1.3", even though the package is in the javax.xml bundle versioned at 1.3.4.

Adding your bundle to the feature.xml

There is a feature in Orbit that is used solely to drive the build. To modify it, checkout the org.eclipse.orbit.build.feature.set1 project. You'd add your bundle to these feature like any other. For example,

   <plugin
        id="javax.servlet"
        download-size="0"
        install-size="0"
        version="2.3.0.qualifier"
        unpack="false"/>

If your bundle should be unzipped when it is installed, you need to omit the unpack="false" attribute, as usual. For example,

  <plugin
        id="org.junit"
        download-size="0"
        install-size="0"
        version="4.8.1.qualifier"/>

Important: If you have added a new bundle to the feature.xml file, then the feature project must be tagged and its map file entry must be updated. See Orbit Bundle Checklist#Update the Map File

Keeping track of legal information

Add a file in org.eclipse.orbit.releng/ip_logs that contains IP log information. See Orbit IP Log for more information.

Adding your bundle to the Orbit Build

See Orbit Builds for Orbit Committers.

Final steps (a.k.a being a good OSS citizen)

It's a good practice to contribute back OSGi packaging - many upstream are not aware of OSGi and will appreciate patches. Having OSGi manifests upstream has a couple of advantages:

  • shared burden of maintenance
  • easier adoption (3rd party software can use newer versions before it is located in Orbit and drive Eclipse migration to latest-and-greatest versions).
  • spreading the word about OSGi

Back to the top