Revision as of 18:38, 19 December 2008

Introduction

This page describes a proposal for adding support for non OSGi version and version ranges in Equinox p2. This page was created as a result of the discussion on the p2 call on Dec 1, 2008. See bug 233699 for discussion.

Background

There are other versioning schemes in wide use that are not compatible with OSGi version and version ranges. The problem is both syntactic and semantic.

Example of semantic issues

Many open source projects do their versioning in a fashion similar to OSGi but with one very significant difference. For two versions that are otherwise equal, a lack of qualifier signifies a higher version then when a qualifier is present. I.e.

1.0.0.alpha 
1.0.0.beta
1.0.0.rc1
1.0.0

The 1.0.0 is the final release. The qualifier happens to be in alphabetical order here but that's not always true.

Mozilla Toolkit versioning has many rules and where each segment has 4 (optional slots; numeric, string, numeric, and string where each slot has a default value of being 0 or "max string" if missing).

1.2a3b.  // yes, a trailing . is allowed and means .0
1.a2

Mozilla also allows bumping the version (using an older Mozilla scheme)

1.0+ 

This means 1.1pre in mozilla.

Example of syntax issue

Here are some examples of versions used in Red Had Fedora distributions.

KDE Admin version 7:4.0.3-3.fc9
Compat libstdc version 33-3.2.3-63
Automake 1.4p6-15.fc7

And here are some mozilla toolkit versions:

1.*.1
1.0+
1.-1  // yes, negative integer version numbers are allowed, the - is not a delimiter
1.2a3b.a

These are not syntactically compatible with OSGi versions.

Current implementation in p2

The current implementation in p2 uses the classes Version and VersionRange to describe the two concepts and these are implementations handling only OSGi version type.

Proposed Solution

Equinox p2 should have one implementation of Version and one of VersionRange (called OmniVersion, and OmniVersionRange in this proposal, but names are placeholders and subject to discussion) capable of capturing the semantics of various version formats. The advantages over previous proposal are that there is no need to dynamically plugin new implementations, and new formats can be more easily be introduced.

Even if the finished solution only requires a single implementation (the OmniVersion), there are other factors to consider. The current p2 SimplePlanner uses the OSGi planner, and it can only understand OSGi versions. There is work being done on SAT4J to enable it being used instead of the OSGi planner (work to handle "explanations" could also be used to handle "attachments" (now being done with OSGi planner). To be able to develop the OmniVersion solution, it needs to coexist with the current SimplePlanner, and thus the current direct use of the classes Version and VersionRange needes to be modified to use IVersion and IVersionRange. (Such an implementation is available in bug 233699 as a patch).

• The interfaces IVersion and IVersionRange should be used throughout the code instead of directly using the corresponding Version and VersionRange classes.
• An IVersion is obtained by calling a factory method such as VersionFactory.create(String versionString)
• An IVersionRange is obtained by a similar factory method
• The version string and version range has a URI scheme like prefix to indicate the version type
• The factory API can naturally contain some options where scheme and version strings are either separate or canonical
• When a version or version range is present without the version type prefix, the default is to use OSGi version type (this preserves backwards compatibility).

It is now possible to replace the SimplePlanner's use of OSGi planner with a similar planner that handles OmniVersion. This can be done by someone that needs support for different versions formats before the SAT4j solution is available.

A note on terminology Although not technically correct, "the type of the version" or "version type", should be read as referring to the "version format".

Version Formats

There are three basic formats the default (current/backwards compatible) osgi string format, a raw canonical format, and fully specified format. A ':' separates the format from the version text.

Note about timestamps An earlier proposal had a 't' rule, but this rule has been deprecated because of the complexity. Instead, the creator of an IU should simply use 's' or 'n' and ensure comparability by using a fixed number of characters when choosing 's' format.

Named Version Formats

Named version formats makes it easier to enter version strings. There should be a number of predefined names as shown in the table below.

The version range delimiters are: '(', ')', '[', ']' and , ',' (comma).

Defining named formats

An IU can define new named formats. The named formats are defined by using a list of defined format names, and then one property per format.

org.equinox.p2.version.formats=<format-name>, <format-name>, ...
org.equinox.p2.version.format.<format-name>=formatstring

Once the format has been specified, it may be used in the IU. The format name should use java package name semantics to ensure unintentional clashes. When using the format names, the user may specify the last part of the name if it is unique. The predefined named formats should not be included in the formats property.

Thus, if an IU introduces the two named formats "org.mycorp.docver", and "org.mycorp.dbdataver" they are described like this:

org.equinox.p2.version.formats=org.mycorp.docver, org.mycorp.dbdataver
org.equinox.p2.version.format.org.mycorp.docver=n.n[s=max;]
org.equinox.p2.version.format.org.mycorp.dbdataver=n.'R'n[.s=ignore;]

When an IU is stored in a repository, the following processing is done:

• The defined formats are extracted from the IU
• If the fomat name does not already exist in the repository, it is added to the repositories list of contained formats.
• If the format name already exists in the contained formats list and the format pattern is the same - nothing needs to be done
• If the format name already exists in the contained formats list and the format pattern for the contained name is different - an exception is thrown

When using a non standard format name in an IU:

• The format definition must also be stored in the IU.

Attempting to redefine pre-defined formats:

• The pre-defined formats have higher precedence - should throw an exception

The user interface can:

• collect all defined formats from all known repositories and present them when the user is defining a version or range
• have a function to define a new format which is stored in the current profile (and thus becomes available for use)

This scheme allows format names to spread virally. The possible downside is potential clashes between repositories (same format name with different definitions in two different repositories) - but this is not a unique problem for version format. A particular IU of a particular version, or a particular artifact of a particular version, could very well be different in two repositories. As an aid/indicator, the UI can flag conflicting formats to the user.

A repository management tool could have a feature to enable modifying/replacing version fomats thus allowing repair.

Omni Version and VersionRange implementation

OmniVersion

The OmniVersion implementation uses a segment Array of Object that is kept in order of significance. The Array can hold Integer, String, Date deprecated, but may be introduced again later, and the special marker classes MaxInteger, and MaxString, and MaxDate deprecated, but may be introduced again later.

The OmniVersion implements Comparable. Segments are compared with the following rules:

• if types are not equal the result is based on the significance of the type; MaxInteger > Integer > MaxString > String > MaxDate > Date deprecated
• if the type is equal (or the max-variant of the type) - they are compared and result is returned
• If all segments are equal up to end of the shortest array, the shorter version is considered smaller (older)

Note:

• In order to be able to present the version in human readable form, the original version string is also kept.
• An OmniVersion can be instantiated from a "version string" as described in the Version Formats section below.
• There is no need to escape version range delimiters (see OmniVersionRange).

OmniVersionRange

The OmniVersionRange holds two OmniVersion, and can bidirectionally convert between string form and OmniVersion. When transforming into string format, all occurences of the range delimiters ", [] ()" in the version strings are escaped with "/". Existing occurences of "/" in the version are also escaped with an "/".

An OmniVersion range in string form consist of an optional version format specification followed by an OSGi type range specification (where version strings are range-safe).

Examples:

• raw:[1.2.3.'r1234',2.0.0]
• [1.2.3.r1234,2.0.0]
• format(a+):[monkey.fred.ate.5.bananas,monkey.fred.ate.10.oranges]
• [1.0.0,2.0.0] equal to osgi:[1.0.0,2.0.0]
• format(s):[andrea doria,titanic]
• rpm:[7:4.0.3-3.fc9,8:1] - an example KDE Admin version 7:4.0.3-3.fc9 to 8:1
• triplet:[1.0.0.RC1,1.0.0]

More examples using 'format'

A version range with format equivalent to OSGi

format(n[.n=0;[.n=0;[.s]]]):[1.0.0.r12345, 2.0.0]

At least one string, and max 5 strings

format(s[.s[.s[.s[.s]]]]):vivaldi.opus.spring.bar5
format(s(.s){0,4}):vivaldi.opus.spring.bar5  => 'vivaldi', 'opus', 'spring', 'bar5'

At least one alpha or numerical with auto format and delimiter

format(a(d?a)*):vivaldi:opus23-spring.bar5  => 'vivaldi', 'opus', 23, 'spring', 'bar', 5

The texts 'opus' and 'bar' should not be included:

format(s['.opus'=ignore;n['.bar'=ignore;n]]):vivaldi.opus23.bar8   => 'vivaldi', 23, 8

Classic SCCS/RCS style:

format(n(.n)*):1.1.1.1.1.1.1.4.5.6.7.8

Max depth 8 of numerical segments (limited classic SCCS/RCS type versions):

format(n(.n){0,7}):1.1.1.1.1.1.1.4

Numeric to optional depth 8, where missing input is set to 0, followed by optional string where 'emtpy > any'

format(n(d?n=0;){0,7}[a=maxs;]):1.1.1.4:beta   => 1,1,1,4,0,0,0,0,'beta'
format(n(d?n=0;){0,7}[a=maxs;]):1.1.1.4   => 1,1,1,4,0,0,0,0,MAXSTRING

Uninterpreted single string range

format(s):[andrea doria,titanic]

Internationalization

The proposed types using alphanumerical segments are assumed to use vanilla string comparison. This does not work so well if versions are expressed in a language where lexical ordering is different. Language specific collation could be supported by combining version type name with the name of a ISO 639 Language code (see java.util.Locale) and where the default would be English. The language could be encoded with a separating '-' e.g. 'format-pt' for collation in Portuguese.

This opens up another can of worms (decomposition strength, comparison of locale and non locale specified types, etc.), and it is probably best to implement just basic string comparison. It is also questionable if internationalization is wanted at all, as "known tools" does not support this, and "correct collation" would thus yield a different result.

Support for internationalized collation is not recommended.

Factory API

The factory API could be something simple like this:

public class VersionFactory
{
    IVersion createVersion(String versionString);
    IVersion createVersion(String versionType, String versionString);
}
public class VersionRangeFactory
{
    IVersionRange createVersionRange(String versionRangeString);
    IVersionRange createVersionRange(String versionType, String versionString);
}

Hard to say how much indirection is required - methods could just be static to keep things simple.

If we want to support the pattern based type, the factory methods needs the pattern as well. To make this generic, it could be seen as a paramter to the version type.

public class VersionFactory
{
    IVersion createVersion(String versionString);
    IVersion createVersion(String versionType, String versionString);
    IVersion createVersion(String versionType, String versionTypeParameter, String versionString);
}
public class VersionRangeFactory
{
    IVersionRange createVersionRange(String versionRangeString);
    IVersionRange createVersionRange(String versionType, String versionString);
    IVersionRange createVersionRange(String versionType, String versionTypeParameter, String versionString);
}

When creating a pattern based version, the versionTypeParameter must be supplied. When creating a pattern based version range, the pattern is optional - the pattern of the individual candidates would then be used to create the canonical form of the upper and lower bounds.

IVersion and IVersionRange API

Basically follows the current Version and VersionRange classes.

Applicability

The generalization of version type applies to objects that by nature may have a different versioning scheme than OSGi. This includes:

• Installable Unit
• Provided Capability
• Required Capability
• Artifact key

These does not need to be generalized:

• File format version numbers (content.xml, artifact.xml, etc)
• Update Descriptor
• Touchpoint version numbers and touchpoint action versions
• Publisher advice versions

Implementation Steps

Enablement of an alternate implementation of Version and VersionType is wanted in 3.5 even if the OSGi resolver is still used. This enablement is available as a patch. On a parallell track - the Omni-version classes are implemented and tested with either a SAT4J based solution or an interims replacement of the OSGi resolver. It is then possible to verify the functionality and performance of the Omni-version implementation.

The Omni-version implementation is not expected to go into 3.5 unless there is a SAT4j solution replacing the OSGi resolver, the functionality and performance is satisfactory, and enough testing has taken place.

The feature to allow introduction of new named formats can wait.

FAQ

Will users just using Eclipse and OSGi bundles be affected?
No, users that only deal within the OSGi domain can continue to use version strings like before, there is no need to specify version formats. The Enablement that is proposed should also be safe as it is only a level of indirection to the current implementation of Version and VersionRange.

How does a user of something know which version type to use? This seems very complicated...
To use some non-osgi component with p2, that component must have been made available in a p2 repository. When it was made available, the publisher must have made it available with a specified version format. The publisher must understand the component's version semantics. A consumer of the component can find the version format in the repository (the user must after all know that a capability is available under a certain name, and certain version range).

Will open (osgi) ranges produce lots of false positives?
Very unlikely. One decision to minimize the risk was to specify that integer segments are considered to be later than string segments. (We also felt that version segments specified with integers are more "precise"). Note that to be included in the range, the required capability would still need to be in a matching name space, and have a matching name. To introduce a false positive, the publisher of the false positive would need to a) publish something already known to others (namespace and name) b) misinterpret how its versioning scheme works, and publishing it with a format of n.n.n.n (or n.n.n.s.<something>), c) having first learned how to actually specify such a format and how to publish it to a p2 repository and d) then persuaded users to use the repository.

What happens when a capability is available with several versioning schemes?
A typical case would be some java package that is versioned at the source using triplet notation, and the same package is also made available using osgi notation (which is a mistake).

As an example, the following capabilities are found:

• org.demo.ships triplet:2.0.0
• org.demo.ships triplet:2.0.0.RC1
• org.demo.ships osgi:2.0.0
• org.demo.ships osgi:2.0.0.RC1

(Reminder: in triplet notation 2.0.0.RC1 is older than 2.0.0).

The raw versions will then look like this:

• 2.0.0.maxs
• 2.0.0.'RC1'
• 2.0.0
• 2.0.0.'RC1'

And the newest is 2.0.0.maxs (which is correct for both OSGi, and triplet). When specifying a range, the outcome may depend on if the range is specified with osgi or triplet notation.

• osgi:(1.0.0,2.0.0) == raw:(1.0.0, 2.0.0) => matches the osgi:2.0.0 version only
• triplet:(1.0.0,2.0.0) == raw:(1.0.0.maxs,2.0.0.maxs) => matches all the versions, and picks 2.0.0.maxs as it is the latest.

i.e. result is correct (assuming the bits are identical as different artifacts would be picked)

Now look at the lower boundary, and assume that the following versions are the (only) available:

• org.demo.ships triplet: 1.0.0 == raw: 1.0.0.maxs
• org.demo.ships triplet: 1.0.0.RC1 == raw:1.0.0.'RC1'
• org.demo.ships osgi: 1.0.0 == raw:1.0.0
• org.demo.ships osgi:1.0.0.RC1 == raw:1.0.0.'RC1'

When specifying ranges:

• osgi:(1.0.0,2.0.0) == raw:(1.0.0, 2.0.0) => matches all the version, and picks 1.0.0.maxs as this is the newest
• triplet:(1.0.0,2.0.0) == raw:(1.0.0.maxs,2.0.0.maxs) results in 1.0.0.maxs as it is the only available version that matches.

i.e. the result is correct and here the exact same version is picked.

The "worst osgi/triplet crime" that can be committed is publishing an unqualified triplet version as an osgi version (if the same version is not also available as a triplet) as this would make that version older than what it is even when queried using a triplet range.

What if the publisher of a component changes versioning scheme - what happens to ranges?
The order among the versions will be correct as long as the versions are published using the correct notation. The only implication is that users must understand that a query for triplet:x.x.x means raw:x.x.x.maxs - e.g. osgi:(1.0.0,2.0.0) != triplet:(1.0.0,2.0.0) (osgi upper range of 2.0.0 would not match triplet published 2.0.0, and triplet lower range of 1.0.0 would not match osgi published 1.0.0).

Why not use regexp instead of the special pattern format?
This was first considered, and would certainly work if the pattern notation was augmented with processing instructions, or if the regexp is specified as a substitution that produces the raw format. Such specifications would typically be much longer and more difficult for humans to read than the proposed format, except possibly for regexp experts :). Another immediate problem is that regexp breaks the current API requirement. It is not included in execution environment CDC-1.1/Foundation-1.1 required by p2.

Pattern parsing looks like it could have performance implications - what are the expectations here?
The intention is to use a mechanism similar to reqular expressions - when a format is first seen it is compiled to an internal structure. The compiled structure is cached and reused for all subsequent occurrences of the same format. A test will be performed to compare current parsing of an OSGi version string with the pattern based parsing. Once parsed, all comparisons are made using the raw vector, which should be comparable in speed to the current implementation.

Why not just let the publisher deal with transforming the version into canonical form?
There are several reasons:

• the original version strings needs to be kept as users would probably not understand the canonical representation in many cases.
• if the transformation pattern is not available a user would not be able to create a request without hand coding the canonical form
• making the transformation logic used by one publisher available to others would mean that all publishers must have extensions that allow plugging in such logic, and the plugins must be made available

Would it be possible to use the current OSGi version as the canonical form?
The long answer is: To be general, the encoding would need to be made in the qualifier string part of the OSGi version. An upper length for segments must be imposed, numerical sections must be left padded with "0" to that length, and string segments must be right padded with space (else string segment parts may overlap integer segments parts). The selected segment length would need to be big enough to allow the longest anticipated string segment. A serializable representation of MAX_STRING must be invented. A different implementation would still be needed to be able to keep the original version strings.
The short answer is: no.

Why not use an escape in string segments to be able to have strings with a mix of quotes? There are several reasons:

• this would mean that the version string would need to be preprocessed as it would not have \ embedded from the start
• all version strings that use \ as a delimiter would need to be pre-processed to escape the the \
• to date, authors of this proposal have not seen a version format that requires a mix of quotes
• parsing performance is affected

External Links

• mozilla toolkit version format
• rpm version comparison
• sun spec version format