Jump to: navigation, search

Difference between revisions of "Equinox/p2/p2.mirrorsURL"

< Equinox‎ | p2
(Enabling your repository to leverage mirrors)
(Where to add?)
Line 15: Line 15:
  
 
===Where to add?===
 
===Where to add?===
Adding this property to <code>artifacts.xml</code> is sufficient, since p2 uses a list of mirrors while retrieving artifacts, not the metadata itself. Obviously, <code>artifacts.jar</code>
+
Adding this property to <code>artifacts.xml</code> is sufficient, since p2 uses a list of mirrors while retrieving artifacts, not the metadata itself. (Hence, no need to have one in content.jar/xml or compositeContent.jar/xml files). Don't forget, <code>artifacts.jar</code>
must be updated or replaced after changing artifact.xml.  
+
must be updated or replaced after changing artifact.xml. (If both exist, artifacts.jar will always be fetched before artifacts.xml.)
  
 
====Should composite repos be treated, too, or just their children (leaves)?====
 
====Should composite repos be treated, too, or just their children (leaves)?====

Revision as of 00:19, 25 February 2012

Disclaimer: this page is not authored by a p2 expert, I merely reproduced some information which I found difficult to collect -- Stephan

Enabling your repository to leverage mirrors

The p2.mirrorsURL should be added to your metadata so that p2 will see the list of available mirrors to choose during installation.

Who should add?

Everyone. Every build. At least, everyone at eclipse.org, that is. Other companies or networks may have their own mirroring system and scripts which are similar to this, but these instructions are specific for the Eclipse Foundation infrastructure.

It is true that not each and every build or repository is mirrored (e.g. integration builds usually are not) but having the property there, even if there are no mirrors, should not hurt anything ... p2 knows to simply fall back to the normal "originally requested" location if no mirrors are found (usually http://download.eclipse.org).

It is also true that those mirroring eclipse downloads and repositories have some choice in what they mirror (there are several options, such as "full", "releases only", "platform only" and similar). So, everyone should add the property, since you can never be sure if/when your repository might be mirrored. If it is not mirrored, by anyone, at some point in time, having the property there should not hurt anything ... p2 knows to simply fall back to the normal "originally requested" location if no mirrors are found (usually http://download.eclipse.org).

Where to add?

Adding this property to artifacts.xml is sufficient, since p2 uses a list of mirrors while retrieving artifacts, not the metadata itself. (Hence, no need to have one in content.jar/xml or compositeContent.jar/xml files). Don't forget, artifacts.jar must be updated or replaced after changing artifact.xml. (If both exist, artifacts.jar will always be fetched before artifacts.xml.)

Should composite repos be treated, too, or just their children (leaves)?

Just the leaves. No need to to specify the property in compositeArtifacts.xml/jar.

What to add?

The p2.mirrorsURL property has the following structure:

  <property name="p2.mirrorsURL" 
          value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
  • replace {repository_path} with the path where your artifacts.jar sits on download.eclipse.org,
    e.g., for http://download.eclipse.org/eclipse/updates/3.6/R-3.6.2-201102101200/artifacts.jar the {repository_path} would be /eclipse/updates/3.6/R-3.6.2-201102101200
  • yes, &amp; is 5 characters, in this context the escaped version is required.
  • the last part, &amp;protocol=http is important, since it restricts the list of mirrors to those that support http. Monitor bug 349679 for status/progress on p2 working with ftp sites. (In general, such as for zip files linked on download pages, and similar, the 'protocol' parameter is optional for the download.php script and when omitted both http and ftp mirrors are returned). bug 349679 seems to have been a red-herring (or some special case) so ftp and http should both work fine, so there is no need to specify the protocol attribute and the mirrors script (download.php) will return a list that contains both type ... the more the merrier.

This p2.mirrorsURL property will end up giving p2 a list of 0 to 80 or so URLs to try and fetch artifacts from. If your project is not mirrored (or, there is some other problem with mirrors) p2 will fall back automatically to using the originally requested site location (usually 'http://download.eclipse.org').

How to add?

If you use a site.xml the word goes that this is where you should specify this property. The Equinox/p2/Publisher should automatically transfer this information into your metadata.

If you don't use a site.xml you may consider manually editing artifacts.xml (which of course can be automated using, e.g., an xsl template, like this p2.xsl from the Mylyn project). When manually editing artifacts.xml add the property to the existing list of properties right at the top of the document. Don't forget to update the size attribute of the enclosing <properties> element.

Others have reported that the p2.mirror ant task can be used to insert properties like p2.mirrorsURL while mirroring the build output to the final destination. The following two snippets may serve for inspiration:

<!-- generate template with proper mirror urls -->
<copy
  file=".../template-artifacts.xml"
  tofile="${buildDirectory}/publishRepoTemplate/artifacts.xml">
  <filterset>
        ... specify string substitutions here ...
  </filterset>
</copy>
<!-- mirror to target (but only the bits that really changed) -->
<p2.mirror ...>
  <source location="..."/>
  <destination .../>
  <destination
    kind="artifact"
        compressed="true"
        name="${p2.repo.name.prefix} (Build ${buildId})"
        location="..."
        append="true"
        format="${templateRepoUrl}"/>
  ...
</p2.mirror>

These snippets were extracted from this build.xml from the gyrex project.

How to test?

Warning2.png
Mirror invalidation
It seems that by modifying artifacts.jar all mirrors of the containing directory will be invalidated. Thus after updating artifacts.jar the mirrors need to catch up before the change can be tested.


First, you may paste the full URL you specified as the value into your browser (now collapse &amp; to &). The result should be an XML file with lots of mirror URLs. Pasting one of these URLs into the browser should in turn take you to the directory holding your artifacts.jar and friends.

Secondly, to test whether install successfully uses mirrors you may use the following options file (stored, e.g., as /tmp/p2options):

 org.eclipse.equinox.p2.core/debug=true
 org.eclipse.equinox.p2.core/artifacts/mirrors=true

(The first line generally enables debugging p2, the second line selects specific tracing for mirror selection)

Then you may want to unpack a fresh Eclipse package and change into the eclipse directory.

Finally, use the Equinox/p2/Director application for installing your feature using this command line:

 ./eclipse -consoleLog -noSplash -debug /tmp/p2options -application org.eclipse.equinox.p2.director \
       -repository http://download.eclipse.org/myproject/myrepo \
       -installIU org.eclipse.myproject.myfeature.group

This should install your feature and print to the console where each artifact was downloaded from. If p2.mirrorsURL is not properly configured no such messages will be shown.

Conversely, you can disable the automatic mirror selection mechanism by setting the VM argument eclipse.p2.mirrors to false (remember that vm args have to be last on the command line preceded by the switch -vmargs).

Moving a repo to archive.eclipse.org

Another situation where redirected downloads are desirable is when a repo is moved from download.eclipse.org to archive.eclipse.org. For files references using the Find a Mirror script (http://www.eclipse.org/downloads/download.php?file=/path/to/a/file.zip) moving to archive is transparently handled by that script. This section is exploring how the same effect can be achieved for p2 repositories.

The obvious way might be through the use of p2.mirrorsURL, however, I couldn't get this to work -- Stephan.

The following strategy seems to do the job (although admittedly, not a candidate for beauty contests):

  • when moving everything to archive.eclipse.org keep a copy of all metadata (artifacts.jar and all those) on download.eclipse.org.
  • extract the .xml files and edit like so:
    for composite repositories
    Ok green.gif edit both compositeContent.xml and compositeArtifacts.xml:
    • simply change all relative paths to absolute paths pointing to http://archive.eclipse.org/something
    for leaf repositories
    Warn.gif edit artifacts.xml (this is the ugly, unofficial approach - use at your own risk):
    • remove any p2.mirrorsURL property (and decrease the size of the enclosing <properties> element accordingly).
    • in the <mappings> element replace all occurrences of ${repoUrl} with the archive path to your repo, like http://archive.eclipse.org/objectteams/updates/ot0.7
    • to be on the safe side increment the p2.timestamp value in the top list of properties

After editing put the .xml files back into their respective jar.

An alternative, cleaner strategy for leaf repositories has been discussed in bug 349141#c3:

Ok green.gif redirect using a composite repository as a proxy:
  • move repository as-is to archive (maybe should remove p2.mirrorsURL at this point?)
  • create new composite repository pointing to archive.eclipse.org/...