Callisto Coordinated Update Sites

From Eclipsepedia

Revision as of 17:12, 4 February 2006 by David williams (Talk | contribs)

Jump to: navigation, search


Purpose of this document

This page is to document processes and procedures for providing improved coordination of update sites provided by the many Eclipse Projects. It's focus is on the 10 or so projects as part of the Callisto simultaneous release but some of the information would be helpful to any Eclipse adopter.

Note: as of this initial writing, this document should be taken as a 'proposal', and as its reviewed and discussed and improved by others, it will become a 'plan'. And then, eventually, once the plan seems to be working, it will become a true "process and procedures" document.

This document was started because of a cross-project agreement reached at the December, 2005, EMO Architecture Council Meeting. There, through their representatives to the Architecture Council, the projects of the Callisto Release agreed to improve the cross-project update experience. The agreement was that I (David Williams) would coordinate the effort ... but the projects still had to do the work!

This document is in no way to "take over" any of the Eclipse base project Update Team's function, proposals, or responsibilities.

It is also not to add to their responsibilities. The goal for Callisto release is to use and "push the limits" of the current capabilities and technologies of the Update Manager. Of course, bugs and feature requests may result, and that is fine, but I wanted to be clear this proposal is not for anything fundamentally new ... it is just to document what's already possible, and make sure it is coordinated and carried out with "best practices" in mind.

If I understand Wiki's at all committers can use the "talk" feature to provide feedback. Others may open a bugzilla on the Cross Project bugzilla component.

Use cases

There are 3 primary use cases that cross-project, coordinated update sites provide:

1. Allow end-users to install some minimum "platform" and from that be able to use Update Manager to install all of the Callisto release, just by going to just one update site and selecting just one thing.

2. Allow committers and developers to install an "SDK" version of Callisto, to be used while developing their own plugins.

3. Allow adopters to provide their own update sites, and "point to" appropriate sites to pick up prerequisite features. While there is nothing in this Callisto effort to support this directly, Callisto should serve as a good example of "best practices" for other projects to follow.

Objectives and Constraints

Besides promoting the use cases given above, there are other objectives and constraints to meet: Infrastructure constraints

  • Disk space.
    • JAR files should not be duplicated to avoid wasting bandwidth and disk space on our mirrors.
    • The site.xml file can be configured to include JARs from various projects and still work with mirrors. See Distributed Bandwidth, below.
  • Mirroring.
    • Callisto must make the best use of our mirror sites.
    • Our mirror sites should have at least 24-hours of exclusive access to the Callisto files before releasing them to the public. This is done by waiting for Webmaster approval before sending files to
    • Webmaster will need to advise mirrors ahead of time, so they can anticipate the rush.
  • Bandwidth.
    • Co-ordinate with the Webmaster to "open the floodgates" on the bandwidth ahead of time.
    • R3.1 required additional bandwidth for a foundation-hosted, separate mirror site. We will need this again.

1. The distribution of Eclipse projects must fit in to its current "mirror system" to allow for distributed bandwidth.

2. There should be something of a "central site" that should make it easy to "get started" with "all" of Callisto. But, in the Eclipse spirit of allowing projects to "do their own thing" each project should also have their own discovery (and update) sites as well. This allows the central "get started" site to be one-size fits all, but still allow projects to offer even more, special features such as tools, examples, etc., that may not be part of their "main' deliverables available via Callisto.

3. Some sites (corporations) can establish an "corporate" update site, which may have its own policies about what updates when, etc. The process and procedures here should do nothing to interfere with that capability.


Distributed Storage

Each project will still have its own disk space and area on Eclipse, and will still be responsible for providing jarred features and plugins there, as they would for their own update site.

Distributed Bandwidth

The features (and archive tags, if needed) will be specified using the new "find the closest mirror" luxury support provided by the download.php script.

Issue: We (any volunteers?) should begin to collect approximate sizes and make rough estimates of required bandwidth for various scenarios, to make sure our mirroring system will suffice ... or we should panic now!?

Here is WebMaster's plan to make this happen:

  • Go mirror shopping to get new mirror sites - especially in US and Asia
  • Substantially increase bandwidth a few days ahead of release
  • Allow mirrors 24 hours to sync up
  • Provision a separate, dedicated high-speed mirror within those 24 hours
  • Boom

The Get Started Download

There will be one ZIP provided (by the Eclipse Project), that corresponds to the minimum amount of Eclipse that enables the update manager to work. This is, literally, the Eclipse Platform ... NOT including JDT and PDE. This feature will have the Callisto site as its implanted discovery site. It will still have 'eclipse' as its update site as it comes from the Eclipse Project (the update site being where "fixes" can be obtained automatically by update manager). All other features, from all other projects, will have their own project's discovery and update sites implanted in their features, not the Callisto site). The few features available from the central Callisto site will have discovery sites for all 10 Callisto projects.

Issue: In this day and age, I'm surprised we do not offer "updates" (or "get started") in the form of a WebStart object. I think we do not only due to people constraints ... so, if there is participation from the community (including the community of Callisto projects :) then we might be able to improve the user's experience so they'd never have to see another zip file again?

Central Callisto Sites

Release Site

For the Callisto release, there will be one central "discovery" site, such as

This one site will have only two features, I propose. Each an "aggregate feature" that only includes other features (no actual code jars itself). One of the features will be the minimum "runtimes" that enables all the functionality of all 10 Callisto projects. The second would be the SDK version of all 10 projects (would include 'source', basic programming examples, etc.).

Issue: I expect this "all or nothing" aspect of this Callisto site proposal to be the most controversial. We will certainly evaluate as we go, but suspect the major driving force will be community feedback. (That is, will Java developers complain if they automatically get C++ related plugins, for example). We (the Eclipse Foundation), do, however, want to make it easier to use all of the Eclipse projects, such as BIRT and TPTP, whose value is often not well understood, if not installed automatically, and made part of a seamless Eclipse experience). Some projects, such as "EMF" and "GEF" are, I believe, exclusively enabling technologies and should never be "visible" to an end user where they have to say they "want" to download EMF or GEF support. The idea is that some developer who wants something very specific, such as GEF and the XSD portion of EMF, will really have no trouble putting the appropriate sites in their update manager and getting exactly what they need, where as more casual users should not even see a hint of the complication of deciding exactly which of the enabling technologies they need. Besides community feedback, we also need to 'get some numbers' and see really how much difference in size there would be.

Issue: We will have to ensure that all licenses are appropriate displayed by update manager, while using this only-a-few-aggregate features strategy. If not, we'll somehow have to come up with one big huge one that covers all aggregated features (yuck).

Milestone Site

For Callisto milestones and release candidates, there will be a separate site

All projects must participate in these milestone and release updates. This requires that all projects begin to use qualified plugin versions.

Weekly Integration Update Sites

For those projects wishing to participate, there will be a site for integration builds:

Working Staging Area

Lastly, there will be a working directory of

that is used exclusively by central "Callisto developers" in the days or weeks ahead of the other update "releases" to test that the central features and site.xml is ready before they are copied over to the "public" sites.

User Interface Consistency

Web site consistency

All update sites will have an "index" file, appropriate for viewing with a web browser ... and we need to work on giving this a consistent look. (currently, some projects have none, some have highly detailed slowly performing ones with jars available for download for some reason I do not understand, but some :) have some really nice looking ones.

Update Manager View Consistency

All update sites will provide a "site.xml" file, with a consistent level of abstraction for separately installable features. (Remember, not literally every feature provided by a project has to be "visible" to the update site user ... many can be "included" as being required by other features, and installed automatically when other features selected).

These will be consistent and follow the following UI guidelines:

Project Name spelled out, followed by initial in parenthesis, if desired. Such as
Graphical Editing Framework (GEF)
Eclipse Modeling Framework (EMF)
Version numbers only at feature level, not project level categories. Such as
Draw2d 3.2 (not GEF 3.2).

Predictable, consistent, long term URL's for update sites

There's currently great inconsistency here. I guess as long as its permanent, and long term, there's no harm. But looks sloppy.

For example, GEF is in Eclipse, no update site of its own.

Some sites have a version number in their URL, such as tools/ve/updates/1.0/ and some do not, such as tools/emf/updates/. The latter is correct, and important with the new qualified versioning conventions.

So, I suggest, this is the time to get consistent, but will need some care and thought to be sure it doesn't "break" anything.

Planned Tests and Trial runs

During February, initial test versions will be created to download "all of Callisto" to see how it does, from several parts of the world. Its not expected these will be "usable" versions of Callisto.

By the end of the M5 period (3/3), there will be a usable version of Callisto available via update manager. (This is the version that can (will) be pointed to during EclipseCon, as an example of what's coming).

After M5, there will be "updates" (what's the right word for updates to updates?) to the milestones site for each Release Candidate leading up to the release.

Note: after M5, projects may elect to use the weekly I-build site. But this part of this proposal is sort of experimental, and will need some investigation, to be sure this is no great extra work for anyone ... but the idea is that for developers that are building on top of other features, it's very handy to be able to update pre-reqs in a frequent, easy way. That is, in other words, these "integration sites" are not literally an end-user requirement, so will get a little less attention.

Project Responsibilities

This document and this "coordinated effort" is in no way meant to substitute for the projects themselves understanding update manger and update sites and actively contributing the necessary people to ensure the correct end result.

The following are some requirements placed on the projects participating in Callisto (most should not be anything "new", but some will be, and feasibility and need will have to be evaluated as we go).

  • provide project update sites, with jars, urls, site.xml files, etc.
  • provide list of minimum features for inclusion in "runtime" version of Callisto (and, please, minimum for "end users" to take advantage of your functionality). It would include 'end user' documentation, but not 'isv documentation nor JavaDoc.
  • provide list of features for inclusion in "SDK" version of Callisto (should be no duplicates with first list, the SDK features should "require" runtime versions, so that someone may install runtime first, then install SDK later, without re-downloading or installing the runtime versions). These SDK features dos not have to be ALL your projects features, but should, at least, include 'source' and 'isv documentation'.
  • provide accurate "feature size" so the update time and progress can be accurate.
  • (maybe) provide signed jars, so users get some assurance they are installing verifiable Callisto features (this is probably more important using the "nearest mirror" techniques, since users may not know exactly where the jar is coming from.

Help Wanted

Besides the main deliverables for Callisto, the participating projects are asked to provide some assistance in testing the Callisto update site and, if possible, there are many tools that could be contributed to help. For example,

  • a "peek" tool which doesn't actually update anything, but which could, say, get the date and size headers for all the jars that would be requested by update manager, just for a warm reassuring feeling that all is (still) available as expected in a quick and painless way.
  • specialist to help with security, signed jars would likely be helpful.
  • specialists to help test and tweak update performance would likely not be refused.
  • etc.

Related Links for Further Enjoyable Reading and Reference

Basics of Update Sites

How To Keep Up To Date

Plugin Versioning Proposal

Update Site Proposal