View source for Callisto Coordinated Update Sites

{{Callisto Deprecation Notice}}
<p>
== Purpose of this document ==

This page is to document processes and procedures for providing improved coordination of update sites provided by the many Eclipse Projects. It's focus is on the 10 or so projects as part of the
[http://www.eclipse.org/projects/callisto.php Callisto simultaneous release] but some of the information would be helpful to any Eclipse adopter.

Note: as of this initial writing, this document should be taken as a 'proposal', and as its reviewed and discussed and improved by others, it will become a 'plan'. And then, eventually, once the plan seems to be
working, it will become a true "process and procedures" document.

This document was started because of a cross-project agreement reached at the December, 2005, EMO Architecture Council Meeting. There, through their representatives to the Architecture Council, the projects of
the Callisto Release agreed to improve the cross-project update experience. The agreement was that I ([[User:David williams|David Williams]]) would coordinate the effort ... but the projects still had to do the work!

This document is in no way to "take over" any of the Eclipse base project [http://www.eclipse.org/platform/index.php Update Team's] function, proposals, or responsibilities.

It is also <b>not to add to their responsibilities</b>. The goal for Callisto release is to use and "push the limits" of the current capabilities and technologies of the Update Manager. Of course, bugs and feature
requests may
result, and that is fine, but I wanted to be clear this proposal is not for anything fundamentally new ... it is just to document what's already possible, and make sure it is coordinated and carried out with "best
practices" in mind.

If I understand Wiki's at all committers can use the "talk" feature to provide feedback. Others may open a bugzilla on the Cross Project bugzilla component.

== Use cases ==

There are 3 primary use cases that cross-project, coordinated update sites provide:

1. Allow end-users to install some minimum "platform" and from that be able to use Update Manager to install all of the Callisto release, just by going to just one update site and selecting just one thing.

<del>2. Allow committers and developers to install an "SDK" version of Callisto, to be used while developing their own plugins.</del>

<del>3. Allow adopters to provide their own update sites, and "point to" appropriate sites to pick up prerequisite features. While there is nothing in this Callisto effort to support this directly, Callisto should serve as a good
example of "best practices" for other projects to follow.</del>

== Objectives and Constraints ==

Besides promoting the use cases given above, there are other objectives and constraints to meet:


=== Eclipse.org Infrastructure constraints ===

The distribution of Eclipse projects must fit in to the current disk space and "mirror system".

* Disk space. 
** JAR files should not be duplicated to avoid wasting bandwidth and disk space on our mirrors.
** The site.xml file can be configured to include JARs from various projects and still work with mirrors. See Distributed Bandwidth, below.

* Mirroring. 
** Callisto must make the best use of our mirror sites.
** Our mirror sites should have '''at least''' 24-hours of exclusive access to the Callisto files before releasing them to the public. This is done by waiting for Webmaster approval before sending files to download.eclipse.org.
** Webmaster will need to advise mirrors ahead of time, so they can anticipate the rush.

* Bandwidth.
** Co-ordinate with the Webmaster to "open the floodgates" on the bandwidth ahead of time.
** R3.1 required additional bandwidth for a foundation-hosted, separate mirror site. We will need this again.


=== Coordination verus Unification ===

For ease-of-use, there should be something of a "central site" that should make it easy to "get started" with "all" of Callisto. But, in the Eclipse spirit of allowing projects to "do their own thing" each project should also have their
own
discovery and update sites as well. This allows the central "get started" site to be one-size fits all, but still allow projects to offer even more, special features such as tools, examples, etc., that may not be part of
their "main' deliverables available via Callisto, not to mention have their own schedule for fixes and point releases. 

=== Update Policies ===

Some sites (corporations) can establish an "corporate" update site, which may have its own policies about what updates when, etc. The process and procedures here should do nothing to interfere with that
capability.

== Fundamentals ==

=== Distributed Storage ===

Each project will still have its own disk space and area on Eclipse, and will still be responsible for providing jarred features and plugins there, as they would for their own update site.

=== Distributed Bandwidth ===

The features (and archive tags, if needed) will be specified using the new "find the closest mirror"  [https://bugs.eclipse.org/bugs/show_bug.cgi?id=99412#c23 luxury support] provided by the Eclipse.org
download.php script.

<cite>
<b>Issue: </b> We (any volunteers?) should begin to collect approximate sizes and make rough estimates of required bandwidth for various scenarios, to make sure our mirroring system will suffice ... or we should panic now!?

Here is WebMaster's plan to make this happen:
* Go mirror shopping to get new mirror sites - especially in US and Asia
* Substantially increase bandwidth a few days ahead of release
* Allow mirrors 24 hours to sync up
* Provision a separate, dedicated high-speed mirror within those 24 hours
* Boom
</cite>

=== The Get-Started Platform Download ===

There will be one ZIP provided (by the Eclipse Project), that corresponds to the minimum amount of Eclipse that enables the update manager to work. This is, literally, the Eclipse Platform ... NOT including JDT and
PDE.  This feature will have the Callisto site as its implanted discovery site. It will still have 'eclipse' as its update site as it comes from the Eclipse Project (the update site being where "fixes" can be obtained
automatically by update manager). All other features, from all other projects, will have their own project's discovery and update sites implanted in their features, not the Callisto site). The few features available from
the central Callisto site will have discovery sites for all 10 Callisto projects.

<cite>
<del><b>Issue: </b>In this day and age, I'm surprised we do not offer "updates" (or "get started") in the form of a WebStart object. I think we do not only due to people constraints ... so, if there is participation from
the community (including the community of Callisto projects :) then we might be able to improve the user's experience so they'd never have to see another zip file again?
</cite></del>

<b>Note: </b>We should be explicit that the Callisto Update Site, contrary to the name, must begin with a 3.2 based Platform Feature ... there is no capability planned to update from a 3.1 based environment. Likewise, there are no plans to update from an 3.2 M4 environment to a 3.2 M5 environment. We will plan to test updating from an M5 environment to an M6 environment.

=== The User's Experience === 

==== Minimum Runtimes versus full SDK's ====

Its been decided that for this Callisto Release, there will not be a 
coordinated discovery site for the "SDK" contributions that projects 
provide. This is partially due to having the focus on simultaneous release itself, 
not necessarily what each user audience might need. Plus, as it turns out, 
the meaning and design of what's in "runtime" and what's in "SDK" is a little different from 
project to project, and will take a longer effort to coordinate an approach that's meaningful to all. 

But, this actually turns out to be a benefit, of sorts, rather than a limitation. 
The expected workflow is that Callisto users can install minimum "runtimes" to get the required or interesting functionality, and as part of those features installing, they each provide their own implanted update or discovery site URL. Hence, it should be easier for a user to install lots of functionality, but then get only the specific SDK portions they need. This will reduce users on-disk footprint, and reduce overall bandwidth on networks. 

However, during development of Callisto, this will make it hard to actually test or exercise this workflow, since the implanted URL's usually refer to official release sites, not the milestone or interim release sites. So, if anyone is interested in testing or using this workflow during Callisto milestones, the following are the milestone or interim sites that projects are currently using. 


 http://download.eclipse.org/birt/update-site/site.xml
 http://cdt.eclipse.org/callisto/site.xml
 http://download.eclipse.org/datatools/updates/site.xml
 http://download.eclipse.org/tools/emf/updates/site-interim.xml
 http://download.eclipse.org/eclipse/updates/3.2milestones/site.xml
 http://download.eclipse.org/tools/gef/update-site/milestones/site.xml
 http://download.eclipse.org/technology/gmf/update-site/callisto/site.xml
 http://download.eclipse.org/tptp/updates/4.2milestones/site.xml
 http://download.eclipse.org/tools/ve/updates/1.0/development/site.xml
 http://download.eclipse.org/webtools/milestones/site.xml



==== The View from Update Manager Dialog ====

The current plan is, for this Callisto Release, to simply list Features "by Project" [[Image:ExampleDialog.gif]].

=== Central Callisto Sites ===

==== Release Site ====
For the Callisto releases and milestones, there will be one central "discovery" site, such as
 http://download.eclipse.org/callisto/releases

Note: Its desriable to avoid any reliance on "aliases" as it makes mirroring more difficult to manage. 

<del>Also, we should be explicit, there is no plans to provide a big-huge-zip that contains all the Callisto projects ... its expected to be provided entirely via update manager.</del> There has been some recent talk of providing "big zips" made easy to get by the use of a form on a web page, and then the user would get, in one download, a zipped up version of their selections. If this is done, would be handled by the Eclipse Foundation (not this cross-project group).



==== Weekly Integration Update Sites ====

For those projects wishing to participate, there will be a site for integration builds:
 http://download.eclipse.org/callisto/interim

==== Working Staging Area ====

Lastly, there will be a working directory of
 http://download.eclipse.org/callisto/staging
that is used exclusively by central "Callisto developers" in the days or weeks ahead of the other update "releases" to test that the central features and site.xml is ready before they are copied over to the "public" sites.

== User Interface Consistency ==

=== Web site consistency ===
All update sites will have an "index" file, appropriate for viewing with a web browser ... and we need to work on giving this a consistent look.
(currently, some projects have none, some have highly detailed slowly performing ones with jars available for download for some reason I do not understand, but [http://download.eclipse.org/webtools/updates/ some :)] have some really nice looking ones.

=== Update Manager View Consistency ===
All update sites will provide a "site.xml" file, with a consistent level of abstraction for separately installable features. (Remember, not literally every feature provided by a project has to be "visible" to the update site
user ... many can be "included" as being required by other features, and installed automatically when other features selected).

These will be consistent and follow the following UI guidelines:

; Project Name spelled out, followed by initial in parenthesis, if desired. Such as
: Graphical Editing Framework (GEF)
: Eclipse Modeling Framework (EMF)

; Version numbers only at feature level, not project level categories. Such as
: Draw2d 3.2 (not GEF 3.2).

=== Predictable, consistent, long term URL's for update sites ===

There's currently great inconsistency here. I guess as long as its permanent, and long term, there's no harm. But looks sloppy.

For example, GEF is in Eclipse, and has no update site of its own. [[https://bugs.eclipse.org/bugs/show_bug.cgi?id=128348 128348]]

Some sites have a version number in their URL, such as tools/ve/updates/1.0/ and some do not, such as
tools/emf/updates/. The latter is correct, and important with the new qualified versioning conventions.

So, I suggest, this is the time to get consistent, but will need some care and thought to be sure it doesn't "break" anything.

== Planned Tests and Trial runs ==

During February, initial test versions will be created to download "all of Callisto" to see how it does,
from several parts of the world. Its not expected these will be "usable" versions of Callisto.

By the end of the M5 period (3/3), there will be a usable version of Callisto available via update manager.
(This is the version that can (will) be pointed to during EclipseCon, as an example of what's coming).

After M5, there will be "updates" (what's the right word for updates to updates?) to the milestones site for each Release Candidate leading up to the release.

Note: after M5, projects may elect to use the weekly I-build site. But this part of this proposal is sort of experimental, and will need some investigation, to be sure this is no great extra work for anyone ... but the idea is that for developers that are building on top of other features, it's very handy to be able to update pre-reqs in a frequent, easy way. That is, in other words, these "integration sites" are not literally an end-user requirement, so will get a little less attention.

== Project Responsibilities ==

This document and this "coordinated effort" is in no way meant to substitute for the projects themselves understanding update manger and update sites and actively
contributing [[callisto_update_people | the necessary people]] to ensure the
correct end result.

The following are some requirements placed on the projects participating in Callisto (most should not be anything "new", but some will be, and feasibility and need will have to be evaluated as we go).

* provide [[project update sites]], with jars, urls, site.xml files, etc. pretty much as has been done before ... but, might be a good time to change past decisions (for example, GEF does not have its own update site, for ... historical reasons, I guess).  The Callisto site.xml file will "point to" these project-level update sites, so, basically, each project must have a working update site before Callisto can have a working update site. See [[project update sites]] for more details.

* provide to "[[callisto build]]" a list of minimum features for inclusion in "runtime" version of Callisto (and, please, minimum for "end users" to take advantage of your functionality). It would include 'end user' documentation, but not 'isv documentation nor JavaDoc.

<del>* provide to [[User:David williams|David williams]] list of features for inclusion in "SDK" version of Callisto (should be no duplicates with first list, the SDK features should "require" runtime versions, so that someone may install runtime first, then install SDK later, without re-downloading or installing the runtime versions). These SDK features dos not have to be ALL your projects features, but should, at least, include 'source' and 'isv documentation'.</del>

* provide accurate "feature size" so the update time and progress can be accurate. This is just good practice, but since I've noticed many don't sizes in their updates, I thought I would call it out here, since this will be more important in Callisto.

* (maybe) provide signed jars, so users get some assurance they are installing verifiable Callisto features. This is more important now, using the "nearest mirror" techniques, since users may not know exactly where the jar is coming from. [I just say "maybe" because I do not think any of us know the technical steps or requirements for this.]

** <b>ISSUE: </b> If we have signed jars, it would be "nice" to have a "signed by Eclipse Foundation" corporate certificate, which I mention here, because I think to get such a certificate is a little time consuming (and not free :)

== Update Manager and Site Optimization ==
The [[Update Site Optimization]] page outlines steps for making update sites more scalable.

== Help Wanted == 

Besides the main deliverables for Callisto, the participating projects are asked to provide some assistance in testing the Callisto update site and, if possible, there are many tools that could be contributed to help. For example, 

* a "peek" tool which doesn't actually update anything, but which could, say, get the date and size headers for all the jars that would be requested by update manager, just for a warm reassuring feeling that all is (still) available as expected in a quick and painless way. 

* specialist to help with security, signed jars would likely be helpful. 

* specialists to help test and tweak update performance would likely not be refused. 

* etc.

== Bugzilla Queries == 

The following are some handy bugzilla queries related to update manager. 

[https://bugs.eclipse.org/bugs/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&classification=Eclipse&product=Platform&component=Update&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailtype1=substring&email1=&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&field0-0-0=noop&type0-0-0=noop&value0-0-0=&order=bugs.bug_severity,bugs.target_milestone%2Cbugs.target_milestone%2Cbugs.bug_severity%2Cmap_components.name%2Cbugs.bug_severity%2Cbugs.target_milestone%2Cbugs.bug_severity%2Cmap_components.name%2Cbugs.bug_severity%2Cmap_reporter.login_name%2Cmap_reporter.login_name%2Cmap_components.name%2Cbugs.target_milestone%2Cbugs.bug_severity%2Cbugs.target_milestone%2Cbugs.resolution%2Cmap_components.name%2Cmap_assigned_to.login_name%2Cbugs.bug_status%2Cbugs.target_milestone%2Cbugs.target_milestone%2Cbugs.target_milestone%2Cbugs.bug_severity%2Cbugs.bug_status%2Cbugs.target_milestone%2Cbugs.priority%2Cbugs.bug_id&query_based_on=  All Opened]


[https://bugs.eclipse.org/bugs/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&classification=Eclipse&product=Platform&component=Update&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailreporter1=1&emailtype1=substring&email1=david_williams&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=   All Opened by David Williams :) since starting to play with Callisto Updates.]

== Tips and Tricks == 

In addition to the written works in the reference section, I often some across little tidbits in bug reports, newsgroups, etc., and hope we can collect some of them on our [[Callisto Build and Update Tips and Tricks]] page. (So I can find them in the future :)

== Related Links for Further Enjoyable Reading and Reference ==

[http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.platform.doc.isv/guide/product_update.htm Basics of Update Sites]

[http://www.eclipse.org/articles/Article-Update/keeping-up-to-date.html How To Keep Up To Date]

[http://www.eclipse.org/equinox/documents/plugin-versioning.html Plugin Versioning Proposal]

[http://www.eclipse.org/eclipse/platform-core/documents/update.html Update Site Proposal]

[http://www.eclipse.org/eclipse/platform-core/documents/3.1/run_from_jars.html Jarring Plugins]

[[Development_Conventions_and_Guidelines]]

[[JAR_Signing]]

[[Update Site Optimization]]

[[Category:Callisto]] [[Category:Coordinated]]