Difference between revisions of "Ganymede/Build"

From Eclipsepedia

Jump to: navigation, search
(Creating Ganymatic for Build.Eclipse.Org)
(Syntax and Semantics)
 
(34 intermediate revisions by 4 users not shown)
Line 7: Line 7:
 
:* [http://build.eclipse.org/ganymede/builds.rss All builds]
 
:* [http://build.eclipse.org/ganymede/builds.rss All builds]
 
:* [http://build.eclipse.org/ganymede/success.rss Successful builds]
 
:* [http://build.eclipse.org/ganymede/success.rss Successful builds]
:* [http://build.eclipse.org/ganymede/fail.rss Failed builds]  
+
:* [http://build.eclipse.org/ganymede/fail.rss Failed builds] -- ''see also [[Ganymede/Build/Troubleshooting|<b style="color:red">Troubleshooting Ganymatic builds</b>]]''
  
 
Of course, there really is no "build" of Ganymede, but only a copying of what is already available on each project's update site, to a central site.
 
Of course, there really is no "build" of Ganymede, but only a copying of what is already available on each project's update site, to a central site.
  
 
This central update site, naturally, is http://download.eclipse.org/releases/ganymede/. To get a sneak peek, point your Update Manager at http://download.eclipse.org/releases/ganymede/staging/.
 
This central update site, naturally, is http://download.eclipse.org/releases/ganymede/. To get a sneak peek, point your Update Manager at http://download.eclipse.org/releases/ganymede/staging/.
 +
 +
==Running a Build==
 +
 +
The Ganymatic runs automatically when it detects changes in CVS that warrant a new build. However, if you're impatient or want to test your latest contribution, you can kick a build yourself using the Cruise Control interface:
 +
 +
* First, go here: http://build.eclipse.org:9777/dashboard/tab/build/detail/ganymatic-R3.0-I (This is linked on the [http://build.eclipse.org/ganymede/ Ganymede dashboard] too.)
 +
 +
* Next, click the icon that looks like an arrow biting its tail. Onmouseover it says "Force build". [[Image:Cruise-Control-force-build-icon.gif]]
 +
 +
* Wait a minute or two, then go back to the main [http://build.eclipse.org/ganymede/ Ganymede dashboard], and wait for your build to complete.
  
 
==The Projects' Roles==
 
==The Projects' Roles==
Line 68: Line 78:
 
http://www.eclipse.org/buckminster/schemas/common-1.0.xsd ,
 
http://www.eclipse.org/buckminster/schemas/common-1.0.xsd ,
 
http://www.eclipse.org/buckminster/schemas/rmap-1.0.xsd
 
http://www.eclipse.org/buckminster/schemas/rmap-1.0.xsd
 +
 +
A note on '''<sc:member''' semantics: please don't use an eclipse -dev mailing list, or any other email address that requires a subscription ... the builder will seldom be subscribed.
  
 
===Project Responsibilities===
 
===Project Responsibilities===
Line 83: Line 95:
 
TBD: how to trigger a new build and how to sign off that the project M-bits are ready to use.
 
TBD: how to trigger a new build and how to sign off that the project M-bits are ready to use.
  
==The Central Ganymatic==
+
==The Production Ganymatic==
  
As of 2007/10/14, the Ganymede builds are being run once a day at 11pm Eastern, 8pm Pacific, 5am Europe, 11am China.
+
The Ganymede builds are triggered automatically when ever someone commits their site contributions to head. In some few cases, for example, if someone subsequently updates or fixes their own update site, but the sc files don't change, then you may have to "touch" your sc file and commit it, just to trigger a build, or, you can "manually" force a new build from the [http://build.eclipse.org:9777/dashboard/tab/builds Orbit Cruise Control page].
  
TBD: how to trigger a new build and how to sign off that the project M-bits are ready to use.
+
===Digests, P2 Metadata, Pack200 and Promotion===
 +
The digest and P2 meta data are created every "build", so they should always be accurate.
 +
But, the site is "packed" (with pack200) only once per day, overnight, since it takes about 2 hours.  
  
===Pack200 and Digests===
+
The implication of this, is that before "promoting" a build, you need to make sure that the packing step has taken place on the latest files in staging. (There will be no pack200.gz files, if it has not ran, yet).  
Currently the pack200 and digest creation is '''not''' part of Ganymatic because there of [https://bugs.eclipse.org/bugs/show_bug.cgi?id=215664 bug 215664]. In the meantime, here is the manual process I use to pack the files:
+
# temp_do_pack200.sh
+
## If that is successful, the bug is fixed
+
## Otherwise...
+
# edit the site.xml to remove the stp features
+
# temp_do_pack200.sh
+
# temp_do_createDigest.sh
+
# edit the site.xml to include the digestURL and mirrorsURL attributes
+
  
===Releasing the Central Staging Bits===
+
If it has not ran yet, and you want to promote, you can "manually" start the pack build from Cruisecontrol, and enjoy your favorite beverage while it cranks away.
On build.eclipse.org as ~bfreeman, first run copyToRelease-1-plugins.sh.  If that is successful, run copyToRelease-2-sitefiles.sh.
+
  
You may want to clean out all the existing files from the release directory before doing the copy (or you may not). If so, use copyToRelease-0-clean.sh
+
===Promoting a build from Staging area to Releases Area===
  
==Running Ganymatic On Your Machine==
+
There is a script, in org.eclipse.ganymede.tools, called 'promote.sh' which performs the usual steps of copying 'staging' bits to 'releases' area. It should be executed from the org.eclipse.ganymede.tools directory.  
# Build a copy of Ganymatic (follow steps 1-4 below).
+
# Check out the org.eclipse.ganymede.sitecontributions and org.eclipse.ganymede.tools from /cvsroot/callisto
+
# Make sure you have a ganymatic.properties in your home directory. See examples in org.eclipse.ganymede.tools/examples
+
# Use ant to run the build.xml from org.eclipse.ganymede.tools
+
# The results (update site and website) are placed in the directories you specified in the ganymatic.properties file.
+
  
===Creating Ganymatic for Build.Eclipse.Org===
+
It removes all the previous 'releases' files (which is appropriate for milestones, and the final release, but will have to be improved for the maintenance releases). After removal, it copies all the files from /releases/ganymede/staging "up to" /releases/ganymede. It also uses a sed script to modify URLs in the site.xml file that contain /releases/ganymede/staging to be shortened to /releases/ganymede. Then, it recreates the the digest.zip file, and the P2 metadata files. These partially have to be re-created because they contain some URLs inside (for the mirror script) which has to be modified from 'staging' to 'releases', but also since as a general principle, they should be recreated, so they are sure to reflect the contents of the 'new' site (even though, most of the time, the contents are identical to the 'old' site).
To create the binaries for Ganymatic:
+
 
# Start with an Eclipse SDK with the latest version of Buckminster [http://www.eclipse.org/projects/project_summary.php?projectid=tools.buckminster] loaded.
+
So, to summarize, the actual mechanics are pretty simple: on build.eclipse.org, perform something like the following, saving the output in case you need to inspect it for errors, etc.
# Check out the org.eclipse.dash.siteassembler project (<nowiki>:pserver:anonymous@dev.eclipse.org:/cvsroot/technology</nowiki>, module: org.eclipse.dash/org.eclipse.dash.siteassembler).
+
 
# Right click on the org.eclipse.dash.siteassembler project and choose Buckminster > Invoke Action... > buckminster.clean
+
  cd /shared/ganymede/workingdir/org.eclipse.ganymede.tools
# Right click on the org.eclipse.dash.siteassembler project and choose Buckminster > Invoke Action... > headless.assembler
+
  ./promote.sh | tee ~/recordOut.txt
#* This generates the org.eclipse.dash.siteassembler/bin/buckminster directory
+
 
#* If there are strange errors, delete the org.eclipse.dash.siteassembler/bin/headless.zip and try again from the buckminster.clean
+
Note, it's not a bad idea to sanity check the files, dates, sizes, etc., in ~/downloads/releases/ganymede.
# Zip up that directory and put it on build.eclipse.org/~bfreeman/ganymede/buckminster/, and unzip it into the bin/buckminster sub-directory
+
They should be similar to the following:
 +
 
 +
-rw-rw-r-- 1 david_williams  98K 2008-06-02 13:21 artifacts.jar
 +
-rw-rw-r-- 1 david_williams 1118K 2008-06-02 13:21 content.jar
 +
-rw-rw-r-- 1 david_williams  129K 2008-06-02 13:08 digest.zip
 +
-rw-rw-r-- 1 david_williams  40K 2008-06-02 13:21 site.xml
 +
 
 +
Plus, it's usually good to read the first of the site.xml file, and make sure the URLs there got changed correctly so they no longer contain 'staging'.
 +
 
 +
Also, if you're curious, there's a utility script, checkStats.sh that will list some summary data, ... to double check things look relatively normal:
 +
 
 +
david_williams@build:~/downloads/releases/ganymede> ./checkStats.sh
 +
 +
  number of features: 433 (11 are not packed)
 +
  number of plugins: 1575 (63 are not packed)
 +
 
 +
==Creating Ganymatic==
 +
 
 +
===For Your Own Local Build Machine===
 +
 
 +
:1. [[#Creating Ganymatic for Build.Eclipse.Org|Build a copy of Ganymatic]] and transfer it to your build server (if not building locally).
 +
 
 +
:2. Check out the org.eclipse.ganymede.sitecontributions and org.eclipse.ganymede.tools projects.
 +
 
 +
{{codeblock|<nowiki>cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/callisto -q co org.eclipse.ganymede.sitecontributions; \
 +
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/callisto -q co org.eclipse.ganymede.tools</nowiki>}}
 +
 
 +
:3. There are two files of variables and properties you need to customize for your machine, with these names:
 +
 
 +
::*[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ganymede.tools/examples/ganymedeConfig/ganymatic.shsource?root=Callisto&view=markup ganymatic.shsource] and
 +
 
 +
::*[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ganymede.tools/examples/ganymatic-bfreeman-build.eclipse.org.properties?root=Callisto&view=markup ganymatic.properties]
 +
 
 +
::It is recommended you create a directory called ''ganymedeConfig'' in your ''$HOME'' directory, and put your two customized files there, since several of the script files will look there automatically and use those files, if they exist.
 +
 
 +
::See other examples in [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ganymede.tools/examples/?root=Callisto org.eclipse.ganymede.tools/examples].
 +
 
 +
::Note, the reason for having your own customizations in a separate location (~/ganymedeConfig) is so that you can easily get a fresh copy of org.eclipse.ganymede.tools without "stepping on" your own customizations. Only the production build machine should depend on the property files located in the org.eclipse.ganymede.tools project.
 +
 
 +
:4. Use ant to run the build.xml from org.eclipse.ganymede.tools. Or, use [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ganymede.tools/do_ganymatic_on_build.eclipse.org_.sh?root=Callisto&view=log this script]:
 +
 
 +
{{codeblock|<nowiki>cd org.eclipse.ganymede.tools; \
 +
/shared/ganymede/apps/apache-ant-1.7.0/bin/ant \
 +
  -Dganymatic.properties.file=/path/to/ganymatic.properties</nowiki>}}
 +
 
 +
:5. The results (update site and website) are placed in the directories you specified in the ganymatic.properties file. Email notification (if any) is also configured in the properties file.
 +
 
 +
===For Build.Eclipse.Org===
 +
 
 +
To create the binaries (headless assembly scripts and runtime) for Ganymatic:
 +
 
 +
:1. Start with an Eclipse 3.3 or 3.4 SDK with the [http://download.eclipse.org/tools/buckminster/updates latest version] of [http://www.eclipse.org/projects/project_summary.php?projectid=tools.buckminster Buckminster] 1.0 installed. You need the two features <i>org.eclipse.buckminster.core</i> and <i>org.eclipse.buckminster.pde</i>. Please note that Buckminster has a lot of optional features. Don't install all of them (some are even mutually exclusive).
 +
 
 +
:2. Check out the org.eclipse.dash.siteassembler project.
 +
{{codeblock|<nowiki>cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/technology -q co \
 +
  -d org.eclipse.dash.siteassembler org.eclipse.dash/org.eclipse.dash.siteassembler</nowiki>}}
 +
 
 +
:3. [This should already be the case, but just in case ... ] Make sure the siteassember project targets a 5.0 JVM. Right-click the checked-out project and select
 +
{{codeblock|Properties > Java Compiler > Compiler compliance level: 1.5}}
 +
 
 +
:4. Right click on the org.eclipse.dash.siteassembler project and choose
 +
{{codeblock|Buckminster > Invoke Action... > buckminster.clean}}
 +
 
 +
:5. Right click on the org.eclipse.dash.siteassembler project and choose  
 +
{{codeblock|Buckminster > Invoke Action... > headless.assembler}}
 +
::* This generates the org.eclipse.dash.siteassembler/bin/buckminster directory
 +
::* If there are strange errors, delete the org.eclipse.dash.siteassembler/bin/headless.zip and try again from the buckminster.clean
 +
 
 +
:6. Zip & transfer the '''bin/buckminster''' directory to build.eclipse.org. Unpack it as ''you''@build.eclipse.org:/shared/ganymede/buckminster/'''bin/buckminster'''.
 +
 
 +
<table><tr><td>
 +
export you='''''yourUserName'''''; \
 +
cd '''''/path/to/workspace'''''/org.eclipse.dash.siteassembler/bin; \
 +
zip -r /tmp/buckminster.zip buckminster; \
 +
ssh ''$you''@build.eclipse.org "mkdir -p ~/ganymede/buckminster/bin/"; \
 +
scp /tmp/buckminster.zip ''$you''@build.eclipse.org:~/ganymede/buckminster/bin/; \
 +
ssh ''$you''@build.eclipse.org "cd ~/ganymede/buckminster/bin/; unzip buckminster.zip"; \
 +
rm -f /tmp/buckminster.zip
 +
</td></tr></table>
 +
 
 +
:7 Unlike the process for your own local build machine, there should '''not''' be any user-defined customization used or required for running Ganymatic on build.eclipse.org. This is by design, so that anyone can take over running the builds on eclipse.org and get the same, repeatable result by simply checking out the org.eclipse.ganymede.tools project and running it (that is, it should not depend on what user id is running the process, for production builds).
 +
 
 +
:Additional tidbits
 +
 
 +
:: If interested in how it works, the 'headless.zip' is the main part of the buckminster code that does headless builds (composition) but it lacks the project specific information that is compiled into org.eclipse.dash.siteassembler jar, which is in the bin/buckminster directory, along with the rest of headless.zip.
 +
 
 +
:: Ever wonder why or when the bin/buckminster code should be regenerated? See [https://bugs.eclipse.org/bugs/show_bug.cgi?id=227693 Bug 227693] but the short answer is whenever there are bug fixes to Buckminster, or the org.eclipse.dash.siteassembler changes.
  
 
==Additional Requirements and Information==
 
==Additional Requirements and Information==
Line 129: Line 215:
 
* [https://bugs.eclipse.org/bugs/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&classification=Eclipse+Foundation&product=Community&component=Cross-Project&classification=Technology&product=Dash&component=Ganymatic&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailtype1=substring&email1=&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0= Search] in Eclipse Foundation > Community > Cross-Project or Technology > Dash > Ganymatic
 
* [https://bugs.eclipse.org/bugs/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&classification=Eclipse+Foundation&product=Community&component=Cross-Project&classification=Technology&product=Dash&component=Ganymatic&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailtype1=substring&email1=&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0= Search] in Eclipse Foundation > Community > Cross-Project or Technology > Dash > Ganymatic
 
* Open a new [https://bugs.eclipse.org/bugs/enter_bug.cgi?assigned_to=cross-project.inbox%40eclipse.org&product=Community&component=Cross-Project Cross-Project] or [https://bugs.eclipse.org/bugs/enter_bug.cgi?assigned_to=cross-project.inbox%40eclipse.org&product=Dash&component=Ganymatic Ganymatic] bug  
 
* Open a new [https://bugs.eclipse.org/bugs/enter_bug.cgi?assigned_to=cross-project.inbox%40eclipse.org&product=Community&component=Cross-Project Cross-Project] or [https://bugs.eclipse.org/bugs/enter_bug.cgi?assigned_to=cross-project.inbox%40eclipse.org&product=Dash&component=Ganymatic Ganymatic] bug  
 +
 +
=== See Also ===
 +
 +
* Creating an [[Ganymede/Build/Archived Update Site|Archived Update Site]].
 +
* [[Ganymede/Build/Troubleshooting|Troubleshooting Ganymatic builds]]
 +
 +
=== Required Patches ===
 +
 +
Some of these patches may be required, depending on which versions of the Platform are being used, to have a perfectly reproducible build as the eclipse production machine. I just wanted a central place to document them, so I wouldn't lose the information. 
 +
 +
* [https://bugs.eclipse.org/bugs/show_bug.cgi?id=229019 bug 229019] contains a patch that allows jars with security exceptions to be identified. This patch was applied to the 3.3.2 version of update.core, exported from developer workspace (not built, per se) and used in place of the one that is used by default in the current Buckminster headless assembly application. The patch simply allows the jar with the error to be listed so it can be tracked down and fixed.
 +
 +
* [https://bugs.eclipse.org/bugs/show_bug.cgi?id=226850 bug 226850] contains a patch that is required to get site optimizer to pack directories. Hopefully this will be in the final 3.4 platform, but if anyone is using a build prior to that, they would have to have this patch applied to get the jars packed.
  
 
[[Category:Ganymede]] [[Category:Coordinated]] [[Category:Releng]]
 
[[Category:Ganymede]] [[Category:Coordinated]] [[Category:Releng]]

Latest revision as of 19:17, 4 September 2008

This page is related to the Ganymede Simultaneous Release.

Contents

[edit] Explanation

The Ganymatic is live and running every day. The status & logs are here: http://build.eclipse.org/ganymede/

In addition to sending emails to project release engineers who are responsible for broken builds, there are three 'Eclipse Build' format RSS feeds available for project teams to monitor. More info on feeds: Getting Started Guide, Feed Schema.

Of course, there really is no "build" of Ganymede, but only a copying of what is already available on each project's update site, to a central site.

This central update site, naturally, is http://download.eclipse.org/releases/ganymede/. To get a sneak peek, point your Update Manager at http://download.eclipse.org/releases/ganymede/staging/.

[edit] Running a Build

The Ganymatic runs automatically when it detects changes in CVS that warrant a new build. However, if you're impatient or want to test your latest contribution, you can kick a build yourself using the Cruise Control interface:

  • Next, click the icon that looks like an arrow biting its tail. Onmouseover it says "Force build". Cruise-Control-force-build-icon.gif
  • Wait a minute or two, then go back to the main Ganymede dashboard, and wait for your build to complete.

[edit] The Projects' Roles

[edit] Pre-Project Configuration Files

The pre-project configuration files are stored in the Callisto CVS:

:extssh:<your-committer-id-goes-here>@dev.eclipse.org:/cvsroot/callisto

Or, general "read" access via

:pserver:anonymous@dev.eclipse.org:/cvsroot/callisto

There is one project:

org.eclipse.ganymede.sitecontributions

[edit] Syntax and Semantics

<?xml version='1.0'?>
<sc:siteContribution
  xmlns="http://www.eclipse.org/buckminster/CSpec-1.0"
  xmlns:sc="http://www.eclipse.org/buckminster/SiteContribution-1.0"
  rmapProviderURL="${downloads}/webtools/milestones/site.xml">
    <sc:member name="Bjorn Freeman-Benson" email="bjorn.freeman-benson@eclipse.org" />
    <sc:member name="Karl Matthias" email="webmaster@eclipse.org" />
    <sc:cspec name="org.eclipse.wtp-sc">
      <dependencies>
        <dependency name="org.eclipse.jpt.feature" 
                    versionDesignator="[1.0.0.v200706250000-77--CYQCCz-CoRPCCCH]" />
        <dependency name="org.eclipse.jst" 
                    versionDesignator="[2.0.0.v200706110805-7B-18dDQISlz0z0cc-oasolrW8jP]" />
        <dependency name="org.eclipse.wst.common_ui.feature" 
                    versionDesignator="[2.0.0.v200706041905-7C5EGzE9RvTVniSrwnf4TgOPe3e9]" />
      </dependencies>
      <groups>
        <public name="Web and JEE Development">
          <attribute component="org.eclipse.jst" />
          <attribute component="org.eclipse.wst.common_ui.feature" />
        </public>
        <public name="Other Development">
          <attribute component="org.eclipse.jpt.feature" />
        </public>
      </groups>
    </sc:cspec>
</sc:siteContribution>
  1. ${downloads}/webtools/milestones/site.xml: The location of the project's update site. It's more efficient to access the site via a file path than a url, but a url will work. ${downloads} refers to the root of the http://download.eclipse.org/ server.
  2. Bjorn Freeman-Benson and bjorn.freeman-benson@eclipse.org: person to notify if this project is responsible for breaking the build. Feel free to have multiple people listed.
  3. org.eclipse.wtp-sc: the unique identifier for this site contribution
  4. org.eclipse.jpt.feature: The feature identifier. One per feature (of course). Features are listed one per <dependency .../> node.
  5. 2.0.0.v200706041905-7C5EGzE9RvTVniSrwnf4TgOPe3e9: The feature version. One per feature (of course). Buckminster (the technology Ganymatic is built on) has a number of ways to specify the version. Most projects will use a single fixed version here, although you can also have an empty string to mean "the latest version found on my update site".
  6. Web and JEE Development: Categories for the Ganymede site.xml file; used to group the features.

Schemas for these files are available at: http://www.eclipse.org/buckminster/schemas/siteContribution-1.0.xsd , http://www.eclipse.org/buckminster/schemas/cspec-1.0.xsd , http://www.eclipse.org/buckminster/schemas/common-1.0.xsd , http://www.eclipse.org/buckminster/schemas/rmap-1.0.xsd

A note on <sc:member semantics: please don't use an eclipse -dev mailing list, or any other email address that requires a subscription ... the builder will seldom be subscribed.

[edit] Project Responsibilities

Each Ganymede project's PMC and/or Project Leader is responsible to keep its own *.sc (site contribution) file up to date as the project generates new milestones and release candidates. The project's project leader will probably delegate to the project's release engineer(s).

Obviously, each project's own update site must already exist (and be working) at the file location or url specified in the site contribution file.

Typically, as projects move from milestone to milestone (or release candidate to release candidate) only the version identifiers (e.g. 1.0.0.v200706250000-77--CYQCCz-CoRPCCCH]) in the site contribution file will need to be changed. At other times, features might be added, removed, or renamed:

  • Adding a feature: add new <dependency> and <attribute> nodes
  • Removing a feature: remove the corresponding <dependency> and <attribute> nodes
  • Renaming a feature: change the feature identifier (in two places) (e.g., org.eclipse.jpt.feature).

Each project should post updates on the Ganymede/Signoffs page after each of their Milestone and Release Candidates is released.

TBD: how to trigger a new build and how to sign off that the project M-bits are ready to use.

[edit] The Production Ganymatic

The Ganymede builds are triggered automatically when ever someone commits their site contributions to head. In some few cases, for example, if someone subsequently updates or fixes their own update site, but the sc files don't change, then you may have to "touch" your sc file and commit it, just to trigger a build, or, you can "manually" force a new build from the Orbit Cruise Control page.

[edit] Digests, P2 Metadata, Pack200 and Promotion

The digest and P2 meta data are created every "build", so they should always be accurate. But, the site is "packed" (with pack200) only once per day, overnight, since it takes about 2 hours.

The implication of this, is that before "promoting" a build, you need to make sure that the packing step has taken place on the latest files in staging. (There will be no pack200.gz files, if it has not ran, yet).

If it has not ran yet, and you want to promote, you can "manually" start the pack build from Cruisecontrol, and enjoy your favorite beverage while it cranks away.

[edit] Promoting a build from Staging area to Releases Area

There is a script, in org.eclipse.ganymede.tools, called 'promote.sh' which performs the usual steps of copying 'staging' bits to 'releases' area. It should be executed from the org.eclipse.ganymede.tools directory.

It removes all the previous 'releases' files (which is appropriate for milestones, and the final release, but will have to be improved for the maintenance releases). After removal, it copies all the files from /releases/ganymede/staging "up to" /releases/ganymede. It also uses a sed script to modify URLs in the site.xml file that contain /releases/ganymede/staging to be shortened to /releases/ganymede. Then, it recreates the the digest.zip file, and the P2 metadata files. These partially have to be re-created because they contain some URLs inside (for the mirror script) which has to be modified from 'staging' to 'releases', but also since as a general principle, they should be recreated, so they are sure to reflect the contents of the 'new' site (even though, most of the time, the contents are identical to the 'old' site).

So, to summarize, the actual mechanics are pretty simple: on build.eclipse.org, perform something like the following, saving the output in case you need to inspect it for errors, etc.

 cd /shared/ganymede/workingdir/org.eclipse.ganymede.tools
 ./promote.sh | tee ~/recordOut.txt 

Note, it's not a bad idea to sanity check the files, dates, sizes, etc., in ~/downloads/releases/ganymede. They should be similar to the following:

-rw-rw-r-- 1 david_williams   98K 2008-06-02 13:21 artifacts.jar
-rw-rw-r-- 1 david_williams 1118K 2008-06-02 13:21 content.jar
-rw-rw-r-- 1 david_williams  129K 2008-06-02 13:08 digest.zip
-rw-rw-r-- 1 david_williams   40K 2008-06-02 13:21 site.xml

Plus, it's usually good to read the first of the site.xml file, and make sure the URLs there got changed correctly so they no longer contain 'staging'.

Also, if you're curious, there's a utility script, checkStats.sh that will list some summary data, ... to double check things look relatively normal:

david_williams@build:~/downloads/releases/ganymede> ./checkStats.sh

 number of features: 433 (11 are not packed)
 number of plugins: 1575 (63 are not packed)

[edit] Creating Ganymatic

[edit] For Your Own Local Build Machine

1. Build a copy of Ganymatic and transfer it to your build server (if not building locally).
2. Check out the org.eclipse.ganymede.sitecontributions and org.eclipse.ganymede.tools projects.
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/callisto -q co org.eclipse.ganymede.sitecontributions; \
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/callisto -q co org.eclipse.ganymede.tools
3. There are two files of variables and properties you need to customize for your machine, with these names:
It is recommended you create a directory called ganymedeConfig in your $HOME directory, and put your two customized files there, since several of the script files will look there automatically and use those files, if they exist.
See other examples in org.eclipse.ganymede.tools/examples.
Note, the reason for having your own customizations in a separate location (~/ganymedeConfig) is so that you can easily get a fresh copy of org.eclipse.ganymede.tools without "stepping on" your own customizations. Only the production build machine should depend on the property files located in the org.eclipse.ganymede.tools project.
4. Use ant to run the build.xml from org.eclipse.ganymede.tools. Or, use this script:
cd org.eclipse.ganymede.tools; \
/shared/ganymede/apps/apache-ant-1.7.0/bin/ant \
  -Dganymatic.properties.file=/path/to/ganymatic.properties
5. The results (update site and website) are placed in the directories you specified in the ganymatic.properties file. Email notification (if any) is also configured in the properties file.

[edit] For Build.Eclipse.Org

To create the binaries (headless assembly scripts and runtime) for Ganymatic:

1. Start with an Eclipse 3.3 or 3.4 SDK with the latest version of Buckminster 1.0 installed. You need the two features org.eclipse.buckminster.core and org.eclipse.buckminster.pde. Please note that Buckminster has a lot of optional features. Don't install all of them (some are even mutually exclusive).
2. Check out the org.eclipse.dash.siteassembler project.
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/technology -q co \
  -d org.eclipse.dash.siteassembler org.eclipse.dash/org.eclipse.dash.siteassembler
3. [This should already be the case, but just in case ... ] Make sure the siteassember project targets a 5.0 JVM. Right-click the checked-out project and select
Properties > Java Compiler > Compiler compliance level: 1.5
4. Right click on the org.eclipse.dash.siteassembler project and choose
Buckminster > Invoke Action... > buckminster.clean
5. Right click on the org.eclipse.dash.siteassembler project and choose
Buckminster > Invoke Action... > headless.assembler
  • This generates the org.eclipse.dash.siteassembler/bin/buckminster directory
  • If there are strange errors, delete the org.eclipse.dash.siteassembler/bin/headless.zip and try again from the buckminster.clean
6. Zip & transfer the bin/buckminster directory to build.eclipse.org. Unpack it as you@build.eclipse.org:/shared/ganymede/buckminster/bin/buckminster.
export you=yourUserName; \
cd /path/to/workspace/org.eclipse.dash.siteassembler/bin; \
zip -r /tmp/buckminster.zip buckminster; \
ssh $you@build.eclipse.org "mkdir -p ~/ganymede/buckminster/bin/"; \
scp /tmp/buckminster.zip $you@build.eclipse.org:~/ganymede/buckminster/bin/; \
ssh $you@build.eclipse.org "cd ~/ganymede/buckminster/bin/; unzip buckminster.zip"; \
rm -f /tmp/buckminster.zip
7 Unlike the process for your own local build machine, there should not be any user-defined customization used or required for running Ganymatic on build.eclipse.org. This is by design, so that anyone can take over running the builds on eclipse.org and get the same, repeatable result by simply checking out the org.eclipse.ganymede.tools project and running it (that is, it should not depend on what user id is running the process, for production builds).
Additional tidbits
If interested in how it works, the 'headless.zip' is the main part of the buckminster code that does headless builds (composition) but it lacks the project specific information that is compiled into org.eclipse.dash.siteassembler jar, which is in the bin/buckminster directory, along with the rest of headless.zip.
Ever wonder why or when the bin/buckminster code should be regenerated? See Bug 227693 but the short answer is whenever there are bug fixes to Buckminster, or the org.eclipse.dash.siteassembler changes.

[edit] Additional Requirements and Information

See Europa Build for how last year's build worked.

[edit] Bugs & Feature Requests

[edit] See Also

[edit] Required Patches

Some of these patches may be required, depending on which versions of the Platform are being used, to have a perfectly reproducible build as the eclipse production machine. I just wanted a central place to document them, so I wouldn't lose the information.

  • bug 229019 contains a patch that allows jars with security exceptions to be identified. This patch was applied to the 3.3.2 version of update.core, exported from developer workspace (not built, per se) and used in place of the one that is used by default in the current Buckminster headless assembly application. The patch simply allows the jar with the error to be listed so it can be tracked down and fixed.
  • bug 226850 contains a patch that is required to get site optimizer to pack directories. Hopefully this will be in the final 3.4 platform, but if anyone is using a build prior to that, they would have to have this patch applied to get the jars packed.