Difference between revisions of "Eclipse Build Available RSS Feeds"

From Eclipsepedia

Jump to: navigation, search
m (Working Examples)
(Working Examples)
Line 124: Line 124:
  
 
These are working examples implemented and in use, based on the samples provided in [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/?hideattic=0&only_with_tag=releng_test org.eclipse.build.tools].  
 
These are working examples implemented and in use, based on the samples provided in [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/?hideattic=0&only_with_tag=releng_test org.eclipse.build.tools].  
 +
 +
In this scenario, EMF publishes a scheduled build every Thursday at 00:00, promotes the build to eclipse.org at 01:00, and then UML2, knowing there's an EMF build every week, starts watching the EMF feed, checking hourly at 02:00, 03:00 and 04:00, in order to pick up the latest driver to do its weekly build. As our build server is at IBM, this delay is necessary to ensure that code committed to eclipse.org has time to propagate across the nodes and to the IBM mirrors. Usually the mirrors replicate between 02:00 and 03:00, causing the UML2 build to start at 03:00.
 +
 +
This is only one use case. For more responsive feed watching, cron-based checks could be done 24h a day, multiple times an hour, and more than once a week.
  
 
== EMF ==
 
== EMF ==
  
Crontab scripts:
+
As noted in the above scenario, publishing an EMF build automatically is done in two steps, both fired by crontab:
  
# EMF M builds (2.2.1)
 
 
 
  # M build every Thu (4); runs as web user, www-data
 
  # M build every Thu (4); runs as web user, www-data
 
  0 0 * * 4 sudo -u www-data /home/www-data/emf-build/scripts/start_cron.sh -proj emf -buildType M \
 
  0 0 * * 4 sudo -u www-data /home/www-data/emf-build/scripts/start_cron.sh -proj emf -buildType M \
 
   -tagBuild true -branchCVS HEAD -noperf -runJDK14Tests -runJDK50Tests -runOldTests > \
 
   -tagBuild true -branchCVS HEAD -noperf -runJDK14Tests -runJDK50Tests -runOldTests > \
 
   $HOME/cron_logs/start_cron.sh.emf.M.txt 2>&1
 
   $HOME/cron_logs/start_cron.sh.emf.M.txt 2>&1
+
 
 +
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/start_cron.sh start_cron.sh] (cron wrapper for start.sh)
 +
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/start.sh start.sh] (start a build)
 +
 
 
  # promote M build + update feed (among other things)
 
  # promote M build + update feed (among other things)
 
  0 1 * * 4 /home/www-data/emf-build/scripts/promoteToEclipse_cron.sh -proj emf -branch 2.2.1 -type M \
 
  0 1 * * 4 /home/www-data/emf-build/scripts/promoteToEclipse_cron.sh -proj emf -branch 2.2.1 -type M \
 
   -user nickb -userIES nboldt -announce > $HOME/cron_logs/promoteToEclipse_cron.sh.emf.M.txt 2>&1
 
   -user nickb -userIES nboldt -announce > $HOME/cron_logs/promoteToEclipse_cron.sh.emf.M.txt 2>&1
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/start_cron.sh start_cron.sh] (cron wrapper for start.sh)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/start.sh start.sh] (start a build)
 
  
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/promoteToEclipse_cron.sh promoteToEclipse_cron.sh] (cron wrapper to publish a build)  
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/promoteToEclipse_cron.sh promoteToEclipse_cron.sh] (cron wrapper to publish a build)  
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/promoteToEclipse.sh promoteToEclipse.sh] (publish a build)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/promoteToEclipse.sh promoteToEclipse.sh] (publish a build)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/promoteToEclipse.emf.properties promoteToEclipse.emf.properties] (properties)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/promoteToEclipse.emf.properties promoteToEclipse.emf.properties] (properties)
 +
 +
One of the steps in promoting a build is to check out the latest copy of the feed from CVS, update it, commit it back to CVS, and scp it to eclipse.org. This is done by having <code>promoteToEclipse.sh</code> fire <code>feedPublish.xml</code>:
  
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/feedPublish.xml feedPublish.xml] (publish a feed)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/feedPublish.xml feedPublish.xml] (publish a feed)
Line 152: Line 156:
 
== UML2 ==
 
== UML2 ==
  
Crontab scripts:
+
In this scenario, UML2 listens from 02:00 to 04:00 by running <code>feedWatch.sh</code>, but has chosen to manually verify and promote builds (also using <code>promoteToEclipse.sh</code>):
  
# UML2 M builds (2.0.1)
 
 
 
  # respond to emf feed change; runs as web user, www-data
 
  # respond to emf feed change; runs as web user, www-data
 
  0 2,3,4 * * 4  cd /home/www-data/emf-build/scripts/ && sudo -u www-data \
 
  0 2,3,4 * * 4  cd /home/www-data/emf-build/scripts/ && sudo -u www-data \
Line 161: Line 163:
 
   $HOME/cron_logs/feedWatch.sh.uml2.txt 2>&1
 
   $HOME/cron_logs/feedWatch.sh.uml2.txt 2>&1
 
   
 
   
# promote M build + update feed :: currently not automated (by choice)
 
 
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/feedWatch.sh feedWatch.sh] (check cached copy of a feed for specific changes on remote host)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/feedWatch.sh feedWatch.sh] (check cached copy of a feed for specific changes on remote host)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/feedWatch.uml2.xml feedWatch.uml2.xml] (run UML2-specific check for upstream EMF feed changes)
 
* [http://dev.eclipse.org/viewcvs/indextools.cgi/*checkout*/emf-home/emf-build/scripts/feedWatch.uml2.xml feedWatch.uml2.xml] (run UML2-specific check for upstream EMF feed changes)
Line 176: Line 176:
 
   </exec>
 
   </exec>
 
  </target>
 
  </target>
 
  
 
= Latest Code =
 
= Latest Code =

Revision as of 04:12, 13 September 2006

Contents

Eclipse Build Available RSS Feeds

Recent changes to this document

Sample Feeds

Support For Eclipse.org Jar Signing

So that we can identify which projects' builds are signed, the <build/> node can take an attribute for jars, thus:

<feed xmlns="http://www.w3.org/2005/Atom">
  ...
  <summary>
    <build 
      xmlns="http://www.eclipse.org/2006/BuildFeed" 
      callisto="COMPLETE" 
      href="http://download.eclipse.org/downloads/drops/S-3.2RC5-200605191206" 
      jars="UNSIGNED" 
      type="S">
    ...
  </summary>
</feed>

I'm thinking of 5 status codes, assuming the current jar signing process continues as proposed:

  • NONE (or "") - no status available or not participating
  • UNSIGNED - no jar signage available or done yet
  • SIGNREADY - jars promoted to eclipse.org, ready for signing
  • BUILDREADY - signed on eclipse.org, ready to be collected and bundled as zips and copied to UM site
  • SIGNED - signed & bundled on download page and on UM site

Support For Build Type

So that we can identify what type of build a given entry is, these values can be specified for a type attribute:

<build 
  xmlns="http://www.eclipse.org/2006/BuildFeed" 
  callisto="COMPLETE" 
  href="http://download.eclipse.org/downloads/drops/S-3.2RC5-200605191206" 
  jars="UNSIGNED"
  type="S">
  • N : Nightly
  • I : Integration
  • M : Maintenance
  • S : Stable (Milestone or Release Candidate)
  • R : Release
  • MC : Maintenance-Callisto
  • SC : Stable-Callisto
  • RC : Release-Callisto

If the property buildType is not specified, the ant task will attempt to guess the build type based on the buildID, like this:

 public void setBuildType(String buildType) 
 {
   if (!isNullString(buildType)) 
   {
     this.buildType = buildType;
   }
   else
   {
     this.buildType = buildID.replaceAll("[^NIMSR]", NS); //$NON-NLS-1$
     if (this.buildType.length()>1) 
     {
       this.buildType=this.buildType.substring(0, 1);
     }
   }
 }

Support For Callisto Status

So that we can identify which projects' have been posted to Callisto, there's an attribute on the <build/> node, thus:

<build 
  xmlns="http://www.eclipse.org/2006/BuildFeed" 
  callisto="COMPLETE" 
  href="http://download.eclipse.org/downloads/drops/S-3.2RC5-200605191206" 
  jars="UNSIGNED"
  type="S">

Status codes here could be:

Code Purpose/Description
NONE (or "") not part of Callisto or unknown status
BUILDCOMPLETE Have you finished your RC1 bits?
2006-05-02T20:50:00Z When do you expect to finish them?
TPTP If you're waiting for another project, which one(s)?
UMSITEREADY Have you placed those bits in your update site?
CALLISTOSITEREADY Have you updated the features.xml file in the Callisto CVS directory?
COMPLETE Are you ready for RC1 to be declared?

Where To Publish

In order to ensure that your feed data is as current as your promoted bits, it's recommended that you publish your feed to the same place you publish your zips/jars. For the EMF example, this is http://download.eclipse.org/tools/emf/feeds/builds.xml, and the CVS archived version of the file is /cvsroot/org.eclipse/www/emf/feeds/builds.xml.

If you plan to publish more than one feed or your project contains subprojects which are built or published separately, it's recommended that you name your feeds with the subproject, branch, and type, eg: http://download.eclipse.org/tools/emf/feeds/builds-emf-2.2-N.xml.

Working Examples

These are working examples implemented and in use, based on the samples provided in org.eclipse.build.tools.

In this scenario, EMF publishes a scheduled build every Thursday at 00:00, promotes the build to eclipse.org at 01:00, and then UML2, knowing there's an EMF build every week, starts watching the EMF feed, checking hourly at 02:00, 03:00 and 04:00, in order to pick up the latest driver to do its weekly build. As our build server is at IBM, this delay is necessary to ensure that code committed to eclipse.org has time to propagate across the nodes and to the IBM mirrors. Usually the mirrors replicate between 02:00 and 03:00, causing the UML2 build to start at 03:00.

This is only one use case. For more responsive feed watching, cron-based checks could be done 24h a day, multiple times an hour, and more than once a week.

EMF

As noted in the above scenario, publishing an EMF build automatically is done in two steps, both fired by crontab:

# M build every Thu (4); runs as web user, www-data
0 0 * * 4 sudo -u www-data /home/www-data/emf-build/scripts/start_cron.sh -proj emf -buildType M \
 -tagBuild true -branchCVS HEAD -noperf -runJDK14Tests -runJDK50Tests -runOldTests > \
 $HOME/cron_logs/start_cron.sh.emf.M.txt 2>&1
# promote M build + update feed (among other things)
0 1 * * 4 /home/www-data/emf-build/scripts/promoteToEclipse_cron.sh -proj emf -branch 2.2.1 -type M \
  -user nickb -userIES nboldt -announce > $HOME/cron_logs/promoteToEclipse_cron.sh.emf.M.txt 2>&1

One of the steps in promoting a build is to check out the latest copy of the feed from CVS, update it, commit it back to CVS, and scp it to eclipse.org. This is done by having promoteToEclipse.sh fire feedPublish.xml:

UML2

In this scenario, UML2 listens from 02:00 to 04:00 by running feedWatch.sh, but has chosen to manually verify and promote builds (also using promoteToEclipse.sh):

# respond to emf feed change; runs as web user, www-data
0 2,3,4 * * 4   cd /home/www-data/emf-build/scripts/ && sudo -u www-data \
 /home/www-data/emf-build/scripts/feedWatch.sh -proj uml2 -branchCVS R2_0_maintenance > \
 $HOME/cron_logs/feedWatch.sh.uml2.txt 2>&1

If a change is seen in the feed, run the respondToFeedChanged target in feedWatch.xml, which has been customized to update dependencies then run a build:

<target name="spawnBuild">
  <echo message="Spawn new build:"/>
  <echo message="/home/www-data/emf-build/scripts/start_cron.sh -proj uml2 ${buildType} ${buildAlias} -branchCVS ${branchCVS} ${runOptions} -URL '${feedWatchAction.TheValue.2}/${feedWatchAction.TheValue.1}' -URL '${feedWatchAction.TheValue.3}'"/>
  <exec spawn="true" executable="bash" resolveexecutable="true">
    <arg line="/home/www-data/emf-build/scripts/start_cron.sh -proj uml2 ${buildType} ${buildAlias} -branchCVS ${branchCVS} ${runOptions} -URL '${feedWatchAction.TheValue.2}/${feedWatchAction.TheValue.1}' -URL '${feedWatchAction.TheValue.3}'"/>
  </exec>
</target>

Latest Code

The latest code can be found in CVS, currently in the releng_test branch:


The latest XML Schema is here:


Additionally, zips are available here (but are NOT as recent as what's in CVS):

  • Shell & Ant tasks to create, update, query & publish feeds (CVS, SCP)
  • Shell & Ant tasks to respond to feed changes (eg., sending an email, firing another ant task, launching a shell process with <exec>)
  • Ant task (and shell wrapper) to validate a feed against the latest schema

Additional Information