Jump to: navigation, search

Difference between revisions of "Eclipse Build Available RSS Feeds"

m (Support For Build Type, Branch & ID)
(Actual Feeds)
 
(15 intermediate revisions by 2 users not shown)
Line 2: Line 2:
  
 
[http://wiki.eclipse.org/index.php?title=Eclipse_Build_Available_RSS_Feeds&action=history Recent changes to this document]
 
[http://wiki.eclipse.org/index.php?title=Eclipse_Build_Available_RSS_Feeds&action=history Recent changes to this document]
 +
 +
==Getting Started With RSS Feeds==
 +
 +
To get started using this code, I'd suggest extracting everything from CVS and having a look at the examples provided:
 +
 +
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse -q checkout -r '''releng_test''' \
 +
-d org.eclipse.build.tools org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools
 +
 +
For details on extracting the code using Eclipse, see '''''[[Eclipse Build Available RSS Feeds Getting Started]]'''''.
 +
 +
 +
Make sure you get the '''releng_test''' branch, not '''HEAD''', if you want the latest implementation.
 +
 +
 +
In there, you'll find '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/scripts_rss/?only_with_tag=releng_test scripts_rss/]''' which includes ant script examples for:
 +
* publishing a feed ('''[http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/scripts_rss/feedPublish.xml?only_with_tag=releng_test feedPublish.xml]''')
 +
* manipulating a feed, eg., updating a feed with a new entry, changing attributes on an existing entry, or just querying a feed for some value ('''[http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/scripts_rss/feedManipulation.xml?only_with_tag=releng_test feedManipulation.xml]''')
 +
* watching a feed and responding to changes ('''[http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/scripts_rss/feedWatch.xml?only_with_tag=releng_test feedWatch.xml]''')
 +
 +
 +
These all use '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/?only_with_tag=releng_test feedTools.jar]''', with sources in '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/src_rss?only_with_tag=releng_test src_rss/]'''. The properties files used by these scripts are in '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/properties/?only_with_tag=releng_test properties/]'''. You'll see three for feedPublish.*.properties (publishing 3 different feeds) and two for feedWatch.*.properties (two projects watching upstream feeds). You'll also find a README in there which provides more details.
 +
 +
 +
You can also look at the sample data in the '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/data/?only_with_tag=releng_test data/]''' folder, which provides three feeds (Eclipse, EMF, UML2).
 +
 +
 +
The feed schema is in the '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/schema/?only_with_tag=releng_test schema/]''' folder. The Ecore model and genmodel created from the schema and used for the feed validator are in '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/model/?only_with_tag=releng_test model/]''', and the feed validator code is best accessed from the '''[http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/scripts_rss/feedValidator.xml?only_with_tag=releng_test feedValidator.xml]''' script in '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/scripts_rss/?only_with_tag=releng_test scripts_rss/]''' and the '''[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/?only_with_tag=releng_test feedValidator.jar]''' in the root.
 +
 +
'''Note:''' The code cited above is the ''example/reference implementation''. For an '''''live working implementation''''', see the [[#Working_Examples | examples below]].
  
 
== Sample Feeds ==
 
== Sample Feeds ==
Line 111: Line 140:
 
= Where To Publish =
 
= Where To Publish =
  
In order to ensure that your feed data is as current as your promoted bits, it's recommended that you publish your feed to the same place you publish your zips/jars. For the EMF example, this is http://download.eclipse.org/tools/emf/feeds/builds.xml, and the CVS archived version of the file is /cvsroot/org.eclipse/www/emf/feeds/builds.xml.
+
In order to ensure that your feed data is as current as your promoted bits, it's recommended that you publish your feed to the same place you publish your zips/jars. For the MDT example, this is [http://download.eclipse.org/modeling/mdt/feeds/builds-uml2.xml http://download.eclipse.org/modeling/mdt/feeds/builds-*.xml], and the CVS archived versions of the files are in [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/mdt/feeds/?root=Eclipse_Website /cvsroot/org.eclipse/www/modeling/mdt/feeds/].
 +
 
 +
It is recommended that you publish one feed per component that people might want to watch uniquely. For MDT, this means [http://www.eclipse.org/modeling/mdt/feeds/ 5 feeds] (at time of writing).
  
If you plan to publish more than one feed or your project contains subprojects which are built or published separately, it's recommended that you name your feeds with the subproject, branch, and type, eg:
 
http://download.eclipse.org/tools/emf/feeds/builds-emf-2.2-N.xml.
 
  
 
== Actual Feeds ==
 
== Actual Feeds ==
  
These are actual feeds, currently being published.  
+
These are actual feeds, currently being published. If you publish a feed that's not listed here, [http://wiki.eclipse.org/index.php?title=Eclipse_Build_Available_RSS_Feeds&action=edit please add it].
 +
 
 +
A sample layout for linking to your feed from your downloads page is here: [http://www.eclipse.org/modeling/mdt/downloads/index.php?project=uml2 UML2 Downloads].
 +
 
 +
A sample icon to use is here: [http://www.eclipse.org/modeling/images/rss-atom10.gif Atom 1.0 icon].
  
* http://download.eclipse.org/eclipse/downloads/builds-eclipse.xml (Eclipse 3.3)
+
* http://download.eclipse.org/eclipse/downloads/builds-eclipse-3.3.xml (Eclipse 3.3.x)
* http://download.eclipse.org/tools/emf/feeds/builds.xml (EMF 2.2.x)
+
* http://www.eclipse.org/modeling/emf/feeds/ (List of EMF feeds)
* http://download.eclipse.org/tools/uml2/feeds/builds.xml (UML2 2.0.x)
+
* http://www.eclipse.org/modeling/mdt/feeds/ (List of MDT feeds)
 +
* http://www.eclipse.org/modeling/m2t/feeds/ (List of M2T feeds)
 +
* http://www.eclipse.org/emft/feeds/ (List of EMFT feeds; components will be removed as they move to MDT, M2T or EMF)
  
 
= Working Examples =
 
= Working Examples =
Line 131: Line 166:
  
 
This is only one use case. For more responsive feed watching, cron-based checks could be done 24h a day, multiple times an hour, and more than once a week.  
 
This is only one use case. For more responsive feed watching, cron-based checks could be done 24h a day, multiple times an hour, and more than once a week.  
 +
 +
The most recent version of the scripts documented here is in [http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/?root=Modeling_Project /cvsroot/modeling/releng-common/tools/scripts/].
 +
 +
== MDT & EMFT Components ==
 +
 +
Currently, MDT & EMFT components only publish RSS, they do not listen to upstream RSS feed(s). Setup is fairly straightforward.
 +
 +
1. <code>[http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/promoteToEclipse.sh?root=Modeling_Project&view=markup promoteToEclipse.sh]</code> (see line 349), is [http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/promoteToEclipse.properties?root=Modeling_Project&view=markup passed the property RSS=1] (or executed using the <code>-rssonly</code> flag).
 +
 +
2. The promote script calls out to <code>[http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/feedPublish.xml?root=Modeling_Project&view=markup feedPublish.xml]</code> using <code>[http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/feedPublish.mdt.properties?root=Modeling_Project&view=markup feedPublish.mdt.properties]</code>.
  
 
== EMF ==
 
== EMF ==
  
As noted in the above scenario, publishing an EMF build automatically is done in two steps, both fired by crontab:
+
Publishing an EMF build automatically is done in two steps, both fired by crontab:
  
 
  # M build every Thu (4); runs as web user, www-data
 
  # M build every Thu (4); runs as web user, www-data
Line 197: Line 242:
  
 
* Here is a list of the bugzillas associated with RSS:
 
* Here is a list of the bugzillas associated with RSS:
** https://bugs.eclipse.org/bugs/show_bug.cgi?id=85485#c17 (recent discussion about Platform's feeds)
+
** https://bugs.eclipse.org/bugs/show_bug.cgi?id=164390 (RSS for eclipse.org releases)
** https://bugs.eclipse.org/bugs/show_bug.cgi?id=116912 (closed)
+
** https://bugs.eclipse.org/bugs/show_bug.cgi?id=141152 (proposed feed enhancements)
** https://bugs.eclipse.org/bugs/show_bug.cgi?id=124172 (EMFT build stack automation)
+
** https://bugs.eclipse.org/bugs/show_bug.cgi?id=130150 ("latest" links for project downloads)
** https://bugs.eclipse.org/bugs/show_bug.cgi?id=141152 (feed enhancements)
+
** <strike>https://bugs.eclipse.org/bugs/show_bug.cgi?id=124172</strike> (build stack automation)
** [https://bugs.eclipse.org/bugs/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=&content=rss more...] (query bugzilla for "rss")
+
** <strike>https://bugs.eclipse.org/bugs/show_bug.cgi?id=116912</strike> (RSS release train)
 +
** <strike>https://bugs.eclipse.org/bugs/show_bug.cgi?id=85485</strike> (Platform RSS feeds)
 +
** [https://bugs.eclipse.org/bugs/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=&content=rss more bugs...] (query bugzilla for "rss")
 +
** [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Platform&component=Releng&assigned_to=codeslave@ca.ibm.com&qa_contact=platform-releng-inbox@eclipse.org&bug_severity=enhancement new bug] (open a new bug in Platform > Releng and assign it to [[User:Nickb | Nick Boldt]])
  
 
= Additional Information =
 
= Additional Information =
Line 207: Line 255:
 
* For more on the Atom 1.0 spec, see [http://www.ietf.org/rfc/rfc4287.txt http://www.ietf.org/rfc/rfc4287.txt].
 
* For more on the Atom 1.0 spec, see [http://www.ietf.org/rfc/rfc4287.txt http://www.ietf.org/rfc/rfc4287.txt].
  
[[Category:Modeling]]
 
 
[[Category:Releng]]
 
[[Category:Releng]]
 +
[[Category:RSS]]
 +
[[Category:Modeling]]

Latest revision as of 13:24, 3 May 2007

Eclipse Build Available RSS Feeds

Recent changes to this document

Getting Started With RSS Feeds

To get started using this code, I'd suggest extracting everything from CVS and having a look at the examples provided:

cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse -q checkout -r releng_test \
-d org.eclipse.build.tools org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools

For details on extracting the code using Eclipse, see Eclipse Build Available RSS Feeds Getting Started.


Make sure you get the releng_test branch, not HEAD, if you want the latest implementation.


In there, you'll find scripts_rss/ which includes ant script examples for:

  • publishing a feed (feedPublish.xml)
  • manipulating a feed, eg., updating a feed with a new entry, changing attributes on an existing entry, or just querying a feed for some value (feedManipulation.xml)
  • watching a feed and responding to changes (feedWatch.xml)


These all use feedTools.jar, with sources in src_rss/. The properties files used by these scripts are in properties/. You'll see three for feedPublish.*.properties (publishing 3 different feeds) and two for feedWatch.*.properties (two projects watching upstream feeds). You'll also find a README in there which provides more details.


You can also look at the sample data in the data/ folder, which provides three feeds (Eclipse, EMF, UML2).


The feed schema is in the schema/ folder. The Ecore model and genmodel created from the schema and used for the feed validator are in model/, and the feed validator code is best accessed from the feedValidator.xml script in scripts_rss/ and the feedValidator.jar in the root.

Note: The code cited above is the example/reference implementation. For an live working implementation, see the examples below.

Sample Feeds

Support For Eclipse.org Jar Signing

So that we can identify which projects' builds are signed, the <build/> node can take an attribute for jars, thus:

<feed xmlns="http://www.w3.org/2005/Atom">
  ...
  <summary>
    <build 
      xmlns="http://www.eclipse.org/2006/BuildFeed" 
      callisto="COMPLETE" 
      href="http://download.eclipse.org/downloads/drops/S-3.2RC5-200605191206" 
      jars="UNSIGNED" 
      type="S">
    ...
  </summary>
</feed>

I'm thinking of 5 status codes, assuming the current jar signing process continues as proposed:

  • NONE (or "") - no status available or not participating
  • UNSIGNED - no jar signage available or done yet
  • SIGNREADY - jars promoted to eclipse.org, ready for signing
  • BUILDREADY - signed on eclipse.org, ready to be collected and bundled as zips and copied to UM site
  • SIGNED - signed & bundled on download page and on UM site

Support For Build Type, Branch & ID

So that we can identify what type of build a given entry is, these values can be specified for a type attribute:

<build 
  xmlns="http://www.eclipse.org/2006/BuildFeed"  
  href="http://download.eclipse.org/downloads/drops/S-3.2RC5-200605191206"
  cvsbranch="HEAD"
  branch="3.2.0"
  datetime="200605191206"
  type="S">
  • N : Nightly
  • I : Integration
  • M : Maintenance
  • S : Stable (Milestone or Release Candidate)
  • R : Release

If the property buildType is not specified, the ant task will attempt to guess the build type based on the buildID, by looking for, like this:

this.buildType = buildID.replaceAll("[^NIMSR]", NS); //$NON-NLS-1$
if (this.buildType.length()>1) 
{
  this.buildType=this.buildType.substring(0, 1);
}

In addition, there are attributes available for branch, cvsbranch and datetime. Branch is a 3-number branch id, like 3.0.0 or 2.2.1. CVS branch is a string like "HEAD" or "R3_2_maintenance", and is optional. Datetime is a 12-digit number defining YYYYMMDDhhmm, the date and time associated with the build.

Support For Coordinated Releases

So that we can identify which projects' have been posted to Callisto/Europa/Whatnot, there's an element in the <build/> node, thus:

<build 
  xmlns="http://www.eclipse.org/2006/BuildFeed" 
  href="http://download.eclipse.org/downloads/drops/S-3.2RC5-200605191206" 
  type="S">
  <coordinated status="COMPLETE">More information here</coordinated>

Status codes are limited to:

Code Purpose/Description
NONE (or "") not part of Callisto or unknown status
BUILDCOMPLETE Have you finished your RC1 bits?
UMSITEREADY Have you placed those bits in your update site?
CALLISTOSITEREADY Have you updated the features.xml file in the Callisto CVS directory?
COMPLETE Are you ready for RC1 to be declared?

Additional information, if applicable, can be placed in the text field of the node, which might include:

Note Purpose/Description
2006-05-02T20:50:00Z When do you expect to finish them?
TPTP If you're waiting for another project, which one(s)?

Where To Publish

In order to ensure that your feed data is as current as your promoted bits, it's recommended that you publish your feed to the same place you publish your zips/jars. For the MDT example, this is http://download.eclipse.org/modeling/mdt/feeds/builds-*.xml, and the CVS archived versions of the files are in /cvsroot/org.eclipse/www/modeling/mdt/feeds/.

It is recommended that you publish one feed per component that people might want to watch uniquely. For MDT, this means 5 feeds (at time of writing).


Actual Feeds

These are actual feeds, currently being published. If you publish a feed that's not listed here, please add it.

A sample layout for linking to your feed from your downloads page is here: UML2 Downloads.

A sample icon to use is here: Atom 1.0 icon.

Working Examples

These are working examples implemented and in use, based on the samples provided in org.eclipse.build.tools.

In this scenario, EMF publishes a scheduled build every Thursday at 00:00, promotes the build to eclipse.org at 01:00, and then UML2, knowing there's an EMF build every week, starts watching the EMF feed, checking hourly at 02:00, 03:00 and 04:00, in order to pick up the latest driver to do its weekly build. As our build server is at IBM, this delay is necessary to ensure that code committed to eclipse.org has time to propagate across the nodes and to the IBM mirrors. Usually the mirrors replicate between 02:00 and 03:00, causing the UML2 build to start at 03:00.

This is only one use case. For more responsive feed watching, cron-based checks could be done 24h a day, multiple times an hour, and more than once a week.

The most recent version of the scripts documented here is in /cvsroot/modeling/releng-common/tools/scripts/.

MDT & EMFT Components

Currently, MDT & EMFT components only publish RSS, they do not listen to upstream RSS feed(s). Setup is fairly straightforward.

1. promoteToEclipse.sh (see line 349), is passed the property RSS=1 (or executed using the -rssonly flag).

2. The promote script calls out to feedPublish.xml using feedPublish.mdt.properties.

EMF

Publishing an EMF build automatically is done in two steps, both fired by crontab:

# M build every Thu (4); runs as web user, www-data
0 0 * * 4 sudo -u www-data /home/www-data/emf-build/scripts/start_cron.sh -proj emf -buildType M \
 -tagBuild true -branchCVS HEAD -noperf -runJDK14Tests -runJDK50Tests -runOldTests > \
 $HOME/cron_logs/start_cron.sh.emf.M.txt 2>&1
# promote M build + update feed (among other things)
0 1 * * 4 /home/www-data/emf-build/scripts/promoteToEclipse_cron.sh -proj emf -branch 2.2.1 -type M \
  -user nickb -userIES nboldt -announce > $HOME/cron_logs/promoteToEclipse_cron.sh.emf.M.txt 2>&1

One of the steps in promoting a build is to check out the latest copy of the feed from CVS, update it, commit it back to CVS, and scp it to eclipse.org. This is done by having promoteToEclipse.sh fire feedPublish.xml:

UML2

In this scenario, UML2 listens from 02:00 to 04:00 by running feedWatch.sh, but has chosen to manually verify and promote builds (also using promoteToEclipse.sh):

# respond to emf feed change; runs as web user, www-data
0 2,3,4 * * 4   cd /home/www-data/emf-build/scripts/ && sudo -u www-data \
 /home/www-data/emf-build/scripts/feedWatch.sh -proj uml2 -branchCVS R2_0_maintenance > \
 $HOME/cron_logs/feedWatch.sh.uml2.txt 2>&1

If a change is seen in the feed, run the respondToFeedChanged target in feedWatch.xml, which has been customized to update dependencies then run a build:

<target name="spawnBuild">
  <echo message="Spawn new build:"/>
  <echo message="/home/www-data/emf-build/scripts/start_cron.sh -proj uml2 ${buildType} ${buildAlias} -branchCVS ${branchCVS} ${runOptions} -URL '${feedWatchAction.TheValue.2}/${feedWatchAction.TheValue.1}' -URL '${feedWatchAction.TheValue.3}'"/>
  <exec spawn="true" executable="bash" resolveexecutable="true">
    <arg line="/home/www-data/emf-build/scripts/start_cron.sh -proj uml2 ${buildType} ${buildAlias} -branchCVS ${branchCVS} ${runOptions} -URL '${feedWatchAction.TheValue.2}/${feedWatchAction.TheValue.1}' -URL '${feedWatchAction.TheValue.3}'"/>
  </exec>
</target>

Latest Code

The latest code - shell and ant scripts, ant task code, XML Schema for the feed, and sample data - can be found in CVS, currently in the releng_test branch. This includes:

Discussion / Bugzilla

Additional Information