Jump to: navigation, search

Difference between revisions of "Modeling Project Releng/Releasing"

m (The First Time)
m (The First Time)
Line 54: Line 54:
===The First Time===
===The First Time===
* First, ensure you've set up all your [[Modeling_Project_Releng/Releasing#SSH_Key_Setup | SSH keys]].
* First, ensure you've set up all your [[#SSH_Key_Setup | SSH keys]], including [[#Connecting_the_web_user_to_your_account|the one for the web user]].
* Next, ensure your [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/promoteToEclipse.ecoretools.properties?root=Modeling_Project&view=markup promoteToEclipse.*.properties] file is up to date. Once your file is correct in CVS, you can this to check out the latest version of that file from CVS:
* Next, ensure your [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/promoteToEclipse.ecoretools.properties?root=Modeling_Project&view=markup promoteToEclipse.*.properties] file is up to date. Once your file is correct in CVS, you can this to check out the latest version of that file from CVS:

Revision as of 18:50, 3 December 2007

Release Process


All components are requested to follow this release schedule:

  • Tuesday/Wednesday night:
    • Your code in CVS must be "compile error free". Feel free to run N builds that afternoon to verify this.
  • Wednesday/Thursday morning:
    • One Integration (I) build from your main branch (HEAD), and/or
    • One Maintenance (M) build from your maintenance branch (eg., R1_0_maintenance).

If your component is participating in a Coordinated release such as Ganymede (June 2008), please be aware of the release schedule for Milestones and Release Candidates, and provide Stable (S) builds in alignment with those dates.

Release Reviews / x.y.0 Final Builds

  • DO NOT PUBLISH an R build until you've been through a release review.
  • DO NOT PUBLISH any build using a vanity name ending in ".0" without a release review.
  • DO PUBLISH as many I or S builds as you want -- ideally, you should be publishing one build a week, but only if there are changes.

You can have unlimited weekly Integration (I) builds, then about once every 6 weeks and at the end of your development cycle, you should publish a Stable (S) build. These are usually vanity-named with "0.8.0M2" or "0.8.0RC" to describe the Milestone or Release Candidate status and number.

At the end of your development cycle, you should do an RC build at least one week before your final planned release date. When you pass your Release Review, you can use the /home/www-data/build/modeling/scripts/renameBuild.sh script to rename your "0.8.0RC" Stable (S) build to a "0.8.0" Release (R) build, and repromote it under the new name. This will ensure the correct metadata is properly updated (RSS feed, release notes).

Once your x.y.0 build is reviewed, passed, and released, you can continue in HEAD working on x.y.1 (or branch immediately). Publish as many (M)aintenance builds as you want, then again, as you get close to the end, a Stable 0.8.1RC can be done. Maintenance branch releases need not be reviewed prior to release.

At some point you'll want to branch HEAD (eg., 0.9.0) and R0_8_maintenance (eg., 0.8.1 or 0.8.2). When you start doing 0.9.0 builds, they will be I builds, interspersed with S milestone builds and RC build(s) towards the end of the cycle. Finally, your last RC will be reviewed and you'll once again rename and repromote.

It is HIGHLY ADVISABLE to not branch midway through a release, in order to preserve accurate release notes.

Please also be aware that your build server is SHARED -- clean up your old builds (especially any -noclean builds) when they are obsoleted by newer ones in order to save space for others.

Renaming a Build / Releasing Your GM Build

  • When releasing your final build, you may decide to rename an RC build if nothing bad has happened since that build was released. For example, you might plan to release your final build on Sept. 28, so you'd do an RC1 build two weeks before. If any stop-ship bugs were found, you'd spin an RC2 build the week after, on the 21st. Then, if no more bugs were found, you'd be able to promote that build as your GM (Golden Master) on Sept. 28 - however, you'd have to rename the zips, md5 files, dependencies listed in build.cfg, etc.
  • The good news is that there's an easy way to do this - renameBuild.sh. This script contains detailed instructions and examples of usage; just run it from the commandline w/o options to see. Additionally, here's an example showing a rename of JET from 0.7.1RC2 to 0.7.1:
sudo -u www-data ./renameBuild.sh -proj jet -verbose -branch 0.7.1 \
-buildID S200609210820=R200609210820 -buildAlias 0.7.1RC2=0.7.1 \
eclipse-SDK-M20060919-1045-linux-gtk.tar.gz=eclipse-SDK-3.2.1-linux-gtk.tar.gz \
M20060919-1045=R-3.2.1-200609210945 \
S200609210005=R200609210005 \
2.2.1RC2=2.2.1 \
fullmoon.torolab.ibm.com=download.eclipse.org \
  • When you're done renaming, you can simply promote the new build as you did the previous week (-buildID S200609210820), using the new ID (-buildID R200609210820), and you'll have your GM build published.
  • After releasing the final build of a given version, you need to remember to upgrade the version of the plugins you change. In other words, if a plugin is not changed then its version remains the same. After upgrading the version of a plugin, you need to ensure that the version of the appropriate features and branding plugins are also upgraded, and that the feature is referring to the new versions of the plugins. Also, if you change a plugin with source code, you may need to upgrade the appropriate documentation plugin (and documentation feature and branding plugin) if the build is adding Javadoc to it.

Promoting A Build

The First Time

  • Next, ensure your promoteToEclipse.*.properties file is up to date. Once your file is correct in CVS, you can this to check out the latest version of that file from CVS:
/home/www-data/build/modeling/scripts/getPromotePropertiesFromCVS.sh emf.ecoretools
  • Review the log file generated by the promote process. Log files are stored in /home/www-data/promo_logs/. If you encounter any errors in the log, review Modeling Project Releng or contact User:Nickb.
  • Should a step fail, you can re-run the whole promote (but you may get duplicate entries in your RSS feed and update site). To avoid this, you can remove the new content before re-running the build, or you can run only the relevant parts that failed via the commandline promote script interface. For example, to re-run the javadoc generation step, use:
cd /home/www-data/build/modeling/scripts/;./promoteToEclipse.sh -sub ecoretools -docsonly -Q -buildID I200712031234 ...
  • To see all the commandline options available to you, run ./promoteToEclipse.sh without options for the inline help.
  • PLEASE NOTE! Once your promote is done, you will not see it on eclipse.org for at least 20-30 minutes as the various eclipse.org cluster nodes replicate.
  • When your build does appear at eclipse.org, download the zip and test it out to ensure it works for you. Then try and install via the update site.
  • When you're satisfied, update the _projectCommon.php file for your project (EMFT, MDT, etc.) to remove your project from the $nodownloads array. Wait 30-60 seconds and then refresh your project's homepage, downloads page, release notes, etc. to ensure your component is listed correctly. See also Website Navigation.
  • If you plan to contribute to a coordinated release such as Ganymede, make sure you've added your component to the list, and that you have write access to /cvsroot/callisto. If you don't, open a bug assigned to webmaster@eclipse.org and cc:'d to your PMC User:Nickb. You should ask:
  • "Please add user yourusername to group callisto-dev for Ganymede .sc file contributions."

Update Manager Sites & Build Types

If you go to http://www.eclipse.org/emft/updates/, you'll see a note about two different UM sites. These two sites are there to split final, release builds from interim builds, so that the two types of users can benefit from the two types of releases.

A release build (R) is anything that's done as a final GM (aka Golden Master) release, eg., 0.7.0, 0.7.1, 1.0.0, 1.0.1. Everything else is considered "interim" - Nightly, Integration, Maintenance, Stable (N, I, M, S).

So, for example, when you publish an I build (eg., 0.7.5 I200610050000), it gets added to site-interim.xml. When you do your final build for that stream, (eg., 0.7.5 R200612020000), it will be added to site.xml.

Note also that the URL to give to Update Manager (or to put in your features) is NOT http://www.eclipse.org/emft/updates/, but download.eclipse.org. www.eclipse.org stores website content only, and provides a location for extracting, updating, and committing changes to the site.xml and site-interim.xml files, in order to provide CVS versioning & history. The actual site is http://download.eclipse.org/technology/emft/updates/site*.xml, which is also benefitted by the fact that it's mirrored (unlike www.eclipse.org).

Generating Update Manager Site Jars

Generating Release Notes

Generating release notes is easy - promoteToEclipse.sh does it for you! However, there are a few simple steps to follow to ensure the generated content is accurate & non-duplicative. Here's the general process for closing a bug and using Bugzilla & CVS to provide the information required to generate release notes.

  • Working in Eclipse, fix your code. Test is as much as possible before committing it to CVS.
  • Commit your change(s) to CVS, using the bug's number in square brackets to begin your commit comment, eg:
[127509] javadocs now available
  • Note that you can include multiple bugs (each number on its own line) and comments are not required. The information that will appear in the release notes comes from the bug's Summary field, not your CVS comments.
  • Browse to the bug's webpage, and assign it the status Assigned. Your bugzilla comment could be something like:
Committed to CVS
  • Produce a build, and promote it. Unless you use the -norss flag:
    • Your RSS feed will be updated,
    • The Modeling database will see that new entry,
    • A corresponding entry will be added to the Releases table, and
    • A refresh of the release notes will now show that new entry.


My new build isn't listed on the website!

Mirror replication takes time. Please try again in 30 mins, or re-read your promote log to verify everything went OK.

My new build isn't listed in the Release Notes!

First, check your RSS feed file to ensure it's been updated. If it hasn't, check your promote log for failures in updating the feed. If it has, please be patient and wait for the next database update. Your RSS feed file controls the list of releases that are imported into the Releases table of the Search CVS database, and thus appear on the Release Notes page.

There are errors in my release notes! How do I edit?

If you want to remove or edit values in your RSS feed and flush old/incorrect values from the database, please edit the file /cvsroot/org.eclipse/www/modeling/projectName/feeds/builds-componentName.xml (eg., projectName = emft; componentName = teneo). When done, send an email to User:nickb requesting a database flush for your component's releases.

Bugs are missing from the Release Notes!

Database updates can take a while. If it's been 24hrs and you *still* are missing data, contact User:nickb.

RSS Update Errors When Promoting
Problem Solution
 [AddEntry] [Fatal Error] builds-teneo.xml:13:2: 
   The content of elements must consist of 
   well-formed character data or markup.
 [AddEntry] org.xml.sax.SAXParseException: 
   The content of elements must consist of 
   well-formed character data or markup.
Check the file on the build server in /var/www/html/emft/feeds/. Is it corrupt? Is it full of invalid XML due to a CVS update/merge? If so, so the following:
cd /var/www/html/emft/feeds/; \
rm -f builds-projectName.xml; \
cvs up -Pd; \
chmod 664 builds-projectName.xml; \
chgrp www builds-projectName.xml;

Then rerun your promote using the -rssonly flag to redo that single step. After that, you can force an update of the database - including loading any new CVS commits - with the -searchcvsonly flag. If you don't need to refresh commits, just the Releases table, you can run this (on emft.eclipse.org only):

cd /shared/modeling/searchcvs; \
php relupdate.php -noheat
Permission denied
Check the file on the build server in /var/www/html/emft/feeds/. Is it writable by you or one of the groups of which you're a member? Try this:
cd /var/www/html/emft/feeds/; \
ls -la; groups; \
chmod 664 *; chgrp www *;

Then rerun your promote using the -rssonly flag to redo that single step. After that, you can force an update of the database - including loading any new CVS commits - with the -searchcvsonly flag. If you don't need to refresh commits, just the Releases table, you can run this (on emft.eclipse.org only):

cd /shared/modeling/searchcvs; \
php relupdate.php -noheat

Automatically Fixing Assigned Bugs

As of 2007-08-29, you can use promo.php or the -bugzonly flag to automatically move your ASSIGNED bugs to FIXED, and place a comment stating in which build the bug was fixed.

See also bug 191571 and bug 206558.

Setup / Troubleshooting

The first time you use this option, or if your bugzilla cookie expires, you will receive an error like this:

[promote] ERROR: login cannot be read from ~/.ssh/UpdateBugStateTask.properties.

To regenerate credentials / cookie information, log in to the build server and run:

cd /home/www-data/build/modeling/scripts; 
./bugzillaLogin.sh 2> ~/.ssh/UpdateBugStateTask.properties

Newsgroup Announcement Setup

Setup for posting automatically to the newsgroup to announce new builds is easy.

1. Copy newsgroup-post.txt into your ~/.ssh folder.

build_id@build_server:~ $ mkdir -p ~/.ssh; cp /home/nickb/newsgroup-post/newsgroup-post.txt ~/.ssh

You are now set up to post automatically to the newsgroup while promoting a build. If you want to post as a specific person, rather than the default name & email, there are two additional steps.

2. Edit org.eclipse.emft/releng/[subproject]/promoteToEclipse.[subproject].properties <example>, placing the correct name and email address for the person whose name will appear on newsgroup posts. This step - and the next one - is ONLY required if the name & email are not the same as the default value recorded here.
newsgroupPublisherEmail="John Q. Committer <john_q_emft_committer_guy@eclipse.org>"
3. Copy org.eclipse.emft/releng/[subproject]/promoteToEclipse.[subproject].properties <example> into /home/www-data/build/emft/scripts.

To announce a build that's already been built, you can use the -announceonly flag to skip all other steps and ONLY post to the newsgroup. Otherwise, adding -announce when promoting will add the extra step of posting a quick note to the newsgroup.

Newsgroup Post Threading

Should you want to thread your announcements so that they can be hidden or categorized, here's how you can do so:

  1. create the parent thread into which subsequent posts will go (just post a message to the newsgroup), then
  2. get the post's Message-ID (using Thunderbird, just turn on all headers to see this), and
  3. update the appropriate entry in your /home/www-data/build/emft/scripts/promoteToEclipse.*.properties file on your build server (emf.torolab or emft.eclipse).

Looking at promoteToEclipse.jet.properties as an example:

# publisher of newsgroup announcements
newsgroupPublisherEmail="John Q. Committer <john_q_emft_committer_guy@eclipse.org>"
# to thread posts under an existing post (or posts) enter the slash-escaped Message-ID for which this is a reply,
# eg: <e0i2gc\$an5\$1@utils.eclipse.org>

If you put the Message-ID in the newsgroupThreadReferences property, announcements about builds will be nested under that existing message.

Note that to edit this file you must sudo to the web user:

  • On emf.toro: sudo -u www-data vi promoteToEclipse.jet.properties
  • On emft.eclipse: sudo -u apache vi promoteToEclipse.cdo.properties

If you'd like to store a copy of your changes in CVS, please contact your releng manager so that your changes can be committed into your releng module for safekeeping/versioning.

SSH Key Setup

Getting Access

You must have ssh access to your build server, download1.eclipse.org, and dev.eclipse.org. If you do not have access, please contact the administrators of these servers to gain access. You need a full login shell (eg., bash), not just a CVS login shell (cvssh). If you cannot ssh into dev.eclipse.org or download1.eclipse.org, please open a bug asking for a full login shell, eg., bug 206146.

To get access to your build server, you must forward your static IP address to the server's administrator. If you do not have a static IP, you can set up your router to broadcast your IP to a domain name registration service like dyndns.org and provide your domain name for access to the build server.

Once you have ssh access, you need to set up limited ssh key access between your build server and the eclipse.org download and cvs servers. This will allow you to run the promote script without being prompted for a password periodically.

Connecting in

To grant yourself ssh access from your build server to eclipse.org, ssh to the build server normally.

Under windows, you can use cygwin or PuTTY. Under Linux, open a shell and type `ssh`.

Generating private and public SSH keys

If you don't already have an ssh key pair created, you will need to generate one.

build_id@build_server:~ $ mkdir -p ~/.ssh; chmod 700 ~/.ssh; cd ~/.ssh; ssh-keygen -b 2048 -t rsa

When prompted for a passphrase, hit enter to skip. Having a passphrase, while more secure than having no passphrase, prevents automated login.

Save your public key in the default place: ~/.ssh/id_rsa.pub.

Then, you have to create the authorized_keys file and append to it your public key :

build_id@build_server:~/.ssh $ touch ~/.ssh/authorized_keys;
build_id@build_server:~/.ssh $ cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys;

Connecting out: authorizing remote hosts to let you in

To copy your local public key to the remote machine, you will need to scp (secure copy) the file. From the build server, copy the file to the remote server, then ssh to that server and combine the new key with the any existing keys in the authorized_keys file. In most cases, build_id == eclipse_id, eg., jlescot == jlescot:

Note that if you have not created yet the .ssh folder on the dev.eclipse.org server, you first need to achieve it using the following instructions :

build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
eclipse_id@dev.eclipse.org:~ $ mkdir -p ~/.ssh; chmod 700 ~/.ssh;
eclipse_id@dev.eclipse.org:~/.ssh $ exit

Then you can execute the following scp command :

build_id@build_server:~/.ssh $ scp id_rsa.pub eclipse_id@dev.eclipse.org:~/.ssh/eclipse_id.temp
build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
(you will still be prompted for a password when connecting)
eclipse_id@dev.eclipse.org:~ $ cd ~/.ssh;
eclipse_id@dev.eclipse.org:~/.ssh $ touch ~/.ssh/authorized_keys;
eclipse_id@dev.eclipse.org:~/.ssh $ cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys_old; # backup
eclipse_id@dev.eclipse.org:~/.ssh $ cat ~/.ssh/eclipse_id.temp ~/.ssh/authorized_keys_old > ~/.ssh/authorized_keys; # prepend new key into old file; save as new file
eclipse_id@dev.eclipse.org:~/.ssh $ rm -fr ~/.ssh/eclipse_id.temp ~/.ssh/authorized_keys_old;
eclipse_id@dev.eclipse.org:~/.ssh $ chmod 644 authorized_keys id_rsa.pub known_hosts; chmod 600 id_rsa; # fix perms
eclipse_id@dev.eclipse.org:~/.ssh $ exit
build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
(this time you should get in using your key instead of needing a password)

Once you have ~/.ssh/authorized_keys and ~/.ssh/known_hosts updated with your public key and server fingerprint(s) (respectively), you should be able to connect to these systems without being prompted, and the promote script should be good to go unassisted.

Connecting out: creating host keys

SSH to your build server (if not already there). Then, one by one, ssh from there to these target systems, accepting whatever yes/no questions are posed about host keys or IP addresses. Once in, exit back to your build server. In most cases, build_id == eclipse_id, eg., jlescot == jlescot:

  • build_id@FQDN, build_id@localhost,
  • eclipse_id@download1.eclipse.org, eclipse_id@dev.eclipse.org

For example:

ssh jlescot@emft.eclipse.org
ssh jlescot@localhost
ssh jlescot@download1.eclipse.org
ssh jlescot@dev.eclipse.org

All of those should work without being prompted for a password. If you are prompted, you need to set an ssh key in your ~/.ssh/authorized_keys file for each user & hostname pair to which you want to connect. The first time you try each of these connections, you should be be prompted to add the host's fingerprint to your ~/.ssh/known_hosts file. This is also required.

Connecting the web user to your account

In order to run builds from the web site on the build server, the web user must be able to ssh to your account on the build server. Put its public key in your authorized_keys file to make this work.


If you require further assistance setting up key access, try here: http://cfm.gs.washington.edu/security/ssh/client-pkauth/.

Promotion FAQ

Many things can go wrong when promoting a build (CVS or SCP errors, artifacts not created, java failures). Most are a result of network or permission problems, and can be fixed by trying again or correcting the permission problem.

RSS Feeds

The log file for my promoted build shows a FileNotFoundException (Permission denied) or NullPointerException when publishing my RSS feed!

Either the permissions on the file do not allow you to edit it...

Buildfile: feedPublish.xml
  [AddEntry] java.io.FileNotFoundException: /var/www/html/modeling/emft/feeds/builds-net4j.xml (Permission denied)

... or your file has been corrupted by a CVS merge/update, and contains "<<<<<<<" and "=======" which makes it an invalid XML file.

To fix this, delete the file on filesystem, refresh from cvs & redo the promote.

Here's how:

$ cd /var/www/html/modeling/mdt/feeds/ 
$ less builds-uml2.xml  
  (note the <<<<<<<<<<<<< and =============== markers that the cvs server added) 
$ rm -fr builds-uml2.xml 
$ cvs up builds-uml2.xml 
  cvs update: warning: `builds-uml2.xml' was lost 
  U builds-uml2.xml 
$ chmod g+w builds-uml2.xml
$ chgrp www builds-uml2.xml

Then rerun the promote to fix the missing feed entry (required for release notes)...

./promoteToEclipse.sh -sub uml2 -Q -buildID somebuildID -rssonly -cvsbranch somebranchvalue ...  

If you decide to repromote completely using the web UI, make sure you've removed this build's site*.xml contribution because a full promote will add it again, slowing down the operation of the update site and presenting users with confusing duplicate entries.

Finally, since the RSS is now updated, you can force the database to be refreshed and have the new release appear in your Release Notes.

Update Site

Please check your promote logs for errors or transient glitches, like CVS disconnects or SCP problems. Sometimes one problem can manifest as several problems, as in the case of this EMFT Validation build's UM jar creation, which threw four warnings that something had gone wrong, all starting from a transient CVS checkout problem (bandwidth overload or connection reset).

[promote] [9] [13:53:18] Create & promote Update Manager jars:
[promote] Running buildUpdate.sh:
[umj-co] [1] Checking out org.eclipse.releng.basebuilder from dev 
  using HEAD
cvs checkout: failed to open /home/vramaswamy/.cvspass for reading: 
  No such file or directory
cvs [checkout aborted]: recv() from server dev.eclipse.org: 
  Connection reset by peer
The java class is not found: org/eclipse/core/launcher/Main
features/*.jar: No such file or directory
plugins/*.jar: No such file or directory
[umj] [7d] [13:54:47] Comparing local and remote folders MD5 sums 
  to ensure SCP completeness...
[compare] Compare md5sums...
[compare] MD5s do not match! Compare failed.
[umj] [13:54:52] ERROR! Script exiting from compareFolders.sh
[promote] [9] [13:54:52] Create & promote Update Manager jars done.

In this case, the problems were:

  • CVS disconnected while checking out required builder code
  • Build failed looking for Eclipse Main, so no build occurred
  • Copying files from temp to /var/www/technology/emft/updates/ failed as they were not created
  • MD5 check failed as nothing was copied to download1.eclipse.org

In this case, the solution was to:

  1. Verify that nothing had been produced on either the build server or download1.eclipse.org.
  2. Confirm that site*.xml had not been changed in CVS, and rollback any changes for this build's ID (if any).
  3. Re-run the promote using the -jarsonly flag in order to bypass all the steps that completed successfully and ONLY do the UM jar creation step.