Jump to: navigation, search

Difference between revisions of "Modeling Project Releng/Releasing"

(Promoting A Build: add new "first time" section)
(The log file for my promoted build shows a FileNotFoundException (Permission denied) or NullPointerException when publishing my RSS feed!)
 
(39 intermediate revisions by 3 users not shown)
Line 11: Line 11:
  
 
If your component is participating in a [[:Category:Coordinated | Coordinated]] release such as [[Ganymede]] (June 2008), please be aware of the release schedule for [[Ganymede#Milestones_and_Release_Candidates | Milestones and Release Candidates]], and provide Stable (S) builds in alignment with those dates.
 
If your component is participating in a [[:Category:Coordinated | Coordinated]] release such as [[Ganymede]] (June 2008), please be aware of the release schedule for [[Ganymede#Milestones_and_Release_Candidates | Milestones and Release Candidates]], and provide Stable (S) builds in alignment with those dates.
 +
 +
===Rampdown to Release===
 +
 +
After the M7/RC0 builds are published, it's usually time to review the copyrights in your sources to ensure they're accurate. You may also be asked at some point to provide all of the about.* files to the Eclipse Management Organization (EMO) legal department so they can review them prior to the release.
 +
 +
You will also be required to provide a slide deck to your PMC, as he will be representing your component at your [http://www.eclipse.org/projects/dev_process/release-review.php Release Review]. For samples of past release review slides, see
 +
* [http://www.eclipse.org/projects/previous-release-reviews.php Previous Release Reviews]
 +
:* [http://www.eclipse.org/projects/slides/EMF-QVT_Europa_Release_Review.pdf EMF QTV Europa Review]
 +
:* [http://www.eclipse.org/projects/slides/EMF_Europa_Review.pdf EMF (Core) Europa Review]
 +
 +
==== Collecting Abouts ====
 +
 +
* First, sources are collected [[CVS_Source_From_Mapfile|using this script]] and the map file listed on your downloads page for your most recent milestone build. (If you don't have access to your own linux box, you can run this on emft.eclipse.org or build.eclipse.org.)
 +
* Next, check out Jeff McAffer's [http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg02409.html org.eclipse.orbit.releng.aboutextractor tool] from CVS and use that to process your sources into a new directory containing only the Abouts.
 +
 +
* Finally, that new directory can be zipped and sent to emo@eclipse.org and/or Janet.Campbell@eclipse.org, as required.
 +
 +
==== Maintaining Your IP Log ====
 +
 +
As part of being an Eclipse project, [http://www.eclipse.org/projects/dev_process/ip-process-in-cartoons.php everyone is responsible for keeping the IP clean]. This includes:
 +
 +
* Contributors: by attaching patches to bugs
 +
* Committers: by committing patches and filing CQs for large contributions, and using the 'contributed' keyword on bugs with contributions which have been committed to your code base
 +
* Project/Component Leads: by enforcing the IP rules for your committers/contributors, updating IP Logs in the event they ever get out of date, and filing CQs on behalf of soon-to-be committers
 +
* PMCs: all of the above, plus liasing with the Foundation in terms of [http://www.eclipse.org/projects/dev_process/release-review.php Release & Graduation Reviews] -- collecting Abouts, Slide Decks, and IP Logs.
 +
 +
 +
Currently, there are [http://www.eclipse.org/modeling/emft/eclipse-project-ip-log.php three ways to log your project or component's IP].
 +
 +
<table border="0"><tr valign="top"><th>Method</th>
 +
<th>Provider</th><th>Example</th><th>How to Update</th></tr>
 +
<tr bgcolor="#EEEEFF"><th>
 +
Dynamic</th><td>Eclipse Foundation</td><td>[http://www.eclipse.org/projects/ip_log.php?projectid=modeling.emft ip_log.php]</td><td>
 +
Contribute to this log by modifying bugs in Bugzilla and by opening CQs in IPZilla.</td></tr>
 +
<tr bgcolor="#FFFFEE"><th>
 +
Half & Half</th><td>Modeling Project</td><td>[http://www.eclipse.org/modeling/emft/project-info/ipquery.php ipquery.php]
 +
</td><td>
 +
Contribute to this log by modifying bugs in Bugzilla and by opening CQs in IPZilla; you can also add static information by editing your [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emf/project-info/ipquery.php?root=Eclipse_Website&content-type=text%2Fplain&view=co ipquery.php] file.</td></tr>
 +
<tr bgcolor="#EEEEFF"><th>
 +
Static</th><td>project/component committers</td><td>[http://www.eclipse.org/modeling/emft/compare/eclipse-project-ip-log.csv eclipse-project-ip-log.csv]
 +
</td><td>
 +
Contribute to this log by editing your [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emft/compare/eclipse-project-ip-log.csv?root=Eclipse_Website&view=co&content-type=text%2Fplain eclipse-project-ip-log.csv] file.</td></tr>
 +
</table>
 +
 +
For more on how to keep your IP Log up to date, see '''''[[Modeling Project Releng/Releasing/IP Log|Modeling Project Releng - Releasing - IP Log]]'''''.
  
 
=== Release Reviews / x.y.0 Final Builds ===
 
=== Release Reviews / x.y.0 Final Builds ===
Line 46: Line 91:
 
  emf.torolab.ibm.com=download.eclipse.org
 
  emf.torolab.ibm.com=download.eclipse.org
  
* When you're done renaming, you can simply promote the new build as you did the previous week (<tt style="color:DarkGreen">-buildID S200609210820</tt>), using the new ID (<tt style="color:DarkGreen">-buildID R200609210820</tt>), and you'll have your GM build published.
+
* When you're done renaming, you can simply promote the new build as you did the previous week (<tt style="color:DarkGreen">-buildID S200609210820</tt>), using the new ID (<tt style="color:DarkGreen">-buildID R200609210820</tt>), and you'll have your GM build published. '''*HOWEVER**''', if you are participating in a coordinated release, you must adhere to the process rules for that release. For example: [[Ganymede/Final_Daze/Modeling_Project|Ganymede - Final Daze - Modeling Project]], [[Ganymede/Final_Daze|Ganymede - Final Daze]].
  
 
* After releasing the final build of a given version, you need to remember to upgrade the version of the plugins you change. In other words, if a plugin is not changed then its version remains the same. After upgrading the version of a plugin, you need to ensure that the version of the appropriate features and branding plugins are also upgraded, and that the feature is referring to the new versions of the plugins. Also, if you change a plugin with source code, you may need to upgrade the appropriate documentation plugin (and documentation feature and branding plugin) if the build is adding Javadoc to it.
 
* After releasing the final build of a given version, you need to remember to upgrade the version of the plugins you change. In other words, if a plugin is not changed then its version remains the same. After upgrading the version of a plugin, you need to ensure that the version of the appropriate features and branding plugins are also upgraded, and that the feature is referring to the new versions of the plugins. Also, if you change a plugin with source code, you may need to upgrade the appropriate documentation plugin (and documentation feature and branding plugin) if the build is adding Javadoc to it.
 +
 +
== Build Naming Conventions ==
 +
 +
Because Modeling consists of many projects with an ever-expanding set of components, it's important that we all adhere to consistent naming rules for each [http://www.eclipse.org/modeling/downloads/build-types.php build type]. It's also important to be consistent within your component or your release notes will be sloppy or incomplete. Build Aliases must conform to one of these patterns in order to be automatically added to the release notes database:
 +
 +
<blockquote><table><tr><td>{{codeblock|/[IM]\d{12}/}}</td><td>  or  </td><td>{{codeblock|/\d+\.\d+\.\d+.*/}}</td></tr></table></blockquote>
 +
 +
When building, use the Build Alias field to name your S and R [http://www.eclipse.org/modeling/downloads/build-types.php type builds].
 +
 +
: [[Image:Uml2tools-build-page.png]]
 +
 +
The rules are very simple.
 +
 +
* If it's a ''Milestone'' (build type '''S'''), call it "x.y.z''M''n", eg., '''0.7.0M6'''
 +
* If it's an ''RC'' (build type '''S'''), call it "x.y.z''RC''n", eg., '''0.8.0RC1''' or '''0.8.2RC'''
 +
* If it's a ''Release'' (build type '''R'''), call it "x.y.z", eg., '''0.9.0''' or '''2.0.2'''
 +
* All other builds ('''I''', '''M''') use no alias and are named using the default pattern of <code>"/[IM]\d{12}/"</code>, eg., '''I200802070245'''
  
 
==Promoting A Build==
 
==Promoting A Build==
Line 54: Line 116:
 
===The First Time===
 
===The First Time===
  
* First, ensure you've set up all your [[Modeling_Project_Releng/Releasing#SSH_Key_Setup | SSH keys]].
+
* First, ensure you've set up all your [[#SSH_Key_Setup | SSH keys]], including [[#Connecting_the_web_user_to_your_account|the one for the web user]].
  
* Next, ensure your [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/promoteToEclipse.ecoretools.properties?root=Modeling_Project&view=markup promoteToEclipse.*.properties] file is up to date. Once your file is correct in CVS, you can this to check out the latest version of that file from CVS:
+
* Next, ensure your [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/promoteToEclipse.ecoretools.properties?root=Modeling_Project&view=markup promoteToEclipse.*.properties] file is up to date. Once your file is correct in CVS, you can run this (sudo'd as the web user, www-data or apache) to check out the latest version of that file from CVS:
  
 
{{codeblock|/home/www-data/build/modeling/scripts/getPromotePropertiesFromCVS.sh emf.ecoretools}}
 
{{codeblock|/home/www-data/build/modeling/scripts/getPromotePropertiesFromCVS.sh emf.ecoretools}}
 +
 +
* Contact [[User:Nickb]], and ask that he:
 +
:* add your component to [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools/scripts_cvs/searchcvs/relupdate.php relupdate.php],
 +
:* add a "0.0.0" release to releases table for your component, and
 +
:* check out your project and releng sources for use with [http://build.eclipse.org/modeling/build/updateSearchCVS.php searchcvs]
  
 
* Load up [http://emft.eclipse.org/modeling/emft/ecoretools/build/promo.php promo.php]. You can only promote I, M, S, and R builds this way. If your branch is not listed, make sure your [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emft/ecoretools/build/_common.php?revision=1.1&root=Eclipse_Website&view=markup _common.php] file is up to date. See also [[Modeling_Project_Releng/Website_Maintenance | Website Maintenance]].  
 
* Load up [http://emft.eclipse.org/modeling/emft/ecoretools/build/promo.php promo.php]. You can only promote I, M, S, and R builds this way. If your branch is not listed, make sure your [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emft/ecoretools/build/_common.php?revision=1.1&root=Eclipse_Website&view=markup _common.php] file is up to date. See also [[Modeling_Project_Releng/Website_Maintenance | Website Maintenance]].  
Line 64: Line 131:
 
* Review the log file generated by the promote process. Log files are stored in <code>/home/www-data/promo_logs/</code>. If you encounter any errors in the log, review [[Modeling Project Releng]] or contact [[User:Nickb]].
 
* Review the log file generated by the promote process. Log files are stored in <code>/home/www-data/promo_logs/</code>. If you encounter any errors in the log, review [[Modeling Project Releng]] or contact [[User:Nickb]].
  
* Should a step fail, you can re-run the whole promote (but you may get duplicate entries in your RSS feed and update site). To avoid this, you can remove the new content before re-running the build, or you can run only the relevant parts that failed via the commandline promote script interface. For example, to re-run the javadoc generation step, use:
+
* Should a step fail, you can re-run the whole promote (but you may get duplicate entries in your [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emft/feeds/?root=Eclipse_Website RSS feed] and [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emft/updates/?root=Eclipse_Website update site]). To avoid this, you can remove the new content before re-running the build, or you can run only the relevant parts that failed via the commandline promote script interface. For example, to re-run the javadoc generation step, use:
  
 
{{codeblock|cd /home/www-data/build/modeling/scripts/;./promoteToEclipse.sh -sub ecoretools -docsonly -Q -buildID I200712031234 ...}}
 
{{codeblock|cd /home/www-data/build/modeling/scripts/;./promoteToEclipse.sh -sub ecoretools -docsonly -Q -buildID I200712031234 ...}}
Line 70: Line 137:
 
* To see all the commandline options available to you, run <code>./promoteToEclipse.sh</code> without options for the inline help.
 
* To see all the commandline options available to you, run <code>./promoteToEclipse.sh</code> without options for the inline help.
  
* '''PLEASE NOTE!''' Once your promote is done, you will not see it on eclipse.org for at least 20-30 minutes as the various eclipse.org cluster nodes replicate.  
+
* '''PLEASE NOTE!''' Once your promote is done, you will not see it on eclipse.org for at least 20-30 minutes as the various eclipse.org cluster nodes replicate. Why?
 +
:* ''download.eclipse.org is served via nfs on all of the nodes from the nfsslave. The master and slave replicate every 4-6 hours depending on system loads.''
 +
::* So, unfortunately, the only recourse is patience.  
  
* When your build does appear at eclipse.org, download the zip and test it out to ensure it works for you. Then try and install via the update site.
+
* When your build does appear at eclipse.org, [http://www.eclipse.org/modeling/emft/downloads/ download] the zip and test it out to ensure it works for you. Then try and install via the [http://www.eclipse.org/modeling/emft/updates/ update site].
  
 
* When you're satisfied, update the [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emft/_projectCommon.php?root=Eclipse_Website&view=markup _projectCommon.php] file for your project (EMFT, MDT, etc.) to remove your project from the <code>$nodownloads</code> array. Wait 30-60 seconds and then refresh your project's homepage, downloads page, release notes, etc. to ensure your component is listed correctly. See also [[Modeling_Project_Releng/Website_Maintenance#Site_Navigation | Website Navigation]].
 
* When you're satisfied, update the [http://dev.eclipse.org/viewcvs/index.cgi/www/modeling/emft/_projectCommon.php?root=Eclipse_Website&view=markup _projectCommon.php] file for your project (EMFT, MDT, etc.) to remove your project from the <code>$nodownloads</code> array. Wait 30-60 seconds and then refresh your project's homepage, downloads page, release notes, etc. to ensure your component is listed correctly. See also [[Modeling_Project_Releng/Website_Maintenance#Site_Navigation | Website Navigation]].
 +
 +
* If you plan to contribute to a coordinated release such as [[Ganymede]], make sure you've added your component to the [[Ganymede#Projects | list]], and that you have write access to <code>/cvsroot/callisto</code>. If you don't, [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=community&component=CVS open a bug] assigned to webmaster@eclipse.org and cc:'d to your PMC [[User:Nickb]]. You should ask:
 +
:* "Please add user ''yourusername'' to group '''callisto-dev''' for Ganymede .sc file contributions."
 +
 +
====Troubleshooting====
 +
 +
* If you don't set up your [[#SSH_Key_Setup | SSH keys]], including [[#Connecting_the_web_user_to_your_account|the one for the web user]] correctly, you may see this in your promo log:
 +
 +
Permission denied, please try again.
 +
Permission denied, please try again.
 +
Permission denied (publickey,gssapi-with-mic,password).
 +
 +
* If you see '''Permission denied''' errors in the promote log, you may not have sufficient permission to create files on the download server.
 +
 +
mkdir: cannot create directory `/home/data/httpd/download.eclipse.org/modeling/m2t/mtl': Permission denied
 +
 +
:* [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=community&component=CVS Open a bug] assigned to webmaster@eclipse.org and cc:'d to your PMC and [[User:Nickb]], asking "Please add user ''yourusername'' to group '''groupname''' for publishing buildname builds."
 +
:* ''Groupname will be in the form '''m2tadmin''' (old style) or '''modeling.emf.website''' (new style).''
 +
 +
===Scheduled Releases (Promoting Automatically)===
 +
 +
See [[Modeling_Project_Releng/Releasing/Scheduled_Releases|Scheduled Releases]].
  
 
===Update Manager Sites &amp; Build Types===
 
===Update Manager Sites &amp; Build Types===
  
If you go to http://www.eclipse.org/emft/updates/, you'll see a note about two different UM sites. These two sites are there to split final, release builds from interim builds, so that the two types of users can benefit from the two types of releases.
+
If you go to http://www.eclipse.org/modeling/emft/updates/, you'll see a note about two different UM sites. These two sites are there to split final, release builds from interim builds, so that the two types of users can benefit from the two types of releases.
  
 
A release build (R) is anything that's done as a final GM (aka ''Golden Master'') release, eg., 0.7.0, 0.7.1, 1.0.0, 1.0.1.
 
A release build (R) is anything that's done as a final GM (aka ''Golden Master'') release, eg., 0.7.0, 0.7.1, 1.0.0, 1.0.1.
 
Everything else is considered "interim" - Nightly, Integration, Maintenance, Stable (N, I, M, S).
 
Everything else is considered "interim" - Nightly, Integration, Maintenance, Stable (N, I, M, S).
  
So, for example, when you publish an I build (eg., 0.7.5 I200610050000), it gets added to [http://download.eclipse.org/technology/emft/updates/site-interim.xml site-interim.xml]. When you do your final build for that stream, (eg., 0.7.5 R200612020000), it will be added to [http://download.eclipse.org/technology/emft/updates/site.xml site.xml].
+
So, for example, when you publish an I build (eg., 0.7.5 I200610050000), it gets added to [http://download.eclipse.org/modeling/emft/updates/site-interim.xml site-interim.xml]. When you do your final build for that stream, (eg., 0.7.5 R200612020000), it will be added to [http://download.eclipse.org/modeling/emft/updates/site.xml site.xml].
  
Note also that the URL to give to Update Manager (or to put in your features) is NOT http://www.eclipse.org/emft/updates/, but download.eclipse.org. www.eclipse.org stores website content only, and provides a location for extracting, updating, and committing changes to the site.xml and site-interim.xml files, in order to provide CVS versioning &amp; history. The actual site is [http://download.eclipse.org/technology/emft/updates/ http://download.eclipse.org/technology/emft/updates/site*.xml], which is also benefitted by the fact that it's mirrored (unlike www.eclipse.org).
+
Note also that the URL to give to Update Manager (or to put in your features) is NOT http://www.eclipse.org/modeling/emft/updates/, but download.eclipse.org. www.eclipse.org stores website content only, and provides a location for extracting, updating, and committing changes to the site.xml and site-interim.xml files, in order to provide CVS versioning &amp; history. The actual site is [http://download.eclipse.org/modeling/emft/updates/ http://download.eclipse.org/modeling/emft/updates/site*.xml], which is also benefitted by the fact that it's mirrored (unlike www.eclipse.org).
  
* For more on build types, see http://www.eclipse.org/emf/downloads/build-types.php.  
+
* For more on build types, see http://www.eclipse.org/modeling/downloads/build-types.php.  
  
 
====Generating Update Manager Site Jars====
 
====Generating Update Manager Site Jars====
  
* For details on generating Update Manager site XML and jars, see: [[Update_Manager_Site_Generation|Update Manager Site Generation]].
+
* For details on generating Update site XML, jars, digest, and metadata, see: [[Modeling_Project_Releng/Releasing/Update Site Generation|Update Site Generation]].
 +
 
 +
* For troubleshooting, see [[Modeling_Project_Releng/Releasing/Update_Site_Generation/Troubleshooting|Troubleshooting]].
 +
 
 +
====Contributing To Ganymede Update Site====
 +
 
 +
<table><tr valign="top"><td>
 +
 
 +
* The [[Ganymede/Build]] collects your *.sc files and turns them into features for the combined Ganymede update site.
 +
 
 +
* To contribute to Ganymede, you must complete these steps:
 +
 
 +
:* Open a bug to get added to group callisto-dev, eg., {{bug|212325}}. Bug should be assigned to '''webmaster@eclipse.org''' and cc'd to '''bjorn.freeman-benson@eclipse.org''' for PMC-like approval.
 +
:* Use your [http://emft.eclipse.org/modeling/emft/compare/build/promo.php promo.php] build page (snapshot at right) to ensure your contribution (/cvsroot/callisto/org.eclipse.ganymede.sitecontributions/*.sc) is generated and committed to CVS
 +
::* promoteToEclipse.sh, called from the promote page, will generate the XML contents of your .sc file. That's it!
 +
 
 +
* To control what features are contributed, tweak your promoteToEclipse.*.properties file. For example:
 +
 
 +
[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.releng/scripts/promoteToEclipse.emf.properties?root=Modeling_Project&view=markup For EMF/SDO/XSD]:
 +
 
 +
# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$"
 +
coordsiteFeaturePattern=".*eclipse/features/org.eclipse..*(emf|emf.sdk|xsd|xsd.sdk|emf.ecore.sdo|emf.ecore.sdo.sdk|emf.common|emf.common.ui|emf.ecore|emf.ecore.edit|emf.edit|emf.edit.ui|xsd.edit)_.*\/$"
 +
# pattern to use when setting feature to include in coordsite's main feature ('Models and Model Development'); all other matching features will go in 'Enabling Features'
 +
coordsiteMainFeaturePattern="org.eclipse.(emf|emf.sdk|xsd|xsd.sdk|emf.ecore.sdo|emf.ecore.sdo.sdk)$"
 +
 
 +
Thus, EMF contributes 6 "main" features:
 +
* emf, emf.sdk, xsd, xsd.sdk, emf.ecore.sdo, emf.ecore.sdo.sdk
 +
And 7 "enabling" features:
 +
* emf.common, emf.common.ui, emf.ecore, emf.ecore.edit, emf.edit, emf.edit.ui, xsd.edit
 +
 
 +
[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.ocl.releng/promoteToEclipse.ocl.properties?root=Modeling_Project&view=markup For OCL]:
 +
 
 +
# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$"
 +
coordsiteFeaturePattern=".*eclipse/features/org.eclipse.(emf.ocl|ocl.all|ocl.all.sdk|ocl|ocl.uml)_.*\/$"
 +
coordsiteMainFeaturePattern="org.eclipse.ocl.(all|all.sdk)"
 +
 
 +
To create aggregate features, such as [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.transaction.releng/promoteToEclipse.transaction.properties?root=Modeling_Project&view=markup EMF Transaction] does to bundle up [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.query.releng/promoteToEclipse.query.properties?root=Modeling_Project&view=markup EMF Query], [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.validation.releng/promoteToEclipse.validation.properties?root=Modeling_Project&view=markup EMF Validation], and EMF Transaction into a single "emfqtv.all.sdk" feature:
 +
 
 +
# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$"
 +
#omit SDK because this is only to create a temp file which will be combined w/ rest of QTV components
 +
coordsiteFeaturePattern=".*eclipse/features/org.eclipse.emf.(emfqtv.all|emfqtv.all.sdk|query|query.ocl|validation|validation.ocl|transaction|workspace)_.*\/$"
 +
coordsiteMainFeaturePattern="org.eclipse.emf.emfqtv.(all|all.sdk)"
 +
 
 +
Then, in the [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.validation.releng/promoteToEclipse.validation.properties?root=Modeling_Project&view=markup EMF Validation] (and [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.query.releng/promoteToEclipse.query.properties?root=Modeling_Project&view=markup EMF Query]) properties files:
 +
 
 +
# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$"
 +
#omit SDK because this is only to create a temp file which will be combined w/ rest of QTV components
 +
coordsiteFeaturePattern=".*eclipse/features/org.eclipse.emf.(validation|validation.ocl)_.*\/$"
 +
coordsiteMainFeaturePattern="NONE"
 +
</td><td>[[Image:EMFT-Build-Page2.png|thumb|right|Build Promotion Page]]</td>
 +
</tr></table>
 +
 
 +
=====Troubleshooting=====
 +
 
 +
See [[Modeling_Project_Releng/Releasing/Update_Site_Generation/Troubleshooting#How_do_I_fix_this.3F|Troubleshooting]].
  
 
===Generating Release Notes===
 
===Generating Release Notes===
Line 101: Line 246:
  
 
  [127509] javadocs now available
 
  [127509] javadocs now available
 +
 +
* If you use Mylyn you can use this template, with a space at the end:
 +
 +
[{$task.id}]
  
 
* Note that you can include multiple bugs (each number on its own line) and comments are not required. The information that will appear in the release notes comes from the bug's <tt style="color: DarkGreen">Summary</tt> field, not your CVS comments.
 
* Note that you can include multiple bugs (each number on its own line) and comments are not required. The information that will appear in the release notes comes from the bug's <tt style="color: DarkGreen">Summary</tt> field, not your CVS comments.
Line 124: Line 273:
  
 
First, check your RSS feed file to ensure it's been updated. If it hasn't, check your promote log for failures in updating the feed. If it has, please be patient and wait for the next database update. Your RSS feed file controls the list of releases that are imported into the Releases table of the Search CVS database, and thus appear on the Release Notes page.
 
First, check your RSS feed file to ensure it's been updated. If it hasn't, check your promote log for failures in updating the feed. If it has, please be patient and wait for the next database update. Your RSS feed file controls the list of releases that are imported into the Releases table of the Search CVS database, and thus appear on the Release Notes page.
 +
 +
There are three rules to consider when building, naming, and publishing maintenance builds:
 +
 +
# If you've branched, you need to tell the build system (and thus the RSS feed generator). Maintenance builds from HEAD are ok, but may confuse the database if you try to branch during a release rather than immediately afterwards. Branching right after your 1.0 release is recommended.
 +
# Maintenance builds on, say, 1.0.0 MUST BE VERSIONED to 1.0.1, 1.0.2: if 1.0.0 has already been released, everything since must be > 1.0.
 +
# Maintenance builds should NOT use an alias of "1.0.0"; they should simply default to MyyyymmddHHMM. You can only have ONE unique "1.0.0" in the database.
  
 
=====There are errors in my release notes! How do I edit?=====
 
=====There are errors in my release notes! How do I edit?=====
Line 169: Line 324:
 
|}
 
|}
  
=== Automatically Fixing Assigned Bugs ===
+
=== Automated 'Fix available in ...' Bugzilla Comments ===
  
As of 2007-08-29, you can use [http://emft.eclipse.org/modeling/emft/compare/build/promo.php promo.php] or the <tt style="color: DarkGreen">-bugzonly</tt> flag to automatically move your ASSIGNED bugs to FIXED, and place a comment stating in which build the bug was fixed.  
+
As of 2008-01-21, you can use [http://emft.eclipse.org/modeling/emft/compare/build/promo.php promo.php] or the <tt style="color: DarkGreen">-bugzonly</tt> flag to automatically move your RESOLVED:FIXED bugs to VERIFIED:FIXED, and place a comment stating in which build & branch the bug was fixed. See also {{bugstrike|206558}}, {{bugstrike|191571}}.
  
See also {{bug|191571}} and {{bug|206558}}.
+
If you prefer, you can enable this step to happen automatically with every promoted build using these parameters in your promoteToEclipse.*.properties file
 +
 
 +
{{codeblock|<nowiki># do search cvs / release notes database update - 1 = do, 0 = skip
 +
searchCVS=1
 +
 +
# how long to block before giving up (in seconds, eg., 1200 for 20 mins wait)
 +
searchCVSTimeout=1200
 +
 +
#automatically move bugzillas from RESOLVED:FIXED to VERIFIED:FIXED - 1 = do, 0 = skip
 +
bugz=1
 +
 +
# check for the existence of the new build in the release notes database
 +
# before trying to close bugs - 1 = do, 0 = skip
 +
checkReleaseExists=1</nowiki>}}
  
 
====Setup / Troubleshooting====
 
====Setup / Troubleshooting====
Line 179: Line 347:
 
The first time you use this option, or if your bugzilla cookie expires, you will receive an error like this:
 
The first time you use this option, or if your bugzilla cookie expires, you will receive an error like this:
  
[promote] ERROR: login cannot be read from ~/.ssh/UpdateBugStateTask.properties.
+
{{codeblock|<nowiki>[promote] ERROR: login cannot be read from ~/.ssh/UpdateBugStateTask.properties.</nowiki>}}
 +
 
 +
If your cookie expires (and one day, it will), you'll see this:
 +
 
 +
{{codeblock|<nowiki>[UpdateBugState] Could not use your login/logincookie credentials to log in.
 +
[UpdateBugState] Bugzilla replied: "I need a legitimate login and password to continue."</nowiki>}}
  
 
To regenerate credentials / cookie information, log in to the build server and run:
 
To regenerate credentials / cookie information, log in to the build server and run:
  
cd /home/www-data/build/modeling/scripts;  
+
{{codeblock|<nowiki>cd /home/www-data/build/modeling/scripts; ./bugzillaLogin.sh</nowiki>}}
./bugzillaLogin.sh 2> ~/.ssh/UpdateBugStateTask.properties
+
 
 +
If you pipe the command's syserr directly to your UpdateBugStateTask.properties file, you won't have to copy & paste (as per the directions in the script).
 +
 
 +
{{codeblock|<nowiki>cd /home/www-data/build/modeling/scripts;  
 +
./bugzillaLogin.sh 2> ~/.ssh/UpdateBugStateTask.properties</nowiki>}}
  
 
===Newsgroup Announcement Setup===
 
===Newsgroup Announcement Setup===
Line 210: Line 387:
 
# create the parent thread into which subsequent posts will go (just post a message to the newsgroup), then  
 
# create the parent thread into which subsequent posts will go (just post a message to the newsgroup), then  
 
# get the post's <tt style="color:DarkGreen">Message-ID</tt> (using Thunderbird, just turn on all headers to see this), and
 
# get the post's <tt style="color:DarkGreen">Message-ID</tt> (using Thunderbird, just turn on all headers to see this), and
# update the appropriate entry in your <tt style="color:DarkGreen">/home/www-data/build/emft/scripts/promoteToEclipse.*.properties</tt> file on your build server (emf.torolab or emft.eclipse).
+
# update the appropriate entry in your <tt style="color:DarkGreen">/home/www-data/build/modeling/scripts/promoteToEclipse.*.properties</tt> file on your build server (emf.torolab or emft.eclipse).
  
 
Looking at promoteToEclipse.jet.properties as an example:
 
Looking at promoteToEclipse.jet.properties as an example:
Line 275: Line 452:
  
 
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~ $</tt> <tt style="color: DarkGreen">cd ~/.ssh; </tt>
 
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~ $</tt> <tt style="color: DarkGreen">cd ~/.ssh; </tt>
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~/.ssh $</tt> <tt style="color: DarkGreen">touch ~/.ssh/authorized_keys; </tt>
+
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~/.ssh $</tt> <tt style="color: DarkGreen">touch ~/.ssh/authorized_keys ~/.ssh/known_hosts; </tt>
 
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~/.ssh $</tt> <tt style="color: DarkGreen">cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys_old; # backup</tt>
 
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~/.ssh $</tt> <tt style="color: DarkGreen">cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys_old; # backup</tt>
 
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~/.ssh $</tt> <tt style="color: DarkGreen">cat ~/.ssh/''eclipse_id''.temp ~/.ssh/authorized_keys_old > ~/.ssh/authorized_keys; # prepend new key into old file; save as new file</tt>
 
:: <tt style="color: #990000; background-color: #FFFFCC;">''eclipse_id''@dev.eclipse.org:~/.ssh $</tt> <tt style="color: DarkGreen">cat ~/.ssh/''eclipse_id''.temp ~/.ssh/authorized_keys_old > ~/.ssh/authorized_keys; # prepend new key into old file; save as new file</tt>
Line 303: Line 480:
 
  ssh jlescot@localhost
 
  ssh jlescot@localhost
 
  exit
 
  exit
  ssh jlescot@download1.eclipse.org
+
  ssh jlescot@download1.eclipse.org ''[publishing build folder, zips, and javadoc]''
 
  exit
 
  exit
  ssh jlescot@dev.eclipse.org
+
  ssh jlescot@dev.eclipse.org       ''[CVS access]''
 +
exit
 +
ssh jlescot@build.eclipse.org    ''[jar signing and update site generation]''
 
  exit
 
  exit
 
</td></tr></table>
 
</td></tr></table>
Line 313: Line 492:
 
====Connecting the web user to your account====
 
====Connecting the web user to your account====
  
In order to run builds from the web site on the build server, the web user must be able to ssh to your account on the build server. Put its public key in your authorized_keys file to make this work.
+
In order to run builds from the web site on the build server, the web user (apache or www-data) must be able to ssh to your account on the build server. Put its public key in your authorized_keys file to make this work.
  
 
====Additional====
 
====Additional====
Line 325: Line 504:
 
===RSS Feeds===
 
===RSS Feeds===
  
=====The log file for my promoted build shows a FileNotFoundException (Permission denied) or NullPointerException when publishing my RSS feed!=====
+
=====The log file for my promoted build shows a FileNotFoundException (Permission denied), NullPointerException, or SAXParseException when publishing my RSS feed!=====
  
 
Either the permissions on the file do not allow you to edit it...
 
Either the permissions on the file do not allow you to edit it...
Line 336: Line 515:
 
... or your file has been corrupted by a CVS merge/update, and contains "<<<<<<<" and "=======" which makes it an invalid XML file.
 
... or your file has been corrupted by a CVS merge/update, and contains "<<<<<<<" and "=======" which makes it an invalid XML file.
  
To fix this, delete the file on filesystem, refresh from cvs & redo the promote.  
+
addEntry:
 +
  [AddEntry] [Fatal Error] builds-gef.xml:7:2: The content of elements must consist of well-formed character data or markup.
 +
  [AddEntry] org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup.
 +
  [AddEntry]    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:264)
 +
 
 +
To fix this, delete the file on filesystem, refresh from cvs & redo the promote. See also {{bugstrike|254645}}
 
   
 
   
 
Here's how:  
 
Here's how:  

Latest revision as of 12:03, 17 November 2008

Release Process

Schedule

All components are requested to follow this release schedule:

  • Tuesday/Wednesday night:
    • Your code in CVS must be "compile error free". Feel free to run N builds that afternoon to verify this.
  • Wednesday/Thursday morning:
    • One Integration (I) build from your main branch (HEAD), and/or
    • One Maintenance (M) build from your maintenance branch (eg., R1_0_maintenance).

If your component is participating in a Coordinated release such as Ganymede (June 2008), please be aware of the release schedule for Milestones and Release Candidates, and provide Stable (S) builds in alignment with those dates.

Rampdown to Release

After the M7/RC0 builds are published, it's usually time to review the copyrights in your sources to ensure they're accurate. You may also be asked at some point to provide all of the about.* files to the Eclipse Management Organization (EMO) legal department so they can review them prior to the release.

You will also be required to provide a slide deck to your PMC, as he will be representing your component at your Release Review. For samples of past release review slides, see

Collecting Abouts

  • First, sources are collected using this script and the map file listed on your downloads page for your most recent milestone build. (If you don't have access to your own linux box, you can run this on emft.eclipse.org or build.eclipse.org.)
  • Next, check out Jeff McAffer's org.eclipse.orbit.releng.aboutextractor tool from CVS and use that to process your sources into a new directory containing only the Abouts.
  • Finally, that new directory can be zipped and sent to emo@eclipse.org and/or Janet.Campbell@eclipse.org, as required.

Maintaining Your IP Log

As part of being an Eclipse project, everyone is responsible for keeping the IP clean. This includes:

  • Contributors: by attaching patches to bugs
  • Committers: by committing patches and filing CQs for large contributions, and using the 'contributed' keyword on bugs with contributions which have been committed to your code base
  • Project/Component Leads: by enforcing the IP rules for your committers/contributors, updating IP Logs in the event they ever get out of date, and filing CQs on behalf of soon-to-be committers
  • PMCs: all of the above, plus liasing with the Foundation in terms of Release & Graduation Reviews -- collecting Abouts, Slide Decks, and IP Logs.


Currently, there are three ways to log your project or component's IP.

Method ProviderExampleHow to Update
DynamicEclipse Foundationip_log.php Contribute to this log by modifying bugs in Bugzilla and by opening CQs in IPZilla.
Half & HalfModeling Projectipquery.php Contribute to this log by modifying bugs in Bugzilla and by opening CQs in IPZilla; you can also add static information by editing your ipquery.php file.
Staticproject/component committerseclipse-project-ip-log.csv Contribute to this log by editing your eclipse-project-ip-log.csv file.

For more on how to keep your IP Log up to date, see Modeling Project Releng - Releasing - IP Log.

Release Reviews / x.y.0 Final Builds

  • DO NOT PUBLISH an R build until you've been through a release review.
  • DO NOT PUBLISH any build using a vanity name ending in ".0" without a release review.
  • DO PUBLISH as many I or S builds as you want -- ideally, you should be publishing one build a week, but only if there are changes.

You can have unlimited weekly Integration (I) builds, then about once every 6 weeks and at the end of your development cycle, you should publish a Stable (S) build. These are usually vanity-named with "0.8.0M2" or "0.8.0RC" to describe the Milestone or Release Candidate status and number.

At the end of your development cycle, you should do an RC build at least one week before your final planned release date. When you pass your Release Review, you can use the /home/www-data/build/modeling/scripts/renameBuild.sh script to rename your "0.8.0RC" Stable (S) build to a "0.8.0" Release (R) build, and repromote it under the new name. This will ensure the correct metadata is properly updated (RSS feed, release notes).

Once your x.y.0 build is reviewed, passed, and released, you can continue in HEAD working on x.y.1 (or branch immediately). Publish as many (M)aintenance builds as you want, then again, as you get close to the end, a Stable 0.8.1RC can be done. Maintenance branch releases need not be reviewed prior to release.

At some point you'll want to branch HEAD (eg., 0.9.0) and R0_8_maintenance (eg., 0.8.1 or 0.8.2). When you start doing 0.9.0 builds, they will be I builds, interspersed with S milestone builds and RC build(s) towards the end of the cycle. Finally, your last RC will be reviewed and you'll once again rename and repromote.

It is HIGHLY ADVISABLE to not branch midway through a release, in order to preserve accurate release notes.

Please also be aware that your build server is SHARED -- clean up your old builds (especially any -noclean builds) when they are obsoleted by newer ones in order to save space for others.

Renaming a Build / Releasing Your GM Build

  • When releasing your final build, you may decide to rename an RC build if nothing bad has happened since that build was released. For example, you might plan to release your final build on Sept. 28, so you'd do an RC1 build two weeks before. If any stop-ship bugs were found, you'd spin an RC2 build the week after, on the 21st. Then, if no more bugs were found, you'd be able to promote that build as your GM (Golden Master) on Sept. 28 - however, you'd have to rename the zips, md5 files, dependencies listed in build.cfg, etc.
  • The good news is that there's an easy way to do this - renameBuild.sh. This script contains detailed instructions and examples of usage; just run it from the commandline w/o options to see. Additionally, here's an example showing a rename of JET from 0.7.1RC2 to 0.7.1:
sudo -u www-data ./renameBuild.sh -proj jet -verbose -branch 0.7.1 \
-buildID S200609210820=R200609210820 -buildAlias 0.7.1RC2=0.7.1 \
eclipse-SDK-M20060919-1045-linux-gtk.tar.gz=eclipse-SDK-3.2.1-linux-gtk.tar.gz \
M20060919-1045=R-3.2.1-200609210945 \
S200609210005=R200609210005 \
2.2.1RC2=2.2.1 \
fullmoon.torolab.ibm.com=download.eclipse.org \
emf.torolab.ibm.com=download.eclipse.org
  • When you're done renaming, you can simply promote the new build as you did the previous week (-buildID S200609210820), using the new ID (-buildID R200609210820), and you'll have your GM build published. *HOWEVER**, if you are participating in a coordinated release, you must adhere to the process rules for that release. For example: Ganymede - Final Daze - Modeling Project, Ganymede - Final Daze.
  • After releasing the final build of a given version, you need to remember to upgrade the version of the plugins you change. In other words, if a plugin is not changed then its version remains the same. After upgrading the version of a plugin, you need to ensure that the version of the appropriate features and branding plugins are also upgraded, and that the feature is referring to the new versions of the plugins. Also, if you change a plugin with source code, you may need to upgrade the appropriate documentation plugin (and documentation feature and branding plugin) if the build is adding Javadoc to it.

Build Naming Conventions

Because Modeling consists of many projects with an ever-expanding set of components, it's important that we all adhere to consistent naming rules for each build type. It's also important to be consistent within your component or your release notes will be sloppy or incomplete. Build Aliases must conform to one of these patterns in order to be automatically added to the release notes database:

/[IM]\d{12}/

or

/\d+\.\d+\.\d+.*/

When building, use the Build Alias field to name your S and R type builds.

Uml2tools-build-page.png

The rules are very simple.

  • If it's a Milestone (build type S), call it "x.y.zMn", eg., 0.7.0M6
  • If it's an RC (build type S), call it "x.y.zRCn", eg., 0.8.0RC1 or 0.8.2RC
  • If it's a Release (build type R), call it "x.y.z", eg., 0.9.0 or 2.0.2
  • All other builds (I, M) use no alias and are named using the default pattern of "/[IM]\d{12}/", eg., I200802070245

Promoting A Build

The First Time

  • Next, ensure your promoteToEclipse.*.properties file is up to date. Once your file is correct in CVS, you can run this (sudo'd as the web user, www-data or apache) to check out the latest version of that file from CVS:
/home/www-data/build/modeling/scripts/getPromotePropertiesFromCVS.sh emf.ecoretools
  • add your component to relupdate.php,
  • add a "0.0.0" release to releases table for your component, and
  • check out your project and releng sources for use with searchcvs
  • Review the log file generated by the promote process. Log files are stored in /home/www-data/promo_logs/. If you encounter any errors in the log, review Modeling Project Releng or contact User:Nickb.
  • Should a step fail, you can re-run the whole promote (but you may get duplicate entries in your RSS feed and update site). To avoid this, you can remove the new content before re-running the build, or you can run only the relevant parts that failed via the commandline promote script interface. For example, to re-run the javadoc generation step, use:
cd /home/www-data/build/modeling/scripts/;./promoteToEclipse.sh -sub ecoretools -docsonly -Q -buildID I200712031234 ...
  • To see all the commandline options available to you, run ./promoteToEclipse.sh without options for the inline help.
  • PLEASE NOTE! Once your promote is done, you will not see it on eclipse.org for at least 20-30 minutes as the various eclipse.org cluster nodes replicate. Why?
  • download.eclipse.org is served via nfs on all of the nodes from the nfsslave. The master and slave replicate every 4-6 hours depending on system loads.
  • So, unfortunately, the only recourse is patience.
  • When your build does appear at eclipse.org, download the zip and test it out to ensure it works for you. Then try and install via the update site.
  • When you're satisfied, update the _projectCommon.php file for your project (EMFT, MDT, etc.) to remove your project from the $nodownloads array. Wait 30-60 seconds and then refresh your project's homepage, downloads page, release notes, etc. to ensure your component is listed correctly. See also Website Navigation.
  • If you plan to contribute to a coordinated release such as Ganymede, make sure you've added your component to the list, and that you have write access to /cvsroot/callisto. If you don't, open a bug assigned to webmaster@eclipse.org and cc:'d to your PMC User:Nickb. You should ask:
  • "Please add user yourusername to group callisto-dev for Ganymede .sc file contributions."

Troubleshooting

Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).
  • If you see Permission denied errors in the promote log, you may not have sufficient permission to create files on the download server.
mkdir: cannot create directory `/home/data/httpd/download.eclipse.org/modeling/m2t/mtl': Permission denied
  • Open a bug assigned to webmaster@eclipse.org and cc:'d to your PMC and User:Nickb, asking "Please add user yourusername to group groupname for publishing buildname builds."
  • Groupname will be in the form m2tadmin (old style) or modeling.emf.website (new style).

Scheduled Releases (Promoting Automatically)

See Scheduled Releases.

Update Manager Sites & Build Types

If you go to http://www.eclipse.org/modeling/emft/updates/, you'll see a note about two different UM sites. These two sites are there to split final, release builds from interim builds, so that the two types of users can benefit from the two types of releases.

A release build (R) is anything that's done as a final GM (aka Golden Master) release, eg., 0.7.0, 0.7.1, 1.0.0, 1.0.1. Everything else is considered "interim" - Nightly, Integration, Maintenance, Stable (N, I, M, S).

So, for example, when you publish an I build (eg., 0.7.5 I200610050000), it gets added to site-interim.xml. When you do your final build for that stream, (eg., 0.7.5 R200612020000), it will be added to site.xml.

Note also that the URL to give to Update Manager (or to put in your features) is NOT http://www.eclipse.org/modeling/emft/updates/, but download.eclipse.org. www.eclipse.org stores website content only, and provides a location for extracting, updating, and committing changes to the site.xml and site-interim.xml files, in order to provide CVS versioning & history. The actual site is http://download.eclipse.org/modeling/emft/updates/site*.xml, which is also benefitted by the fact that it's mirrored (unlike www.eclipse.org).

Generating Update Manager Site Jars

Contributing To Ganymede Update Site

  • The Ganymede/Build collects your *.sc files and turns them into features for the combined Ganymede update site.
  • To contribute to Ganymede, you must complete these steps:
  • Open a bug to get added to group callisto-dev, eg., bug 212325. Bug should be assigned to webmaster@eclipse.org and cc'd to bjorn.freeman-benson@eclipse.org for PMC-like approval.
  • Use your promo.php build page (snapshot at right) to ensure your contribution (/cvsroot/callisto/org.eclipse.ganymede.sitecontributions/*.sc) is generated and committed to CVS
  • promoteToEclipse.sh, called from the promote page, will generate the XML contents of your .sc file. That's it!
  • To control what features are contributed, tweak your promoteToEclipse.*.properties file. For example:

For EMF/SDO/XSD:

# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$" 
coordsiteFeaturePattern=".*eclipse/features/org.eclipse..*(emf|emf.sdk|xsd|xsd.sdk|emf.ecore.sdo|emf.ecore.sdo.sdk|emf.common|emf.common.ui|emf.ecore|emf.ecore.edit|emf.edit|emf.edit.ui|xsd.edit)_.*\/$"
# pattern to use when setting feature to include in coordsite's main feature ('Models and Model Development'); all other matching features will go in 'Enabling Features'
coordsiteMainFeaturePattern="org.eclipse.(emf|emf.sdk|xsd|xsd.sdk|emf.ecore.sdo|emf.ecore.sdo.sdk)$"

Thus, EMF contributes 6 "main" features:

  • emf, emf.sdk, xsd, xsd.sdk, emf.ecore.sdo, emf.ecore.sdo.sdk

And 7 "enabling" features:

  • emf.common, emf.common.ui, emf.ecore, emf.ecore.edit, emf.edit, emf.edit.ui, xsd.edit

For OCL:

# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$" 
coordsiteFeaturePattern=".*eclipse/features/org.eclipse.(emf.ocl|ocl.all|ocl.all.sdk|ocl|ocl.uml)_.*\/$"
coordsiteMainFeaturePattern="org.eclipse.ocl.(all|all.sdk)"

To create aggregate features, such as EMF Transaction does to bundle up EMF Query, EMF Validation, and EMF Transaction into a single "emfqtv.all.sdk" feature:

# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$" 
#omit SDK because this is only to create a temp file which will be combined w/ rest of QTV components
coordsiteFeaturePattern=".*eclipse/features/org.eclipse.emf.(emfqtv.all|emfqtv.all.sdk|query|query.ocl|validation|validation.ocl|transaction|workspace)_.*\/$"
coordsiteMainFeaturePattern="org.eclipse.emf.emfqtv.(all|all.sdk)"

Then, in the EMF Validation (and EMF Query) properties files:

# pattern to use when searching for features to include in coordsite; defaults to ".*eclipse/features/org.eclipse..*(${subprojectName}|${subprojectName}.sdk)_.*\/$" 
#omit SDK because this is only to create a temp file which will be combined w/ rest of QTV components
coordsiteFeaturePattern=".*eclipse/features/org.eclipse.emf.(validation|validation.ocl)_.*\/$"
coordsiteMainFeaturePattern="NONE"
Build Promotion Page
Troubleshooting

See Troubleshooting.

Generating Release Notes

Generating release notes is easy - promoteToEclipse.sh does it for you! However, there are a few simple steps to follow to ensure the generated content is accurate & non-duplicative. Here's the general process for closing a bug and using Bugzilla & CVS to provide the information required to generate release notes.

  • Working in Eclipse, fix your code. Test is as much as possible before committing it to CVS.
  • Commit your change(s) to CVS, using the bug's number in square brackets to begin your commit comment, eg:
[127509] javadocs now available
  • If you use Mylyn you can use this template, with a space at the end:
[{$task.id}] 
  • Note that you can include multiple bugs (each number on its own line) and comments are not required. The information that will appear in the release notes comes from the bug's Summary field, not your CVS comments.
  • Browse to the bug's webpage, and assign it the status Assigned. Your bugzilla comment could be something like:
Committed to CVS
  • Produce a build, and promote it. Unless you use the -norss flag:
    • Your RSS feed will be updated,
    • The Modeling database will see that new entry,
    • A corresponding entry will be added to the Releases table, and
    • A refresh of the release notes will now show that new entry.

Troubleshooting

My new build isn't listed on the website!

Mirror replication takes time. Please try again in 30 mins, or re-read your promote log to verify everything went OK.

My new build isn't listed in the Release Notes!

First, check your RSS feed file to ensure it's been updated. If it hasn't, check your promote log for failures in updating the feed. If it has, please be patient and wait for the next database update. Your RSS feed file controls the list of releases that are imported into the Releases table of the Search CVS database, and thus appear on the Release Notes page.

There are three rules to consider when building, naming, and publishing maintenance builds:

  1. If you've branched, you need to tell the build system (and thus the RSS feed generator). Maintenance builds from HEAD are ok, but may confuse the database if you try to branch during a release rather than immediately afterwards. Branching right after your 1.0 release is recommended.
  2. Maintenance builds on, say, 1.0.0 MUST BE VERSIONED to 1.0.1, 1.0.2: if 1.0.0 has already been released, everything since must be > 1.0.
  3. Maintenance builds should NOT use an alias of "1.0.0"; they should simply default to MyyyymmddHHMM. You can only have ONE unique "1.0.0" in the database.
There are errors in my release notes! How do I edit?

If you want to remove or edit values in your RSS feed and flush old/incorrect values from the database, please edit the file /cvsroot/org.eclipse/www/modeling/projectName/feeds/builds-componentName.xml (eg., projectName = emft; componentName = teneo). When done, send an email to User:nickb requesting a database flush for your component's releases.

Bugs are missing from the Release Notes!

Database updates can take a while. If it's been 24hrs and you *still* are missing data, contact User:nickb.

RSS Update Errors When Promoting
Problem Solution
addEntry:
 [AddEntry] [Fatal Error] builds-teneo.xml:13:2: 
   The content of elements must consist of 
   well-formed character data or markup.
 [AddEntry] org.xml.sax.SAXParseException: 
   The content of elements must consist of 
   well-formed character data or markup.
Check the file on the build server in /var/www/html/emft/feeds/. Is it corrupt? Is it full of invalid XML due to a CVS update/merge? If so, so the following:
cd /var/www/html/emft/feeds/; \
rm -f builds-projectName.xml; \
cvs up -Pd; \
chmod 664 builds-projectName.xml; \
chgrp www builds-projectName.xml;

Then rerun your promote using the -rssonly flag to redo that single step. After that, you can force an update of the database - including loading any new CVS commits - with the -searchcvsonly flag. If you don't need to refresh commits, just the Releases table, you can run this (on emft.eclipse.org only):

cd /shared/modeling/searchcvs; \
php relupdate.php -noheat
Permission denied
Check the file on the build server in /var/www/html/emft/feeds/. Is it writable by you or one of the groups of which you're a member? Try this:
cd /var/www/html/emft/feeds/; \
ls -la; groups; \
chmod 664 *; chgrp www *;

Then rerun your promote using the -rssonly flag to redo that single step. After that, you can force an update of the database - including loading any new CVS commits - with the -searchcvsonly flag. If you don't need to refresh commits, just the Releases table, you can run this (on emft.eclipse.org only):

cd /shared/modeling/searchcvs; \
php relupdate.php -noheat

Automated 'Fix available in ...' Bugzilla Comments

As of 2008-01-21, you can use promo.php or the -bugzonly flag to automatically move your RESOLVED:FIXED bugs to VERIFIED:FIXED, and place a comment stating in which build & branch the bug was fixed. See also bug 206558, bug 191571.

If you prefer, you can enable this step to happen automatically with every promoted build using these parameters in your promoteToEclipse.*.properties file

# do search cvs / release notes database update - 1 = do, 0 = skip
searchCVS=1
 
# how long to block before giving up (in seconds, eg., 1200 for 20 mins wait)
searchCVSTimeout=1200
 
#automatically move bugzillas from RESOLVED:FIXED to VERIFIED:FIXED - 1 = do, 0 = skip
bugz=1
 
# check for the existence of the new build in the release notes database 
# before trying to close bugs - 1 = do, 0 = skip
checkReleaseExists=1

Setup / Troubleshooting

The first time you use this option, or if your bugzilla cookie expires, you will receive an error like this:

[promote] ERROR: login cannot be read from ~/.ssh/UpdateBugStateTask.properties.

If your cookie expires (and one day, it will), you'll see this:

[UpdateBugState] Could not use your login/logincookie credentials to log in.
[UpdateBugState] Bugzilla replied: "I need a legitimate login and password to continue."

To regenerate credentials / cookie information, log in to the build server and run:

cd /home/www-data/build/modeling/scripts; ./bugzillaLogin.sh

If you pipe the command's syserr directly to your UpdateBugStateTask.properties file, you won't have to copy & paste (as per the directions in the script).

cd /home/www-data/build/modeling/scripts; 
./bugzillaLogin.sh 2> ~/.ssh/UpdateBugStateTask.properties

Newsgroup Announcement Setup

Setup for posting automatically to the newsgroup to announce new builds is easy.

1. Copy newsgroup-post.txt into your ~/.ssh folder.

build_id@build_server:~ $ mkdir -p ~/.ssh; cp /home/nickb/newsgroup-post/newsgroup-post.txt ~/.ssh

You are now set up to post automatically to the newsgroup while promoting a build. If you want to post as a specific person, rather than the default name & email, there are two additional steps.

2. Edit org.eclipse.emft/releng/[subproject]/promoteToEclipse.[subproject].properties <example>, placing the correct name and email address for the person whose name will appear on newsgroup posts. This step - and the next one - is ONLY required if the name & email are not the same as the default value recorded here.
newsgroupPublisherEmail="John Q. Committer <john_q_emft_committer_guy@eclipse.org>"
3. Copy org.eclipse.emft/releng/[subproject]/promoteToEclipse.[subproject].properties <example> into /home/www-data/build/emft/scripts.

To announce a build that's already been built, you can use the -announceonly flag to skip all other steps and ONLY post to the newsgroup. Otherwise, adding -announce when promoting will add the extra step of posting a quick note to the newsgroup.

Newsgroup Post Threading

Should you want to thread your announcements so that they can be hidden or categorized, here's how you can do so:

  1. create the parent thread into which subsequent posts will go (just post a message to the newsgroup), then
  2. get the post's Message-ID (using Thunderbird, just turn on all headers to see this), and
  3. update the appropriate entry in your /home/www-data/build/modeling/scripts/promoteToEclipse.*.properties file on your build server (emf.torolab or emft.eclipse).

Looking at promoteToEclipse.jet.properties as an example:

# publisher of newsgroup announcements
newsgroupPublisherEmail="John Q. Committer <john_q_emft_committer_guy@eclipse.org>"
newsgroup=eclipse.technology.emft
# to thread posts under an existing post (or posts) enter the slash-escaped Message-ID for which this is a reply,
# eg: <e0i2gc\$an5\$1@utils.eclipse.org>
newsgroupThreadReferences=""

If you put the Message-ID in the newsgroupThreadReferences property, announcements about builds will be nested under that existing message.

Note that to edit this file you must sudo to the web user:

  • On emf.toro: sudo -u www-data vi promoteToEclipse.jet.properties
  • On emft.eclipse: sudo -u apache vi promoteToEclipse.cdo.properties

If you'd like to store a copy of your changes in CVS, please contact your releng manager so that your changes can be committed into your releng module for safekeeping/versioning.

SSH Key Setup

Getting Access

You must have ssh access to your build server, download1.eclipse.org, and dev.eclipse.org. If you do not have access, please contact the administrators of these servers to gain access. You need a full login shell (eg., bash), not just a CVS login shell (cvssh). If you cannot ssh into dev.eclipse.org or download1.eclipse.org, please open a bug asking for a full login shell, eg., bug 206146.

To get access to your build server, you must forward your static IP address to the server's administrator. If you do not have a static IP, you can set up your router to broadcast your IP to a domain name registration service like dyndns.org and provide your domain name for access to the build server.

Once you have ssh access, you need to set up limited ssh key access between your build server and the eclipse.org download and cvs servers. This will allow you to run the promote script without being prompted for a password periodically.

Connecting in

To grant yourself ssh access from your build server to eclipse.org, ssh to the build server normally.

Under windows, you can use cygwin or PuTTY. Under Linux, open a shell and type `ssh`.

Generating private and public SSH keys

If you don't already have an ssh key pair created, you will need to generate one.

build_id@build_server:~ $ mkdir -p ~/.ssh; chmod 700 ~/.ssh; cd ~/.ssh; ssh-keygen -b 2048 -t rsa

When prompted for a passphrase, hit enter to skip. Having a passphrase, while more secure than having no passphrase, prevents automated login.

Save your public key in the default place: ~/.ssh/id_rsa.pub.

Then, you have to create the authorized_keys file and append to it your public key :

build_id@build_server:~/.ssh $ touch ~/.ssh/authorized_keys;
build_id@build_server:~/.ssh $ cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys;

Connecting out: authorizing remote hosts to let you in

To copy your local public key to the remote machine, you will need to scp (secure copy) the file. From the build server, copy the file to the remote server, then ssh to that server and combine the new key with the any existing keys in the authorized_keys file. In most cases, build_id == eclipse_id, eg., jlescot == jlescot:

Note that if you have not created yet the .ssh folder on the dev.eclipse.org server, you first need to achieve it using the following instructions :

build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
eclipse_id@dev.eclipse.org:~ $ mkdir -p ~/.ssh; chmod 700 ~/.ssh;
eclipse_id@dev.eclipse.org:~/.ssh $ exit

Then you can execute the following scp command :

build_id@build_server:~/.ssh $ scp id_rsa.pub eclipse_id@dev.eclipse.org:~/.ssh/eclipse_id.temp
build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
(you will still be prompted for a password when connecting)
eclipse_id@dev.eclipse.org:~ $ cd ~/.ssh;
eclipse_id@dev.eclipse.org:~/.ssh $ touch ~/.ssh/authorized_keys ~/.ssh/known_hosts;
eclipse_id@dev.eclipse.org:~/.ssh $ cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys_old; # backup
eclipse_id@dev.eclipse.org:~/.ssh $ cat ~/.ssh/eclipse_id.temp ~/.ssh/authorized_keys_old > ~/.ssh/authorized_keys; # prepend new key into old file; save as new file
eclipse_id@dev.eclipse.org:~/.ssh $ rm -fr ~/.ssh/eclipse_id.temp ~/.ssh/authorized_keys_old;
eclipse_id@dev.eclipse.org:~/.ssh $ chmod 644 authorized_keys id_rsa.pub known_hosts; chmod 600 id_rsa; # fix perms
eclipse_id@dev.eclipse.org:~/.ssh $ exit
build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
(this time you should get in using your key instead of needing a password)

Once you have ~/.ssh/authorized_keys and ~/.ssh/known_hosts updated with your public key and server fingerprint(s) (respectively), you should be able to connect to these systems without being prompted, and the promote script should be good to go unassisted.

Connecting out: creating host keys

SSH to your build server (if not already there). Then, one by one, ssh from there to these target systems, accepting whatever yes/no questions are posed about host keys or IP addresses. Once in, exit back to your build server. In most cases, build_id == eclipse_id, eg., jlescot == jlescot:

  • build_id@FQDN, build_id@localhost,
  • eclipse_id@download1.eclipse.org, eclipse_id@dev.eclipse.org

For example:

ssh jlescot@emft.eclipse.org
exit
ssh jlescot@localhost
exit
ssh jlescot@download1.eclipse.org [publishing build folder, zips, and javadoc]
exit
ssh jlescot@dev.eclipse.org       [CVS access]
exit
ssh jlescot@build.eclipse.org     [jar signing and update site generation]
exit

All of those should work without being prompted for a password. If you are prompted, you need to set an ssh key in your ~/.ssh/authorized_keys file for each user & hostname pair to which you want to connect. The first time you try each of these connections, you should be be prompted to add the host's fingerprint to your ~/.ssh/known_hosts file. This is also required.

Connecting the web user to your account

In order to run builds from the web site on the build server, the web user (apache or www-data) must be able to ssh to your account on the build server. Put its public key in your authorized_keys file to make this work.

Additional

If you require further assistance setting up key access, try here: http://cfm.gs.washington.edu/security/ssh/client-pkauth/.

Promotion FAQ

Many things can go wrong when promoting a build (CVS or SCP errors, artifacts not created, java failures). Most are a result of network or permission problems, and can be fixed by trying again or correcting the permission problem.

RSS Feeds

The log file for my promoted build shows a FileNotFoundException (Permission denied), NullPointerException, or SAXParseException when publishing my RSS feed!

Either the permissions on the file do not allow you to edit it...

Buildfile: feedPublish.xml
publishFeed:
addEntry:
  [AddEntry] java.io.FileNotFoundException: /var/www/html/modeling/emft/feeds/builds-net4j.xml (Permission denied)

... or your file has been corrupted by a CVS merge/update, and contains "<<<<<<<" and "=======" which makes it an invalid XML file.

addEntry:
  [AddEntry] [Fatal Error] builds-gef.xml:7:2: The content of elements must consist of well-formed character data or markup.
  [AddEntry] org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup.
  [AddEntry]     at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:264)

To fix this, delete the file on filesystem, refresh from cvs & redo the promote. See also bug 254645

Here's how:

$ cd /var/www/html/modeling/mdt/feeds/ 
$ less builds-uml2.xml  
  (note the <<<<<<<<<<<<< and =============== markers that the cvs server added) 
$ rm -fr builds-uml2.xml 
$ cvs up builds-uml2.xml 
  cvs update: warning: `builds-uml2.xml' was lost 
  U builds-uml2.xml 
$ chmod g+w builds-uml2.xml
$ chgrp www builds-uml2.xml

Then rerun the promote to fix the missing feed entry (required for release notes)...

./promoteToEclipse.sh -sub uml2 -Q -buildID somebuildID -rssonly -cvsbranch somebranchvalue ...  

If you decide to repromote completely using the web UI, make sure you've removed this build's site*.xml contribution because a full promote will add it again, slowing down the operation of the update site and presenting users with confusing duplicate entries.

Finally, since the RSS is now updated, you can force the database to be refreshed and have the new release appear in your Release Notes.

Update Site

Please check your promote logs for errors or transient glitches, like CVS disconnects or SCP problems. Sometimes one problem can manifest as several problems, as in the case of this EMFT Validation build's UM jar creation, which threw four warnings that something had gone wrong, all starting from a transient CVS checkout problem (bandwidth overload or connection reset).

[promote] [9] [13:53:18] Create & promote Update Manager jars:
[promote] Running buildUpdate.sh:
[umj-co] [1] Checking out org.eclipse.releng.basebuilder from dev 
  using HEAD
cvs checkout: failed to open /home/vramaswamy/.cvspass for reading: 
  No such file or directory
cvs [checkout aborted]: recv() from server dev.eclipse.org: 
  Connection reset by peer
The java class is not found: org/eclipse/core/launcher/Main
features/*.jar: No such file or directory
plugins/*.jar: No such file or directory
[umj] [7d] [13:54:47] Comparing local and remote folders MD5 sums 
  to ensure SCP completeness...
[compare] Compare md5sums...
[compare] MD5s do not match! Compare failed.
[umj] [13:54:52] ERROR! Script exiting from compareFolders.sh
[promote] [9] [13:54:52] Create & promote Update Manager jars done.

In this case, the problems were:

  • CVS disconnected while checking out required builder code
  • Build failed looking for Eclipse Main, so no build occurred
  • Copying files from temp to /var/www/technology/emft/updates/ failed as they were not created
  • MD5 check failed as nothing was copied to download1.eclipse.org

In this case, the solution was to:

  1. Verify that nothing had been produced on either the build server or download1.eclipse.org.
  2. Confirm that site*.xml had not been changed in CVS, and rollback any changes for this build's ID (if any).
  3. Re-run the promote using the -jarsonly flag in order to bypass all the steps that completed successfully and ONLY do the UM jar creation step.