Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "EDT:Release Engineering"

(Maintaining Project Metadata)
 
(28 intermediate revisions by the same user not shown)
Line 4: Line 4:
  
 
= Background Information =
 
= Background Information =
 +
 
== General Information ==
 
== General Information ==
The [[Main Page|Eclipse Wiki]] is sometimes useful, and sometimes outdated or missing info you think it should have...it may drive you insane.
 
  
The Eclipse sysadmins can be reached at webmaster@eclipse.org.  They're very helpful.  Before writing to them, check their FAQ at [[Webmaster_FAQ]] and the other resources on this page.
+
The [[Main_Page|Eclipse Wiki]] is sometimes useful, and sometimes outdated or missing info you think it should have...it may drive you insane.  
  
You can learn a lot by reading [[Development_Resources]] and [[IT_Infrastructure_Doc]].
+
The Eclipse sysadmins can be reached at webmaster@eclipse.org. They're very helpful. Before writing to them, check their FAQ at [[Webmaster FAQ]] and the other resources on this page.
 +
 
 +
You can learn a lot by reading [[Development Resources]] and [[IT Infrastructure Doc]].  
  
 
== Developing EDT ==
 
== Developing EDT ==
To be the release engineer, you have to know the EDT development process in addition to the build process.
+
 
* [[EDT:Developer's Guide to Getting Started on EDT]]
+
To be the release engineer, you have to know the EDT development process in addition to the build process.  
* [[EDT:Downloading and installing EDT builds]]
+
 
* [[EDT:How to commit code to EDT]]
+
*[[EDT:Developer's Guide to Getting Started on EDT]]  
* [[EDT:How to add new EDT plugins]]
+
*[[EDT:Downloading and installing EDT builds]]  
 +
*[[EDT:How to commit code to EDT]]  
 +
*[[EDT:How to add new EDT plugins]]
  
 
= Releng Resources and Information =
 
= Releng Resources and Information =
 +
 
== Bugs ==
 
== Bugs ==
The Bugzilla component for release engineering is EDTBuilds, though some people might report problems on the Website component. See [[EDT:How to be notified of EDT bugs]].
+
 
 +
The Bugzilla component for release engineering is EDTBuilds, though some people might report problems on the Website component. See [[EDT:How to be notified of EDT bugs]].  
  
 
== Hudson ==
 
== Hudson ==
Our builds run in Eclipse's Hudson system.  The build job is at [https://hudson.eclipse.org/hudson/job/cbi-edt-nightly/ https://hudson.eclipse.org/hudson/job/cbi-edt-nightly/].
 
  
Once you log in, you can disable/enable nightly builds with the Disable/Enable Project button. Click the Configure link to edit the build schedule and many other parameters. One thing you '''must''' do is enter your email address in the Email Notification box at the bottom of the Config page. That way you'll know when builds fail.
+
Our builds run in Eclipse's Hudson system. The build job is at [https://hudson.eclipse.org/hudson/job/cbi-edt-nightly/ https://hudson.eclipse.org/hudson/job/cbi-edt-nightly/]. Despite having "nightly" in its name, we use the job for nightly, release, and milestone builds.
 +
 
 +
Once you log in, you can disable/enable builds with the Disable/Enable Project button. Click the Configure link to edit the build schedule and many other parameters. One thing you '''must''' do is enter your email address in the Email Notification box at the bottom of the Config page. That way you'll know when builds fail.
  
 
== org.eclipse.edt.releng ==
 
== org.eclipse.edt.releng ==
Some aspects of our builds are configured in Hudson, and the rest are configured by files in the project named org.eclipse.edt.releng. It's a regular Eclipse project, not a plugin. It contains the map file, and the main build.properties and build.xml.
+
 
 +
Some aspects of our builds are configured in Hudson, and the rest are configured by files in the project named org.eclipse.edt.releng. It's a regular Eclipse project, not a plugin. It contains the map file, and the main build.properties and build.xml.  
  
 
== build.eclipse.org ==
 
== build.eclipse.org ==
Eclipse provides a machine called build.eclipse.org where you can do various build tasks. Any committer can log in, but the default shell only lets you run one command and then logs you out (to limit the number of people who can screw up the system). The EDT release engineer needs a real shell. Send a request to webmaster@eclipse.org .
+
 
 +
Eclipse provides a machine called build.eclipse.org where you can do various build tasks. Any committer can log in, but the default shell only lets you run one command and then logs you out (to limit the number of people who can screw up the system). The EDT release engineer needs a real shell. Send a request to webmaster@eclipse.org .  
  
 
== Downloads and the Archive ==
 
== Downloads and the Archive ==
The last step in our build process makes the new build available for download from download.eclipse.org/edt/.  You can access and modify the downloadable files when you're logged in to build.eclipse.org, by going to /home/data/httpd/download.eclipse.org/edt/.
 
  
People are encouraged to use a mirror when downloading files.  So rather than giving people a direct link like http://download.eclipse.org/edt/A/B/C, we should tell them to use http://www.eclipse.org/downloads/download.php?file=/edt/A/B/C.  The download.php script lets people choose a mirror.  (But do NOT use this kind of URL for a P2 update site.)
+
The last step in our build process makes the new build available for download from download.eclipse.org/edt/. You can access and modify the downloadable files when you're logged in to build.eclipse.org, by going to /home/data/httpd/download.eclipse.org/edt/.  
  
Space on the Eclipse download server is limited. It has the latest release, the latest milestone of the next release, and the latest nightly builds.  Older nightly builds are deleted, but older milestone and release builds are moved to the archive instead. Its URL is [http://archive.eclipse.org/edt/ http://archive.eclipse.org/edt/]. When you're logged in to build.eclipse.org, our archive can be found at /home/data/httpd/archive.eclipse.org/edt/.
+
People are encouraged to use a mirror when downloading files. So rather than giving people a direct link like http://download.eclipse.org/edt/A/B/C, we should tell them to use http://www.eclipse.org/downloads/download.php?file=/edt/A/B/C. The download.php script lets people choose a mirror. (But do NOT use this kind of URL for a P2 update site.)
  
The download.php script will look in the archive if it can't find the file you want on the download server. This means it's important for the archive to have the same directory structure as the main downloads area, so a URL like http://www.eclipse.org/downloads/download.php?file=/edt/A/B/C works even after the file is archived.
+
Space on the Eclipse download server is limited. It has the latest release, the latest milestone of the next release, and the latest nightly builds. Older nightly builds are deleted, but older milestone and release builds are moved to the archive instead. Its URL is [http://archive.eclipse.org/edt/ http://archive.eclipse.org/edt/]. When you're logged in to build.eclipse.org, our archive can be found at /home/data/httpd/archive.eclipse.org/edt/.  
  
== The MyFoundation Portal ==
+
The download.php script will look in the archive if it can't find the file you want on the download server. This means it's important for the archive to have the same directory structure as the main downloads area, so a URL like http://www.eclipse.org/downloads/download.php?file=/edt/A/B/C works even after the file is archived.  
The MyFoundation portal at https://dev.eclipse.org/portal/myfoundation/portal/portal.php lets you view and modify various kinds of meta-data about the EDT project.
+
  
In the Eclipse Projects box, choose 'view' next to tools.edt, and click '[maintain]' to edit the project meta-data.  Fields that may need to be updated from time to time are Bugzilla, Source Repository, and Release.
+
== Maintaining Project Metadata ==
  
If you choose '[tools]' next to tools.edt in the Eclipse Projects box, there are links for 'download stats', or to manage Bugzilla stuff use 'Bugzilla components, targets, versions'You can add milestones, releases, components, etc. Be very careful when working with Bugzilla because many changes are permanent.
+
Two sites let you view and modify various kinds of meta-data about the EDT project.  One is EDT's PMI page at http://projects.eclipse.org/projects/tools.edt (PMI stands for Project Management Infrastructure).  The other is the MyFoundation portal at https://dev.eclipse.org/portal/myfoundation/portal/portal.php .  You'll need to use these sites to update things such as the current releaseN.B. As of February 2013, functionality is being migrated to PMI and out of the older MyFoundation portal.
  
 
== cross-project-issues-dev ==
 
== cross-project-issues-dev ==
Eclipse's cross-project-issues-dev mailing list is for announcements and problems that affect many projects. You should subscribe to it by going to https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
+
 
 +
Eclipse's cross-project-issues-dev mailing list is for announcements and problems that affect many projects. You should subscribe to it by going to https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev  
  
 
== Our website ==
 
== Our website ==
The files of www.eclipse.org/edt are stored in CVS. To view them, make a repository connection like :extssh:<USER>@dev.eclipse.org:/cvsroot/org.eclipse, and go to the folder www/edt/.
+
 
 +
The files of www.eclipse.org/edt are stored in Git. To work with them, make a repository connection like&nbsp;ssh://committer_id@git.eclipse.org/gitroot/www.eclipse.org/edt.git, and import the project named edt.
 +
 
 +
== Download Statistics ==
 +
 
 +
You can find out how often EDT builds have been downloaded at https://dev.eclipse.org/committers/committertools/stats.php .  Enter a path in the Partial File Name field, and optionally select a start and/or end date.  All of our builds have a path that begins with "/edt/".  To view download stats for release builds, use "/edt/releases/".
  
 
= Starting a Build =
 
= Starting a Build =
Log in to Hudson and click Build Now. The single build parameter identifies the kind of build: use N for nightly, M for milestone, and R for release. Nothing special needs to be done for nightly builds, but see below for special instructions for milestone and release builds.
+
 
 +
Log in to Hudson and click Build Now. The single build parameter identifies the kind of build: use N for nightly, M for milestone, and R for release. Nothing special needs to be done for nightly builds, but see below for special instructions for milestone and release builds.  
  
 
= Outputs of a Build =
 
= Outputs of a Build =
Builds use a workspace directory on one of the Hudson slave machines.  The workspace can be viewed through Hudson, but it can't be accessed from build.eclipse.org.
 
  
After a successful build, the new update sites are accessible from build.eclipse.org. The zipped update site (and JUnit results when we get those going) is in ''downloads''/edt/downloads/drops/''BuildTypeName''/''Version''/''BuildType''_ThisBuild, where
+
Builds use a workspace directory on one of the Hudson slave machines. The workspace can be viewed through Hudson, but it can't be accessed from build.eclipse.org.  
  
* ''downloads'' is /home/data/httpd/download.eclipse.org/
+
After a successful build, the new update sites are accessible from build.eclipse.org. The zipped update site (and JUnit results when we get those going) is in ''Downloads''/edt/downloads/drops/''BuildTypeName''/''Version''/''BuildType''_ThisBuild, where
* ''BuildTypeName'' is nightly, milestones, or release
+
* ''Version'' is a three-part version number
+
* ''BuildType'' is a letter: N for nightly, M for milestone, R for release
+
  
Examples: <BR>
+
*''Downloads'' is /home/data/httpd/download.eclipse.org/  
''downloads''/edt/downloads/drops/nightly/0.8.0/N_ThisBuild <BR>
+
*''BuildTypeName'' is nightly, milestones, or release
''downloads''/edt/downloads/drops/milestones/0.8.0/M_ThisBuild
+
*''Version'' is a three-part version number
 +
*''BuildType'' is a letter: N for nightly, M for milestone, R for release
  
The build process also unzips the update site.  The update site for a nightly build is unzipped into ''downloads''/edt/updates/nightly/.  A milestone build is unzipped into ''downloads''/edt/milestones/1.0/, and a release build is unzipped into ''downloads''/edt/updates/1.0/.  See [[EDT:Update_Site_Guidelines]] for more info about these update sites.
+
Examples: <br> ''Downloads''/edt/downloads/drops/nightly/0.8.0/N_ThisBuild <br> ''Downloads''/edt/downloads/drops/milestones/0.8.0/M_ThisBuild
  
We give Hudson permission to write into our downloads area using ACLs. The Hudson user is hudsonBuild. If you change the existing structure or make a new directory, but sure to modify its ACL.
+
The "_ThisBuild" directory is created under the workspace/build directory on the slave machine, and it's copied to the download location at the end of the build. The previous build (if there is one) is preserved in the workspace. It has the same path as the new build, except it ends with _OlderBuild not _ThisBuild.  
  
= How To Deal With a Failed Build =
+
The build process also unzips the update site. The update site for a nightly build is unzipped into ''Downloads''/edt/updates/nightly/. Milestone and release builds are unzipped into ''Downloads''/edt/private-test/. We use private-test for milestone and release builds so we can do installation testing on a milestone or release candidate build without modifying the update sites. This prevents users from installing a candidate build before we finish testing it.
When a build fails, Hudson will send and email to you and everyone who committed a change in that build.  The message lists the changes and shows the tail of the build log.
+
  
Errors and warnings from javac are shown as numbered lists in the build log.  To find a compiler error, search for ".&nbsp;ERROR".  The word "error" will show up in several other places, but putting the period and space in front will take you to the compiler's output.
+
We give Hudson permission to write into our downloads area using ACLs. The Hudson user is hudsonBuild. If you change the existing structure or make a new directory, but sure to modify its ACL.  
  
If the build failed because something didn't compile, you probably don't need to remind the developer to fix the problem or help them, unless the failure was a missing plugin. Missing plugins are discussed at [[EDT:How_to_add_new_EDT_plugins]].
+
= How To Deal With a Failed Build =
  
If the build failed because of a Hudson problem, check Bugzilla. Hudson bugs have Classification=Eclipse Foundation, Product=Community, and Component=Hudson. If there's no bug for the problem, open one (you can use this link [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=Hudson https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=Hudson]).
+
When a build fails, Hudson will send an email to you, and everyone who committed a change in that build. The message lists the changes and shows the tail of the build log.
 +
 
 +
Errors and warnings from javac are shown as numbered lists in the build log. To find a compiler error, search for ".&nbsp;ERROR". The word "error" will show up in several other places, but putting the period and space in front will take you to the compiler's output.
 +
 
 +
If the build failed because something didn't compile, you probably don't need to remind the developer to fix the problem or help them, unless the failure was a missing plugin. Missing plugins are discussed at [[EDT:How to add new EDT plugins]].
 +
 
 +
If the build failed because of a Hudson problem, check Bugzilla. Hudson bugs have Classification=Eclipse Foundation, Product=Community, and Component=Hudson. If there's no bug for the problem, open one (you can use this link [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=Hudson https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&amp;component=Hudson]).  
  
 
= Starting a New Release =
 
= Starting a New Release =
Only one change needs to be made: update the value of the '''version''' property in org.eclipse.edt.releng\build.properties.
+
 
 +
Only a few changes need to be made. Update the value of the '''version''' property in org.eclipse.edt.releng\build.properties, and update the version numbers in our features. Developers are responsible for updating the version numbers of their plugins as they make changes.
 +
 
 +
We may also need a new Target Milestone in Bugzilla.
  
 
= Procedure for Milestone Builds =
 
= Procedure for Milestone Builds =
Prior to running a milestone build, we'll probably disable nightly builds and build on-demand.  We'll probably be in some kind of lockdown, so you might need to check that nobody snuck anything into CVS that shouldn't be in the milestone build.
 
  
There are two special tasks to perform if this is the first build of this milestone.
+
Prior to running a milestone build, we'll probably disable nightly builds and build on-demand. We'll probably be in some kind of lockdown, so you might need to check that nobody snuck anything into Git that shouldn't be in the milestone build.  
# Move the previous milestone's zipped update site from downloads/edt/milestones/1.0 to the archive.  Be sure to put a link to the newly-archived build in the archive's index.html and describe it in the index.html for the newly-archived build's directory. Delete the other contents of the milestone build directory.
+
# Run the copyright tool.  We have our own specialized version of Eclipse's tool.  It has some bug fixes, knows how to recognize RBD-style copyright comments, and operates on more kinds of files.  It's in the IBM CVS repository, project org.eclipse.wst.common.utils under the CopyrightTools folder, in the EDT branch.  Check it out, then export it as a deployable jar, and put it in your dropins folder.  Restart EDT.  You can delete the project.  To use the copyright tool, select a file/folder/project and choose Update EDT/RBD Copyrights.  Verify that nothing is screwed up before you commit the changes.  If the tool detects problems, it will write into a file called copyrightLog.txt in the root of your workspace.
+
  
Start a build with type=M. It will take longer than a nightly build because the plugins will be signed.
+
If this is the first build of this milestone, run the copyright tool. We have our own specialized version of Eclipse's tool. It has some bug fixes, knows how to recognize RBD-style copyright comments, and operates on more kinds of files. It's in the IBM CVS repository, project org.eclipse.wst.common.utils under the CopyrightTools folder, in the EDT branch. Check it out, then export it as a deployable jar, and put it in your dropins folder. Restart EDT. You can delete the project. To use the copyright tool, select a file/folder/project and choose Update EDT/RBD Copyrights. '''Important!''' After you run the tool, inspect the changes to verify that nothing is screwed up before you push them upstream. If the tool detects problems, it will write into a file called copyrightLog.txt in the root of your workspace. The tool isn't able to detect every kind of problem, and it tends to make mistakes on files that contain non-IBM copyrights.
  
Move the milestone zip to downloads/edt/milestones/1.0/. Rename the zip and its md5 file to something like edt-Update-R080M2.zip and edt-Update-R080M2.zip.md5 (replace "080M2" with the proper version and milestone).  Edit the md5 file.  Put in the new name for the zip.
+
Update the associate.sites setting in org.eclipse.edt.releng/build.properties according to the comments in the file.
  
Now unzip artifacts.jar.  It contains only one file: artifacts.xml.  Edit artifacts.xml.  First, turn on download statistics tracking.  At the top of the file is a repository tag, which contains a properties tag, which contains several property tags.  Increment the value of the properties tag's size attribute and add another property tag: <property name='p2.statsURI' value='http://download.eclipse.org/stats'/>  Now find the artifact tag for org.eclipse.edt.feature.  Increment its size attribute and add this property tag below it: <property name='download.stats' value='/edt/milestones/1.0'/>  (the value should match the directory).  More info about download tracking is at [[Equinox_p2_download_stats]] and [[Project Download Stats]].
+
Ensure that downloads/edt/private-test/ is empty and start a build with type=M. It will take longer than a nightly build because the plugins will be signed.  
  
Consider adding another property to artifacts.xml, so people can install from a mirror of their choice. We haven't done this yet. Info is at [[Equinox/p2/p2.mirrorsURL]].
+
Move the milestone zip to downloads/edt/private-test/. Rename the zip and its md5 file to something like edt-Update-R080M2.zip and edt-Update-R080M2.zip.md5 (replace "080M2" with the proper version and milestone). Edit the md5 file. Put in the new name for the zip.
  
Put the new artifacts.xml into artifacts.jar with ''zip -f artifacts.jar'', then delete artifacts.xml.
+
Notify the people doing installation testing that the build at downloads/edt/private-test/ is ready.
  
Finally, send an email to edt-dev so everyone knows the M build is ready for final testing.
+
When installation testing is complete, move the previous milestone's zipped update site from downloads/edt/milestones/1.0 to the archive. Be sure to put a link to the newly-archived build in the archive's index.html and describe it in the index.html for the newly-archived build's directory. Delete the other contents of the milestone build directory and move the contents of downloads/edt/private-test/ to downloads/edt/milestones/1.0/.
  
When testing is complete, tag the files in CVS. Use a tag name like "EDT080M2". Tag the plugins and features listed in the map file, plus the following four projects. They're used by customBuildCallbacks.xml in org.eclipse.edt.ide.ui.resources to create our RUI widget proejcts during the build.
+
Now unzip artifacts.jar. It contains only one file: artifacts.xml. Edit artifacts.xml. First, turn on download statistics tracking. At the top of the file is a repository tag, which contains a properties tag, which contains several property tags. Increment the value of the properties tag's size attribute and add another property tag: &lt;property name='p2.statsURI' value='http://download.eclipse.org/stats'/&gt; Now find the artifact tag for org.eclipse.edt.feature. Increment its size attribute and add this property tag below it: &lt;property name='download.stats' value='/edt/milestones/1.0'/&gt; (the value should match the directory). More info about download tracking is at [[Equinox p2 download stats]] and [[Project Download Stats]].
<PRE>widgets/org.eclipse.edt.rui.dojo.runtime.local  
+
 
 +
Consider adding another property to artifacts.xml, so people can install from a mirror of their choice. We haven't done this yet. Info is at [[Equinox/p2/p2.mirrorsURL]].
 +
 
 +
Put the new artifacts.xml into artifacts.jar with ''zip -f artifacts.jar'', then delete artifacts.xml.
 +
 
 +
We should do another quick test to verify that the build can be installed from its new location. Once we're satisfied, tag the files in Git. Use a tag name like "EDT080M2". Tag the plugins and features listed in the map file, plus org.eclipse.edt.releng and the following four projects. They're used by customBuildCallbacks.xml in org.eclipse.edt.ide.ui.resources to create our RUI widget projects during the build.  
 +
<pre>widgets/org.eclipse.edt.rui.dojo.runtime.local  
 
widgets/org.eclipse.edt.rui.dojo.runtime.remote  
 
widgets/org.eclipse.edt.rui.dojo.runtime.remote  
 
widgets/org.eclipse.edt.rui.dojo.widgets  
 
widgets/org.eclipse.edt.rui.dojo.widgets  
widgets/org.eclipse.edt.rui.widgets</PRE>
+
widgets/org.eclipse.edt.rui.widgets</pre>  
 +
 
 +
Use the same tag on important test projects, such as the official EUnits, so we can easily track which version of a test was run with each version of EDT.
 +
 
 +
If necessary, create a Target Milestone in Bugzilla for the next milestone.
 +
 
 +
Finally, send an email to edt-dev so everyone knows the build is done.
 +
 
 +
Change the associate.sites setting in org.eclipse.edt.releng/build.properties back to its previous value, and turn the nightly builds back on.
  
 
= Procedure for Release Builds =
 
= Procedure for Release Builds =
Building a release is very similar to building a milestone, so I'll only note the differences here.  Obviously some of the paths are different, and you use R for the buildtype when starting the build.
 
  
Before the first R build, archive the last milestone build according to the instructions in the previous section. Then archive the previous release build. In addition to the release's zipped update site in downloads/edt/updates/1.0/, also archive its (non-zipped) update site and all-in-one zips from downloads/edt/releases/1.0/.
+
Building a release is very similar to building a milestone, so I'll only note the differences here. Obviously some of the paths are different, and you use R for the buildtype when starting the build.  Also, when the build is done, create new all-in-one zips according to the instructions in org.eclipse.epp.package.edt.feature/README.txt .
 +
 
 +
When installation testing is complete, and we have permission from Eclipse to make the release public, archive the last milestone build according to the instructions in the previous section. Then archive the previous release build. In addition to the release's zipped update site in downloads/edt/updates/1.0/, also archive its (non-zipped) update site and all-in-one zips from downloads/edt/releases/1.0/. Make the links on the EDT Downloads page point to the archived files so people can still find them while we're finishing the release.
 +
 
 +
If any sample projects had to change for the new release, move their old versions to the archive too.
 +
 
 +
Move the zipped update site to downloads/edt/releases/1.0/, and move the other contents of downloads/edt/private-test/ to downloads/edt/updates/1.0/.  Put links to the files in download.eclipse.org/edt/milestones/1.0, so that people can update from a milestone build to the release.
  
When testing is complete, tag the files in CVS with a name like "EDT080GM".  Also, create new all-in-one zips according to the instructions in org.eclipse.epp.package.edt.feature/README.txt
+
When tagging files in Git, use a tag like "EDT080GM".  Make the links on the EDT Downloads page point to the new files.  
  
 
[[Category:EDT]]
 
[[Category:EDT]]

Latest revision as of 08:52, 20 February 2013

Release Engineering for EDT

This page contains hints, tips, links, and instructions for the EDT project's release engineer.

Background Information

General Information

The Eclipse Wiki is sometimes useful, and sometimes outdated or missing info you think it should have...it may drive you insane.

The Eclipse sysadmins can be reached at webmaster@eclipse.org. They're very helpful. Before writing to them, check their FAQ at Webmaster FAQ and the other resources on this page.

You can learn a lot by reading Development Resources and IT Infrastructure Doc.

Developing EDT

To be the release engineer, you have to know the EDT development process in addition to the build process.

Releng Resources and Information

Bugs

The Bugzilla component for release engineering is EDTBuilds, though some people might report problems on the Website component. See EDT:How to be notified of EDT bugs.

Hudson

Our builds run in Eclipse's Hudson system. The build job is at https://hudson.eclipse.org/hudson/job/cbi-edt-nightly/. Despite having "nightly" in its name, we use the job for nightly, release, and milestone builds.

Once you log in, you can disable/enable builds with the Disable/Enable Project button. Click the Configure link to edit the build schedule and many other parameters. One thing you must do is enter your email address in the Email Notification box at the bottom of the Config page. That way you'll know when builds fail.

org.eclipse.edt.releng

Some aspects of our builds are configured in Hudson, and the rest are configured by files in the project named org.eclipse.edt.releng. It's a regular Eclipse project, not a plugin. It contains the map file, and the main build.properties and build.xml.

build.eclipse.org

Eclipse provides a machine called build.eclipse.org where you can do various build tasks. Any committer can log in, but the default shell only lets you run one command and then logs you out (to limit the number of people who can screw up the system). The EDT release engineer needs a real shell. Send a request to webmaster@eclipse.org .

Downloads and the Archive

The last step in our build process makes the new build available for download from download.eclipse.org/edt/. You can access and modify the downloadable files when you're logged in to build.eclipse.org, by going to /home/data/httpd/download.eclipse.org/edt/.

People are encouraged to use a mirror when downloading files. So rather than giving people a direct link like http://download.eclipse.org/edt/A/B/C, we should tell them to use http://www.eclipse.org/downloads/download.php?file=/edt/A/B/C. The download.php script lets people choose a mirror. (But do NOT use this kind of URL for a P2 update site.)

Space on the Eclipse download server is limited. It has the latest release, the latest milestone of the next release, and the latest nightly builds. Older nightly builds are deleted, but older milestone and release builds are moved to the archive instead. Its URL is http://archive.eclipse.org/edt/. When you're logged in to build.eclipse.org, our archive can be found at /home/data/httpd/archive.eclipse.org/edt/.

The download.php script will look in the archive if it can't find the file you want on the download server. This means it's important for the archive to have the same directory structure as the main downloads area, so a URL like http://www.eclipse.org/downloads/download.php?file=/edt/A/B/C works even after the file is archived.

Maintaining Project Metadata

Two sites let you view and modify various kinds of meta-data about the EDT project. One is EDT's PMI page at http://projects.eclipse.org/projects/tools.edt (PMI stands for Project Management Infrastructure). The other is the MyFoundation portal at https://dev.eclipse.org/portal/myfoundation/portal/portal.php . You'll need to use these sites to update things such as the current release. N.B. As of February 2013, functionality is being migrated to PMI and out of the older MyFoundation portal.

cross-project-issues-dev

Eclipse's cross-project-issues-dev mailing list is for announcements and problems that affect many projects. You should subscribe to it by going to https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

Our website

The files of www.eclipse.org/edt are stored in Git. To work with them, make a repository connection like ssh://committer_id@git.eclipse.org/gitroot/www.eclipse.org/edt.git, and import the project named edt.

Download Statistics

You can find out how often EDT builds have been downloaded at https://dev.eclipse.org/committers/committertools/stats.php . Enter a path in the Partial File Name field, and optionally select a start and/or end date. All of our builds have a path that begins with "/edt/". To view download stats for release builds, use "/edt/releases/".

Starting a Build

Log in to Hudson and click Build Now. The single build parameter identifies the kind of build: use N for nightly, M for milestone, and R for release. Nothing special needs to be done for nightly builds, but see below for special instructions for milestone and release builds.

Outputs of a Build

Builds use a workspace directory on one of the Hudson slave machines. The workspace can be viewed through Hudson, but it can't be accessed from build.eclipse.org.

After a successful build, the new update sites are accessible from build.eclipse.org. The zipped update site (and JUnit results when we get those going) is in Downloads/edt/downloads/drops/BuildTypeName/Version/BuildType_ThisBuild, where

  • Downloads is /home/data/httpd/download.eclipse.org/
  • BuildTypeName is nightly, milestones, or release
  • Version is a three-part version number
  • BuildType is a letter: N for nightly, M for milestone, R for release

Examples:
Downloads/edt/downloads/drops/nightly/0.8.0/N_ThisBuild
Downloads/edt/downloads/drops/milestones/0.8.0/M_ThisBuild

The "_ThisBuild" directory is created under the workspace/build directory on the slave machine, and it's copied to the download location at the end of the build. The previous build (if there is one) is preserved in the workspace. It has the same path as the new build, except it ends with _OlderBuild not _ThisBuild.

The build process also unzips the update site. The update site for a nightly build is unzipped into Downloads/edt/updates/nightly/. Milestone and release builds are unzipped into Downloads/edt/private-test/. We use private-test for milestone and release builds so we can do installation testing on a milestone or release candidate build without modifying the update sites. This prevents users from installing a candidate build before we finish testing it.

We give Hudson permission to write into our downloads area using ACLs. The Hudson user is hudsonBuild. If you change the existing structure or make a new directory, but sure to modify its ACL.

How To Deal With a Failed Build

When a build fails, Hudson will send an email to you, and everyone who committed a change in that build. The message lists the changes and shows the tail of the build log.

Errors and warnings from javac are shown as numbered lists in the build log. To find a compiler error, search for ". ERROR". The word "error" will show up in several other places, but putting the period and space in front will take you to the compiler's output.

If the build failed because something didn't compile, you probably don't need to remind the developer to fix the problem or help them, unless the failure was a missing plugin. Missing plugins are discussed at EDT:How to add new EDT plugins.

If the build failed because of a Hudson problem, check Bugzilla. Hudson bugs have Classification=Eclipse Foundation, Product=Community, and Component=Hudson. If there's no bug for the problem, open one (you can use this link https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=Hudson).

Starting a New Release

Only a few changes need to be made. Update the value of the version property in org.eclipse.edt.releng\build.properties, and update the version numbers in our features. Developers are responsible for updating the version numbers of their plugins as they make changes.

We may also need a new Target Milestone in Bugzilla.

Procedure for Milestone Builds

Prior to running a milestone build, we'll probably disable nightly builds and build on-demand. We'll probably be in some kind of lockdown, so you might need to check that nobody snuck anything into Git that shouldn't be in the milestone build.

If this is the first build of this milestone, run the copyright tool. We have our own specialized version of Eclipse's tool. It has some bug fixes, knows how to recognize RBD-style copyright comments, and operates on more kinds of files. It's in the IBM CVS repository, project org.eclipse.wst.common.utils under the CopyrightTools folder, in the EDT branch. Check it out, then export it as a deployable jar, and put it in your dropins folder. Restart EDT. You can delete the project. To use the copyright tool, select a file/folder/project and choose Update EDT/RBD Copyrights. Important! After you run the tool, inspect the changes to verify that nothing is screwed up before you push them upstream. If the tool detects problems, it will write into a file called copyrightLog.txt in the root of your workspace. The tool isn't able to detect every kind of problem, and it tends to make mistakes on files that contain non-IBM copyrights.

Update the associate.sites setting in org.eclipse.edt.releng/build.properties according to the comments in the file.

Ensure that downloads/edt/private-test/ is empty and start a build with type=M. It will take longer than a nightly build because the plugins will be signed.

Move the milestone zip to downloads/edt/private-test/. Rename the zip and its md5 file to something like edt-Update-R080M2.zip and edt-Update-R080M2.zip.md5 (replace "080M2" with the proper version and milestone). Edit the md5 file. Put in the new name for the zip.

Notify the people doing installation testing that the build at downloads/edt/private-test/ is ready.

When installation testing is complete, move the previous milestone's zipped update site from downloads/edt/milestones/1.0 to the archive. Be sure to put a link to the newly-archived build in the archive's index.html and describe it in the index.html for the newly-archived build's directory. Delete the other contents of the milestone build directory and move the contents of downloads/edt/private-test/ to downloads/edt/milestones/1.0/.

Now unzip artifacts.jar. It contains only one file: artifacts.xml. Edit artifacts.xml. First, turn on download statistics tracking. At the top of the file is a repository tag, which contains a properties tag, which contains several property tags. Increment the value of the properties tag's size attribute and add another property tag: <property name='p2.statsURI' value='http://download.eclipse.org/stats'/> Now find the artifact tag for org.eclipse.edt.feature. Increment its size attribute and add this property tag below it: <property name='download.stats' value='/edt/milestones/1.0'/> (the value should match the directory). More info about download tracking is at Equinox p2 download stats and Project Download Stats.

Consider adding another property to artifacts.xml, so people can install from a mirror of their choice. We haven't done this yet. Info is at Equinox/p2/p2.mirrorsURL.

Put the new artifacts.xml into artifacts.jar with zip -f artifacts.jar, then delete artifacts.xml.

We should do another quick test to verify that the build can be installed from its new location. Once we're satisfied, tag the files in Git. Use a tag name like "EDT080M2". Tag the plugins and features listed in the map file, plus org.eclipse.edt.releng and the following four projects. They're used by customBuildCallbacks.xml in org.eclipse.edt.ide.ui.resources to create our RUI widget projects during the build.

widgets/org.eclipse.edt.rui.dojo.runtime.local 
widgets/org.eclipse.edt.rui.dojo.runtime.remote 
widgets/org.eclipse.edt.rui.dojo.widgets 
widgets/org.eclipse.edt.rui.widgets

Use the same tag on important test projects, such as the official EUnits, so we can easily track which version of a test was run with each version of EDT.

If necessary, create a Target Milestone in Bugzilla for the next milestone.

Finally, send an email to edt-dev so everyone knows the build is done.

Change the associate.sites setting in org.eclipse.edt.releng/build.properties back to its previous value, and turn the nightly builds back on.

Procedure for Release Builds

Building a release is very similar to building a milestone, so I'll only note the differences here. Obviously some of the paths are different, and you use R for the buildtype when starting the build. Also, when the build is done, create new all-in-one zips according to the instructions in org.eclipse.epp.package.edt.feature/README.txt .

When installation testing is complete, and we have permission from Eclipse to make the release public, archive the last milestone build according to the instructions in the previous section. Then archive the previous release build. In addition to the release's zipped update site in downloads/edt/updates/1.0/, also archive its (non-zipped) update site and all-in-one zips from downloads/edt/releases/1.0/. Make the links on the EDT Downloads page point to the archived files so people can still find them while we're finishing the release.

If any sample projects had to change for the new release, move their old versions to the archive too.

Move the zipped update site to downloads/edt/releases/1.0/, and move the other contents of downloads/edt/private-test/ to downloads/edt/updates/1.0/. Put links to the files in download.eclipse.org/edt/milestones/1.0, so that people can update from a milestone build to the release.

When tagging files in Git, use a tag like "EDT080GM". Make the links on the EDT Downloads page point to the new files.

Back to the top