Revision as of 12:33, 6 October 2011

{{#eclipseproject:technology.linux-distros}}

Linux Tools Release Engineering Procedures

p2 Repositories

• Indigo nightly (what Git master targets as of 2011-03-31): http://download.eclipse.org/technology/linuxtools/updates-nightly
• Helios nightly (what Git "stable-0.7.0" branches target): http://download.eclipse.org/technology/linuxtools/updates-nightly-helios
• Signed builds towards Indigo train (like a staging area): http://download.eclipse.org/technology/linuxtools/update-indigo
• Releases (latest is always http://download.eclipse.org/technology/linuxtools/update):
  • 0.1: http://download.eclipse.org/technology/linuxtools/update-0.1
  • 0.2: http://download.eclipse.org/technology/linuxtools/update-0.2 (shows as Access Forbidden in a browser due to no site.xml; using p2 Update Manager works fine)
  • 0.2.1: http://download.eclipse.org/technology/linuxtools/update-0.2.1 (shows as Access Forbidden in a browser due to no site.xml; using p2 Update Manager works fine)
  • 0.3: http://download.eclipse.org/technology/linuxtools/update-0.3
  • 0.4: http://download.eclipse.org/technology/linuxtools/update-0.4
  • 0.4.1: http://download.eclipse.org/technology/linuxtools/update-0.4.1
  • 0.5: http://download.eclipse.org/technology/linuxtools/update-0.5
  • 0.5.1: http://download.eclipse.org/technology/linuxtools/update-0.5.1
  • 0.6: http://download.eclipse.org/technology/linuxtools/update-0.6
  • 0.6.1: http://download.eclipse.org/technology/linuxtools/update-0.6.1
  • 0.7: http://download.eclipse.org/technology/linuxtools/update-0.7
  • 0.8: http://download.eclipse.org/technology/linuxtools/update-0.8

Hudson builds

Hudson runs builds of our project every 6 hours (0 */6 * * *) if something has changed in Git.

For builds towards Helios service releases (SR1, SR2), ensure the code is on the stable-0.7 Git branch. For builds towards our Indigo contributions, ensure the code is on the master branch.

As of 2010-03-31 we have two active Hudson jobs:

https://hudson.eclipse.org/hudson/job/cbi-linuxtools-Indigo (built from master)

https://hudson.eclipse.org/hudson/job/cbi-linuxtools-Helios (built from stable-0.7 branch)

Releases

We have nightly builds happening from master. When it comes time for a release, we will:

1. Write Linux_Tools_Project/Releng#New_and_Noteworthy items as new noteworthy stuff is committed. Put this into new-{next release} in our website CVS location (:ext:youruserid@dev.eclipse.org:/cvsroot/org.eclipse).
2. Create a branch (ex. stable-0.8)
3. On master, update the version number in the pom.xml files (for f in autotools/ changelog/ gcov/ gprof/ libhover/ lttng/ man/ oprofile/ profiling/ releng/ rpm rpmstubby/ systemtap/ valgrind/; do pushd $f; mvn3 org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=${to-be-released-version + 1}-SNAPSHOT; popd; done ... commit and push the resulting changes)
4. Alter the Hudson job configuration (Helios or Indigo) to use the branch
5. Continue to have automated builds with test runs from the branch
6. Continue to commit any work not targetted at the release to master
7. Commit changes specifically targetted towards the release only on master
8. Commit changes targetted at both the release and trunk on both the branch and master (cherry-pick from master to stable-XX)
9. Ensure the repos listed in our parent pom.xml contain the dependencies of the correct versions against which we should be building/testing <--- this is important!
10. When all sub-project leads are satisfied that the branch is stable, remove the "-SNAPSHOT" from pom.xml versions (for f in autotools/ changelog/ gcov/ gprof/ libhover/ lttng/ man/ oprofile/ profiling/ releng/ rpm rpmstubby/ systemtap/ valgrind/; do pushd $f; mvn3 org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=${to-be-released-version}; popd; done; mvn3 org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=${to-be-released-version} ... commit and push the resulting changes) In the case of a release matching a simultaneous release, this step needs to be done in advance of the RC4 build.
11. Ensure "-P build-server" is in the Hudson job configuration's maven arguments line to get a signed build
12. With the version set properly in the repository's pom.xml files, push a build. Note that if the release is also targeted for a simultaneous release, then the last build submitted for RC4 is the release build.
13. Generate source tarballs (mvn package assembly:assembly) and upload to http://download.eclipse.org/technology/linuxtools/${version}-sources/, copying index.php from a previous release
14. Copy the format of a previous release's mediawiki file in doc/ and modify it for this release's documentation
    1. Generate HTML from the mediawiki file using Mylyn WikiText and save both the generated HTML and mediawiki file in CVS
15. Go over IP Contribution Review to ensure patches have iplog+ where necessary (note: I've gone over everything earlier than bug #315815 and concluded we don't need to do anything -- Andrew Overholt The IP Log needs to be submitted well in advance of the release date (2 -3 weeks) so all bugs targeted for the release must be frozen by then as well.
16. Submit IP log with tool The tool allows you to exclude bugs. Exclude any bugs that pertain to eclipse-build or are not part of the release such as bugs that are targeting a future release and not the present one.
17. After IP log has been cleared, save a copy in the website CVS (the legal team usually emails a PDF or HTML snapshot to the IP log approval requestor)
18. Email technology-pmc@eclipse.org requesting permission to release (only required for a major (x.0) or minor ({*.x) release)
    1. Include links to the HTML for the release documentation and IP log and N&N
19. Email emo@eclipse.org requesting scheduling of a release review (only required for a major (x.0) or minor ({*.x) release)
20. If things are acceptable with the signed build, we will tag the Git repo with a tag of the format vMajor.Minor.Micro (ex. v0.7.1). Use the -a flag on the tag. You need to specify --tags on the git push command to ensure that the tag is visible to others. Do not tag until the IP Log is completed and the repository has been tested. If you tag too early, you can tag again using -a -f to force the re-tag, however, if you re-tag, you need to send a note out to indicate this has happened. Once complete, an announcement note should be sent out to tell the list that the tag has been created and for what commit SHA-1 the tag is for.
21. The Hudson job will then be switched back to use the master or stable-*** branch
22. The Hudson job that will become the build should be locked to prevent automatic deletion (ex. build #263) and a description added (ex. "0.7.1 release")
23. Ensure the archive of the entire build is placed into the downloads directory (~/downloads/technology/linuxtools) (Jeff Johnston, Andrew Overholt, and Alex Kurtakov are currently the only committers with access to this area)
    1. ex. wget --no-check-certificate https://hudson.eclipse.org/hudson/job/cbi-linuxtools-Helios/413/artifact/*zip*/archive.zip
    2. Rename archive.zip to linuxtools-RELEASE-incubation.zip
    3. Run md5sum linuxtools-RELEASE-incubation.zip >linuxtools-RELEASE-incubation.zip.md5.
    4. remove the previous linuxtools-RELEASE-1-incubation.zip and md5
    5. chgrp technology.linux-distros linuxtools-RELEASE-incubation.zip*
24. Ensure the release's Linux_Tools_Project/Releng#New_and_Noteworthy is ready to go
25. Update linuxtools/{index.php,downloads.php,new} to reflect the release
26. Add a news item to the main wiki page
27. Copy our update site to a versioned copy to archive it (ex. cp -rp updates-nightly update-0.5)
28. Verify the update site and all download links (including zip of update site) work
29. Ensure Equinox/p2/p2.mirrorsURL is set in artifacts.xml (in artifacts.jar)
30. Add release to list of versions in bugzilla (in portal)
31. Add release.1 to list of target milestones in bugzilla (in portal)
32. Mark release as complete in list of releases in portal
33. Add next version to list of releases in portal
34. Add next version to list of target milestones in bugzilla (in portal)
35. Announce release on mailing list, newsgroup and blog(s)

JAR Signing

Our Tycho builds use the Dash signing/re-packing plugin written by Jesse Mcconnell and Dave Carver. This takes care of signing and re-packing the p2 repository. If this plugin were to stop working for us or if we had to manually sign a build, this is how to do it:

1. ensure the build is not already signed
2. take a zip of the p2 repository (artifacts.jar, content.jar, features/, plugins/) and put it in /home/data/httpd/download-staging.priv/commonBuild/ on dev.eclipse.org
3. call the signer: /usr/bin/sign /home/data/httpd/download-staging.priv/commonBuild/(filename of zip)
4. output of /usr/bin/sign --help will guide you with what log to tail, etc. to see when it's finished
5. move the signed zip out of the staging area and scp it to your local machine
6. use the p2.process.artifacts ant task to re-pack the archive
   1. ant -f build.xml where build.xml contains something like:

<?xml version="1.0" encoding="UTF-8"?>
<project name="process signed zip" default="processZip">
    <target name="processZip">
        <p2.process.artifacts normalize="true" repositorypath="file:///tmp/explodedZipOfp2Repo" pack="true"/>
    </target>
</project>

1. verify you can install from the re-packed repository and that the plugins are all signed (the little certificate icons under Help->About->Installation Details->Plug-ins tab should not be broken)
2. upload the resulting re-packed repository to a download area somewhere

Simultaneous Release Inclusion

As of Helios, Linux Tools is a part of the annual Eclipse simultaneous release. The simultaneous release aggregator takes content from the p2 repos that are listed in our b3aggrcon files Indigo, Juno). Builds that we would like to promote to the simultaneous release must:

• be signed
• not necessarily be in category.xml
• exist in the p2 repository listed in the linuxtools.b3aggrcon file with the exact same feature versions/qualifiers

Note also that categories for the main simultaneous release are different from our p2 repository categories (which are set in our category.xml). Categories for the release train are defined in separate files (Indigo, Juno and the order of the features must match that in our contribution files (Indigo, Juno). Use the b3 aggregator editor to make any adjustments here and read the relevant documentation!

The builds must remain in the p2 repo until we change the feature versions/qualifiers in the b3aggrcon file. This will prevent future aggregation runs from failing. More information can be found here:

• Indigo/Contributing_to_Indigo_Build <-- This one contains the CVS location for org.eclipse.indigo.build where the linuxtools.b3aggrcon file lives and will need to be updated to new qualifiers [and versions] when we want to include a new release in Indigo (for SR1 and SR2)
• Juno/Contributing_to_Juno_Build <-- This one contains the CVS location for org.eclipse.juno.build where the linuxtools.b3aggrcon file lives and will need to be updated to new qualifiers [and versions] when we want to include a new release in Indigo (for M2, M3, M4, etc.)
• Indigo/Simultaneous_Release_Plan <-- Indigo Dates
• Juno/Simultaneous_Release_Plan <-- Juno Dates
• Calendar of dates!
• Eclipse_b3/aggregator/manual

The cross-project-issues-dev mailing list must be monitored by project leads and release engineers. This mailing list can help with any problems with the simultaneous release as well as with EPP packages. David Williams leads the simultaneous release creation and Markus Knauer coordinates the EPP packages including "ours" (Indigo).

Note that during the "quiet week" between RC4 and the final release, our release bits must be put into their final place but not be made "visible". Read the Final Daze document for guidelines and pointers to FAQs on how to make things invisible, etc.

Builds and how they get places

New and Noteworthy

New and noteworthy (N&N) items should be written as soon as possible after the item has been committed. This will prevent a frantic scramble at the end of the release cycle. For example, for the 0.3.0 release, we have http://www.eclipse.org/linuxtools/new-0.3/ in CVS (:ext:youruserid@dev.eclipse.org:/cvsroot/org.eclipse and module: www/linuxtools). Please add new items under the various headings. Feel free to add new headings as necessary. The parenthesized numbers in the pseudo-index at the top of the file indicate the number of N&N items in that section. There is a template to be used for each release at new-template. Copy this for releases and be sure all recent releases are listed in the table and list at the top. Be sure to add details such as the release version and date, bug count (with a link to the query), a few sentences documenting non-committer contributions, etc.

N&N images

Screenshots should be cropped to show only the pertinent region of the screen and so the N&N page doesn't appear too wide. Use the gimp to add drop shadows (pick the default values for radius and opacity). Save images as PNGs.

Accepting Patches

Patches contributed by non-committers must have the iplog flag set to +1 on the attachment. The flag should be set on the patch attachment itself and not the bug.

Adding a new component

Our build process is pom.xml-driven but we have a mirrored feature structure for use by p2 repos. In order for a new component to be built, it needs to fit into the hierarchy somewhere. When adding a new top-level sub-project feature (sub-features of existing features should get added to their containing feature), follow these steps:

Code-level checklist

1. Create your feature and containing plugins
   1. Name the feature(s) org.eclipse.linuxtools.mycoolstuff and put it in Git as org.eclipse.linuxtools.mycoolstuff-feature (replacing "mycoolstuff", obviously)
   2. Name the plugin(s) org.eclipse.linuxtools.mycoolstuff.core, org.eclipse.linuxtools.mycoolstuff.ui, etc.
   3. Do not put _ characters in the bundle ID (this breaks the Maven signing plugin we're using)
   4. Ensure your packages are all in the org.eclipse.linuxtools namespace (and in the .mycoolstuff.core, .mycoolstuff.ui packages where appropriate)
   5. Ensure your strings are externalized
   6. Ensure your feature and plugin provider fields are set to "Eclipse Linux Tools" (no quotes)
2. Either copy over existing pom.xml files and manually edit them or generate using Tycho and manually edit
3. Create your JUnit test plugins
   1. Name your test plugin the same as your functional plugin but with a ".tests" tacked onto the end
   2. Ensure your test bundle's pom.xml looks like an existing test bundle's pom.xml

Git-level checklist

1. If this is a new sub-project, create a directory to contain it, like "oprofile" or "autotools"
2. Check all of your new stuff into Git master

Build-related checklist

1. Add any new top-level feature to the top-level pom.xml in your Git clone
2. If your sub-project has dependencies outside the existing ones (BIRT, EMF, DTP, CDT, GEF), notify Andrew Overholt
   1. Hopefully this will have been caught in the CQ (legal review)
3. Ensure your BREEs are correct in your plugin MANIFEST.MF files
4. Ensure the version on your feature ends with ".qualifier" (without the quotation marks)
5. Ensure the versions in your MANIFEST.MF and feature.xml files match those in your pom.xml files
6. Add new features to our p2 repo's category.xml file
7. Run a local build with mvn -fae clean install to ensure the build still works

Building locally

• ensure you have a recent Maven 3 release on your path

• clone our Git repository (perhaps read our Git instructions)

• cd into your clone and run mvn -fae clean install

• follow this guide for debugging tests being run by tycho

Adding a new test plugin to those that are run during the automated build

• Create test plugin(s) (ex. org.eclipse.linuxtools.mycoolfeature.ui.tests)
• Copy an existing test plugin's pom.xml (this is used when the automated build is run)
• Add your test plugin to the parent pom.xml
• Check your plugin(s) into Git
• Verify that your tests are built/run with a local build (see instructions on this page)

The next time a build happens, your test plugin(s) will be built and run. If you need a build pushed sooner than the next 6 hour mark when our scheduled builds happen, speak with the project leads via linuxtools-dev@eclipse.org or #eclipse-linux on Freenode.

Creating Source Tarballs

You may need to create tarballs of some of the sub-projects found in Linux Tools. To create a source tarball of a subproject,

• in your git repository, change into the sub-project (e.g. gprof, autotools)
• there, run: mvn assembly:single (note requires maven3)
• the src tarball will be found in the target directory
• rename as desired since the tarball will be given the same name each time

To get a snapshot for a particular commit:

• clone the Linux Tools git repository
• git checkout -b LOCAL_BRANCH_NAME commit-hash-number
• follow the steps above to get a src tarball of a particular sub-project

eclipse-build tarballs

Tarballs for Helios go here and Galileo here. Older releases should be moved here. Don't forget to update md5sums.txt and sha1sums.txt!

• Tag the Git hash
• Generate with buildEclipseBuildSource.sh (ensure proper tag is used!): ./buildEclipseBuildSource.sh -eclipseBuildTag 0.4
• Upload: scp to downloads/technology/linuxtools/eclipse-build/whereyouwantittogo and chgrp technology.linux-distros; chmod 775 fileYouJustUploaded; md5sum fileYouJustUploaded >> md5sums.txt; sha1sum fileYouJustUploaded >> sha1sums.txt. For scp access to download.eclipse, a committer must be in special groups in the Eclipse Foundation systems. If you require access, please check with Andrew Overholt or Alex Kurtakov and then contact the Eclipse webmaster via a bug.