Jump to: navigation, search

Linux Tools Project/Releng

{{#eclipseproject:technology.linux-distros}}

Linux Tools
Website
Download
Community
Mailing ListForumsIRC
Bugzilla
Open
Help Wanted
Bug Day
Contribute
Browse Source

p2 Repositories

Hudson builds

Hudson runs builds of our project every 6 hours (0 */6 * * *) if something has changed in Git.

For builds towards the next Juno service release (SR2), ensure the code is already on the stable-1.2 Git branch. For builds towards our Kepler contributions, ensure the code is on the master branch.

As of 2012-11-20 we have two active Hudson jobs:

https://hudson.eclipse.org/hudson/job/linuxtools-juno (built from stable-1.2)

https://hudson.eclipse.org/hudson/job/linuxtools-master (built from master branch)

Release HOWTO

As code is being worked on

  1. Write Linux_Tools_Project/Releng#New_and_Noteworthy items as new noteworthy stuff is committed. Put this into new-{next release} in our website git location (ssh://your_committer_id@git.eclipse.org/gitroot/www.eclipse.org/linuxtools.git).
  2. Ensure the project plan (project-info/plan.xml in website repository) is kept up to date with release deliverables and dates

ASAP when a release date is known

  1. Let project committers and contributors know the cut-off date for accepting patches from non-committers. Reminders on the dates from the table should be send to the mailing list too.

Proposed freeze periods:

Phase Period
Feature Complete (all new functionality is in git) 3 weeks prior to release date
Code Freeze (approval by one other committer) last 3 weeks prior to release date
Strict Code Freeze (approval by the release engineer) last 10 days prior to release date

3 weeks before the planned release date or RC4 milestone

  1. Let the project know that the date has now arrived that new contributions from non-committers cannot be accepted for this release.
  2. Email the EMO (emo@eclipse.org) to request a release review
  3. Go over IP Contribution Review to ensure patches have iplog+ where necessary (note: I've gone over everything earlier than bug #315815 and concluded we don't need to do anything -- Andrew Overholt)
  4. Submit IP log with the IP log tool.
  5. Prepare the release review "docuware". Base it on the previous release's *.mediawiki in our website repository's 'doc' sub-directory. Further details regarding release "docuware" can be found here: Development_Resources/HOWTO/Release_Reviews. Since the release review docuware will point to the N&N, ensure that is finished.
  6. Create a branch (ex. stable-2.1) from master and push it to the main git repository

2 weeks before the planned release date or RC4 milestone

  1. Alter the Hudson job configuration for maintenance releases (e.g. for releases based on Kepler) to pull from the correct git branch
  2. On the stable-x branch
    1. modify releng/org.eclipse.linuxtools.releng-site/pom.xml so the build output is written to updates-nightly-<release> (e.g. kepler) instead of updates-nightly (reserved for the bleeding-edge master)
    2. modify the top-level pom.xml so the mirror-repo-name property has a value of update-<version> (e.g. update-0.10) instead of updates-nightly (reserved for the bleeding-edge master)
  3. On master, update the version number in the pom.xml files (ex. mvn3 org.eclipse.tycho:tycho-versions-plugin:0.17.0:set-version -DnewVersion=${to-be-released-version + 1}-SNAPSHOT; commit and push the resulting changes)
    1. Check if all versions were updated correctly. Usually maven updates more files than we want. We need in this step to update the versions from the pom.xml file from the root directory and from pom.xml files located in 1st level of directories. We don't need to update the version in plugins and features in this step.
  4. Ensure the repos listed in our parent pom.xml contain the dependencies of the correct versions against which we should be building/testing <--- this is important!
  5. Commit any work not targetted at the release to master
  6. Commit changes specifically targetted towards the release only on the release branch (ex. stable-0.9)
  7. Commit changes targetted at both the release and trunk on master and then EGit/User_Guide#Cherry_Picking the commit(s) from master to the branch

10 days before the planned release date or RC4 milestone

  1. After IP log has been cleared, save a copy in the website repository (the legal team usually emails a PDF or HTML snapshot to the IP log approval requestor).
  2. Use Mylyn WikiText to transform the now-complete release review docuware from .mediawiki to .html
    1. To do that, create a new eclipse project, import the .mediawiki file, right click it and select WikiText->Generate HTML option.
    2. Save both the generated HTML and MediaWiki files in the repository alongside previous versions.
  3. Email our PMC to request approval for release. A previous request looks like this. This is only required for a major (x.0) or minor ({*.x) release.
    1. Include links to the HTML for the release documentation, IP log and N&N
  4. Await PMC approval on tools-pmc mailing list. For the Tools PMC, there is no single approval. What happens is that one or more committers will +1 the request and if no one disputes, then consider these +1s as approvals.
  5. Once the PMC has approved the release, email the resulting release review docuware HTML, the IP log snapshot, and a link to the PMC approval email to emo@eclipse.org. If the emo has opened a release tracker bug, you may consider the release approved if the bug has been closed and you have sent the appropriate documentation. In most cases, you will already have supplied the docuware and ip log to the bug.
    1. The EMO may need a week to review the documents, so this step should attempt to finish at least one week to the release day. The emo team usually start the review process at Wednesdays. So it is a good idea to finish this step before the Wednesday that it at least one week from the release date. If this won't be possible, ask them about deadlines.

A few days before the release or RC4 milestone

  1. All component leads should now be satisfied that the branch is stable and all unit tests should be passing
  2. Remove the "-SNAPSHOT" from pom.xml versions (mvn3 org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=${to-be-released-version} ... commit and push the resulting changes)
    1. The maven command is removing -SNAPSHOT from places that it shouldn't. So it is necessary to check if everything is correct before committing. -SNAPSHOT should be removed from the root directory and from pom.xml files located in 1st level of directories. -SNAPSHOTS from plugins and features shouldn't be removed. The maven command will erroneously match plug-ins and features that have the same release number as the overall release and this will end up stripping the -SNAPSHOT and more importantly, the .qualifier from the version numbers. You can spot this by looking at the build for features/plug-ins that are missing the date stamp.
  3. Ensure all p2.inf files found in features, point to the Linux Tools stable update site. To do a mass change, one could do: find . -name p2.inf | xargs sed -i -e 's/updates-nightly/update/g'. The master branch will point to updates-nightly by default. A branch already used for a stable release will likely already be pointed to the correct update site and no action needs to be taken. Look in an existing feature such as autotools/org.eclipse.linuxtools.autotools-feature for confirmation.
    1. In the case of a release which will be contributed to a simultaneous release, this step needs to be done in advance of the RC4 contribution build.
  4. Ensure "-P build-server" is in the Hudson job configuration's maven arguments line to get a signed build
  5. With the version set properly in the repository's pom.xml files, push a build in Hudson. Note that if the release is also targeted for a simultaneous release, then the last build submitted for RC4 must be the release build.
  6. Verify the p2 repository
    1. A handy way to list the contents of a p2 repository is: eclipse -consolelog -nosplash -application org.eclipse.equinox.p2.director -repository <url> -list
    2. Ensure all qualifiers are the same and are what you expect
    3. Using the p2 UI, ensure the IUs are categorized properly
  7. Perform a quick smoke test to ensure the build is acceptable. Zero code changes should have happened since the SHA-1 that was used for the previous build.
  8. Ensure Equinox/p2/p2.mirrorsURL has been automatically set in artifacts.xml (in artifacts.jar) and that the p2.index file exists.
  9. If things are acceptable with the signed build, tag the Git repo:
    1. use a tag of the format vMajor.Minor.Micro (ex. v0.7.1)
    2. use the -a flag on the tag
    3. specify --tags on the git push command to ensure that the tag is visible to others
    4. do not tag until the IP Log is completed and the repository has been tested
    5. if you tag too early, you can tag again using -a -f to force the re-tag, however, if you re-tag, you need to send a note out to indicate this has happened
    6. once complete, an announcement note should be sent out to tell the list that the tag has been created and for what commit SHA-1 the tag is for.
  10. Lock the Hudson job that was used for the release build to prevent automatic deletion (ex. build #263) and add a description (ex. "0.7.1 release")
  11. Save the archive of the entire build somewhere before it gets over-written (it will not be stored across the next build)
    1. ex. wget --no-check-certificate https://hudson.eclipse.org/hudson/job/linuxtools-juno/471/artifact/*zip*/archive.zip
  12. Get source tarballs:
    1. Go to http://git.eclipse.org/c/linuxtools/org.eclipse.linuxtools.git
    2. Download the .tar.bz2 file for the matching tag or last commit
    3. rename the tarball to linuxtools-${version}.tar.bz2
    4. upload the source tarball to http://download.eclipse.org/technology/linuxtools/${version}-sources/, copying and modifying index.php from a previous release
    5. chmod 755 the source tarball

Release day

  1. Put an archive of the entire build into the downloads directory (~/downloads/technology/linuxtools) (Jeff Johnston, Andrew Overholt, and Alex Kurtakov are currently the only committers with access to this area)
    1. Unzip the archive.zip saved after last successful build
    2. cd to sub-directory containing actualy repository and re-zip contents to linuxtools-RELEASE.zip in the downloads directory
    3. Run md5sum linuxtools-RELEASE-incubation.zip >linuxtools-RELEASE.zip.md5.
    4. move the previous linuxtools-RELEASE.zip and md5 to /home/data/httpd/archive.eclipse.org/linuxtools
    5. chgrp technology.linux-distros linuxtools-RELEASE.zip*
    6. chmod 755 linuxtools-RELEASE.zip*
  2. Make any final changes to the release's Linux_Tools_Project/Releng#New_and_Noteworthy
  3. Update linuxtools/new to reflect the release
    1. Keep a new-{version-number} with the updated n&n for the release and edit the new/index.php file to point to the new-{version-number}/index.html file.
  4. Update the link on downloads.php to point to the correct listing of source tarballs for this release
  5. Update the "Unit test results" link on downloads.php to point to the Hudson job that was used
  6. Add a news item to the main wiki page
  7. Ensure download links -- including the zip of the repository -- are correct
  8. Re-verify the p2 repository
    1. A handy way to list the contents of a p2 repository is: eclipse -consolelog -nosplash -application org.eclipse.equinox.p2.director -repository <url> -list
    2. Ensure all qualifiers are the same and are what you expect
    3. Using the p2 UI, ensure the IUs are categorized properly
  9. Copy our p2 repository to a versioned copy to archive it (ex. cp -rp updates-nightly-juno update-0.5)
  10. Announce release on mailing list, newsgroup and blog(s)
  11. Add this release to list of versions in bugzilla (in portal)
  12. Add release.1 to list of target milestones in bugzilla or add a new major release if the next minor release won't occur (in portal)
  13. Mark release as complete in list of releases in portal
  14. Add next version to list of releases in portal
  15. Add next version to list of target milestones in bugzilla (in portal)

Day after the release

  1. Relax :)
  2. Decide whether or not older releases should be moved to archive.eclipse.org (take note of p2.mirrorsURL modification)
  3. Switch the Hudson job back to use the master or stable-*** branch
  4. Start on the next release

JAR Signing

Our Tycho builds use the Dash signing/re-packing plugin written by Jesse Mcconnell and Dave Carver. This takes care of signing and re-packing the p2 repository. If this plugin were to stop working for us or if we had to manually sign a build, this is how to do it:

  1. ensure the build is not already signed
  2. take a zip of the p2 repository (artifacts.jar, content.jar, features/, plugins/) and put it in /home/data/httpd/download-staging.priv/commonBuild/ on dev.eclipse.org
  3. call the signer: /usr/bin/sign /home/data/httpd/download-staging.priv/commonBuild/(filename of zip)
  4. output of /usr/bin/sign --help will guide you with what log to tail, etc. to see when it's finished
  5. move the signed zip out of the staging area and scp it to your local machine
  6. use the p2.process.artifacts ant task to re-pack the archive
    1. ant -f build.xml where build.xml contains something like:
<?xml version="1.0" encoding="UTF-8"?>
<project name="process signed zip" default="processZip">
    <target name="processZip">
        <p2.process.artifacts normalize="true" repositorypath="file:///tmp/explodedZipOfp2Repo" pack="true"/>
    </target>
</project>
  1. verify you can install from the re-packed repository and that the plugins are all signed (the little certificate icons under Help->About->Installation Details->Plug-ins tab should not be broken)
  2. upload the resulting re-packed repository to a download area somewhere

Simultaneous Release Inclusion

As of Helios, Linux Tools is a part of the annual Eclipse simultaneous release. The simultaneous release aggregator takes content from the p2 repos that are listed in our b3aggrcon files Kepler, Juno). Builds that we would like to promote to the simultaneous release must:

  • be signed
  • not necessarily be in category.xml
  • exist in the p2 repository listed in the linuxtools.b3aggrcon file with the exact same feature versions/qualifiers

Note also that categories for the main simultaneous release are different from our p2 repository categories (which are set in our category.xml). Categories for the release train are defined in separate files (Indigo, Juno, Kepler and the order of the features must match that in our contribution files (Indigo, Juno, Kepler). Use the b3 aggregator editor to make any adjustments here and read the relevant documentation!

The builds must remain in the p2 repo until we change the feature versions/qualifiers in the b3aggrcon file. This will prevent future aggregation runs from failing. More information can be found here:

The cross-project-issues-dev mailing list must be monitored by project leads and release engineers. This mailing list can help with any problems with the simultaneous release as well as with EPP packages. David Williams leads the simultaneous release creation and Markus Knauer coordinates the EPP packages including "ours" (Indigo).

Note that during the "quiet week" between RC4 and the final release, our release bits must be put into their final place but not be made "visible". Read the Final Daze document for guidelines and pointers to FAQs on how to make things invisible, etc.

Builds and how they get places

Git Mechanism for it to get to next site p2 Repository Mechanism for it to get to next site Staging site for simultaneous release contributions Simultaneous release site EPP packages
master with the Eclipse jar signing plugin, the resulting p2 repository gets written to updates-nightly automatically updates-nightly (~/downloads/linuxtools/updates-nightly on dev.eclipse.org) N/A N/A N/A N/A
stable-2.1 branch with the Eclipse jar signing plugin, the resulting p2 repository gets written to updates-nightly-kepler automatically updates-nightly-kepler (~/downloads/linuxtools/updates-nightly-kepler on dev.eclipse.org) N/A N/A N/A N/A
master (for Luna inclusion) Manual cp -rp on dev.eclipse.org from p2 repository containing signed build into ~/downloads/linuxtools/update-kepler-MILESTONE on dev.eclipse.org update-luna-m3 (~/downloads/linuxtools/update-luna-MILESTONE on dev.eclipse.org) (only signed builds!) B3 aggregator (pulls from site listed in linuxtools.b3aggrcon) http://download.eclipse.org/releases/staging (maybe?) N/A until release Pulled from simultaneous release site. org.eclipse.epp.package.cpp.feature/p2.inf contains the list of Linux Tools features in the CPP package.
stable-2.1 branch (for Kepler service releases) Manual cp -rp on dev.eclipse.org from p2 repository containing signed build into ~/downloads/linuxtools/update-kepler-sr2 on dev.eclipse.org update-kepler-sr2 (~/downloads/linuxtools/update-kepler-sr2 on dev.eclipse.org) (only signed builds!) B3 aggregator (pulls from site listed in linuxtools.b3aggrcon) http://download.eclipse.org/releases/maintenance (maybe?) N/A until release Pulled from simultaneous release site. org.eclipse.epp.package.cpp.feature/p2.inf contains the list of Linux Tools features in the CPP package.

New and Noteworthy

New and noteworthy (N&N) items should be written as soon as possible after the item has been committed. This will prevent a frantic scramble at the end of the release cycle. For example, for the 0.3.0 release, we have http://www.eclipse.org/linuxtools/new-0.3/ in GIT (ssh://youruserid@git.eclipse.org/gitroot/www.eclipse.org/linuxtools.git). Please add new items under the various headings. Feel free to add new headings as necessary. The parenthesized numbers in the pseudo-index at the top of the file indicate the number of N&N items in that section. There is a template to be used for each release at new-template. Copy this for releases and be sure all recent releases are listed in the table and list at the top. Be sure to add details such as the release version and date, bug count (with a link to the query), a few sentences documenting non-committer contributions, etc.

N&N images

Screenshots should be cropped to show only the pertinent region of the screen and so the N&N page doesn't appear too wide. Use the gimp to add drop shadows (pick the default values for radius and opacity). Save images as PNGs.

Accepting Patches

Patches contributed by non-committers must have the iplog flag set to +1 on the attachment. The flag should be set on the patch attachment itself and not the bug.

Adding a new component

Our build process is pom.xml-driven but we have a mirrored feature structure for use by p2 repos. In order for a new component to be built, it needs to fit into the hierarchy somewhere. When adding a new top-level sub-project feature (sub-features of existing features should get added to their containing feature), follow these steps:

Code-level checklist

  1. Create your feature and containing plugins
    1. Name the feature(s) org.eclipse.linuxtools.mycoolstuff and put it in Git as org.eclipse.linuxtools.mycoolstuff-feature (replacing "mycoolstuff", obviously)
    2. Name the plugin(s) org.eclipse.linuxtools.mycoolstuff.core, org.eclipse.linuxtools.mycoolstuff.ui, etc.
    3. Do not put _ characters in the bundle ID (this breaks the Maven signing plugin we're using)
    4. Ensure your packages are all in the org.eclipse.linuxtools namespace (and in the .mycoolstuff.core, .mycoolstuff.ui packages where appropriate)
    5. Ensure your strings are externalized
    6. Ensure your feature and plugin provider fields are set to "Eclipse Linux Tools" (no quotes)
  2. Either copy over existing pom.xml files and manually edit them or generate using Tycho and manually edit
  3. Copy over existing p2.inf file from any other feature and add p2.inf to your build.properties binary files
  4. Create your JUnit test plugins
    1. Name your test plugin the same as your functional plugin but with a ".tests" tacked onto the end
    2. Ensure your test bundle's pom.xml looks like an existing test bundle's pom.xml
  5. Enable API Tools on non-example/test/feature plugins

Git-level checklist

  1. If this is a new sub-project, create a directory to contain it, like "oprofile" or "autotools"
  2. Check all of your new stuff into Git master

Build-related checklist

  1. Add any new top-level feature to the top-level pom.xml in your Git clone
  2. If your sub-project has dependencies outside the existing ones (BIRT, EMF, DTP, CDT, GEF), notify the mailing list and project leads
    1. Hopefully this will have been caught in the CQ (legal review)
  3. Ensure your BREEs are correct in your plugin MANIFEST.MF files
  4. Ensure the version on your feature ends with ".qualifier" (without the quotation marks)
  5. Ensure the versions in your MANIFEST.MF and feature.xml files match those in your pom.xml files
  6. Add new features to our p2 repo's category.xml file
  7. Ensure your pom.xml files have the same source plugin and feature bits as the others
  8. Run a full local build from the top-level of your git clone with mvn -fae clean install to ensure the build still works

Building locally

  • ensure you have a recent Maven 3 release on your path
  • clone our Git repository (perhaps read our Git instructions)
  • cd into your clone and run mvn -fae clean install
  • You can add the parameter "-Dmaven.test.skip=true" for debug purposes, but remember that no contribution that breaks the unit tests will be accepted.
  • Some components have dependencies between then, for example Valgrind plug-ins. In that case, always build the whole Valgrind suite instead of one particular plug-in.
  • follow this guide for debugging tests being run by Tycho

Adding a new test plugin to those that are run during the automated build

  • Create test plugin(s) (ex. org.eclipse.linuxtools.mycoolfeature.ui.tests)
  • Copy an existing test plugin's pom.xml (this is used when the automated build is run)
  • Add your test plugin to the parent pom.xml
  • Check your plugin(s) into Git
  • Verify that your tests are built/run with a local build (see instructions on this page)

The next time a build happens, your test plugin(s) will be built and run. If you need a build pushed sooner than the next 6 hour mark when our scheduled builds happen, speak with the project leads via linuxtools-dev@eclipse.org or #eclipse-linux on Freenode.

Creating Source Tarballs

You may need to create tarballs of some of the sub-projects found in Linux Tools. To create a source tarball of a subproject,

  • in your git repository, change into the sub-project (e.g. gprof, autotools)
  • there, run: mvn assembly:single (note requires maven3)
  • the src tarball will be found in the target directory
  • rename as desired since the tarball will be given the same name each time

To get a snapshot for a particular commit:

  • clone the Linux Tools git repository
  • git checkout -b LOCAL_BRANCH_NAME commit-hash-number
  • follow the steps above to get a src tarball of a particular sub-project

eclipse-build tarballs

Tarballs are organized into directories based on version (ex. Indigo tarballs go here). Older releases should be moved here. Don't forget to update md5sums.txt and sha1sums.txt!

  • Tag the Git hash
  • Generate with buildEclipseBuildSource.sh (ensure proper tag is used!): ./buildEclipseBuildSource.sh -eclipseBuildTag 0.4
  • Upload: scp to downloads/technology/linuxtools/eclipse-build/whereyouwantittogo and chgrp technology.linux-distros; chmod 775 fileYouJustUploaded; md5sum fileYouJustUploaded >> md5sums.txt; sha1sum fileYouJustUploaded >> sha1sums.txt. For scp access to download.eclipse, a committer must be in special groups in the Eclipse Foundation systems. If you require access, please check with Andrew Overholt or Alex Kurtakov and then contact the Eclipse webmaster via a bug.