Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Linux Tools Project/Releng

< Linux Tools Project
Revision as of 06:52, 13 January 2022 by Akurtako.redhat.com (Talk | contribs) (p2 Repositories)

Linux Tools
Website
Download
Community
Mailing ListForumsIRCmattermost
Issues
OpenHelp WantedBug Day
Contribute
Browse Source

p2 Repositories

Hudson builds

Hudson runs builds of our project every 6 hours (0 */6 * * *) if something has changed in Git.

For builds towards our next release, ensure the code is on the master branch.

As of 2015-05-01 we have four active Hudson build jobs:

https://hudson.eclipse.org/linuxtools/job/linuxtools-kepler (built from stable-2.2)

https://hudson.eclipse.org/linuxtools/job/linuxtools-luna (built from stable-3.2)

https://hudson.eclipse.org/linuxtools/job/linuxtools-mars (built from stable-4.2)

https://hudson.eclipse.org/linuxtools/job/linuxtools-neon (built from stable-5.1)

https://hudson.eclipse.org/linuxtools/job/linuxtools-master (built from master branch)

Release HOWTO

As code is being worked on

  1. Write Linux_Tools_Project/Releng#New_and_Noteworthy items as new noteworthy stuff is committed. Put this into new-{next release} in our website git location (ssh://your_committer_id@git.eclipse.org/gitroot/www.eclipse.org/linuxtools.git).
  2. Ensure the project plan is kept up to date with release deliverables and dates. The next release should always be set up after the previous one completes. To add a new release, go to the project plan web page and expand the Committer Tools section. There, you will find Create New Release under the Releases group. Create the release and set the release date according to the release train final release date. The release should be major if APIs are being removed or changed. Otherwise a minor release or bug-fix release is sufficient. A bug-fix only release does not require a review.

ASAP when a release date is known

  1. Let project committers and contributors know the cut-off date for accepting patches from non-committers. Reminders on the dates from the table should be send to the mailing list too.

Proposed freeze periods:

Phase Period
Feature Complete (all new functionality is in git) 3 weeks prior to release date
Code Freeze (approval by one other committer) last 3 weeks prior to release date
Strict Code Freeze (approval by the release engineer) last 10 days prior to release date

3 weeks before the planned release date or RC4 milestone

  1. Let the project know that the date has now arrived that new contributions from non-committers cannot be accepted for this release.
  2. Email the EMO (emo@eclipse.org) to request a release review and he/she will create a bug
  3. Go over IP Contribution Review to ensure patches have iplog+ where necessary (note: I've gone over everything earlier than bug #315815 and concluded we don't need to do anything -- Andrew Overholt)
  4. Submit IP log with the IP log tool.
  5. Prepare the release review documentation and edit the online release plan for the specified release. A lot of information can be copied over from the previous release online documentation. In the past, this was done as a "docuware" document. Those older documents can be found in our website repository's 'doc' sub-directory. The release documentation should have a pointer to our N&N wiki page: https://wiki.eclipse.org/Linux_Tools_Project/News/NewInXX where XX is the release number.
  6. Create a branch (ex. stable-2.1) from master and push it to the main git repository

2 weeks before the planned release date or RC4 milestone

  1. Alter the Hudson job configuration for maintenance releases (e.g. for releases based on Kepler) to pull from the correct git branch
  2. On the stable-x branch
    1. modify releng/org.eclipse.linuxtools.releng-site/pom.xml so the build output is written to updates-nightly-<release> (e.g. kepler) instead of updates-nightly (reserved for the bleeding-edge master)
    2. modify the top-level pom.xml so the mirror-repo-name property has a value of update-<version> (e.g. update-0.10) instead of updates-nightly (reserved for the bleeding-edge master)
  3. On master, update the version number in the pom.xml files (ex. mvn3 org.eclipse.tycho:tycho-versions-plugin:0.17.0:set-version -DnewVersion=${to-be-released-version + 1}-SNAPSHOT; commit and push the resulting changes)
    1. Check if all versions were updated correctly. Usually maven updates more files than we want. We need in this step to update the versions from the pom.xml file from the root directory and from pom.xml files located in 1st level of directories. We don't need to update the version in plugins and features in this step.
  4. Ensure the repos listed in our parent pom.xml contain the dependencies of the correct versions against which we should be building/testing <--- this is important!
  5. Commit any work not targetted at the release to master
  6. Commit changes specifically targetted towards the release only on the release branch (ex. stable-0.9)
  7. Commit changes targetted at both the release and trunk on master and then EGit/User_Guide#Cherry_Picking the commit(s) from master to the branch

10 days before the planned release date or RC4 milestone

  1. After IP log has been cleared, save a copy in the website repository (the legal team usually emails a PDF or HTML snapshot to the IP log approval requestor).
  2. Email our PMC to request approval for release. A previous request looks like this. This is only required for a major (x.0) or minor ({*.x) release.
    1. Include links to the HTML for the release documentation, IP log and N&N
  3. Await PMC approval on tools-pmc mailing list. For the Tools PMC, there is no single approval. What happens is that one or more committers will +1 the request and if no one disputes, then consider these +1s as approvals.
  4. Once the PMC has approved the release, email the resulting release review docuware HTML, the IP log snapshot, and a link to the PMC approval email to emo@eclipse.org. If the emo has opened a release tracker bug, you may consider the release approved if the bug has been closed and you have sent the appropriate documentation. In most cases, you will already have supplied the docuware and ip log to the bug.
    1. The EMO may need a week to review the documents, so this step should attempt to finish at least one week to the release day. The emo team usually start the review process at Wednesdays. So it is a good idea to finish this step before the Wednesday that it at least one week from the release date. If this won't be possible, ask them about deadlines.

A few days before the release or RC4 milestone

  1. All component leads should now be satisfied that the branch is stable and all unit tests should be passing
  2. Remove the "-SNAPSHOT" from pom.xml versions (find . -name pom.xml | xargs sed -i '/-parent/,+3 s/-SNAPSHOT//1')
    • We used to use mvn3 org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=${to-be-released-version} but the maven command removes -SNAPSHOT from feature and plugin versions and their corresponding .qualifier references in feature.xml and MANIFEST.MF. Regardless, you should look at the build to make sure all features/plug-ins have date stamps.
  3. Ensure all p2.inf files found in features, point to the Linux Tools stable update site. To do a mass change, one could do: find . -name p2.inf | xargs sed -i -e 's/updates-nightly/update/g'; find . -name p2.inf | xargs sed -i -e 's/updates-docker-nightly/update-docker/g'. The master branch will point to updates-nightly and updates-docker-nightly by default. A branch already used for a stable release will likely already be pointed to the correct update site and no action needs to be taken. Look in an existing feature such as autotools/org.eclipse.linuxtools.autotools-feature for confirmation.
    • In the case of a release which will be contributed to a simultaneous release, this step needs to be done in advance of the RC4 contribution build.
  4. Ensure "-P build-server" is in the Hudson job configuration's maven arguments line to get a signed build
  5. With the version set properly in the repository's pom.xml files, push a build in Hudson. Note that if the release is also targeted for a simultaneous release, then the last build submitted for RC4 must be the release build.
  6. Verify the p2 repository
    • A handy way to list the contents of a p2 repository is: eclipse -consolelog -nosplash -application org.eclipse.equinox.p2.director -repository <url> -list
    • Ensure all qualifiers are the same and are what you expect
    • Using the p2 UI, ensure the IUs are categorized properly
  7. Perform a quick smoke test to ensure the build is acceptable. Zero code changes should have happened since the SHA-1 that was used for the previous build.
  8. Ensure Equinox/p2/p2.mirrorsURL has been automatically set in artifacts.xml (in artifacts.jar) and that the p2.index file exists.
  9. If things are acceptable with the signed build, tag the Git repo:
    • use a tag of the format vMajor.Minor.Micro (ex. v0.7.1)
    • use the -a flag on the tag
    • specify --tags on the git push command to ensure that the tag is visible to others
    • do not tag until the IP Log is completed and the repository has been tested
    • if you tag too early, you can tag again using -a -f to force the re-tag, however, if you re-tag, you need to send a note out to indicate this has happened
    • once complete, an announcement note should be sent out to tell the list that the tag has been created and for what commit SHA-1 the tag is for.
  10. Lock the Hudson job that was used for the release build to prevent automatic deletion (ex. build #263) and add a description (ex. "0.7.1 release")
  11. Save the archive of the entire build somewhere before it gets over-written (it will not be stored across the next build)
  12. Get source tarballs:
    1. Go to http://git.eclipse.org/c/linuxtools/org.eclipse.linuxtools.git
    2. Download the .tar.gz file for the matching tag or last commit
    3. rename the tarball to linuxtools-${version}.tar.gz
    4. upload the source tarball to http://download.eclipse.org/technology/linuxtools/${version}-sources/, copying and modifying index.php from a previous release
    5. chmod 755 the source tarball

Release day

  1. Put an archive of the entire build into the downloads directory (~/downloads/technology/linuxtools) (Jeff Johnston, Andrew Overholt, and Alex Kurtakov are currently the only committers with access to this area)
    1. Unzip the archive.zip saved after last successful build
    2. cd to sub-directory containing actualy repository and re-zip contents to linuxtools-RELEASE.zip in the downloads directory
    3. Run md5sum linuxtools-RELEASE-incubation.zip >linuxtools-RELEASE.zip.md5.
    4. move the previous linuxtools-RELEASE.zip and md5 to /home/data/httpd/archive.eclipse.org/linuxtools
    5. chgrp technology.linux-distros linuxtools-RELEASE.zip*
    6. chmod 755 linuxtools-RELEASE.zip*
  2. Make any final changes to the release's Linux_Tools_Project/Releng#New_and_Noteworthy
  3. Update linuxtools/new to reflect the release
    1. Keep a new-{version-number} with the updated n&n for the release and edit the new/index.php file to point to the new-{version-number}/index.html file.
  4. Update the link on downloads.php to point to the correct listing of source tarballs for this release
  5. Update the "Unit test results" link on downloads.php to point to the Hudson job that was used
  6. Add a news item to the main wiki page
  7. Ensure download links -- including the zip of the repository -- are correct
  8. Re-verify the p2 repository
    1. A handy way to list the contents of a p2 repository is: eclipse -consolelog -nosplash -application org.eclipse.equinox.p2.director -repository <url> -list
    2. Ensure all qualifiers are the same and are what you expect
    3. Using the p2 UI, ensure the IUs are categorized properly
  9. Copy our p2 repository to a versioned copy to archive it (ex. cp -rp updates-nightly-juno update-0.5)
  10. Update any relevant Eclipse Marketplace pages
    1. https://marketplace.eclipse.org/content/eclipse-docker-tooling
    2. https://marketplace.eclipse.org/content/eclipse-vagrant-tooling
  11. Announce release on mailing list, newsgroup and blog(s)
  12. Add this release to list of versions in bugzilla (in portal)
  13. Add release.1 to list of target milestones in bugzilla or add a new major release if the next minor release won't occur (in portal)
  14. Mark release as complete in list of releases in portal
  15. Add next version to list of releases in portal
  16. Add next version to list of target milestones in bugzilla (in portal)

Day after the release

  1. Relax :)
  2. Decide whether or not older releases should be moved to archive.eclipse.org (take note of p2.mirrorsURL modification)
  3. Switch the Hudson job back to use the master or stable-*** branch
  4. Start on the next release

JAR Signing

Our Tycho builds use the Dash signing/re-packing plugin written by Jesse Mcconnell and Dave Carver. This takes care of signing and re-packing the p2 repository. If this plugin were to stop working for us or if we had to manually sign a build, this is how to do it:

  1. ensure the build is not already signed
  2. take a zip of the p2 repository (artifacts.jar, content.jar, features/, plugins/) and put it in /home/data/httpd/download-staging.priv/commonBuild/ on dev.eclipse.org
  3. call the signer: /usr/bin/sign /home/data/httpd/download-staging.priv/commonBuild/(filename of zip)
  4. output of /usr/bin/sign --help will guide you with what log to tail, etc. to see when it's finished
  5. move the signed zip out of the staging area and scp it to your local machine
  6. use the p2.process.artifacts ant task to re-pack the archive
    1. ant -f build.xml where build.xml contains something like:
<?xml version="1.0" encoding="UTF-8"?>
<project name="process signed zip" default="processZip">
    <target name="processZip">
        <p2.process.artifacts normalize="true" repositorypath="file:///tmp/explodedZipOfp2Repo" pack="true"/>
    </target>
</project>
  1. verify you can install from the re-packed repository and that the plugins are all signed (the little certificate icons under Help->About->Installation Details->Plug-ins tab should not be broken)
  2. upload the resulting re-packed repository to a download area somewhere

Simultaneous Release Inclusion

As of Helios, Linux Tools is a part of the annual Eclipse simultaneous release. The simultaneous release aggregator takes content from the p2 repos that are listed in our b3aggrcon files Kepler, Juno). Builds that we would like to promote to the simultaneous release must:

  • be signed
  • not necessarily be in category.xml
  • exist in the p2 repository listed in the linuxtools.b3aggrcon file with the exact same feature versions/qualifiers

Note also that categories for the main simultaneous release are different from our p2 repository categories (which are set in our category.xml). Categories for the release train are defined in separate files (Indigo, Juno, Kepler and the order of the features must match that in our contribution files (Indigo, Juno, Kepler). Use the b3 aggregator editor to make any adjustments here and read the relevant documentation!

The builds must remain in the p2 repo until we change the feature versions/qualifiers in the b3aggrcon file. This will prevent future aggregation runs from failing. More information can be found here:

The cross-project-issues-dev mailing list must be monitored by project leads and release engineers. This mailing list can help with any problems with the simultaneous release as well as with EPP packages. David Williams leads the simultaneous release creation and Markus Knauer coordinates the EPP packages including "ours" (Indigo).

Note that during the "quiet week" between RC4 and the final release, our release bits must be put into their final place but not be made "visible". Read the Final Daze document for guidelines and pointers to FAQs on how to make things invisible, etc.

Builds and how they get places

Git Mechanism for it to get to next site p2 Repository Mechanism for it to get to next site Staging site for simultaneous release contributions Simultaneous release site
master with the Eclipse jar signing plugin, the resulting p2 repository gets written to updates-nightly and updates-docker-nightly automatically updates-nightly (~/downloads/linuxtools/updates-nightly on dev.eclipse.org) and

updates-docker-nightly (~/downloads/linuxtools/updates-docker-nightly on dev.eclipse.org)

N/A N/A N/A
stable-4.0 branch with the Eclipse jar signing plugin, the resulting p2 repository gets written to updates-nightly-mars and updates-docker-nightly-mars automatically updates-nightly-mars (~/downloads/linuxtools/updates-nightly-mars on dev.eclipse.org) and

updates-docker-nightly-mars (~/downloads/linuxtools/updates-docker-nightly-mars on dev.eclipse.org)

N/A N/A N/A
stable-3.2 branch (for Luna service releases) Manual cp -rp on dev.eclipse.org from p2 repository containing signed build into ~/downloads/linuxtools/update-luna-sr2 on dev.eclipse.org update-luna-sr2 (~/downloads/linuxtools/update-luna-sr2 on dev.eclipse.org) (only signed builds!) B3 aggregator (pulls from site listed in linuxtools.b3aggrcon) http://download.eclipse.org/releases/maintenance (maybe?) N/A until release

New and Noteworthy

New and noteworthy (N&N) items should be written as soon as possible after the item has been committed. This will prevent a frantic scramble at the end of the release cycle. As of 3.0.0, Linux Tools has switched to use a wiki page for N&N. The 3.0.0 version is: https://wiki.eclipse.org/Linux_Tools_Project/News/NewIn30 as a sample page. Please add new items under the various headings. Feel free to add new headings as necessary. Use the previous wiki as a template to be used for the next release.

N&N images

Screenshots should be cropped to show only the pertinent region of the screen and so the N&N page doesn't appear too wide. Use the gimp to add drop shadows (pick the default values for radius and opacity). Save images as PNGs.

Accepting Patches

Patches contributed by non-committers must have the iplog flag set to +1 on the attachment. The flag should be set on the patch attachment itself and not the bug.

Adding a new component

Our build process is pom.xml-driven but we have a mirrored feature structure for use by p2 repos. In order for a new component to be built, it needs to fit into the hierarchy somewhere. When adding a new top-level sub-project feature (sub-features of existing features should get added to their containing feature), follow these steps:

Code-level checklist

  1. Create your feature and containing plugins
    1. Name the feature(s) org.eclipse.linuxtools.mycoolstuff and put it in Git as org.eclipse.linuxtools.mycoolstuff-feature (replacing "mycoolstuff", obviously)
    2. Name the plugin(s) org.eclipse.linuxtools.mycoolstuff.core, org.eclipse.linuxtools.mycoolstuff.ui, etc.
    3. Do not put _ characters in the bundle ID (this breaks the Maven signing plugin we're using)
    4. Ensure your packages are all in the org.eclipse.linuxtools namespace (and in the .mycoolstuff.core, .mycoolstuff.ui packages where appropriate)
    5. Ensure your strings are externalized
    6. Ensure your feature and plugin provider fields are set to "Eclipse Linux Tools" (no quotes)
  2. Either copy over existing pom.xml files and manually edit them or generate using Tycho and manually edit
  3. Copy over existing p2.inf file from any other feature and add p2.inf to your build.properties binary files
  4. Create your JUnit test plugins
    1. Name your test plugin the same as your functional plugin but with a ".tests" tacked onto the end
    2. Ensure your test bundle's pom.xml looks like an existing test bundle's pom.xml
  5. Enable API Tools on non-example/test/feature plugins

Git-level checklist

  1. If this is a new sub-project, create a directory to contain it, like "oprofile" or "autotools"
  2. Check all of your new stuff into Git master

Build-related checklist

  1. Add any new top-level feature to the top-level pom.xml in your Git clone
  2. If your sub-project has dependencies outside the existing ones (BIRT, EMF, DTP, CDT, GEF), notify the mailing list and project leads
    1. Hopefully this will have been caught in the CQ (legal review)
  3. Ensure your BREEs are correct in your plugin MANIFEST.MF files
  4. Ensure the version on your feature ends with ".qualifier" (without the quotation marks)
  5. Ensure the versions in your MANIFEST.MF and feature.xml files match those in your pom.xml files
  6. Add new features to our p2 repo's category.xml file
  7. Ensure your pom.xml files have the same source plugin and feature bits as the others
  8. Run a full local build from the top-level of your git clone with mvn -fae clean install to ensure the build still works

Building locally

  • ensure you have a recent Maven 3 release on your path
  • clone our Git repository (perhaps read our Git instructions)
  • cd into your clone and run mvn -fae clean install
  • You can add the parameter "-Dmaven.test.skip=true" for debug purposes, but remember that no contribution that breaks the unit tests will be accepted.
  • Some components have dependencies between then, for example Valgrind plug-ins. In that case, always build the whole Valgrind suite instead of one particular plug-in.
  • follow this guide for debugging tests being run by Tycho

Adding a new test plugin to those that are run during the automated build

  • Create test plugin(s) (ex. org.eclipse.linuxtools.mycoolfeature.ui.tests)
  • Copy an existing test plugin's pom.xml (this is used when the automated build is run)
  • Add your test plugin to the parent pom.xml
  • Check your plugin(s) into Git
  • Verify that your tests are built/run with a local build (see instructions on this page)

The next time a build happens, your test plugin(s) will be built and run. If you need a build pushed sooner than the next 6 hour mark when our scheduled builds happen, speak with the project leads via linuxtools-dev@eclipse.org or #eclipse-linux on Freenode.

Creating Source Tarballs

You may need to create tarballs of some of the sub-projects found in Linux Tools. To create a source tarball of a subproject,

  • in your git repository, change into the sub-project (e.g. gprof, autotools)
  • there, run: mvn assembly:single (note requires maven3)
  • the src tarball will be found in the target directory
  • rename as desired since the tarball will be given the same name each time

To get a snapshot for a particular commit:

  • clone the Linux Tools git repository
  • git checkout -b LOCAL_BRANCH_NAME commit-hash-number
  • follow the steps above to get a src tarball of a particular sub-project

eclipse-build tarballs

Tarballs are organized into directories based on version (ex. Indigo tarballs go here). Older releases should be moved here. Don't forget to update md5sums.txt and sha1sums.txt!

  • Tag the Git hash
  • Generate with buildEclipseBuildSource.sh (ensure proper tag is used!): ./buildEclipseBuildSource.sh -eclipseBuildTag 0.4
  • Upload: scp to downloads/technology/linuxtools/eclipse-build/whereyouwantittogo and chgrp technology.linux-distros; chmod 775 fileYouJustUploaded; md5sum fileYouJustUploaded >> md5sums.txt; sha1sum fileYouJustUploaded >> sha1sums.txt. For scp access to download.eclipse, a committer must be in special groups in the Eclipse Foundation systems. If you require access, please check with Andrew Overholt or Alex Kurtakov and then contact the Eclipse webmaster via a bug.

Back to the top