Platform-releng/Checklist

This page is to provide a checklist of "release engineering" tasks that must be done at various stages of development.

Note: this page is "just getting started" and will be added to incrementally, as phases are gone through and new things remembered to document. Also, as it develops the structure may change substantially, so "header links" may not stabilize for a while, and hence would be best not to use headers in links ... at least for a while, until it takes it's final form.

After release

Update no-version-in-url repository composites

• Update the repo composites that have no version in URL (I-builds, milestones)

Update build scripts, branding, and test jobs

• See bug 496465 for one of the latest bugs that outline what is required. The outline below is an approximate list of the largest steps. Please expand and add to this list as things are noticed missing.

• Coordinated change: The parent pom's version needs to be incremented and all other pom's that refer to this parent need to be updated. This is true of "master" and the R4_x_maintenance branch.

- FYI: This alone requires all repositories create an R4_x_maintenance branch.

• the "streams/repositories-${branch} files need to be updated and/or created.
• the "bootstrap" files need to be updated and/or created to set the correct version and branch.
• Create new p2 repositories to "seed" the comparator process. The initial contents are typically copied from the previous final released build.
• The reference repository in the build-individual-bundles should be changed to "M-builds" in the branch stream of parent pom.
• files such as eclipse.platform.releng/features/org.eclipse.platform-feature/rootfiles/.eclipseproduct need to be updated with correct version, in both streams.
• splash screen should be updated in both streams.
• product numbers should be updated in both streams.
• "previous release" is used in several places in the scripts. Sometimes simply as the "solid" base for when we need an Eclipse SDK to perform some task, but many other places as the reference to compare against, such as for "repository reports", "API analysis", and "performance baselines". So at major releases, those should all be changed. For example, bug 498137 documents when for "4.7" and "4.6.1" we updated all occurrences of "4.5.2" to "4.6".

• Check for all pom files of bundles that specity <skipAPIAnalysis>true</skipAPIAnalysis>. For those that were released for the first time in previous release, then API Analysis can be enabled and this property can be removed.

Gerrit and Hudson changes

• The main (production) unit tests need to be copied with new names such as "ep46I-unit-*" would become "ep47I-unit-*" and any previous "ep45M-unit-*" jobs need to change to "ep46M-unit*". Each "test job" is nearly identical, and the appropriate version and branch and "passed in" as variables, but the name itself is significant as it is used both for improved Hudson bookkeeping and used later to "retrieve" the correct results for a given build.

• New jobs needs to be created or modified to deplore "parent pom" and "prereq sdks" from master and newly created R4_x_maintenance branch.

• Each Hudson job "triggered" from Gerrit changes need to change from be triggered from "master" and R4_n_maintenance to be triggered from "master" and R4_n+1_maintenance. Or, better yet, the Gerrit jobs should be always set to build from "**" branch, in which case it will always build from the branch the patch was in. In cases where one job builds multiple branches, it is typically important that the "localMavenRepo" be 'cleaned' (deleted) each build.

- Plus, be aware that branches older than current (master) and "current-1" (such as R4_6_maintenance, when 4.7 is master, may need adjustments to the parent pom so the build-individual-bundle profile has the correct reference repository for that branch. Typically, for the master branch the reference repository is 'N-builds" or "I-builds" and for n-1 branch the reference repository is M-builds. Further back than that, though, the reference repo would typically be the "released" repository (such as .../updates/4.5/) or if lots of patches made for LTS a special repository created for that purpose.

Remove old, no longer needed repositories

As an example, once we start 4.7 builds, we no longer need 4.6-I-builds. Note: 4.6milestones is a special case since that is where we put RC candidates for the 4.6 Update Releases, so it can remain.

Typically we would do this a "few weeks" after a release (and after starting new builds) to avoid immediate breakage of someone else's build, but after we have new builds working, removing the old ones can be a gentle reminder to update builds to get current content.

Do not forget cleanup scripts: There are some scripts, such as 'dailyCleanBuildMachine.sh' which loops through each "type" of the repositories and removes old builds from them on the build machine and downloads server. These typically have values for "minor" which are passed to another bash function which does the cleaning so several lines will need to be updated.

After any Release

• Prepare bugzilla to guide and remind us to "Tag a Release".
• Prepare bugzilla (and code) to update the parent pom to "next version". That is, when a "next version" is planned. For example, after "4.5.2" we do not automatically change to "4.5.3". Any long term maintenance is done is seen as "patch builds" of 4.5.2 (usually denoted 4.5.2+ in bugzilla) -- it is not a "new release stream".
• Update any "info center" bugs with our latest collection of 'doc bundles'. As one example of one, see bug 477616.

And since I have made this error several times, the correct pattern to copy our doc bundles in a "p2 repo" plugins directory is (thanks to Markus Keller):

- *.doc.*.jar is the pattern that picks the right 5 bundles

- Not *.doc*.jar, which picks up a 6th one, org.eclipse.ua.tests.doc_1.0.x.q.jar, which is literally "tests", not "docs".

• Add the N&N, acknowledgments and readme links to the download page. These are automated in the "promote script" to avoid manual edits, but will describe here in case there are issues. In short, the promote script adds three variables to the buildproperties.php file when it is renaming/promoting to an "R-build" : $NEWS_ID, $README_ID, $ACK_ID. And, from those, the index.php file has conditions based on 'isset' that will compute the full URL. For example, for any Release in the 4.5.x stream, the promote script adds the following variable=value pairs buildproperties.php:

 $NEWS_ID = "4.5";
 $ACK_ID = "4.5";
 $README_ID = "4.5";

• Copy the release to "archives". While it would seem not to be required until it is removed from "downloads" page, it is a good idea to do it early. This is partially just so it does not have to be done later when it is removed from DL page. But more important, some adopters may like to use the "archives" URL in their own builds simply so that their build script never has to change. This, I have heard, is a common practice at Apache sites. [Note: this is automated for Equinox, so it is done by the same script that promotes Equinox releases but is not yet automated for Eclipse.]

• There are a few places in our builds where "previous release" is specified. That should be updated to the new "previous release". (It is best to search the aggregator project for the previous release version to find the spots that need to be updated.)

The following are the files that needed modifying after the 4.5.2 release (but, things may change over time). See this commit log entry for more details of this specific case.

eclipse.platform.releng.tychoeclipsebuilder/eclipse-junit-tests/src/main/resources/equinoxp2tests.properties
eclipse.platform.releng.tychoeclipsebuilder/eclipse-junit-tests/src/main/resources/label.properties
eclipse.platform.releng.tychoeclipsebuilder/eclipse-junit-tests/src/main/scripts/getPreviousRelease.sh
eclipse.platform.releng.tychoeclipsebuilder/eclipse-junit-tests/src/main/scripts/runtests.sh
production/build-functions.shsource
production/sdk/miscTools/proxyRelated/getBaseBuilderAndTools.xml
production/testScripts/configuration/sdk.tests/testScripts/runtests.sh
production/testScripts/configuration/streamSpecific.properties
production/testScripts/updateTestResultsPages.sh

Milestones

Build Calendar and reminders

• Be sure build calendar is up to date (ideally, a few weeks before the milestone).
• Send a reminder to platform-releng-dev with the following information. A good time is right after the next-to-last I-build before stabilization begins, so there is about 10 days notice.

- The build schedule.

- Remember that "stabilization week" means "code freeze" (except for regression fixes) with a link to Eclipse/Rhythm#Milestones

- The "all teams, all day" test on Tuesday should be taken seriously by all Platform committers. Testing is more important than writing code during stabilization week -- we are, after all, the Platform! Our quality effects nearly every other Eclipse project.

- Sign-off day is on Thursday. Project leads or their delegate should be sure to reserve time to handle that in a timely fashion.

• From time to time, can expand on the above with a more detailed version:

• The rule is that on Monday we have the last day of development (and even then, no "big changes"). The goal to have a good build for the full day test pass on Tuesday. After Monday, no feature work or unrelated fixes are allowed -- only regression fixes.
• On Tuesday no one should develop or fix anything, but literally spend the entire day testing. Ideally, this would not be only their component but also Eclipse as a whole. Ideally, project leads will have a brief list of test activities, such as new features to test, bugs to verify, etc. (but, admittedly, each component is different).
• Wednesday is the fix-day with a focus on fixing regressions found during the test-day.
• The last Wednesday build is the release candidate every team signs-off on Thursday. Signing-off may involve some re-testing, or at least confirming no changes have been made to your components code since the last time the component was tested well.
• No new code changes can be made until the milestone is officially declared. That is, it is not enough that your component has signed off. There is always a chance a re-build will be required at the last minute.
• Also, while not stabilization week specific, all Platform committers must subscribe to platform-releng-dev and eclipse-dev (as well as their own component's mailing lists). Component leads should communicate this to new committers. Please spread the word if you think there are current committers do not know this.

- The platform-releng-dev list is used to routinely communicate information about builds, timing, rebuilds, off-schedule builds, special activities that must be coordinated, etc.

- The eclipse-dev list is where we officially announce milestones, release candidates, and releases, in addition to our official plans, and other information the community needs to know.

Create and monitor "sign-off" bug

• Typically created after the last I-build is complete that is the release candidate, but can be created earlier, such as if there are vacations or holidays on the critical days.
• Project leads (and others) are added to the CC list in the bug
• Typically a note with the bug number is sent to platform-releng-dev to serve as both an added reminder and also so everyone in the community can know about it and follow it.

Do special releng tests

Some of these may be automated eventually ... but, until then ...

• Confirm all jars are signed
• sanity check version numbers, names, etc. (i.e. run "repo tests" locally, until automated)
• confirm can "update" from previous milestone

Promote the milestone

Happens after signed off ... though some initial steps can be done early if fairly confident all is well because the build remains "hidden" until the deferred steps are performed.

Initial step

• add "testNotes.html" to buildDirectory, with sign-off bug number. [TODO: automate]
• Initial step: run the "promote" job from appropriate directory, with appropriate changes for the I-build to promote, what labels should be. This puts artifacts "in place" but, "invisible" until deferred step. (This allows the promotion to be done "early" possible before all sign-offs, but also to allow some mirroring to occur, before being made visible.)

Deferred step

• Deferred step: run the "deferred steps" script (which would have been created as part of the initial promotion job). This makes things visible and tags the aggregator.