Revision as of 17:08, 16 March 2012

Transition Plans for Platform builds after Juno M6

This page is to give a centralized place to document the plan and known issues about transitions taking place in the Eclipse Platform releng build after M6. It is not meant to take the place of bugzilla entries, etc., but often, its easier to understand "the big picture" from a little prose, rather than list of of bugs. First of all, we will be transitioning off Kim Moir's long and tender-loving-care of the builds as she moves on to other exciting opportunities. ^[1]

While I (David Williams) will do my best to (partially) take her place (along with other skilled team members) I am sure I could never "fill her shoes". So, be patient, I'm sure I'll break a many builds and take longer to do things than you are used to ... but I have every confidence we will get the job done as a team.

But, this page is not to document my transition, but instead the transition of the Eclipse builds from running on IBM Internal hardware, to running on the Eclipse Foundation's build.eclipse.org infrastructure. Luckily, Kim has been working on this for a while, and it is "nearly done" now and expected to be "essentially done" by the time she formally leaves. We will be working furiously on it during the week of 3/19 to 3/23.

The big picture, annotated

4.2 builds to be "built from scratch"

Currently, 3.8 is built from scratch, and 4.2 built as a delta of that. 3.8 will continue to be "built from scratch", but 4.2 will start to be built from scratch and be the primary build (you can monitor progress in bug 355430). 4.2 that will become the "nightly" build, instead of 3.8. It may seem kind of "wasteful" to build both "from scratch", but, part of the reason is that to have a better, more componitized build, "building part from scratch and share the rest" would a) be a lot of work, and b) might be more work than its worth, at least for now, since . After Juno, 3.8 will possibly might become a minor "maintenance branch", not needing as many builds as 4.x stream. Even the exact schedule of 3.8 maintenance builds is not decided yet. The dates might correspond to the "coordinated" SR1 and SR2 releases, but at least some chance they might not. More importantly, the Platform hopes to be able to move to the CBI build some time in the months ahead so there is no sense in investing in something that won't be used for very long.

Platform builds will (still) be started from cronjobs

At least for now, until after Juno Release, the platform builds will still be started from cronjobs. Since there are still some issues occasionally with Hudson, it is thought safest to "evolve" this part of the build, and get the cronjob approach replicated and working first, and experiment with stressing Hudson later, after the Juno Release. Plus, issues of "tagging the build" and "moving do downloads" will have to be solved before running under "hudsonBuilder". All solvable, but will take some time/effort, so one thing at a time.

So, one point is, everyone should be aware, even though not seen in Hudson, the build machine itself will likely be much busier, and much more stressed, once the platform builds are running there everyday (not to mention, the unit tests running at Eclipse, and on Hudson slaves).

Platform unit tests will be Hudson jobs

The current plan is once a build finishes, it will trigger a hudson job to run the unit tests on appropriate slave machines (Linux, Windows, and Mac). And progress of those jobs and tests can (eventually) be watched at

https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/

Known Issues and loss of test function

Likely to be many unit test failures initially

These failures may take many weeks if not the whole M7 milestone to get straight. In some cases, this may require a change to the test, where some machine/infrastructure assumptions were implicitly being made, In some cases, it may take changes in infrastructure or "installed platform" on a Hudson slave, so a test "has what it needs". Many of these issues have already been fixed thanks to Kim and the webmasters good work, but some are still not well understood, such as bug 372880. The current plan is to focus on the builds first, making sure they are working, and then turn attention to unit tests.

No performance tests for Juno

There are several issues with running performance tests on Eclipse.org infrastructure (some tracked in bug 374441) that could all probably be solved eventually, or alternatives found ... but, we don't expect it to happen by Juno release. Unless someone comes forward to volunteer some major commitment of time and effort to take that over. In fact, even after Juno release, we may need to depend on someone volunteering to "take over" that part of the build routine, e.g. having a separate Hudson job, that runs performance tests once a week, or something.

No translation enablement checking

While running on IBM hardware and network, the builds used a proprietary tool to check for potential "translation enablement" problems, called checkPII. Despite its name, it checked both PII but also regular documentation for problem, such as encoding, or HTML issues that would cause translations to not be possible, or not work correctly. While we may eventually find some sort of Open Source alternatives for these types of tools or checks, we don't expect to find them by the Juno Release. There is some chance some IBM committers may find or make time to run the internal tool occasionally and open bugs if problems found ... but ... not being automated will be a known loss of function as we move to build.eclipse.org, and will take a fair amount of effort to replace. (tracking in bug 374456).

No "test coverage" statistics for Juno

It basically hasn't work right for a while, and recent attempts to fix seemed to interfere with regular unit tests. And no time or resource to track down those issues. Note: developers can still use what ever tools they'd like in their workspaces to check their own test coverage, but there will be nothing Eclipse-wide in the automated tests. (See bug 373594)

Minor changes for simplification

No more "source tar balls"

It is basically broken now (after transition to git), and while it could probably be fixed with enough effort, there might be other better approaches. (Follow bug 374428 for all the late breaking news and progress).

No more "platform SDK" downloadable zip

Download numbers indicates almost never downloaded ... so, while we are trying to simplify, makes sense to to start reducing deliverables that are not required. (See bug 374436).

Final remarks

As always, as issues found, send notes to eclipse-releng-dev and/or open appropriate bugs.

Eclipse committers might want to "watch" this wiki page, in case "plans change", but hopefully this page won't be needed for long.

And, if anyone wants to help out with some of the releng tasks that the core team does not have time to do, please feel free to check out the helpwanted releng bugs. Some, admittedly, would take a large effort, some would take an ongoing commitment, but ... if it is important to you ... it is an open source project!

Notes

1. ↑ If this is new news to you then a) where have you been? and b) see Kim's blog for details.