Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Platform-releng/Releng Plan for CBI adoption

< Platform-releng
Revision as of 12:17, 16 April 2013 by David williams.acm.org (Talk | contribs) (Towards M7)

Purpose

This document is the plan, and working notes for moving to CBI-based builds. There is still lots to do, and many changes to make, so want to be sure as much is communicated as best as possible.

We want to be aggressive in adopting CBI based builds for Kepler, but at the same time, responsible. That is, to make sure it is correct -- or, correct enough -- that adopters (or, ourselves!) are not broken by any changes. And, that means "correct enough" from one I-build to the next, not "just by the time we release". This is a new area for most of us so will take a lot of focus.

The following are the major criteria to judge if/when we are ready to adopt as our primary build and/or release. Any others?

Main Criteria for moving to CBI based build for Kepler

  1. Have same output deliverables as current build: zips, repository, etc. (For Equinox and Eclipse)
  2. (Be able to reproduce exact same build, given a tag (or tags) to start the build with.)
  3. Same warnings (and compile errors, if any) as PDE based build.
  4. Be able to run our JUnit tests, with same results, as PDE based build.
  5. Produces the same Java Doc and Help documentation.
  6. Run a binary comparator against the bundles to ensure that they are the same binary content as the regular bundles.
    1. [Not sure how to do this since "qualifier algorithm" is different].
  7. Qualifiers not change (except for branding bundles) if the content has not changed.
  8. All bundles signed.
  9. Final I-build type repository have same content and metadata as PDE based build (nothing missing, nothing extra).
  10. Be easy to fit in to current workflow of automated builds and tests of "nightlies", I-builds, milestones (i.e. committers have to know what to do to "release" something for a build, how to "freeze" changes at a certain point).
  11. Don't create maintenance burden by duplicating information (bug 387802)

Bug queries and reference documents

  • URL for CBI based builds (temporary, just until we "switch"):
http://download.eclipse.org/eclipse/downloads/cbibasedindex.html

Significant builds and repositories

The last PDE based build

  • I20130219-1600 is the final "PDE Based build". It is recommended for those "building on" Eclipse to move up to that level now, so once they try a "CBI based build", they will have a good comparison if "all is ok" or if something surprising changed, or broke (if so, open a bug, if one doesn't exist; if one exists, add comments so we can better understand impact and priorities).
   http://download.eclipse.org/eclipse/downloads/drops4/I20130219-1600/

after M6 this was moved to archives site, but you'll need to know this exact URL to "see" it (not displayed on archives summary page). We will leave it there for remainder of Kepler, in case needs to be consulted for testing, etc., but then remove it.

   http://archive.eclipse.org/eclipse/downloads/drops4/I20130219-1600/
  • Repository. Since we are still evaluating version numbering (to make sure they always increase compared to previous I-builds) the repo site at http://download.eclipse.org/eclipse/updates/4.3-I-builds/ may not always give the result from the most recent build (for we hope only a short time). Therefore, the repo that corresponds to that "last PDE build", in non-composite form, is documented here:
   http://download.eclipse.org/eclipse/updates/4.3-I-builds/I20130219-1600/

after M6 this was moved to archives site, but you'll need to know this exact URL to "see" it (not displayed on archives summary page). We will leave it there for remainder of Kepler, in case needs to be consulted for testing, etc., but then remove it.

   http://archive.eclipse.org/eclipse/downloads/drops4/I20130219-1600/repository/

The first Tycho/Maven based Milestone build

  • S-4.3M6-201303141330 is the "first" Tycho/Maven based milestone (stable) build. Its distributions such as "SDK" and "Platform", tests framework, etc., all appear to be perfectly usable, though there remains some issues. The following limitations are known and being worked on next:
Some of the "zipped repositories" are not complete.
No Equinox drops
   http://download.eclipse.org/eclipse/downloads/drops4/S-4.3M6-201303141330/
  • The p2 repository for above site was published at the normal milestones sites.
     Composite: http://download.eclipse.org/eclipse/updates/4.3milestones/

   Simple site: http://download.eclipse.org/eclipse/updates/4.3milestones/S-4.3M6-201303141330/

Rollout and deadlines

Weeks below are numbered in "milestone weeks", beginning with "towards M6" ... there are only 6 weeks for M6.

Towards M6

Week 2: (2/11 - 15)

Begin to produce daily automated CBI based builds and run unit tests in parallel with PDE-based builds.
I'll schedule these to start at 8:11 PM, same time as PDE-based Nightlies and same time as weekly I-builds.
Initially, all CBI based builds (even "nightlies") will be "I-builds" to better assess versioning, and more work is need to implement "nightly vs. integration" builds.
I builds will build against master, controlled by repositories.txt
(Unit tests on Linux only, until those work as expected, to save Windows and Mac machine resources).
During this week "announce" the CBI based builds available, get community/adopter help in testing adequacy.
Committers should begin to use SDK's produced, and evaluate for correctness. In particular:
Check that version numbers change (or don't change) as expected. (I recommend you make some change, even trivial, to make sure it "gets in to" a build, if not otherwise planning changes this week).
Those with "binary bundles" check that they are correct ones.
Help evaluate failing JUnit suites. Some may be "releng" issues, but some may be project pom issues?
Initial Limitations: (meaning of priorities: high, need before making primary; medium, need before release; low, probably need at some point)
there will be no "autotagging" of aggregator or repositories, until we are more confident. (medium priority)
there will be no "equinox drops". (low/medium priority)
there will be no cbibased "I-build repo" (high priority)
"zipped repos" are known not to be quite right yet (low prioriy)
no ecj.jar produced (in hands of JDT Team? To tweak their "custom script"?) (low priority)
no "auto update" of pom's, based on manifest ... such as committers have to update version number in both, make "build.properties" changes in both. (low priority)

Week 3: (2/18 - 22)

Cut over to CBI based builds as primary (unless someone raising a true, blocking objection). [done. 2/20]
Begin to tag aggregator upon each build, (with "commits" of latest submodules added to aggregator). [done. 2/20]
(This allows "local build" to duplicate our build following directions given at Platform Build wiki)
TODO: someone (Thanh?) should confirm it gives same "target" results?)

Week 4: (2/25 - 3/1)

Begin to produce "4.3-I-build" repositories at .../eclipse/updates bug 393927.
Will make "massive" coordinated change to update all pom groupIds bug 397850 and the eclipse parent pom renamed bug 394831.
Some projects may need to "touch" their projects to get qualifiers "in order" (there is a difference in heuristics used for qualifiers so some in CBI build might be "lower than" those produced by PDE, in theory).
Will "back port" some changes to 422+ branches, since even though we will not be be releasing anything after 4.2.2, one of the reasons we committed to doing this was to support the LTS efforts, which needs the changes too.

Week 5: (3/4 - 8)

Possibly this week start to have true "CBI nightlies" vs. "Integration".

Week 6: (3/11 - 15) <== 3/15 is M6+0 day.

Towards M7

Items are listed in rough order of priority. If I've missed any large or important items, let me know.

Week 1: (3/18 - 3/22)

triage/planning
initial 'nightlies' bug 398256
who needs "master" vs. "integration" branches?

Week 2: (3/25 - 3/29)

Create Equinox drop site bug 398257
finish delta pack bug 402343 [Paul, and Tom, with Curtis; using antrun]
improve build error checking bug 400633
automate build failure notification
include POM Updates in build notifications
more could be done ... but, basic, common cases are handled
fix or ensure all warnings/compile errors are shown in summary bug 400753
figure out how to not "fail on error"? That is, "failOnError=false".
Turns out, this is not possible with Tycho (though, it is with Maven and Ant). I've opened bug 404634 against Tycho.
move pack.gz step to the end of production builds bug 403805 but may be complicated by lack/change in eclipse.inf files bug 403589?
Turns out, for our solution, lack of eclipse.inf did not matter, much, but that lack (bug 403589) might still matter to some? For example, mattered to creating final "common repo" with b3 aggregator.
begin building our "publishing tools" separately and stop using "basebuilder" bug 324682
Could more easily fix or improve "publishing" issues, etc.
some progress: Building the "core build tools", with Maven/Tycho, on Hudson, in their own build-job.
more progress: Could start using not-basebuilder (but instead "Platform binary + tools" for builds ... but, a little more work needed before using to run Unit Tests

Week 3: (4/01 - 4/05)

And/remove pack.gz files (bug 403805)

Week 4: (4/08 - 4/12)

Week 5: (4/15 - 4/19)

Add "chmod" logic where needed bug 402661
Add post-processing step (using p2.process.artifacts)
add traditional comparator step for sanity check (and fringe cases) bug 402448
add md5 checksums bug 402908
remove "Uncategorized" category bug 404103
sliced, zipped repos
Do we really need all those zipped repos? See bug 403811
make accurate. Bugs numbers?

Week 6: (4/22 - 4/26)

add .target specification to builds more reproducible/predictable bug 400518
remove the artificial "master features" bug 402647
add, fix, improve "releng tests" [Bug numbers?]

Week 7: (4/29 - 5/03) <== 5/03 is M7 +0 day

Milestone stabilization week

Post Kepler

Over time, there may be efforts to make the build more "pure maven" (it still has some ant tasks, bash scripts, and still not ran on Hudson itself, for example), be less monolithic (its still pretty much "all or nothing", from a clean state), etc. BUT this will only be done with (continued) community help and involvement. There are no specific plans or resources to apply in this area, and this is just mentioned here for those from the community who read our "transition plan", wondering what the long term plan is.

Back to the top