Building ECF Release 3.2

From Eclipsepedia

Jump to: navigation, search

Return to ECF.

Please use this page for planning and discussing build issues related to the upcoming ECF 3.2 Release.

We're targeting February 19, 2010 for the ECF 3.2 release. What does this mean?

It means on the morning (PST which is -8 hours UTC) of Feb 19, we have a built, signed, and tested set of bundles. So actually this means Feb 18 for Europe.

In the discussion below we refer to the "Bucky Book." The Bucky Book is Eclipse Buckminster: The Definitive Guide by Henrik Lindberg and Thomas Hallgren. The current version is version 0.6. You can download it from the Buckminster site .

Contents

How to follow what's building

Builds (success and failure) are tweeted to ecfbuildmonkey. Failures and failure-fixes are mailed to ecf-build@eclipse.org. Consider subscribing to this list if you are interested in our builds. It is not a discussion list, just a notification list.

Where does the release build go?

The build is uploaded (by our build script) to dev.eclipse.org. It goes to downloads/rt/ecf/3.2Test/3.6/site.p2. Currently, it really goes to downloads/rt/ecf/3.2Test/3.5/site.p2, but that will be changed shortly. What's key to note here is that the directory is 3.2Test, not 3.2. The Test here does not mean that there is actually any ECF testing performed on dev.eclipse.org. When we're making release builds, we want to first test that the build is made correctly and gets uploaded correctly. Scott looks at the 3.2Test directory; and if he likes what he sees, he moves the files over to 3.2.

Past experience has shown that sometimes the release build just doesn't work. The network may have timed out and only a subset of the files got uploaded. Or the permissions on the dev.eclipse.org directories were not correct. Or the certificate we use to upload can't be accessed. Or a variety of reasons that have nothing to do with the correctness of the ECF software. And we don't want to clutter up the real 3.2 directory (and overwrite the good but old stuff that may be there) until we know that we have a complete build.

When does the release build happen?

It's forced. It does not occur on a schedule. You have to use the Hudson web interface to build.ecf-project.org to kick off a build. And you have to have an account there. https://build.ecf-project.org/hudson/

What needs to be done

We need to decide what builds will have tests attached to them. We need to figure out how to have those tests run automatically with the build. We need to bring in our OSU builds into our Hudson/Buckminster configuration.

We need to attach tests to the build

We have the tests. This task is basically figuring out how to modify the Hudson/Buckminster configuration to perform the tests. Currently we know how to link Hudson projects to each other. By that I mean when a build is started and completes, another (the linked build, which contains the tests also builds). We need to understand how to make those tests run as part of the build.

Which builds should we attach tests to?

We have lots of Hudson projects, but they fall into four categories: Continuous, Nightly, Platform, and Release.

  • Continuous builds occur whenever there is a change to our CVS repository. These builds are not uploaded to dev.eclipse.org. We don't make them available to users; they are considered throwaway builds.
  • Nightly (or daily depending on your time zone) occur once a day. The build is uploaded to dev.eclipse.org and available for users who want the very latest.
  • Platform builds occur once a week. They occur automatically once a week at 7:12 UTC on Mondays.
  • Release builds occur when forced.

We could test all or a subset of these builds. Each of these builds may not have the same set of tests.

We need to get the OSU builds up and working

The OSU builds are plugins that have not yet gone through the eclipse IP process. This means that their CVS repositories are on ecf1.osuosl.org and not dev.eclipse.org. I think eventually they will move to dev.eclipse.org, but this transition is not our current concern. The current concern is making a Hudson project for each of these builds. Scott, Markus, and myself are currently figuring out how to do this, but help and advice from the owners of these projects is needed.

Here is a preliminary list of OSU projects and their owners. ECF committers, please look at this list and verify or update your information.

This means making a Hudson project for each of the OSU builds. There are several issues with the OSU builds.

One important issue is the fact that these projects require ECF to be present to be built. I think this is all of the OSU projects, although I believe it is only a subset of ECF that these projects need. Is it the same subset for all projects and if not, what are the subsets?

Should we rebuild ECF before building each of these projects?

Then, how do we ensure that the OSU project uses the ECF that we just built? Note that our Hudson/Buckminster configuration uses a headless eclipse that comes with Buckminster. Is there a way of updating this eclipse, that is, of adding plugins to it or updating its version. In fact, there does not seem to be a way of even finding out what eclipse version this is, even by looking in its config.ini. The inability to determine the version of eclipse used by Buckminster is bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=298241

Should we look into using the delta pack?

There seems to be some issue about running the delta pack in a headless eclipse. Can it be done? http://aniefer.blogspot.com/2009/06/using-deltapack-in-eclipse-35.html Why are we interested in the delta pack? Our Skype plugin requires that we build it with a 32-bit version of java. Our build machine (ecf2.osuosl.org) uses 64-bit java. So that's why we used to build it on ecf1.osuosl.org, which is a 32-bit machine. And we have to build different versions of the Skype plugin for Linux and Windows. There is no Mac version.

Thomas Hallgren does not recommend using the delta pack with Buckminster. See the forum entry, http://www.eclipse.org/forums/index.php?t=msg&goto=505226& Instead he recommends that we

"set up our target platform for the build so that it contains the multi-platform setup that you use. This is normally the case when you use target.os, target.arch, and target.ws set to '*'. You can also achieve this by installing this feature into your TP using a 'Software Site'. The buckminster launcher doesn't care about the delta-pack at all unless you mix your runtime with your target platform (not recommended). Buckminster comes with two launchers, one shell script for Linux and a bat file for Windows. You normally don't need an executable to launch Buckminster."

Can someone provide more details on how we would do this?

Adding a project to Hudson/Buckminster

The link below contains instructions on how to add a project to Hudson/Buckminster control. This entails creating Buckminster configuration files, checking them into our ecf repository, and creating a Hudson project using those files with the Hudson web interface on ecf2.

What we need to put a project under Hudson/Buckminster control

Should we change our builds to check out code using extssh instead of pserver?

There's now a time lag between extssh and pserver. Although we can't be 100% sure, we do think we may have seen some build failures that resulted from recent commits not being picked up by the build.

Make the Release_3_2 branch

Currently the release build is made from HEAD. We have to create a branch called Release_3_2. Making the branch doesn't mean that we cannot make changes and fix bugs, but it does mean that we are not adding new features. When should we do this? When the branch is created, we'll change the build scripts to use the branch.

The Buckminster configuration files

The links here go to a discussion of Buckminster configuration files that we currently use. The intent is to understand these files in enough detail so that we can construct similar files for all our ECF/OSU projects. The intent is not to show the very latest version of these files (although they may very well be the latest) but rather to illustrate the concepts governing the file's definition.

The rmap File

For a Buckminster build, the first thing to set up is an rmap. Here is the rmap file that we use with the ECF SDK.

releng/org.eclipse.ecf.releng.bm/ecf.rmap

The cquery Files

The cquery files work with the rmap file. For your queries to resolve against repositories (source code or binary), you need to construct a Buckminster Resource Map (rmap) (Bucky Book, page 46).

What's inside the cquery file? You can use the Component query editor (part of Eclipse's Buckminster plugin) to view the cquery file. Or you can just read and edit the xml.

  • Identify the rmap if you are using one (we are).
  • Give the component a name (org.eclipse.ecf.sdk) and a type (eclipse.feature).
  • We are not specifying a version designator. I mean we specify ?==Version but then we do not identity Version.
  • Note that target.arch, target.os, and target,.ws have values of *. These values may become important later when we figure out how to use different target platforms.
  • Lots of advisor nodes. These provide advice to the Buckminster resolution process. Order is important for the advisor nodes. An advisor node is identified by a name pattern, which “is a regular expression pattern that is used to select the components that should receive the advice provided in the advisor node. “ (Bucky Book, page 50).

The cquery creates a bill of materials and materializes them. What is materialization? Materializing means creating material (files?) in your workspace, installing material (plugins?) into your Eclipse IDE or target platform, or placing files in your file system.

Consider the two projects: C-HEAD-sdk_feature and C-HEAD-platform_feature. These are continuous projects, meaning they get fired off whenever there is a committed change to our source repository.

Note that the sdk project also has an N version (nightly) and and R version (released). Note that the platform project also has an I version (integration bundles provided to the platform) and is linked to an L version (test bundles linked to the platform builds).

The links below go to discussions about the cquery files. If you look in our repository, you see the following cquery files. Go to the link for information about each file.

releng/features/org.eclipse.ecf.sdk_feature/org.eclipse.ecf.sdk.cquery
releng/features/org.eclipse.ecf.platform-feature/org.eclipse.ecf.platform.cquery
releng/features/org.eclipse.ecf.tests_feature/org.eclipse.ecf.tests.cquery
releng/features/org.eclipse.ecf.tests-platform_feature/org.eclipse.ecf.tests-platform.cquery

The cspex Files

We got cspex files, not cspec files. You only need to create a cspec file for components that are not automatically translated into the cspec model.

An eclipse component is a collection of plugins and features. Buckminster uses a component reader to interpret the component's meta data and translate it into a Buckminster Component Specification (CSPEC). (Bucky Book, page 20).

Buckminster comes with component readers for several meta data formats. Buckminster currently automatically translates the following component types: osgi.bundle, eclipse.feature, jar, maven, buckminster, and bom. (Bucky book, page 39). We're building features, so we don't need to create a cspec file. The cspec itself is automatically generated and not persisted.

" ...there is nothing you have to do if the software unit you are interested in already has meta data for which there is a component reader available (as it is for all the Eclipse related types; bundle/plugin, feature, and product)." (Bucky Book, page 22).

But we can and do have cspex files. "It is possible to extend/decorate an automatically generated CSPEC by placing a file called buckminster.cspex in the component’s root." (Bucky Book, page 63). You can only use a cspex file with an automatically generated cspec.

Ans so what do we use our cspex files for? We want to add dependencies to underspecified components. (Bucky Book, page 23). So does this mean that if our components were specified more completely, we would not need cspex files?

If you open our buckminster.cspex file with the Buckminster CSPEX editor, you just see what is essentially a color-coded xml editor. There is no graphical assistance.

Our buckminster.cspex file is just the default with some specified dependencies.

The links below go to discussions about the cspex files. If you look in our repository, you see the following cspex files. Go to the link for information about each file. Both are called buckminster.cspex. They are identical.

releng/features/org.eclipse.ecf.core-feature/buckminster.cspex
releng/features/org.eclipse.platform-feature/buckminster.cspex

The mspec Files

"By default (when executing a query with “Resolve and Materialize”), the materialization goes into the workspace."(Bucky Book, page 93).

But in the mspec files listed below, what we have is a list of namePatterns with the attribute materializer="workspace". Isn't this the default? Why must we specify it?

Unless what's really important is the namePattern .* which sets the attribute installLocation="${targetPlatormPath}".

The links below go to discussions about the mspec files. If you look in our repository, you see the following mspec files. Go to the link for information about each file.

releng/features/org.eclipse.ecf.platform-feature/org.eclipse.ecf.platform.mspec
releng/features/org.eclipse.ecf.sdk-feature/org.eclipse.ecf.sdk.mspec
releng/features/org.eclipse.ecf.tests-feature/org.eclipse.ecf.tests.mspec
releng/features/org.eclipse.ecf.tests-platform-feature/org.eclipse.ecf.tests-platform.mspec