Difference between revisions of "WTP/Build/Accounting for history during builds"

Revision as of 02:29, 13 October 2010

Introduction

By using p2 (and related tools) during builds, it is possible to better account for history when delivering builds of our code.

In short, if a bundle has the exact same version as it did from previous builds, then that previously built bundle should be assumed to be the same, and that previously built bundle reused and it be delivered instead of the recently built one. This is kind of counter intuitive, but this is the way a p2 based install would work; if someone already had that bundle installed, with the exact same version, then they would never get a new one installed.

Of course, the immediate problem is that sometimes the byte codes do change even though the source code doesn't change (so hence the version doesn't change). In general this happens because we version code based on our source, but other factors can effect the built code ... for example, a constant in a pre-req might change, the compiler might change, etc. For a complete overview and introduction, see Andrew Niefer's blog entry.

In particular, for WTP, we have had a recurring problem, where a feature version (with its build-time computed suffix) stays exactly the same, but the contents of the feature changes (and then, the problem is actually, it is not noticed until weeks or months later). This is due to the computed suffix not reflecting enough significant digits to fully account for its contained bundles; see bug bug 208143. The only solution for this bug, since we do not want to make the suffixes extremely long, is to detect the problem, and take appropriate action. While it'd be ideal to do that "real time", during the build, the next best alternative is to at least detect it shortly after a build, and correct it before the code is delivered.

Process during WTP Builds

The following image will be used to explain the parts of this process as used by WTP builds.

The flow is basically left to right, over time or steps of the build.

• The build makes use of PDE's "gather repo" flags to create the 'buildrepo', which is literally the byte codes as built, in the form of a handy repo. (Bundles are signed, by this point, when doing a production build).

• In a "create final repo" ant script in our builds, the p2.mirror operation is used to created the final 'repository'. It mirrors content from the 'buildrepo', but if it happens to find an IU (bundle or feature jar) with the exact same version as in the 'reference repositories', then it uses the one from the 'reference repository'.

• During that mirror operation, if it finds two IUs with the same version, then in addition to using the old one for the final repository, it also uses the jar comparator task from p2 team, to see if there is some difference in content. If not, fine, but if it does, then it prints that information to the 'comparator.log'.

• In an ideal world, that'd be it; we'd just look if the comparator log had any messages, to know if there were issues to address. But the reality is there are actually lots of "differences" found ... most of which we do not care about. Therefore, we run a release engineering test to filter out expected messages from unexpected messages and throw an error if there are any unexpected messages.

• Not shown in diagram, any zips created or tests are ran with the final repository (not the 'build repo').

• The 'repository' shows a dashed line back to the 'reference repositories', as once a build is declared, then it becomes one of the reference repositories. The reference repositories are a composite, made up of previous release and subsequent builds. This currently takes a "manual" step to add a repo to the reference repository. Note: on the production build machine, where all builds are accessible via the file system, the composite reference repo simply uses the file system. If someone were to do a "local" build, they'd likely want to create the reference repository ahead of time on their local system, for efficiency during the build itself. But, note, if there is no reference repository found (such as for a local, test build), then the operation continues and essentially the final 'repository' consists of what ever was in the 'buildrepo', which is probably fine for a test build.

Differences found for which no correction is needed.

Note: it will take some time an experience to fully understand all "differences" found and whether or not significant, so some cases will be documented here to share knowledge and increase our group learning curve.

All the "known cases" are codified in a property file called comparatorfilter.properties that is used during the releng test to filter out some messages. If filtered out, the message is still printed in a file called "excludedMessages" on our test results page. This can be pretty much ignored ... but wouldn't hurt for committers to examine at some points to see if anything is mistakenly being filtered out.

• aboutmappings

In about.mappings, the property "0" has different values: "<date1>" and "<date2>".

One of the most common "differences" found is similar to following. The dates in about.mappings are literally the date of the build, and is a good example of when we would prefer the old bundle be used, to maintain the "original build" date, instead of changing it each time, even though the date was the only thing changed.

• testjars

IOException comparing ... Error opening zip file  

We have some cases where "test jars" are purposely invalid, to make sure our code handles invalid input ... but they cause a message to be logged, so we ignore those messages.

• docbuildxml

Binary file build.xml: sizes differ by ... [10 bytes]

There's several little differences found in "doc" bundles. Honestly do not know exactly what these mean ... but, assuming for now its just dates or similar. [Note: the Eclipse Platform automatically excludes all "doc" bundles from comparator examination, presumably since there are frequently small unimportant changes].

Significant, unexpected differences and what to do about them.

The "fix" for nearly any "real" problem found in comparator is to re-tag the bundle or feature so it appears different in version number, even if the source did not actually change.

The hard part, and the reason for long drawn-out explanation and graphic in previous section is that if there is a difference to be investigated, then that bundle is no longer "in" the build distribution -- the "old" one was put in our final repository and zips, so the 'new' bundle (with exact same version) must be obtained from the literal 'buildrepo' on the build machine (and before it is deleted for subsequent build ... though that wouldn't happen until a build is declared).

• differences in feature contents

Difference found for canonical:
org.eclipse.update.feature,org.eclipse.wst.xml_sdk.feature,3.3.0.v201007311522-7A78-8DXJQUlJHRDD2LBB_qiiymz between 
file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.3.0/S-3.3.0M2-20100923155521/repository/ and
file:/shared/webtools/projects/wtp-R3.3.0-I/workdir/I-3.3.0-20101007023510/buildrepository/wst-sdk

The entry "Feature: org.eclipse.wst.xml_ui.feature
3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk5b6p2Mb" is not present in both features.

The entry "Feature: org.eclipse.wst.xml_ui.feature.source
3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk5b6p2Mb" is not present in both features.

This type of messages occurs for the case where a feature suffix doesn't get computed accurately enough to reflect the difference in content. See bug 327176 for a detailed explanation of this particular case.

Note the repositories referred to. In this case, one version was in M2 ... and that's the one that would have been "delivered" to the build's repository and zip files. So, to see the "differences" one has to get a copy of the "new" one in "buildrepository/wst-sdk". On the build machine, it is possible to drill down into a build's directories, using a web browser, but might be easier for some to use shell account or scp to copy down the file.