Platform-releng/Platform Build Comparator Logs

Platform Build Comparator Logs

The comparator in Tycho/Maven builds is similar in concept to the comparator in the old PDE builds. The fundamental goal is that if a version/qualifier does not change from previous build, then we want to use the artifact and metadata from the previous build, not the newly produced one. This is simply because there is a type of "contract" that if version and qualifier is the same (say from two repos) then the bits are identical.

In PDE builds, we used to do this "at the end" of a build, during a mirroring step. The Tycho/Maven build does this as it builds each bundle or feature. In general, instead of map files, Tycho/Maven computes the qualifier of a bundle based on the last change made to the code, as logged by Git. For features, it is a combination of if the feature project itself changes, or, if the bundles that make up the feature have changed qualifiers.

In any case, once the qualifier is determined, Tycho/Maven builds will look in "reference repository" (such as our .../eclipse/updates/4.3-I-build) and see if there is a bundle with same version/qualifier. If there is not, then business as usual, the new artifact becomes part of the build and distribution. If there is one with matching version/qualifier, then its assume the bits are identical and the old one is taken as the one to use in current build and distribution.

As a sanity check, if debug (-X) is turned on, Tycho/Maven uses the comparator code to determine exactly what's different between the two. In some cases, there might be no difference. In many many cases, it will find that the "signature timestamp" is different. This is a case where the difference is insignificant, and we want the "old" version to be used. In other cases, maybe only the POM file changed, which in most cases will be insignificant (if there are no other changes detected). But, in other cases, it will detect *.class files are different. These are the problematic cases where a bundle likely needs to be "touch" (so its qualifier will increase, and be unique) and then a re-build requested.

What to look for

The full "debug log" is parsed by one of our custom tools, and divided up into more meaning files, so its easier to focus on different cases.

• buildtimeComparatorUnanticipated.log.txt: This is the most important one to look at. It includes cases that were not expected, and will be the one that will show cases where a change in a pre-req (such as for an in-lined constant) or perhaps a change in compiler options change what is produced, even though there is no change in the code that is in Git. If you see *.class files in there ... you likely need a rebuild so the "new" version gets included, not the old one.

• buildtimeComparatorSignatureOnly.log.txt: This one, based on some simple heuristics, is where the tool has determined only the Eclipse signature changed, and therefore are not significant and no reason for a rebuild.

• buildtimeComparatorDocBundle.log.txt: This one splits out the "doc bundles" into their own file. The reason is simply that JavaDoc changes nearly every build, and typically there is no reason to "touch" and rebuild every week, just for a JavaDoc change. The doc bundles are, though, touched as we near a milestone, RC, or Release, to make sure their qualifier increases and therefore the most accurate JavaDoc is in those builds.

• buildtimeComparatorFull.log.txt: This is simply the combination of all the other logs. Essentially its the comparator output from the huge debug log, pulled into its own file. Its main purpose is to make sure the custom tool is working correctly and accounting for all cases, one way or another. We may stop displaying it in the future.

• others: in the future, we may improve the tool to sort out "pom.xml changes only" or "feature.xml changes only", etc.

How to do a deep investigation

Sometimes, you may need to "dig deep" to figure out what a difference is, and if is significant. To do this, you basically need to look fairly soon, before the next build, directly on build machine, so you can see what was in the target (the current, "new" bundle) vs. what was in the "old" bundle that is part of the distribution. The URL to the "top" of the build location, for an I build is currently

http://build.eclipse.org/eclipse/builds/4I/gitCache/eclipse.platform.releng.aggregator/

And from there, drill down into your repository, and find the project in question, and its 'target' directory.

For example, I looked once when the comparator said two feature.xml files were different. It turned out, in the distributed (old) feature, it listed the download size of one bundle as 209 bytes. In the target (the new feature) is said the download bytes was 210 bytes. Not sure how it computed different values ... but seemed insignificant. [Follow-up: Filed Tycho bug 509983 for this.]

In addition, we currently run with -Dtycho.debug.artifactcomparator which will produce for each class file that is different between baseline and current, two bytecode dumps, one with -build and one with -baseline suffix in project target/ folder.

To diff the baseline class files against the respective build files in target/artifactcomparison, you can use this shell command:

$ find . -name "*.class-baseline" -exec bash -c 'build=`echo {} | sed -r "s/(.*)\.class-baseline/\1.class-build/g"` ;  git diff --no-index --color=always {} $build' \; | less -R

It compares equivalently-named files and presents all differences in color.

What some terms and common phrases mean

• "no-classifier": the main (code) artifact produced by the build (normally, the most important one to consider if a rebuild is needed).

• "classifier-sources": is the sources jar produced

Comparator options

The comparator is "turned on" via the <baselineReplace> element in the parent POM. There are three possible values:

• none: comparisons are still done (if plugin is enabled) but no replacements are done ... always takes and distributes current build. (perhaps appropriate for "CI builds", or similar).

• all: always replaces all matching "old" artifacts for the "new" ones, where the artifacts must consist of a "matching set" of artifacts: the actual jar artifact, the pack.gz file, and/or the source bundle. The build will fail if it finds the current build, say, produces a jar file, but not a pack.gz file. Cases like that probably means something went wrong with current (or previous) build, and deserves a rebuild once the problem is understood. May require the project in Git repo to be "touched".

• common: as above, but it will continue, even if if there are missing artifacts from that triplet of artifacts. This is rarely what's desired, and likely indicates a problem with a build setup. If 'common' is used, for example, it might get the old jar, but the new pack.gz file. (and they might not necessarily match). [Actually, at the time of this writing, we are using 'common', due to bug 405911].

Can the comparator logs be wrong?

At times, the reason for "comparator log errors" may not be obvious and may not be due to "current build". If something is wrong with a previous build, then the current build may be comparing current, correct build against "garbage". If the previous, incorrect build is detected early enough, it can be removed from the comparator repo (such as 4.4-I-builds) so that the new, correct build will not show any "comparator errors" and current, correct bundles will not be replaced with "garbage". This can happen, for example, if there is a build that does not use BREEs to compile bundles. These builds will complete without error, but some of the byte codes will be incorrect, compared to when BREEs are used, as they should be. See bug 420587#c5 which found such a case caused by bug 420478.