WTP/Build/Accounting for history during builds
By using p2 (and related tools) during builds, it is possible to better account for history when delivering builds of our code.
In short, if a bundle has the exact same version as it did from previous builds, then that previously built bundle should be assumed to be the same, and that previously built bundle reused and it be delivered instead of the recently built one. This is kind of counter intuitive, but this is the way a p2 based install would work; if someone already had that bundle installed, with the exact same version, then they would never get a new one installed.
Of course, the immediate problem is that sometimes the byte codes do change even though the source code doesn't change (so hence the version doesn't change). In general this happens because we version code based on our source, but other factors can effect the built code ... for example, a constant in a pre-req might change, the compiler might change, etc. For a complete overview and introduction, see Andrew Niefer's blog entry.
In particular, for WTP, we have had a recurring problem, where a feature version (with its build-time computed suffix) stays exactly the same, but the contents of the feature changes (and then, the problem is actually, it is not noticed until weeks or months later). This is due to the computed suffix not reflecting enough significant digits to fully account for its contained bundles; see bug bug 208143. The only solution for this bug, since we do not want to make the suffixes extremely long, is to detect the problem, and take appropriate action. While it'd be ideal to do that "real time", during the build, the next best alternative is to at least detect it shortly after a build, and correct it before the code is delivered.
Process during WTP Builds
The following image will be used to explain the parts of this process as used by WTP builds.
The flow is basically left to right, over time or steps of the build.
- The build makes use of PDE's "gather repo" flags to create the 'buildrepo', which is literally the byte codes as built, in the form of a handy repo. (Bundles are signed, by this point, when doing a production build).
- During the "create final repo" task in our build scripts, the p2.mirror operation is used to created the final 'repository'. It primarily mirrors content from the 'buildrepo', but if it happens to find an IU (bundle or feature jar) with the exact same version in the reference repositories, then it uses the one from the reference repository.
- During that mirror operation, while it is comparing IUs, using the comparator task, if it finds two IUs have the same version, but have some difference in content, then it prints that information to the 'comparator.log'.
- In an ideal world, that'd be it; we'd just look if the comparator log had any messages, to know if there were issues to address. But the reality is there are actually lots of "differences" found ... most of which we do not care about. Therefore, we run a release engineering test to filter out expected messages from unexpected messages and throw an error if there are any unexpected messages.
- Not shown in diagram, any zips created or tests are ran with the final repository (not the 'build repo').
- The 'repository' shows a dashed line back to the 'reference repositories', as once it is declared, then it becomes one of the reference repositories. The reference repositories are a composite, made up of previous release and subsequent builds.