Difference between revisions of "OTBuilding"

Revision as of 11:29, 30 June 2018

OT builds are executed on https://hudson.eclipse.org/objectteams/

The Object Teams Development Tooling is built in two main stages:

Building and Testing with PDE/Build

This is basically a plain vanilla PDE/Build setup with the following customizations:

• the bootstrap script of the Hudson job buildAndTest exports the releng folder from git by saying:

git archive --remote=file://localhost/gitroot/objectteams/org.eclipse.objectteams.git ${branch} releng | tar xv

• build is then invoked via otdt_runtests-hipp.sh

which consumes a few parameters from the parameterized job

• a top-level run.xml ant file initializes a myriad of properties and then
  1. prepare building:
     • unzip a drop of the Eclipse SDK
     • put org.eclipse.egit.fetchfactory into dropins (version ≥ 0.12.0.201201141247 to include the fix for bug 365944 and more)
  2. create the OT/J compiler by 1. invocation of PDE/Build
     • action after this stage
       • install the org.eclipse.objectteams.otdt.core.patch feature into base
  3. build the OTDT and tests by 2. invocation of PDE/Build
     • preparations before this stage
       • unpack eclipse-test-framework
       • install the org.eclipse.test from the previous step into base, to make it available while compiling tests
       • pre-load the output repository location from the previous build including category-less metadata
     • polish after this stage
       • patch content.xml re version range in feature reference from patch feature (see also below, Publishing step 5)
  4. delegate to the next level for testing
• the main build.xml is a fairly thin layer for steps 2 & 3 above: it handles some properties and then delegates to one of two final PDE/Build stages:
  • OT-Compiler: a PDE/Build with a generated build.xml and these parameters from customTargets.xml:
    • copy otdt.map from the exported releng/maps/
    • build feature org.eclipse.objectteams.otdt.core.patch
    • assemble as a p2 repo at file://${otdtUpdatesDir}
  • OTDT: a second PDE/Build with a generated build.xml and these customizations:
    • at this stage, the jdt.core is our version, so we can actually compiler OT/J code
    • copy otdt.map from the exported releng/maps/
    • by setting p2.gathering=true in build.properties we trigger a p2-based build.
• the final test.xml drives setup and execution of tests:
  • create the software-under-test:
    • unzip the base Eclipse SDK
    • call p2 to install the OTDT from testrun/otdtUpdatesDir
  • call p2 to install the tests from testrun/updateSiteTests/eclipse
  • invoke all tests in sequence, has support for parallization (currently unused)

Unit test source and statistics

• Source for all unit tests are here
  • 2 suites (plug-ins) of jdt tests are maintained with slight modifications
  • 4 more suites of jdt tests are used unmodified
  • 6 test suites have been developed specifically for Object Teams.
  • A grand total of more than 89000 tests are used.
• As of 2.6M3 a total of 47 failures are known to happen:
  • 44 failures in OTJLD tests (4 x 11), see bug 484164
  • 2 failures in builder tests (unadapted), unexpected build order
  • org.eclipse.objectteams.otdt.debug.tests.core.StratumTests.testDefaultStratum

Publishing and Promoting a p2 Repository

After building and testing were successful job createRepo performs the following step (using shell script createRepository-hipp.sh):

1. Jar signing
   • condition jars using org.eclipse.equinox.p2.jarprocessor
   • use curl to invoke the signing service on each relevant jar
   • (see bug 521263 for EOL of command line singing)
2. Populate a new repository stagingRepo
   • symlink artifacts from existing update site -- this ensures exact same bits, so unchanged plugins can be re-used during updating

     would we copy these files they were more difficult to distinguish/skip in subsequent phases

   • physically copy bcel jar

     how can we ensure that clients in all update-scenarii will find bcel from original orbit repo?

3. Pack200 all jar files
4. Generate p2 metadata
5. Patch content.xml as to widen the version range of references to org.eclipse.jdt.core.feature.group:

   The patch feature org.eclipse.objectteams.otdt.core.patch must refer to an exact version of the feature which it patches, but even the same jdt bits will create different qualifiers in subsequent builds, so when we build against jdt RC3 it wouldn't be compatible with the final release even if jdt has no changes. - see bug 304156

   This is done using patch-content-xml.xsl

6. Archive category-less metadata as the basis for future builds

   cumulative repositories and categories don't play well together, because a category, once created cannot be augmented see bug 251888#c11

7. Generate the category
8. Add download statistics capability

   use XSL transformation addDownloadStats.xsl, which may not be a very safe way of doing this

9. Jar-up metadata
10. Remove symlinks which were needed to resolve dependencies on previous builds
11. Promote

    if parameter promote was set to true the resulting repository will be promoted to the download server as directory objectteams/updates/ot2.x/timestamp

The script accepts these parameters:

createRepository.sh updateMasterRelativePath [ -nosign ] [ statsRepoId statsVersionId ]

• baseRepo:

relative path under downloads/objectteams/updates pointing to a repository from which to resolve unchanged bundles/features

specify none for an un-parented repository

• statsRepoId

this identifies the repository in the download statistics, currently either 2.6 or unstable

• statsVersionId

this identifies the version in the downloading statistics, currently 2.6M3

• promote

if set to true promote the build result to the download server

• sign

if set to nosign signing is skipped

Starting a new maintenance branch

• Create new branch in GIT under branches/maintenance.
  • otdt.map: add ,tag=branches/maintenance/MYBRANCH to all plugins (incl. tests) that are not locked to a specific build (tags/builds/..)
  • exportReleng.sh: create a copy using the corresponding branch
  • test.properties.in: adjust hard-coded major.minor.micro versions of test plug-ins

Updating to a new Eclipse version

Throughout the build process various version dependencies must be observed, thus moving to a new version of Eclipse to build against involves a number of adjustments:

• See hints on branching above.
• The exact version of a downloadable SDK build must be selected
  • this version goes into file otdt_prerequisites.sh, variables EVERSION, SDK_QUALIFIER and possibly DROP (the latter should be derivable from the former).
• Use job extractVersions passing the relative directory that holds the SDK downloads. This creates versioned text snippets for these files:
  • run.properties: versions of
    • pde.build
    • equinox.launcher
    • jdt feature (short format, exact and next - to define a minimal matching range)
  • feature.xml of otdt.core.patch
    • long form of jdt feature version
  • test.properties
    • qualifier changes are automatically updated, but major.minor.micro of test dependencies are currently hard-coded!
• paste, commit and push these changes
• finally run the Hudson job buildAndTest