Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

MoDisco/Releng/Athena/How it Works

Warning2.png
This page explains how MoDisco was built using Athena. MoDisco is now built with Buckminster


This page describes the build process of the MoDisco component, from a build engineering standpoint. For help on merely using the build, see How to Use?.

Hudson

Nightly builds job (cbi-modisco-nightly)

  • Everything starts from Hudson (see the nightly Hudson Job configuration [1]), which runs on the Eclipse build server (build.eclipse.org).
  • The build is started on a defined schedule (every 6 hours), but can also be started manually.
  • Hudson first exports the contents of the MoDisco SVN [2] to its Workspace (/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there.
  • The Hudson job is configured to execute a shell script that:
    • Checks out the MoDisco plugins and features from the head of the trunk into /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace/MoDiscoTree
    • imports and runs the /opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh script
    • reads the next build number from /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/nextBuildNumber, and decrements it to find the current build number
    • looks at the log /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/builds/$buildNumber/log to determine if the build failed or succeeded
    • exits with an error code if the build failed
    • touches the /opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN file if the build succeeded

Integration builds job (cbi-modisco-integration)

A little different from the nightly build job (apart from the file paths):

  • Configuration : [3]
  • The build is started once a week
  • Hudson first exports the contents of the Releng project [4] to its Workspace (/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/workspace). If the build was started automatically and nothing changed in the Releng project since the last build, then the build job stops there. This allows the integration job to start if the map files have changed.
  • touches the /opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalI file if the build succeeded

Athena CBI

The MoDisco project is built using Athena Common Build.

The run.sh script mentioned above does some setup, and then runs /opt/public/cbi/build/org.eclipse.dash.common.releng/tools/scripts/start.sh.

The start.sh script does some more setup (creates and sets up the build directory, retrieves the releng project, installs support for SVN,...), and then starts an Eclipse instance with the org.eclipse.ant.core.antRunner application, on /opt/public/cbi/build/org.eclipse.dash.common.releng/buildAll.xml, which is the Athena Common Builder main build script.

The buildAll.xml script in turn invokes other Athena ant scripts, and makes use of PDE Build functionality to:

  • fetch sources from the repository according to map files in the Releng project (unless for nightly builds, where a local source checkout directory is used instead, to always pull the head of the trunk instead of the tags or branches from the maps)
  • download build dependencies (Eclipse + other needed plugins) specified in the build.properties file in the Releng project, and install them in a new Eclipse installation that is used specifically for the build
  • build plugins, features and source plugins with a headless invocation of PDE Build
  • build tests and run them under a new Eclipse instance
  • create a p2 update site
  • compress the update site with pack200
  • sign the plugins with the Eclipse Foundation certificate, apart for nightly builds (signing takes a while)
  • package several zips : runtime, SDK, examples, tests, all in one update site
  • clean up the build directory at the end, unless there were errors

Athena makes use of these elements in the Releng project:

  • build.properties : the most important configuration file : specifies the steps to do, the feature to build, the build dependencies, the Java paths, plus a few other parameters, some being optional.
  • the map files : define where to get plugins and features from (CVS or SVN path) and which version to take for each one (head, tag, or branch). The maps are used by the integration job, but not the nightly job, which always builds from the trunk.
  • testing.properties : defines the tests to run
  • buildExtra.xml : optional file containing ant targets that will be called by Athena at pre-defined points during the build. This is useful to do custom packaging, fetch extra dependencies, compile javadoc, launch code generators, etc.

Promote

Once the Hudson job is finished, the build results are found in /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/lastSuccessful/ (or /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/lastSuccessful/).

The following entry was added to a crontab (crontab -e): 

*/5 * * * *  /opt/public/modeling/gmt/modisco/cronPromote.sh

So that the cronPromote.sh executes every five minutes on the build server.

This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build with the date of a reference file touched at the end of each promote (/opt/public/modeling/gmt/modisco/lastPromoteRefN and lastPromoteRefI).

If a new build is detected this way, the script then promotes the result of this build by calling ant on the promote.xml of the Releng project, passing it the promote-N.properties for a nightly build or promote-I.properties for an integration build.

The promote is not done directly from Hudson because it does not have the necessary rights.

The promote-N.properties and promote-I.properties define where to get the files from, where to put the archives and update site, and whether to add to or replace the existing contents.

The cronPromote.sh script logs everything it does to /opt/public/modeling/gmt/modisco/log-cronPromote, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in /opt/public/modeling/gmt/modisco/promo_logs.

Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.

Hudson configuration

Hudson saves its configuration in /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/config.xml for the nightly job and /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/config.xml for the integration job.

Hudson important parameters

Files to archive = build/?20*/** : archive the results of the build

Excludes (below "Files to archive", click on Advanced) = build/?20*/eclipse, build/?20*/eclipse/**, build/?20*/testing, build/?20*/testing/**, build/?20*/*-AllFeaturesAndPlugins-*.zip*, build/?20*/*-Master-*.zip*, build/?20*/noclean : don't archive work directories, that are left over by Athena when the build failed.

Test report XMLs = build/?20*/testresults/xml/*.xml : the test results, that will be displayed in Hudson

Repository URL =

Hudson script (nightly job)

# configuration
export PROJRELENGROOT='-projRelengRoot svn://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk'
export PROJRELENGPATH='-projRelengPath org.eclipse.gmt.modisco.releng'
promoteSignal=/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN
hudsonWorkspace=/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace
SVN=/usr/local/bin/svn

rm -rf $hudsonWorkspace/MoDiscoTree
mkdir $hudsonWorkspace/MoDiscoTree

$SVN export svn://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/features/trunk $hudsonWorkspace/MoDiscoTree/features
$SVN export svn://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/tests/trunk/ $hudsonWorkspace/MoDiscoTree/plugins --force
$SVN export svn://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/tests/trunk/org.eclipse.gmt.modisco.tests.feature $hudsonWorkspace/MoDiscoTree/features/org.eclipse.gmt.modisco.tests.feature --force
$SVN export svn://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/doc/trunk/ $hudsonWorkspace/MoDiscoTree/plugins --force

#plugins are already exported by Hudson SCM Poll (see Job configuration)
cp -r $hudsonWorkspace/trunk/* $hudsonWorkspace/MoDiscoTree/plugins

# run the build
. /opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh

read nextBuildNumber < /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/nextBuildNumber
buildNumber=$((nextBuildNumber-1))

# Test if build succeeded. If not, then exit with error 1.
# Copy the log, because the commands appear in the grepped log...
tmpLog=`mktemp` || exit 1
cat /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/builds/$buildNumber/log > $tmpLog
grep "BUILD FAILED"  $tmpLog && exit 1
grep "BUILD SUCCESSFUL" $tmpLog || exit 1

#triggers promoting the build by a script waiting for the modification date of this file to change
touch $promoteSignal

cronPromote.sh

#!/bin/bash

# This script compares the date of a signal file touched by the hudson build
# with the date of a reference file touched at the end of each promote.
# It then promotes the result of the build. The promote is not done from
# Hudson because it does not have the necessary rights.

lastPromoteFileN=/opt/public/modeling/gmt/modisco/lastPromoteRefN
lastPromoteFileI=/opt/public/modeling/gmt/modisco/lastPromoteRefI
promoteSignalN=/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN
promoteSignalI=/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalI
logFile=/opt/public/modeling/gmt/modisco/log-cronPromote

ANT=/opt/public/common/apache-ant-1.7.1/bin/ant
RELENG_DIR=/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace/build/org.eclipse.modisco.releng
PROMO_LOGS_DIR=/opt/public/modeling/gmt/modisco/promo_logs
BUILDS_DIR=/home/data/httpd/download.eclipse.org/modeling/gmt/modisco/downloads/drops/0.8.0

DATE=`date +%Y%m%d-%H%M`

# trim the log : keep about last 10 days of log
function trimLog {
        tail -n 3000 $logFile > ${logFile}Trimmed && mv ${logFile}Trimmed $logFile
}


function prune() {
        # prefix: N for nightlies, I for integration, ...
        kind=$1
        filesPath=$2
        nBuildsToKeep=$3
        
        allFiles=$(ls -trd $filesPath/${kind}20*)
        nFiles=$(ls -1d $filesPath/${kind}20* | wc -l)
        nToDelete=$((nFiles-nBuildsToKeep))

        if [ $nToDelete -gt 0 ]; then
                for file in $allFiles; do
                        echo "$DATE: deleting $file" >> $logFile 
                        rm -rf $file
                        nToDelete=$((nToDelete-1))
                        if [ $nToDelete -le 0 ]; then break; fi
                done
        fi
}



if [ ! -e $lastPromoteFileN ]; then touch $lastPromoteFileN; fi
if [ ! -e $lastPromoteFileI ]; then touch $lastPromoteFileI; fi
if [ ! -e $promoteSignalN ]; then echo "$DATE: ERROR: $promoteSignalN not found" >> $logFile; exit 1; fi
if [ ! -e $promoteSignalI ]; then echo "$DATE: ERROR: $promoteSignalI not found" >> $logFile; exit 1; fi

signalDateN=`stat --format=%Y $promoteSignalN`
signalDateI=`stat --format=%Y $promoteSignalI`
lastPromoteDateN=`stat --format=%Y $lastPromoteFileN`
lastPromoteDateI=`stat --format=%Y $lastPromoteFileI`

if [ $signalDateN -gt $lastPromoteDateN ]; then 
        echo "$DATE: removing old nightlies" >> $logFile
        prune N $BUILDS_DIR 4   
        echo "$DATE: publishing nightly build ..." >> $logFile
        mkdir -p $PROMO_LOGS_DIR
        $ANT -f $RELENG_DIR/promote.xml -Dpromote.properties=$RELENG_DIR/promote-N.properties 2>/dev/null 1> $PROMO_LOGS_DIR/cbi-modisco-nightly-promo-${DATE}.txt 
        touch $lastPromoteFileN
        echo "$DATE: done" >> $logFile
        trimLog
fi


if [ $signalDateI -gt $lastPromoteDateI ]; then
        echo "$DATE: removing old integration builds" >> $logFile
        prune I $BUILDS_DIR 4
        echo "$DATE: publishing integration build ..." >> $logFile
        mkdir -p $PROMO_LOGS_DIR
        $ANT -f $RELENG_DIR/promote.xml -Dpromote.properties=$RELENG_DIR/promote-I.properties 2>/dev/null 1> $PROMO_LOGS_DIR/cbi-modisco-integration-promo-${DATE}.txt
        touch $lastPromoteFileI
        echo "$DATE: done" >> $logFile
        trimLog
fi

Some help with ssh and scp

On Windows, use Cygwin for ssh and scp.

Logging in to the build server: 

ssh <committerid>@build.eclipse.org
<type password...>

Copying a file from the build server (example): 

scp <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/config.xml .

Retrieving the last successful build from the build server: 

scp -r <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/lastSuccessful/archive/build/* .

Sending a file to the build server: 

scp archive.tgz <committerid>@build.eclipse.org:

Sending a directory to the build server: 

scp -r directory <committerid>@build.eclipse.org:
Retrieved from "http://wiki.eclipse.org/index.php?title=MoDisco/Releng/Athena/How_it_Works&oldid=212256"

This page was last modified 04:00, 21 July 2010 by Eclipsepedia anonymous user Unnamed Poltroon.

Back to the top