Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "MoDisco/Releng/How it Works"

(broken link)
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page describes the build process of the [http://www.eclipse.org/gmt/modisco/ MoDisco] component, from a build engineering standpoint. For help on merely using the build, see [[MoDisco/Releng/How to Use|How to Use?]].
+
This page describes the build process of the [http://www.eclipse.org/MoDisco/ MoDisco] component, from a build engineering standpoint. For help on merely using the build, see [[MoDisco/Releng/How to Use|How to Use?]].
  
== Hudson ==
+
<span style="color:#606060">(for old Athena build, see [[MoDisco/Releng/Athena/How_it_Works|this page]])</span>
=== Nightly builds job (cbi-modisco-nightly) ===
+
* Everything starts from Hudson (see the nightly Hudson Job configuration [https://build.eclipse.org/hudson/job/cbi-modisco-nightly/configure]), which runs on the Eclipse build server (build.eclipse.org).
+
* The build is started on a defined schedule (every 6 hours), but can also be started manually.
+
* Hudson first exports the contents of the MoDisco SVN [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco] to its Workspace (<code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace</code>). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there.
+
* The Hudson job is configured to execute a shell script that:
+
** imports and runs the </code>/opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh</code> script
+
** reads the next build number from <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/nextBuildNumber</code>, and decrements it to find the current build number
+
** looks at the log <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/builds/$buildNumber/log</code> to determine if the build failed or succeeded
+
** exits with an error code if the build failed
+
** touches the <code>/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN</code> file if the build succeeded
+
  
=== Integration builds job (cbi-modisco-integration) ===
+
== Build jobs ==
A little different from the nightly build job (apart from the file paths):
+
* Everything starts from Hudson (see the nightly Hudson Job configuration [https://build.eclipse.org/hudson/job/modisco-nightly/configure]), which runs on the Eclipse build server (build.eclipse.org).
 +
* The build is started on a defined schedule (every 6 hours) for the nightly build, or once a week for the integration build, but can also be started manually.
 +
* For the nightly build, Hudson first exports the contents of the MoDisco SVN [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.modisco] to its Workspace (<code>/opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/workspace</code> on build1). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there.
 +
* The Hudson job is configured to:
 +
** Check out the releng project from <code>$PROJRELENGROOT/org.eclipse.gmt.modisco.releng.buckminster</code> (PROJRELENGROOT is a build parameter).
 +
** Remove the previous build results and work area
 +
** Create a <code>build.properties</code> by concatenating the <code>modisco-build.properties</code> from the releng project with properties set or found in the Hudson job.
 +
** Call Buckminster:
 +
*** materializes the target platform and workspace using <code>modisco.mspec</code> from the releng project
 +
*** builds the workspace (using Eclipse builders)
 +
*** runs JUnit tests by using the <code>org.eclipse.gmt.modisco.tests/AllTests.launch</code> Eclipse launcher
 +
*** builds the packed (with pack200) and optionally signed update site
 +
** Call ant on <code>other.ant</code> in the releng project, to create a JUnit report as HTML pages
 +
** zip the update site, and put that in another zip with the build log and test results
 +
** touch a file on the build server to trigger a promote script which is waiting for this file to change (<code>cronPromote.sh</code> in <code>/opt/public/modeling/mdt/modisco</code>)
  
* Configuration : [https://build.eclipse.org/hudson/job/cbi-modisco-nightly/configure]
 
* The build is started once a week
 
* Hudson first exports the contents of the Releng project [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng] to its Workspace (<code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/workspace</code>). If the build was started automatically and nothing changed in the Releng project since the last build, then the build job stops there. This allows the integration job to start if the map files have changed.
 
* touches the <code>/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalI</code> file if the build succeeded
 
 
== Athena CBI ==
 
The MoDisco project is built using [http://wiki.eclipse.org/Category:Athena_Common_Build Athena Common Build].
 
 
The <code>run.sh</code> script mentioned above does some setup, and then runs <code>/opt/public/cbi/build/org.eclipse.dash.common.releng/tools/scripts/start.sh</code>.
 
 
The <code>start.sh</code> script does some more setup (creates and sets up the build directory, retrieves the releng project, installs support for SVN,...), and then starts an Eclipse instance with the <code>org.eclipse.ant.core.antRunner</code> application, on <code>/opt/public/cbi/build/org.eclipse.dash.common.releng/buildAll.xml</code>, which is the Athena Common Builder main build script.
 
 
The <code>buildAll.xml</code> script in turn invokes other Athena ant scripts, and makes use of PDE Build functionality to:
 
* fetch sources from the repository according to map files in the Releng project
 
* download build dependencies (Eclipse + other needed plugins) specified in the [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/build.properties build.properties file in the Releng project], and install them in a new Eclipse installation that is used specifically for the build
 
* build plugins, features and source plugins with a headless invocation of PDE Build
 
* build tests and run them under a new Eclipse instance
 
* create a p2 update site
 
* compress the update site with pack200
 
* sign the plugins with the Eclipse Foundation certificate, apart for nightly builds
 
* package several zips : runtime, SDK, examples, tests, all in one update site
 
* clean up the build directory at the end, unless there were errors
 
 
Athena makes use of these elements in the Releng project:
 
* [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/build.properties build.properties] : the most important configuration file : specifies the steps to do, the feature to build, the build dependencies, the Java paths, plus a few other parameters, some being optional.
 
* [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/maps the map files] : define where to get plugins and features from (CVS or SVN path) and which version to take for each one (head, tag, or branch).
 
* [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/testing.properties testing.properties] : defines the tests to run
 
*  [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/buildExtra.xml buildExtra.xml] : optional file containing ant targets that will be called by Athena at pre-defined points during the build. This is useful to do custom packaging, fetch extra dependencies, compile javadoc, launch code generators, etc.
 
  
 
== Promote ==
 
== Promote ==
Once the Hudson job is finished, the build results are found in <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/lastSuccessful/</code> (or <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/lastSuccessful/</code>).
 
 
 
The following entry was added to a crontab (<code>crontab -e</code>):
 
The following entry was added to a crontab (<code>crontab -e</code>):
  */5 * * * *  /opt/public/modeling/gmt/modisco/cronPromote.sh
+
  */5 * * * *  /opt/public/modeling/mdt/modisco/cronPromote.sh
 
So that the [[#cronPromote.sh|<code>cronPromote.sh</code>]] executes every five minutes on the build server.
 
So that the [[#cronPromote.sh|<code>cronPromote.sh</code>]] executes every five minutes on the build server.
  
This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build.
+
This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build with the date of a reference file touched at the end of each promote (<code>/opt/public/modeling/gmt/modisco/lastPromoteRefN</code> and <code>lastPromoteRefI</code>).
with the date of a reference file touched at the end of each promote (<code>/opt/public/modeling/gmt/modisco/lastPromoteRefN</code> and <code>lastPromoteRefI</code>).
+
  
If a new build is detected this way, it then promotes the result of this build by calling ant on [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote.xml the promote.xml of the Releng project], passing it the [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote-N.properties promote-N.properties] for a nightly build or [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote-I.properties promote-I.properties] for an integration build.
+
If a new build is detected this way, the script then promotes the result of this build.
  
 
The promote is not done directly from Hudson because it does not have the necessary rights.
 
The promote is not done directly from Hudson because it does not have the necessary rights.
  
The <code>promote-N.properties</code> and <code>promote-I.properties</code> define where to get the files from, where to put the archives and update site, and whether to add to or replace the existing contents.
+
The <code>cronPromote.sh</code> script logs everything it does to <code>/opt/public/modeling/mdt/modisco/log-cronPromote</code>, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in <code>/opt/public/modeling/mdt/modisco/promo_logs</code>.
 
+
The <code>cronPromote.sh</code> script logs everything it does to <code>/opt/public/modeling/gmt/modisco/log-cronPromote</code>, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in <code>/opt/public/modeling/gmt/modisco/promo_logs</code>.
+
  
 
Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.
 
Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.
  
== Additional information ==
+
== Hudson configuration ==
 +
Hudson saves its configuration in [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng.buckminster/serverConfiguration/configN.xml <code>/opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/config.xml</code>] for the nightly job and [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/serverConfiguration/configI.xml <code>/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml</code>] for the integration job.
 +
 
 +
 
 +
== Some help with ssh and scp ==
 
On Windows, use [http://www.cygwin.com/ Cygwin] for ssh and scp.
 
On Windows, use [http://www.cygwin.com/ Cygwin] for ssh and scp.
  
Line 73: Line 48:
  
 
Copying a file from the build server (example):
 
Copying a file from the build server (example):
  scp <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/config.xml .
+
  scp <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml .
  
 
Retrieving the last successful build from the build server:
 
Retrieving the last successful build from the build server:
  scp -r <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/lastSuccessful/archive/build/* .
+
  scp -r <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/lastSuccessful/archive/build/* .
  
 
Sending a file to the build server:
 
Sending a file to the build server:
Line 83: Line 58:
 
Sending a directory to the build server:
 
Sending a directory to the build server:
 
  scp -r directory <committerid>@build.eclipse.org:
 
  scp -r directory <committerid>@build.eclipse.org:
 
== Hudson configuration ==
 
Hudson saves its configuration in <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/config.xml</code> for the nightly job and <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/config.xml</code> for the integration job.
 
 
=== Hudson important parameters ===
 
 
'''Files to archive''' = <code>build/?20*/**</code> : archive the results of the build
 
 
'''Excludes''' (below "Files to archive", click on '''Advanced''') = <code>build/?20*/eclipse, build/?20*/eclipse/**, build/?20*/testing, build/?20*/testing/**, build/?20*/*-AllFeaturesAndPlugins-*.zip*, build/?20*/*-Master-*.zip*, build/?20*/noclean</code> : don't archive work directories, that are left over by Athena when the build failed.
 
 
'''Test report XMLs''' = <code>build/?20*/testresults/xml/*.xml</code> : the test results, that will be displayed in Hudson
 
 
'''Repository URL''' =
 
* <code>http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco</code> for the nighty build, so that a nightly build starts if the MoDisco repository changed
 
* <code>http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng</code> for the integration build, so that a build starts only if the releng project (that is, maps) changed
 
 
=== Hudson script ===
 
 
<pre>
 
# configuration
 
export PROJRELENGROOT='-projRelengRoot svn://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk'
 
export PROJRELENGPATH='-projRelengPath org.eclipse.gmt.modisco.releng'
 
promoteSignal=/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN
 
 
# parameters
 
#export SNAPSHOT="true"
 
 
# run the build
 
. /opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh
 
 
read nextBuildNumber < /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/nextBuildNumber
 
buildNumber=$((nextBuildNumber-1))
 
 
# Test if build succeeded. If not, then exit with error 1.
 
# Copy the log, because the commands appear in the grepped log...
 
tmpLog=`mktemp` || exit 1
 
cat /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/builds/$buildNumber/log > $tmpLog
 
grep "BUILD FAILED"  $tmpLog && exit 1
 
grep "BUILD SUCCESSFUL" $tmpLog || exit 1
 
 
#triggers promoting the build by a script waiting for the modification date of this file to change
 
touch $promoteSignal
 
</pre>
 
 
== cronPromote.sh ==
 
<pre>
 
#!/bin/bash
 
 
# This script compares the date of a signal file touched by the hudson build
 
# with the date of a reference file touched at the end of each promote.
 
# It then promotes the result of the build. The promote is not done from
 
# Hudson because it does not have the necessary rights.
 
 
lastPromoteFileN=/opt/public/modeling/gmt/modisco/lastPromoteRefN
 
lastPromoteFileI=/opt/public/modeling/gmt/modisco/lastPromoteRefI
 
promoteSignalN=/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN
 
promoteSignalI=/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalI
 
logFile=/opt/public/modeling/gmt/modisco/log-cronPromote
 
 
ANT=/opt/public/common/apache-ant-1.7.1/bin/ant
 
RELENG_DIR=/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace/build/org.eclipse.modisco.releng
 
PROMO_LOGS_DIR=/opt/public/modeling/gmt/modisco/promo_logs
 
BUILDS_DIR=/home/data/httpd/download.eclipse.org/modeling/gmt/modisco/downloads/drops/0.8.0
 
 
DATE=`date +%Y%m%d-%H%M`
 
 
# trim the log : keep about last 10 days of log
 
function trimLog {
 
        tail -n 3000 $logFile > ${logFile}Trimmed && mv ${logFile}Trimmed $logFile
 
}
 
 
 
function prune() {
 
        # prefix: N for nightlies, I for integration, ...
 
        kind=$1
 
        filesPath=$2
 
        nBuildsToKeep=$3
 
       
 
        allFiles=$(ls -trd $filesPath/N20*)
 
        nFiles=$(ls -1d $filesPath/N20* | wc -l)
 
        nToDelete=$((nFiles-nBuildsToKeep))
 
 
        if [ $nToDelete -gt 0 ]; then
 
                for file in $allFiles; do
 
                        echo "$DATE: deleting $file" >> $logFile
 
                        rm -rf $file
 
                        nToDelete=$((nToDelete-1))
 
                        if [ $nToDelete -le 0 ]; then break; fi
 
                done
 
        fi
 
}
 
 
 
 
if [ ! -e $lastPromoteFileN ]; then touch $lastPromoteFileN; fi
 
if [ ! -e $lastPromoteFileI ]; then touch $lastPromoteFileI; fi
 
if [ ! -e $promoteSignalN ]; then echo "$DATE: ERROR: $promoteSignalN not found" >> $logFile; exit 1; fi
 
if [ ! -e $promoteSignalI ]; then echo "$DATE: ERROR: $promoteSignalI not found" >> $logFile; exit 1; fi
 
 
signalDateN=`stat --format=%Y $promoteSignalN`
 
signalDateI=`stat --format=%Y $promoteSignalI`
 
lastPromoteDateN=`stat --format=%Y $lastPromoteFileN`
 
lastPromoteDateI=`stat --format=%Y $lastPromoteFileI`
 
 
if [ $signalDateN -gt $lastPromoteDateN ]; then
 
        echo "$DATE: removing old nightlies" >> $logFile
 
        prune N $BUILDS_DIR 4 
 
        echo "$DATE: publishing nightly build ..." >> $logFile
 
        mkdir -p $PROMO_LOGS_DIR
 
        $ANT -f $RELENG_DIR/promote.xml -Dpromote.properties=$RELENG_DIR/promote-N.properties 2>/dev/null 1> $PROMO_LOGS_DIR/cbi-modisco-nightly-promo-${DATE}.txt
 
        touch $lastPromoteFileN
 
        echo "$DATE: done" >> $logFile
 
        trimLog
 
fi
 
 
 
if [ $signalDateI -gt $lastPromoteDateI ]; then
 
        echo "$DATE: removing old integration builds" >> $logFile
 
        prune I $BUILDS_DIR 4 
 
        echo "$DATE: publishing integration build ..." >> $logFile
 
        mkdir -p $PROMO_LOGS_DIR
 
        $ANT -f $RELENG_DIR/promote.xml -Dpromote.properties=$RELENG_DIR/promote-I.properties 2>/dev/null 1> $PROMO_LOGS_DIR/cbi-modisco-integration-promo-${DATE}.txt
 
        touch $lastPromoteFileI
 
        echo "$DATE: done" >> $logFile
 
        trimLog
 
fi
 
</pre>
 
  
 
[[Category:MoDisco]]
 
[[Category:MoDisco]]
 
[[Category:Releng]]
 
[[Category:Releng]]
[[Category:Athena Common Build]]
 

Latest revision as of 04:55, 21 July 2010

This page describes the build process of the MoDisco component, from a build engineering standpoint. For help on merely using the build, see How to Use?.

(for old Athena build, see this page)

Build jobs

  • Everything starts from Hudson (see the nightly Hudson Job configuration [1]), which runs on the Eclipse build server (build.eclipse.org).
  • The build is started on a defined schedule (every 6 hours) for the nightly build, or once a week for the integration build, but can also be started manually.
  • For the nightly build, Hudson first exports the contents of the MoDisco SVN [2] to its Workspace (/opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/workspace on build1). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there.
  • The Hudson job is configured to:
    • Check out the releng project from $PROJRELENGROOT/org.eclipse.gmt.modisco.releng.buckminster (PROJRELENGROOT is a build parameter).
    • Remove the previous build results and work area
    • Create a build.properties by concatenating the modisco-build.properties from the releng project with properties set or found in the Hudson job.
    • Call Buckminster:
      • materializes the target platform and workspace using modisco.mspec from the releng project
      • builds the workspace (using Eclipse builders)
      • runs JUnit tests by using the org.eclipse.gmt.modisco.tests/AllTests.launch Eclipse launcher
      • builds the packed (with pack200) and optionally signed update site
    • Call ant on other.ant in the releng project, to create a JUnit report as HTML pages
    • zip the update site, and put that in another zip with the build log and test results
    • touch a file on the build server to trigger a promote script which is waiting for this file to change (cronPromote.sh in /opt/public/modeling/mdt/modisco)


Promote

The following entry was added to a crontab (crontab -e):

*/5 * * * *  /opt/public/modeling/mdt/modisco/cronPromote.sh

So that the cronPromote.sh executes every five minutes on the build server.

This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build with the date of a reference file touched at the end of each promote (/opt/public/modeling/gmt/modisco/lastPromoteRefN and lastPromoteRefI).

If a new build is detected this way, the script then promotes the result of this build.

The promote is not done directly from Hudson because it does not have the necessary rights.

The cronPromote.sh script logs everything it does to /opt/public/modeling/mdt/modisco/log-cronPromote, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in /opt/public/modeling/mdt/modisco/promo_logs.

Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.

Hudson configuration

Hudson saves its configuration in /opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/config.xml for the nightly job and /opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml for the integration job.


Some help with ssh and scp

On Windows, use Cygwin for ssh and scp.

Logging in to the build server:

ssh <committerid>@build.eclipse.org
<type password...>

Copying a file from the build server (example):

scp <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml .

Retrieving the last successful build from the build server:

scp -r <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/lastSuccessful/archive/build/* .

Sending a file to the build server:

scp archive.tgz <committerid>@build.eclipse.org:

Sending a directory to the build server:

scp -r directory <committerid>@build.eclipse.org:

Back to the top