Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "MoDisco/Releng/How it Works"
| Line 2: | Line 2: | ||
== Hudson == | == Hudson == | ||
| − | === Nightly builds === | + | === Nightly builds job (cbi-modisco-nightly) === |
* Everything starts from Hudson (see the nightly Hudson Job configuration [https://build.eclipse.org/hudson/job/cbi-modisco-nightly/configure]), which runs on the Eclipse build server (build.eclipse.org). | * Everything starts from Hudson (see the nightly Hudson Job configuration [https://build.eclipse.org/hudson/job/cbi-modisco-nightly/configure]), which runs on the Eclipse build server (build.eclipse.org). | ||
* The build is started on a defined schedule (every 6 hours), but can also be started manually. | * The build is started on a defined schedule (every 6 hours), but can also be started manually. | ||
| Line 12: | Line 12: | ||
** exits with an error code if the build failed | ** exits with an error code if the build failed | ||
** touches the <code>/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN</code> file if the build succeeded | ** touches the <code>/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN</code> file if the build succeeded | ||
| + | |||
| + | === Integration builds job (cbi-modisco-integration) === | ||
| + | A little different from the nightly build job (apart from the file paths): | ||
| + | |||
| + | * Configuration : [https://build.eclipse.org/hudson/job/cbi-modisco-nightly/configure] | ||
| + | * The build is started once a week | ||
| + | * Hudson first exports the contents of the Releng project [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng] to its Workspace (<code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/workspace</code>). If the build was started automatically and nothing changed in the Releng project since the last build, then the build job stops there. This allows the integration job to start if the map files have changed. | ||
| + | * touches the <code>/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalI</code> file if the build succeeded | ||
== Promote == | == Promote == | ||
| − | Once the Hudson job is finished, the build results are found in <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/lastSuccessful/</code>. | + | Once the Hudson job is finished, the build results are found in <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/lastSuccessful/</code> (or <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/lastSuccessful/</code>). |
The following entry was added to a crontab (<code>crontab -e</code>): | The following entry was added to a crontab (<code>crontab -e</code>): | ||
| Line 20: | Line 28: | ||
So that the <code>cronPromote.sh</code> executes every five minutes on the build server. | So that the <code>cronPromote.sh</code> executes every five minutes on the build server. | ||
| − | This script compares the date of the signal | + | This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build. |
| − | with the date of a reference file touched at the end of each promote (<code>/opt/public/modeling/gmt/modisco/lastPromoteRefN</code>). | + | with the date of a reference file touched at the end of each promote (<code>/opt/public/modeling/gmt/modisco/lastPromoteRefN</code> and <code>lastPromoteRefI</code>). |
| − | It then promotes the result of the build by calling ant on [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote.xml the promote.xml of the Releng project], passing it the [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote-N.properties promote-N.properties] for a nightly build or [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote- | + | It then promotes the result of the build by calling ant on [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote.xml the promote.xml of the Releng project], passing it the [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote-N.properties promote-N.properties] for a nightly build or [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote-I.properties promote-I.properties] for an integration build. |
The promote is not done directly from Hudson because it does not have the necessary rights. | The promote is not done directly from Hudson because it does not have the necessary rights. | ||
| + | |||
| + | The <code>promote-N.properties</code> and <code>promote-I.properties</code> define where to get the files from, where to put the archives and update site, and whether to add to or replace the existing contents. | ||
The <code>cronPromote.sh</code> script logs everything it does to <code>/opt/public/modeling/gmt/modisco/log-cronPromote</code>, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in <code>/opt/public/modeling/gmt/modisco/promo_logs</code>. | The <code>cronPromote.sh</code> script logs everything it does to <code>/opt/public/modeling/gmt/modisco/log-cronPromote</code>, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in <code>/opt/public/modeling/gmt/modisco/promo_logs</code>. | ||
Revision as of 08:37, 9 October 2009
This page describes the build process of the MoDisco component, from a build engineering standpoint. For help on merely using the build, see How to Use?.
Contents
Hudson
Nightly builds job (cbi-modisco-nightly)
- Everything starts from Hudson (see the nightly Hudson Job configuration [1]), which runs on the Eclipse build server (build.eclipse.org).
- The build is started on a defined schedule (every 6 hours), but can also be started manually.
- Hudson first exports the contents of the MoDisco SVN [2] to its Workspace (
/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there. - The Hudson job is configured to execute a shell script that:
- imports and runs the </code>/opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh</code> script
- reads the next build number from
/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/nextBuildNumber, and decrements it to find the current build number - looks at the log
/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/builds/$buildNumber/logto determine if the build failed or succeeded - exits with an error code if the build failed
- touches the
/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalNfile if the build succeeded
Integration builds job (cbi-modisco-integration)
A little different from the nightly build job (apart from the file paths):
- Configuration : [3]
- The build is started once a week
- Hudson first exports the contents of the Releng project [4] to its Workspace (
/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/workspace). If the build was started automatically and nothing changed in the Releng project since the last build, then the build job stops there. This allows the integration job to start if the map files have changed. - touches the
/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalIfile if the build succeeded
Promote
Once the Hudson job is finished, the build results are found in /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/lastSuccessful/ (or /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/lastSuccessful/).
The following entry was added to a crontab (crontab -e):
*/5 * * * * /opt/public/modeling/gmt/modisco/cronPromote.sh
So that the cronPromote.sh executes every five minutes on the build server.
This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build.
with the date of a reference file touched at the end of each promote (/opt/public/modeling/gmt/modisco/lastPromoteRefN and lastPromoteRefI).
It then promotes the result of the build by calling ant on the promote.xml of the Releng project, passing it the promote-N.properties for a nightly build or promote-I.properties for an integration build.
The promote is not done directly from Hudson because it does not have the necessary rights.
The promote-N.properties and promote-I.properties define where to get the files from, where to put the archives and update site, and whether to add to or replace the existing contents.
The cronPromote.sh script logs everything it does to /opt/public/modeling/gmt/modisco/log-cronPromote, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in /opt/public/modeling/gmt/modisco/promo_logs.
Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.
Additional information
On Windows, use [www.cygwin.com/ Cygwin] for ssh and scp.
Logging in to the build server:
ssh <committerid>@build.eclipse.org <type password...>
Copying a file from the build server (example):
scp <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/config.xml .
Retrieving the last successful build from the build server:
scp -r <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/lastSuccessful/archive/build/* .
Sending a file to the build server:
scp archive.tgz <committerid>@build.eclipse.org:
Sending a directory to the build server:
scp -r directory <committerid>@build.eclipse.org:
Hudson configuration
Hudson saves its configuration in /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/config.xml for the nightly job and /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-integration/config.xml for the integration job.
Hudson important parameters
Files to archive = build/?20*/** : archive the results of the build
Excludes (below "Files to archive", click on Advanced) = build/?20*/eclipse, build/?20*/eclipse/**, build/?20*/testing, build/?20*/testing/**, build/?20*/*-AllFeaturesAndPlugins-*.zip*, build/?20*/*-Master-*.zip*, build/?20*/noclean : don't archive work directories, that are leftover by Athena when the build failed.
Test report XMLs = build/?20*/testresults/xml/*.xml : the test results, that will be displayed in Hudson
Repository URL =
-
http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modiscofor the nighty build, so that a nightly build starts if the MoDisco repository changed -
http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.relengfor the integration build, so that a build starts only if the releng project (that is, maps) changed
Hudson script
# configuration export PROJRELENGROOT='-projRelengRoot svn://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk' export PROJRELENGPATH='-projRelengPath org.eclipse.gmt.modisco.releng' promoteSignal=/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN # parameters #export SNAPSHOT="true" # run the build . /opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh read nextBuildNumber < /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/nextBuildNumber buildNumber=$((nextBuildNumber-1)) # Test if build succeeded. If not, then exit with error 1. # Copy the log, because the commands appear in the grepped log... tmpLog=`mktemp` || exit 1 cat /opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/builds/$buildNumber/log > $tmpLog grep "BUILD FAILED" $tmpLog && exit 1 grep "BUILD SUCCESSFUL" $tmpLog || exit 1 #triggers promoting the build by a script waiting for the modification date of this file to change touch $promoteSignal