Jump to: navigation, search

Difference between revisions of "MoDisco/Releng/How it Works"

(New page: This page describes the build process of the [http://www.eclipse.org/gmt/modisco/ MoDisco] component, from a build engineering standpoint. For help on merely using the build, see [[MoDisco...)
 
 
(22 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page describes the build process of the [http://www.eclipse.org/gmt/modisco/ MoDisco] component, from a build engineering standpoint. For help on merely using the build, see [[MoDisco/Releng/How to Use|How to Use?]].
+
This page describes the build process of the [http://www.eclipse.org/MoDisco/ MoDisco] component, from a build engineering standpoint. For help on merely using the build, see [[MoDisco/Releng/How to Use|How to Use?]].
  
== Hudson ==
+
<span style="color:#606060">(for old Athena build, see [[MoDisco/Releng/Athena/How_it_Works|this page]])</span>
=== Nightly builds ===
+
* Everything starts from Hudson (see the nightly Hudson Job configuration [https://build.eclipse.org/hudson/job/cbi-modisco-nightly/configure]).
+
* The build is started on a defined schedule (every 6 hours), but can also be started manually.
+
* Hudson first exports the contents of the MoDisco SVN [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco] to its Workspace (<code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/workspace</code>). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there.
+
* The Hudson job is configured to execute a shell script that:
+
** imports and runs the </code>/opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh</code> script
+
** reads the next build number from <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/nextBuildNumber</code>, and decrements it to find the current build number
+
** looks at the log <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/builds/$buildNumber/log</code> to determine if the build failed or succeeded
+
** exits with an error code if the build failed
+
** touches the <code>/opt/public/modeling/gmt/modisco/modiscoBuildPromoteSignalN</code> file if the build succeeded
+
  
== Promote ==
+
== Build jobs ==
Once the Hudson job is finished, the build results are found in <code>/opt/users/hudsonbuild/.hudson/jobs/cbi-modisco-nightly/lastSuccessful/</code>.
+
* Everything starts from Hudson (see the nightly Hudson Job configuration [https://build.eclipse.org/hudson/job/modisco-nightly/configure]), which runs on the Eclipse build server (build.eclipse.org).
 +
* The build is started on a defined schedule (every 6 hours) for the nightly build, or once a week for the integration build, but can also be started manually.
 +
* For the nightly build, Hudson first exports the contents of the MoDisco SVN [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.modisco] to its Workspace (<code>/opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/workspace</code> on build1). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there.
 +
* The Hudson job is configured to:
 +
** Check out the releng project from <code>$PROJRELENGROOT/org.eclipse.gmt.modisco.releng.buckminster</code> (PROJRELENGROOT is a build parameter).
 +
** Remove the previous build results and work area
 +
** Create a <code>build.properties</code> by concatenating the <code>modisco-build.properties</code> from the releng project with properties set or found in the Hudson job.
 +
** Call Buckminster:
 +
*** materializes the target platform and workspace using <code>modisco.mspec</code> from the releng project
 +
*** builds the workspace (using Eclipse builders)
 +
*** runs JUnit tests by using the <code>org.eclipse.gmt.modisco.tests/AllTests.launch</code> Eclipse launcher
 +
*** builds the packed (with pack200) and optionally signed update site
 +
** Call ant on <code>other.ant</code> in the releng project, to create a JUnit report as HTML pages
 +
** zip the update site, and put that in another zip with the build log and test results
 +
** touch a file on the build server to trigger a promote script which is waiting for this file to change (<code>cronPromote.sh</code> in <code>/opt/public/modeling/mdt/modisco</code>)
  
 +
 +
== Promote ==
 
The following entry was added to a crontab (<code>crontab -e</code>):
 
The following entry was added to a crontab (<code>crontab -e</code>):
  */5 * * * *  /opt/public/modeling/gmt/modisco/cronPromote.sh
+
  */5 * * * *  /opt/public/modeling/mdt/modisco/cronPromote.sh
So that the <code>cronPromote.sh</code> executes every five minutes on the build server.
+
So that the [[#cronPromote.sh|<code>cronPromote.sh</code>]] executes every five minutes on the build server.
  
This script compares the date of the signal file (modiscoBuildPromoteSignalN) touched by the Hudson script at the end of each build
+
This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build with the date of a reference file touched at the end of each promote (<code>/opt/public/modeling/gmt/modisco/lastPromoteRefN</code> and <code>lastPromoteRefI</code>).
with the date of a reference file touched at the end of each promote (<code>/opt/public/modeling/gmt/modisco/lastPromoteRefN</code>).
+
  
It then promotes the result of the build by calling ant on [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote.xml the promote.xml of the Releng project], passing it the [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote-N.properties promote-N.properties] for a nightly build or [http://dev.eclipse.org/svnroot/modeling/org.eclipse.gmt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/promote-N.properties promote-I.properties] for an integration build.  
+
If a new build is detected this way, the script then promotes the result of this build.
  
 
The promote is not done directly from Hudson because it does not have the necessary rights.
 
The promote is not done directly from Hudson because it does not have the necessary rights.
  
The <code>cronPromote.sh</code> script logs everything it does to <code>/opt/public/modeling/gmt/modisco/log-cronPromote</code>, and trims this log file regularly to keep about the last 10 days of log.
+
The <code>cronPromote.sh</code> script logs everything it does to <code>/opt/public/modeling/mdt/modisco/log-cronPromote</code>, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in <code>/opt/public/modeling/mdt/modisco/promo_logs</code>.
  
 
Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.
 
Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.
 +
 +
== Hudson configuration ==
 +
Hudson saves its configuration in [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng.buckminster/serverConfiguration/configN.xml <code>/opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/config.xml</code>] for the nightly job and [http://dev.eclipse.org/svnroot/modeling/org.eclipse.mdt.modisco/releng/trunk/org.eclipse.gmt.modisco.releng/serverConfiguration/configI.xml <code>/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml</code>] for the integration job.
 +
 +
 +
== Some help with ssh and scp ==
 +
On Windows, use [http://www.cygwin.com/ Cygwin] for ssh and scp.
 +
 +
Logging in to the build server:
 +
ssh <committerid>@build.eclipse.org
 +
<type password...>
 +
 +
Copying a file from the build server (example):
 +
scp <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml .
 +
 +
Retrieving the last successful build from the build server:
 +
scp -r <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/lastSuccessful/archive/build/* .
 +
 +
Sending a file to the build server:
 +
scp archive.tgz <committerid>@build.eclipse.org:
 +
 +
Sending a directory to the build server:
 +
scp -r directory <committerid>@build.eclipse.org:
 +
 +
[[Category:MoDisco]]
 +
[[Category:Releng]]

Latest revision as of 03:55, 21 July 2010

This page describes the build process of the MoDisco component, from a build engineering standpoint. For help on merely using the build, see How to Use?.

(for old Athena build, see this page)

Build jobs

  • Everything starts from Hudson (see the nightly Hudson Job configuration [1]), which runs on the Eclipse build server (build.eclipse.org).
  • The build is started on a defined schedule (every 6 hours) for the nightly build, or once a week for the integration build, but can also be started manually.
  • For the nightly build, Hudson first exports the contents of the MoDisco SVN [2] to its Workspace (/opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/workspace on build1). If the build was started automatically and nothing changed on the SVN since the last build, then the build job stops there.
  • The Hudson job is configured to:
    • Check out the releng project from $PROJRELENGROOT/org.eclipse.gmt.modisco.releng.buckminster (PROJRELENGROOT is a build parameter).
    • Remove the previous build results and work area
    • Create a build.properties by concatenating the modisco-build.properties from the releng project with properties set or found in the Hudson job.
    • Call Buckminster:
      • materializes the target platform and workspace using modisco.mspec from the releng project
      • builds the workspace (using Eclipse builders)
      • runs JUnit tests by using the org.eclipse.gmt.modisco.tests/AllTests.launch Eclipse launcher
      • builds the packed (with pack200) and optionally signed update site
    • Call ant on other.ant in the releng project, to create a JUnit report as HTML pages
    • zip the update site, and put that in another zip with the build log and test results
    • touch a file on the build server to trigger a promote script which is waiting for this file to change (cronPromote.sh in /opt/public/modeling/mdt/modisco)


Promote

The following entry was added to a crontab (crontab -e):

*/5 * * * *  /opt/public/modeling/mdt/modisco/cronPromote.sh

So that the cronPromote.sh executes every five minutes on the build server.

This script compares the date of the signal files (modiscoBuildPromoteSignalN and modiscoBuildPromoteSignalI) touched by the Hudson script at the end of each build with the date of a reference file touched at the end of each promote (/opt/public/modeling/gmt/modisco/lastPromoteRefN and lastPromoteRefI).

If a new build is detected this way, the script then promotes the result of this build.

The promote is not done directly from Hudson because it does not have the necessary rights.

The cronPromote.sh script logs everything it does to /opt/public/modeling/mdt/modisco/log-cronPromote, and trims this log file regularly to keep about the last 10 days of log. It also logs the ant output in a file in /opt/public/modeling/mdt/modisco/promo_logs.

Additionally, it removes old nightly and integration builds on the download server, keeping the last 5.

Hudson configuration

Hudson saves its configuration in /opt/users/hudsonbuild/.hudson/jobs/modisco-nightly/config.xml for the nightly job and /opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml for the integration job.


Some help with ssh and scp

On Windows, use Cygwin for ssh and scp.

Logging in to the build server:

ssh <committerid>@build.eclipse.org
<type password...>

Copying a file from the build server (example):

scp <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/config.xml .

Retrieving the last successful build from the build server:

scp -r <committerid>@build.eclipse.org:/opt/users/hudsonbuild/.hudson/jobs/modisco-integration/lastSuccessful/archive/build/* .

Sending a file to the build server:

scp archive.tgz <committerid>@build.eclipse.org:

Sending a directory to the build server:

scp -r directory <committerid>@build.eclipse.org: