Jump to: navigation, search

Difference between revisions of "Common Build Infrastructure/Publishing"

(Example: VE Nightly and Integration Builds: update script to set JAVA_HOME and ANT_HOME)
 
(16 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
{{:DocumentationGuidelines/DraftHeader}}
 +
 
OK, so you've built your SDK or runtime zip, your examples, your docs, your tests, and a p2 repo / update site. Great. How do you get those bits out to the world?
 
OK, so you've built your SDK or runtime zip, your examples, your docs, your tests, and a p2 repo / update site. Great. How do you get those bits out to the world?
  
Well, for the simple case of "built on build.eclipse.org, published over NFS to download.eclipse.org", you can use a simple Ant script, run from your crontab on build.eclipse.org.
+
Well, for the simple case of "built on build.eclipse.org, published over NFS to download.eclipse.org", you can use a simple Ant script, run from your [http://en.wikipedia.org/wiki/Cron#crontab_syntax crontab] on build.eclipse.org.
 +
 
 +
# Copy sample promote.xml and promote.properties from org.eclipse.ve.releng or other Athana project.
 +
# Update the promote.properties to fit your needs.
 +
# Add entry in your crontab that will invoke the promote.xml Ant script.
  
 
=== Example: VE Nightly and Integration Builds ===
 
=== Example: VE Nightly and Integration Builds ===
Line 13: Line 19:
 
(To save time, create the Nightly job, then copy it to produce the Integration build, passing in slightly different parameters like <code>BUILDTYPE = N</code> or <code>BUILDTYPE = I</code>.)
 
(To save time, create the Nightly job, then copy it to produce the Integration build, passing in slightly different parameters like <code>BUILDTYPE = N</code> or <code>BUILDTYPE = I</code>.)
  
Next, create crontab entries which match the frequency of your N and I builds. In this example, the latest Nightly VE build will be published every day at 15h40 EST. The latest Integration will be published every Tuesday at 16h00 EST. Note that the promote.xml is identical, but the properties file differs.
+
Next, ssh to build.eclipse.org using your committer id and type <code>crontab -e</code> to create crontab entries which match the frequency of your N and I builds. In this example, the latest Nightly VE build will be published every day at 15h40 EST. The latest Integration will be published every Tuesday at 16h00 EST. Note that the promote.xml is identical, but the properties file differs.
  
 
  # daily N builds
 
  # daily N builds
  40 15 * * * /opt/public/common/apache-ant-1.7.1/bin/ant -f
+
  40 15 * * * JAVA_HOME=/shared/common/ibm-java-x86_64-60 ANT_HOME=/shared/common/apache-ant-1.7.1/ ant -f
 
   /opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-nightly-Ganymede/workspace/build/org.eclipse.ve.releng/promote.xml
 
   /opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-nightly-Ganymede/workspace/build/org.eclipse.ve.releng/promote.xml
 
     -Dpromote.properties=/opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-nightly-Ganymede/workspace/build/org.eclipse.ve.releng/promote-N.properties
 
     -Dpromote.properties=/opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-nightly-Ganymede/workspace/build/org.eclipse.ve.releng/promote-N.properties
Line 22: Line 28:
  
 
  # weekly I build
 
  # weekly I build
  00 16 * * 2 /opt/public/common/apache-ant-1.7.1/bin/ant -f
+
  00 16 * * 2 JAVA_HOME=/shared/common/ibm-java-x86_64-60 ANT_HOME=/shared/common/apache-ant-1.7.1/ ant -f
 
   /opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-integration-Ganymede/workspace/build/org.eclipse.ve.releng/promote.xml
 
   /opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-integration-Ganymede/workspace/build/org.eclipse.ve.releng/promote.xml
 
     -Dpromote.properties=/opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-integration-Ganymede/workspace/build/org.eclipse.ve.releng/promote-I.properties
 
     -Dpromote.properties=/opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-integration-Ganymede/workspace/build/org.eclipse.ve.releng/promote-I.properties
Line 30: Line 36:
  
 
In addition to publishing your zips, the promote.xml script also unpacks your Update zip into the path you specify, so you will have a daily/weekly up to date update site.
 
In addition to publishing your zips, the promote.xml script also unpacks your Update zip into the path you specify, so you will have a daily/weekly up to date update site.
 +
 +
===Specifying Source Directory===
 +
 +
Your promote.*.properties file needs to define the parent folder in which to find a build folder. For example:
 +
 +
sourceZipsDir=/tmp/build
 +
 +
if building locally, or
 +
 +
sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/'''HUDSON_JOB_NAME'''/lastSuccessfulBuild/archive/build
 +
 +
if building on build.eclipse.org, where '''HUDSON_JOB_NAME''' = cbi-ve-1.5-Helios, cbi-tmf-xtext-0.7, etc.
 +
 +
However, since lastSuccessfulBuild or lastStableBuild symlinks may not work with the latest version of Hudson, a better way is needed ({{bug|302170}}). For a <code>sourceZipsDir</code> such as:
 +
 +
sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/'''HUDSON_JOB_NAME'''/
 +
 +
Athena will attempt to find your build dir based on three well-known patterns, for all <code>buildTypes</code> specified (eg., N, I, M, S and/or R):
 +
 +
  /opt/users/hudsonbuild/.hudson/jobs/'''HUDSON_JOB_NAME'''/N201001031122
 +
  /opt/users/hudsonbuild/.hudson/jobs/'''HUDSON_JOB_NAME'''/2010-01-03_11-22-33/archive/build/N201001031122
 +
  /opt/users/hudsonbuild/.hudson/jobs/'''HUDSON_JOB_NAME'''/workspace/build/N201001031122
 +
 +
If more than one match is found, the newest directory by name (most recent build) will be selected.
 +
 +
As a result, if your lastest build contains test errors/failures, you will need to ensure that you do not accidentally promote that build (unless you want to do so). To avoid this, simply set a more explicit <code>sourceZipsDir</code> such as:
 +
 +
sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/'''HUDSON_JOB_NAME'''/2010-01-03_11-22-33/archive/build
 +
 +
If you always want to publish your latest build, use:
 +
 +
sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/'''HUDSON_JOB_NAME'''/workspace/build
  
 
===References===
 
===References===
  
 +
* {{bug|273302}}, [https://bugs.eclipse.org/bugs/show_bug.cgi?id=270849#c37 bug 270849], {{bug|284593}}, {{bug|302170}}
 +
* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.common.releng/promote-wrapper.xml?root=Technology_Project per-project promote.xml wrapper]
 
* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.common.releng/promote.xml?root=Technology_Project common promote.xml]
 
* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.common.releng/promote.xml?root=Technology_Project common promote.xml]
 
* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/?root=Tools_Project org.eclipse.ve.releng]
 
* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/?root=Tools_Project org.eclipse.ve.releng]
 +
:* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/promote.xml?root=Tools_Project&view=markup per-project promote.xml wrapper]
 
:* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/promote-I.properties?root=Tools_Project&view=markup promote-I.properties]
 
:* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/promote-I.properties?root=Tools_Project&view=markup promote-I.properties]
 
:* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/promote-N.properties?root=Tools_Project&view=markup promote-N.properties]
 
:* [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/promote-N.properties?root=Tools_Project&view=markup promote-N.properties]
 +
 +
[[Category:Athena Common Build]]
 +
[[Category:Draft_Documentation]]

Latest revision as of 01:17, 14 December 2010

Warning2.png
Draft Content
This page is currently under construction. Community members are encouraged to maintain the page, and make sure the information is accurate.


OK, so you've built your SDK or runtime zip, your examples, your docs, your tests, and a p2 repo / update site. Great. How do you get those bits out to the world?

Well, for the simple case of "built on build.eclipse.org, published over NFS to download.eclipse.org", you can use a simple Ant script, run from your crontab on build.eclipse.org.

  1. Copy sample promote.xml and promote.properties from org.eclipse.ve.releng or other Athana project.
  2. Update the promote.properties to fit your needs.
  3. Add entry in your crontab that will invoke the promote.xml Ant script.

Example: VE Nightly and Integration Builds

Hudson-Config-Parameterized-Build.png

Say you want to run Nightlies every day, and a single Integration build once a week. First, set up two jobs in Hudson:

  1. https://build.eclipse.org/hudson/view/Athena%20CBI/job/cbi-ve-1.4-nightly-Ganymede/
  2. https://build.eclipse.org/hudson/view/Athena%20CBI/job/cbi-ve-1.4-integration-Ganymede/

(To save time, create the Nightly job, then copy it to produce the Integration build, passing in slightly different parameters like BUILDTYPE = N or BUILDTYPE = I.)

Next, ssh to build.eclipse.org using your committer id and type crontab -e to create crontab entries which match the frequency of your N and I builds. In this example, the latest Nightly VE build will be published every day at 15h40 EST. The latest Integration will be published every Tuesday at 16h00 EST. Note that the promote.xml is identical, but the properties file differs.

# daily N builds
40 15 * * *  JAVA_HOME=/shared/common/ibm-java-x86_64-60 ANT_HOME=/shared/common/apache-ant-1.7.1/ ant -f
  /opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-nightly-Ganymede/workspace/build/org.eclipse.ve.releng/promote.xml
    -Dpromote.properties=/opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-nightly-Ganymede/workspace/build/org.eclipse.ve.releng/promote-N.properties
      2>/dev/null 1> ~/promo_logs/cbi-ve-1.4-nightly-Ganymede-promo.txt
# weekly I build
00 16 * * 2 JAVA_HOME=/shared/common/ibm-java-x86_64-60 ANT_HOME=/shared/common/apache-ant-1.7.1/ ant -f
  /opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-integration-Ganymede/workspace/build/org.eclipse.ve.releng/promote.xml
    -Dpromote.properties=/opt/users/hudsonbuild/.hudson/jobs/cbi-ve-1.4-integration-Ganymede/workspace/build/org.eclipse.ve.releng/promote-I.properties
      2>/dev/null 1> ~/promo_logs/cbi-ve-1.4-integration-Ganymede-promo.txt

In the N build properties file, we use "add" mode to incrementally publish N builds as they happen. In the I properties file, we use "replace" mode to scrub any accumulated builds and publish only the latest I build. So, once a week, we automatically purge the old garbage and get a fresh build.

In addition to publishing your zips, the promote.xml script also unpacks your Update zip into the path you specify, so you will have a daily/weekly up to date update site.

Specifying Source Directory

Your promote.*.properties file needs to define the parent folder in which to find a build folder. For example:

sourceZipsDir=/tmp/build

if building locally, or

sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/HUDSON_JOB_NAME/lastSuccessfulBuild/archive/build

if building on build.eclipse.org, where HUDSON_JOB_NAME = cbi-ve-1.5-Helios, cbi-tmf-xtext-0.7, etc.

However, since lastSuccessfulBuild or lastStableBuild symlinks may not work with the latest version of Hudson, a better way is needed (bug 302170). For a sourceZipsDir such as:

sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/HUDSON_JOB_NAME/

Athena will attempt to find your build dir based on three well-known patterns, for all buildTypes specified (eg., N, I, M, S and/or R):

 /opt/users/hudsonbuild/.hudson/jobs/HUDSON_JOB_NAME/N201001031122
 /opt/users/hudsonbuild/.hudson/jobs/HUDSON_JOB_NAME/2010-01-03_11-22-33/archive/build/N201001031122 
 /opt/users/hudsonbuild/.hudson/jobs/HUDSON_JOB_NAME/workspace/build/N201001031122

If more than one match is found, the newest directory by name (most recent build) will be selected.

As a result, if your lastest build contains test errors/failures, you will need to ensure that you do not accidentally promote that build (unless you want to do so). To avoid this, simply set a more explicit sourceZipsDir such as:

sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/HUDSON_JOB_NAME/2010-01-03_11-22-33/archive/build

If you always want to publish your latest build, use:

sourceZipsDir=/opt/users/hudsonbuild/.hudson/jobs/HUDSON_JOB_NAME/workspace/build

References