Difference between revisions of "Common Build Infrastructure/Getting Started"

From Eclipsepedia

Jump to: navigation, search
(In Eclipse With AntRunner)
(Run A Build)
Line 53: Line 53:
  
 
== Run A Build ==
 
== Run A Build ==
 +
 +
* You can build in Eclipse using Ant and a build.xml script, or headlessly via commandline using start.sh (which in turn launches Eclipse's AntRunner too).
 +
 +
* You can have requirements in your workspace (o.e.releng.basebuilder, o.e.d.common.releng, o.e.myproject.releng), or in /opt/public/cbi/build/, or in /tmp/build/. As long as they're on disk or you specify where to put them, they'll be used or checked out then used. We've also provided [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/psf/?root=Technology_Project a few sample .pcf files] to help you share this configuration with your team.
 +
 +
* The SVN PDE plugin will be automatically downloaded and unpacked into o.e.releng.basebuilder if the thirdPartyDownloadLicenseAcceptance flag is properly set in the project's build.properties; this is a bit of a legal hack for making people accept that they need some non-EPL stuff to build from SVN sources. Alternatively, you can download this plugin yourself and add it to basebuilder by hand.
 +
 +
* Ant-Contrib is required to run the build. You can put this jar in one of 4 different places, at your discretion:
 +
 +
<taskdef resource="net/sf/antcontrib/antlib.xml">
 +
  <classpath>
 +
    <pathelement path="${classpath}" />
 +
    <pathelement location="${relengCommonBuilderDir}/lib/ant-contrib.jar" />
 +
    <pathelement location="${relengBuilderDir}/lib/ant-contrib.jar" />
 +
    <pathelement location="${thirdPartyJarsDir}/ant-contrib.jar" />
 +
    <pathelement location="/usr/share/java/ant-contrib.jar" />
 +
  </classpath>
 +
</taskdef>
 +
 +
 +
* You can build in, for example, /opt/public/cbi/build/tools/gef/downloads/drops/3.4.0/N200902260105/, or in /tmp/build/tools/gef/downloads/drops/3.4.0/N200902260105/ or in /tmp/build/N200902260105 - whatever you want you can do.
 +
 +
* You can build with Hudson - there are [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/hudson/?root=Technology_Project sample wrapper scripts for start.sh] to pass though Hudson parameters. You can either copy the whole script into the hudson config field, or if the script is elsewhere on the disk, you can simply point at it like this:
 +
 +
export BUILDTYPE="${BUILDTYPE}"
 +
export EXTRAFLAGS="${EXTRAFLAGS}"
 +
. /opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/gef-3.4.x-nightly.sh
  
 
===Commandline Headless Build===
 
===Commandline Headless Build===
Line 58: Line 85:
 
Builds are run though [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/tools/scripts/start.sh?root=Technology_Project&view=markup start.sh], which sets up configuration variables and invokes Eclipse's AntRunner to run the build. Testing is done by having this ant thread call into the shell to start a new Eclipse thread on a different display (or one day, a remote machine). As such, there is still some dependency on bash scripting for the build system to work. We plan to remove this dependency over time.
 
Builds are run though [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/tools/scripts/start.sh?root=Technology_Project&view=markup start.sh], which sets up configuration variables and invokes Eclipse's AntRunner to run the build. Testing is done by having this ant thread call into the shell to start a new Eclipse thread on a different display (or one day, a remote machine). As such, there is still some dependency on bash scripting for the build system to work. We plan to remove this dependency over time.
  
Here's a sample invokation of start.sh -- read the source for other examples, or check out the README.txt files in the [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/?root=Technology_Project other sample projects]:
+
Here's a sample invokation of start.sh -- run the script without any commandline options for other examples, or check out the README.txt files in the [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/?root=Technology_Project other sample projects]:
  
  cd /opt/public/cbi/build/org.eclipse.dash.common.releng_HEAD; \
+
  cd /opt/public/cbi/build/org.eclipse.dash.common.releng; \
  cvs up -Pd; cd tools/scripts;./start.sh -projectid tools.gef -version 3.4.0 -buildType N \
+
  cvs up -Pd; cd tools/scripts; ./start.sh -projectid tools.gef -version 3.4.0 \
 
   -projRelengRoot ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology' \
 
   -projRelengRoot ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology' \
   -projRelengPath 'org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng' \
+
   -projRelengPath 'org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng'
  -basebuilderBranch R35_M2 -javaHome /opt/public/common/ibm-java2-142 \
+
  -URL http://download.eclipse.org/eclipse/downloads/drops/R-3.4-200806172000/eclipse-SDK-3.4-linux-gtk.tar.gz \
+
 
   2>&1 | tee /opt/public/cbi/logs/buildlog_`date +%H%M%S`.txt
 
   2>&1 | tee /opt/public/cbi/logs/buildlog_`date +%H%M%S`.txt
  
* To run locally, see examples in [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/tools/scripts/start.sh?root=Technology_Project&view=markup start.sh].
+
* To run locally, check out org.eclipse.releng.basebuilder, org.eclipse.dash.common.releng, and the [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng/?root=Technology_Project org.eclipse.gef.releng project in the Dash Athena project], then run the Ant script build.xml. You can do all three checkouts using [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/psf/gef.psf?root=Technology_Project&view=markup this psf file]
  
 
===In Eclipse With AntRunner===
 
===In Eclipse With AntRunner===
 
<b style="color:red">'''''This information is subject to change.'''''</b>
 
  
 
You can create a build.xml in your project.releng folder, and use that to run a headless build from within your Eclipse workspace. [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng/build.xml?root=Technology_Project&revision=1.2.2.4&content-type=text%2Fplain&pathrev=nickb_runInEclipse Here's a sample].
 
You can create a build.xml in your project.releng folder, and use that to run a headless build from within your Eclipse workspace. [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng/build.xml?root=Technology_Project&revision=1.2.2.4&content-type=text%2Fplain&pathrev=nickb_runInEclipse Here's a sample].
 
Note also that this requires the "nickb_runInEclipse" branch of org.eclipse.dash.commonbuilder.releng, which has not yet been merged with HEAD.
 
  
 
===In Hudson===
 
===In Hudson===
 
<b style="color:red">'''''This information is subject to change.'''''</b>
 
  
 
The quickest way to get started w/ Hudson is to copy an existing configuration, because not everything in Hudson can [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/hudson/?root=Technology_Project be scripted and stored in VCS].
 
The quickest way to get started w/ Hudson is to copy an existing configuration, because not everything in Hudson can [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/hudson/?root=Technology_Project be scripted and stored in VCS].

Revision as of 02:19, 26 February 2009

This information is preliminary and subject to change. If you see an error or omission, please feel free to edit it.

Contents

Create Configuration Project

The first step is to create a project (often called a .releng project) to house your configuration settings. Here are some sample projects to get you started.

At its simplest, this is:

  • build.properties -- settings for the build
  • maps/build.map -- where to fetch sources (you can have as many .map files as you need)

However, you will likely need to feed in extra dependencies to the build, if it requires more than just the Eclipse "Classic" SDK to build (ie, JDT, PDE, CVS).

  • buildExtra.xml -- extra inputs and packaging steps

You will likely also want to run automated tests as part of your build. For this you'll need:

  • testing.properties -- properties specific to running tests

Finally, to control how your build is published, you'll need another properties file:

  • promote.properties -- properties specific to publishing builds

Check all these files into CVS or SVN.

Set Up Locally

To set up the local bootstrapping required to run a build, you can run this script:

Note that this script is currently only supported on Fedora 10. We're looking for testers to try it out on other linuxes and on Windows via cygwin. If you can't get it working, use it as a checklist for the manual steps required to get your local machine set up to build.

Once you've set up your environment, you should have something like this:

3rdPartyJars/ [libraries and 3rd party jars, including ant-contrib.jar]
downloads/    [dependencies, cached and reused]

scripts/      [build system scripts & tools]

org.eclipse.dash.common.releng_HEAD/ [a specific version of o.e.d.c.releng]
org.eclipse.dash.common.releng/      [a symlink to the above]

org.eclipse.releng.basebuilder_R35_M2/ [a specific version of o.e.r.bb]
org.eclipse.releng.basebuilder/        [a symlink to the above]

org.eclipse.pde.build.svn-1.0.1RC2.zip [required for making PDE understand SVN map files]

Run A Build

  • You can build in Eclipse using Ant and a build.xml script, or headlessly via commandline using start.sh (which in turn launches Eclipse's AntRunner too).
  • You can have requirements in your workspace (o.e.releng.basebuilder, o.e.d.common.releng, o.e.myproject.releng), or in /opt/public/cbi/build/, or in /tmp/build/. As long as they're on disk or you specify where to put them, they'll be used or checked out then used. We've also provided a few sample .pcf files to help you share this configuration with your team.
  • The SVN PDE plugin will be automatically downloaded and unpacked into o.e.releng.basebuilder if the thirdPartyDownloadLicenseAcceptance flag is properly set in the project's build.properties; this is a bit of a legal hack for making people accept that they need some non-EPL stuff to build from SVN sources. Alternatively, you can download this plugin yourself and add it to basebuilder by hand.
  • Ant-Contrib is required to run the build. You can put this jar in one of 4 different places, at your discretion:
<taskdef resource="net/sf/antcontrib/antlib.xml">
  <classpath>
    <pathelement path="${classpath}" />
    <pathelement location="${relengCommonBuilderDir}/lib/ant-contrib.jar" />
    <pathelement location="${relengBuilderDir}/lib/ant-contrib.jar" />
    <pathelement location="${thirdPartyJarsDir}/ant-contrib.jar" />
    <pathelement location="/usr/share/java/ant-contrib.jar" />
  </classpath>
</taskdef>


  • You can build in, for example, /opt/public/cbi/build/tools/gef/downloads/drops/3.4.0/N200902260105/, or in /tmp/build/tools/gef/downloads/drops/3.4.0/N200902260105/ or in /tmp/build/N200902260105 - whatever you want you can do.
  • You can build with Hudson - there are sample wrapper scripts for start.sh to pass though Hudson parameters. You can either copy the whole script into the hudson config field, or if the script is elsewhere on the disk, you can simply point at it like this:
export BUILDTYPE="${BUILDTYPE}"
export EXTRAFLAGS="${EXTRAFLAGS}"
. /opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/gef-3.4.x-nightly.sh

Commandline Headless Build

Builds are run though start.sh, which sets up configuration variables and invokes Eclipse's AntRunner to run the build. Testing is done by having this ant thread call into the shell to start a new Eclipse thread on a different display (or one day, a remote machine). As such, there is still some dependency on bash scripting for the build system to work. We plan to remove this dependency over time.

Here's a sample invokation of start.sh -- run the script without any commandline options for other examples, or check out the README.txt files in the other sample projects:

cd /opt/public/cbi/build/org.eclipse.dash.common.releng; \
cvs up -Pd; cd tools/scripts; ./start.sh -projectid tools.gef -version 3.4.0 \
 -projRelengRoot ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology' \
 -projRelengPath 'org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng'
  2>&1 | tee /opt/public/cbi/logs/buildlog_`date +%H%M%S`.txt

In Eclipse With AntRunner

You can create a build.xml in your project.releng folder, and use that to run a headless build from within your Eclipse workspace. Here's a sample.

In Hudson

The quickest way to get started w/ Hudson is to copy an existing configuration, because not everything in Hudson can be scripted and stored in VCS.

In the above configs, a pre-checked-out cache of basebuilder and common.releng are available in build.eclipse.org:/opt/public/cbi/build/ to speed up builds and to reduce CVS checkout churn.

  • As a developer working on the common.releng code, you will need to cvs up in there to ensure your checked in changes are pushed to the build system; however, bear in mind that your changes will be public immediately and anything you broke will be immediately obvious... so play nice (test locally first)!
  • As a user running builds using the common.releng code, you can choose to checkout or export from a given CVS tag for common.releng and basebuilder rather than using symlinks if you prefer to not live on the bleeding edge. When you want to move up to a new stable tag, you can clear your Hudson workspace and start fresh. In this case your startup script will need to be a little different.

Debug Build

Here are some general tips for finding log files and sifting the build's tea leaves:

  • Always log to a file, preferably a unique one. You can watch console output AND log to file (standard out and standard error) with this:
start.sh ... 2>&1 | tee /opt/public/cbi/logs/buildlog_`date +%H%M%S`.txt
  • You can also use this, which will dump a log in your build's folder for you:
start_logger.sh
  • Logs can be found in the build's folder like this:
find /path/to/build/folder -type f -name "*log" -o -name "*log.txt" -o -name "*log.zip"
  • Search logs for keywords like "ERROR" and "FAIL" to see what went wrong
  • Chances are if something really bad happened, it occurred in the first or last 100 lines of the log.

See Also