Jump to: navigation, search

Difference between revisions of "Orbit/Builds"

(Building Orbit In Your Workspace)
m (committer area link corrected)
 
(31 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
This page is the starting point for information on Orbit builds.  
 
This page is the starting point for information on Orbit builds.  
  
Orbit builds are a bit different from others, in that nothing is really "built" in the traditional since of determining dependencies and compiling. The builds are more of a "copy and assemble" operation. One reason the builds are still very important, though, is that the resulting jars are "conditioned" for packing. These means that as others use the bundles, they are assured of getting the exact same bits, whether those bits come from a downloaded bundle, some assembled zip file, or some update site.  
+
Orbit builds are a bit different from others, in that nothing is really "built" in the traditional sense of determining dependencies and compiling. The builds are more of a "copy and assemble" operation. One reason the builds are still very important, though, is that the resulting jars are "conditioned" for packing. These means that as others use the bundles, they are assured of getting the exact same bits, whether those bits come from a downloaded bundle, some assembled zip file, or some update site.  
  
 
== Orbit Builds for Consumers ==  
 
== Orbit Builds for Consumers ==  
Line 11: Line 11:
 
http://download.eclipse.org/tools/orbit/committers/
 
http://download.eclipse.org/tools/orbit/committers/
  
Once committers are happy with the bundles produced, the bundles in the
+
See [[Orbit/Promotion%2C_Release%2C_and_Retention_Policies]]
committers area will be moved to the downloads area. But, naturally,
+
consumers are encouraged to use "early" versions from the committers area, and
+
be sure the bundles work as they expect.
+
  
The bundles are made available in two forms. First, all in one big zip file. Second, as individual bundles. The individual bundles are either a jar file, or a zip file. The jar files are ready to use as-is. That is, they are intended to be used in the jarred form. The zip files are bundles that are intended to be unzipped before use.  
+
The bundles are made available in several forms;
 +
: '''as p2 repositories.''' The repository can be used in batch builds, or scripts that use p2 operations. Or, for developers, you can use the Orbit repository in your PDE target runtime, and specify which bundle(s) you want to take advantage of.
 +
: '''downloadable one-by-one.''' The bundles/jars can be downloaded one by one, and placed in dropins folder, for example, but this is not the recommended way. Care is needed to know if the bundle should be unzipped as a directory, or left in jar format, for example.  
  
There are no features from Orbit, for consumers to use. The goal is that these bundles can be included in other project's features as needed. While this results in some duplication in projects zips or update sites, since all projects use exactly the same bundles from Orbit, there should be no duplication once installed.  
+
There are no features from Orbit, for consumers to use or refer to in their own features. The intent is that the bundles can be included in other project's features and redistributed by those other projects (after they submit appropriate CQs, of course). While this results in some duplication in the various projects zips or update sites, since all projects use exactly the same bundles from Orbit, there should be no duplication once installed. For this reason, it is important that for new development/releases, that all projects use the very latest Orbit deliveries so that these common bundles use exactly the version and qualifier. 
  
There's been some discussion on the orbit-dev mailing list on methods to include bundles in consumers features. One way is to include all the bundles in the "baseLocation" of PDE's build, and then they can be referred to in features and are included in packaged zips and update sites as needed (or determined) by PDE build. Another technique is to use the feature custom callback mechanism to copy the bundle to the directory being built or assembled.
+
There are a couple of methods to include bundles in consumer's features. One way is to include all the bundles in the "baseLocation" of PDE's build, and then they can be referred to in features and are included in packaged zips and update sites as needed (or determined) by PDE build. Another technique is to use the feature custom callback mechanism to copy the bundle to the directory being built or assembled.
  
For consumers who wish to fetch Orbit bundles as part of their build process, the Orbit build produces a map file containing URL GET entries which point directly to the bundles in Orbit. These entries should replace the CVS GET entry currently in the consumer's map files with the new ones. For convenience, you can also just remove the entries from your map files and include the whole Orbit map file in your build. (since the build is feature-driven the extra entries in the map file will be ignored)
+
For consumers to fetch Orbit bundles as part of their build process, and include in the "baseLocation", and ultimately their own features, the easiest way is to use the map files produced by the Orbit build process. 
 +
 
 +
 
 +
:'''CVS-TEXT style map files [deprecated].''' Orbit produces a map file, named directory.txt, containing CVS fetch entries which point directly to the versioned bundles in Orbit's repository. [This style of map file should be considered deprecated; we recommend the p2-style be used instead. In fact, as long as there is no "anonymous" access to Orbit's CVS repo, these maps are not usable, as is. For us to use our maps on build.eclipse.org, we use :local: instead of :pserver:anonymous:@dev.eclipse.org: (the replacement being done automatically during our build). Therefore, if anyone DID insist on using them, you'd have to do a similar replacement, either to ":local", if on build.eclipse.org, or to :ext:<committerid>:@dev.eclipse.org if you are committer on a machine besides build.eclipse.org.]
 +
 
 +
:'''GET-HTTP style map files [deprecated].''' Orbit produces a map file containing URL GET entries which point directly to the bundles in Orbit. These entries should replace the CVS GET entry currently in the consumer's map files with the new ones. For convenience, you can also just remove the entries from your map files and include the whole Orbit map file in your build, since the build is feature-diven the extra entries in the map file will be ignored. [This style of map file should be considered deprecated; we recommend the p2-style be used instead.]
 +
 
 +
:'''P2 style map files.''' Orbit also map files in a "p2 style". In this case, the PDE build knows how to read the map entries and use p2 to fetch the bundles from the orbit build repository. To do this, the only change in build scripts, from other style map files, is to specify the 'transformedRepoLocation' property, say in the overall ''build.properties'' file such as below (where ''${buildDirectory}'' is an already required location property. And ... not sure required ... but also might need ''<mkdir ${transformedRepoLocation}/>'' in scripts, to make sure that 'transfromedRepo' directory exists before PDE build needs it. 
 +
 
 +
transformedRepoLocation=${buildDirectory}/transformedRepo
 +
 
 +
For more information about specifying bundles to be included in features, see  [[Easy_Bake_Builds_with_Orbit_Bundles#Include_Orbit_bundles_into_a_feature| Include Orbit bundles into a feature]].
  
 
== Orbit Builds for Orbit Committers ==
 
== Orbit Builds for Orbit Committers ==
Line 31: Line 41:
 
in their own features. But, we'd still like to use the PDE build process, as much as possible, so  
 
in their own features. But, we'd still like to use the PDE build process, as much as possible, so  
 
some build-features are still needed just to drive the build process. These should have pretty neutral names,  
 
some build-features are still needed just to drive the build process. These should have pretty neutral names,  
such as "set1", to emphasize there is no particular conceptual organization. There are several reasons we can not
+
such as "set1", to emphasize there is no particular conceptual organization. At some point, we may decide not  
use just one big feature set:
+
to continually rebuild the "old" set1, and put new bundles in set2.
 
+
#There can be bundles with identical symbolic names, but different versions (such as org.junit, versions 3.8.1 and 4.1.0.1). These can not be built in "one pass" of PDE builder. So, ideally, the first version of a bundle would go in "set 1", and subsequent versions would go in "set 2".
+
#Sometimes, PDE "thinks" there are dependencies between bundles. Well, there are dependencies, but they do not really matter at build time, since we are not really compiling anything. This may eventually be improved, see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=169991 bug 169991] but in the meantime, we do need to group bundles that depend on one another.
+
#Lastly, we want separate build-features, since at some point, some of them would be ready to release, but not necessarily all. So, we can "release" build-feature by build-feature and still leave all the "old" features in place, in case they ever need to be re-done.  
+
 
+
  
 
=== Steps ===  
 
=== Steps ===  
  
 
There's probably several ways to do these steps, but I think of it just other Eclipse project builds I've seen, where projects are "released" to a build, via the platform's releng team's releng tool. In many of the following steps, if you are doing something for the first time, you can usually use what already exists as examples, and in most cases, this will suffice. Long term, we may detail (or point to) more of the details, if people need it.  
 
There's probably several ways to do these steps, but I think of it just other Eclipse project builds I've seen, where projects are "released" to a build, via the platform's releng team's releng tool. In many of the following steps, if you are doing something for the first time, you can usually use what already exists as examples, and in most cases, this will suffice. Long term, we may detail (or point to) more of the details, if people need it.  
 +
NOTE: the platform's releng tool does not currently work if there are more than one version of the same bundle in a map file.
  
 
* Once your bundle is ready to put into a build, make sure you have that version (that is, branch) loaded in your workspace, as you want it released ... this means, usually, all the most recent code in that branch.  
 
* Once your bundle is ready to put into a build, make sure you have that version (that is, branch) loaded in your workspace, as you want it released ... this means, usually, all the most recent code in that branch.  
Line 49: Line 55:
 
** Be sure the unpack attribute is set as you desire. Also, set the the download/install sizes ... a rough estimate is fine, say rounded up to within 10% of actually size. In the feature, leave the version as a "0.0.0" and it will later be computed from the data in the bundle itself.  
 
** Be sure the unpack attribute is set as you desire. Also, set the the download/install sizes ... a rough estimate is fine, say rounded up to within 10% of actually size. In the feature, leave the version as a "0.0.0" and it will later be computed from the data in the bundle itself.  
  
* Be sure the map 'branch' is loaded in your workspace that matches the name of the feature, set1, set2, etc. The maps are contained in the project named 'org.eclipse.orbit.releng'.  
+
* The maps are contained in the project named 'org.eclipse.orbit.releng'. Be sure you have the HEAD version loaded in your workspace.
  
 
** If not already done, be sure there is an entry in the map file that has your bundle info. Initially, you can use a "fake" cvs version number (or, just use "HEAD") as it will be corrected in the next step.  
 
** If not already done, be sure there is an entry in the map file that has your bundle info. Initially, you can use a "fake" cvs version number (or, just use "HEAD") as it will be corrected in the next step.  
  
* Use Team, Release to release the bundle to the build. This tags the project with a CVS tag, updates the map file, and commits the map file to that map files branch.  
+
* Use Team, Release to release the bundle to the build. This tags the project with a CVS tag, updates the map file, and commits the map file to that map files branch. If there's two versions of a bundle, you can not use the releng tool, so you need to "manually" tag your bundle project, and paste that value in the map entry, and then commit the map file to HEAD.
  
 
** By convention, use CVS tags of the form vYYYYMMDDHHMM, where the date and time is in UTC time.  
 
** By convention, use CVS tags of the form vYYYYMMDDHHMM, where the date and time is in UTC time.  
Line 59: Line 65:
 
** The feature that has your bundle must also be released, similar to the above step. This only has to be done whenever the feature actually changes, which is likely to be just once, when you first add the bundle to the feature.  
 
** The feature that has your bundle must also be released, similar to the above step. This only has to be done whenever the feature actually changes, which is likely to be just once, when you first add the bundle to the feature.  
  
* Once released, the build system should notice this, and starts a new build. It will upload the results to the [http://download.eclipse.org/tools/orbit/committers/ committers area]. The build loop currently is set to 1 hour, so if you do not see an updated build on the committers area within an hour and a half, please contact one of the Orbit release engineers.
+
* Once released, the build system should notice this, and starts a new build. It will upload the results to the [http://build.eclipse.org/orbit/committers/ committers area]. The build loop currently is set to 1 hour, so if you do not see an updated build on the committers area within, say, an hour and a half, please contact one of the Orbit release engineers.
  
 
== Orbit Builds for Orbit Builds-masters ==
 
== Orbit Builds for Orbit Builds-masters ==
Line 65: Line 71:
 
The build system uses CruiseControl to drive the build-loop.  
 
The build system uses CruiseControl to drive the build-loop.  
  
The [http://build.eclipse.org:9777/cruisecontrol/ status] is displayed via the web, and for those with the right "build-master" password, there is an [http://build.eclipse.org:9000/ agent] that can control the builds. If either of these links are dead, it means the CruiseControl server is not running and needs to be restarted.
+
The [http://build.eclipse.org:9777/cruisecontrol/ status] is displayed via the web, and for those with the right "build-master" password, there is an [http://build.eclipse.org:9000/ agent] that can control the builds. If either of these links are dead, it means the CruiseControl server is not running and needs to be restarted. With the introduction of CruiseControl 2.7.x, there is also a [http://build.eclipse.org:9777/dashboard/tab/builds "dashboard" interface] to CruiseControl.  
  
 
This system was created by using the WTP build as a starting point, and then simplified and changed as needed. So, in some cases, there is likely old stuff left over from the WTP process that is not really needed in Orbit. Feel free to clean up as you find them.  
 
This system was created by using the WTP build as a starting point, and then simplified and changed as needed. So, in some cases, there is likely old stuff left over from the WTP process that is not really needed in Orbit. Feel free to clean up as you find them.  
Line 79: Line 85:
 
**The 'build.xml' file in the root of the project is the main entry point, and is called from the releng.control project.  
 
**The 'build.xml' file in the root of the project is the main entry point, and is called from the releng.control project.  
 
* 'org.eclipse.orbit.releng.control': this project holds the code that controls the build-loop, downloads fresh builders, etc.  
 
* 'org.eclipse.orbit.releng.control': this project holds the code that controls the build-loop, downloads fresh builders, etc.  
* 'org.eclipse.orbit.releng.dropsite': this project holds the code that makes up the download site build pages.
+
* 'org.eclipse.orbit.releng.downloadsites': this project holds the code that makes up the download site build pages, both on download server, and committers area of orbits build server.
  
 
=== How to update the base builder ===
 
=== How to update the base builder ===
Line 89: Line 95:
 
Like everything else, the base builder we use is specified in various property files and ant tasks. For our "production builds" we need to specify a tagged version of the base builder (not just "HEAD") so that our builds are reproducible. It is not often, if ever, we actually do reproduce a build ... but, that is simply a required software engineering practice.
 
Like everything else, the base builder we use is specified in various property files and ant tasks. For our "production builds" we need to specify a tagged version of the base builder (not just "HEAD") so that our builds are reproducible. It is not often, if ever, we actually do reproduce a build ... but, that is simply a required software engineering practice.
  
We accomplish this in sort of an indirect way (for ease of use, mostly). Our own orbit builder specifies which version of the base builder we want to use, and our orbit builder is the one that is  tagged and documented for each build. The specification of which orbit builder to use is in the HEAD version of build.cfg, but then that releng project is tagged for each build, so the information is retrievable from CVS after the fact.  [Note: in writing this up, I'd discovered this "principle" is not actually true yet ... I've somehow lost that in the translation (will track in [https://bugs.eclipse.org/bugs/show_bug.cgi?id=171904 bug 171904])].
+
We accomplish this in sort of an indirect way (for ease of use, mostly). Our own orbit builder specifies which version of the base builder we want to use, and our orbit builder is the one that is  tagged and documented for each build. The specification of which orbit builder to use is in the HEAD version of build.cfg, but then that releng project is tagged for each build, so the information is retrievable from CVS after the fact.
  
 
==== Step by Step ====
 
==== Step by Step ====
Line 103: Line 109:
 
So, change it to be the tag desired, found above in the first step. For example
 
So, change it to be the tag desired, found above in the first step. For example
 
  eclipse.builder.version=M4_33  
 
  eclipse.builder.version=M4_33  
 +
<li>
 +
Update the 'eclipse.base.version' property in the build.properties file to be the version of org.eclipse.pde.build that is in the base builder. This is used to help build the path to the build scripts.
 +
</li>
 
<li>
 
<li>
 
Commit and tag the builder project ('org.eclipse.orbit.releng.builder')
 
Commit and tag the builder project ('org.eclipse.orbit.releng.builder')
Line 120: Line 129:
 
</ol>
 
</ol>
  
=== How to declare a build ===
+
=== How to declare or promote a build ===
  
To "Declare a build" means, basically, everyone agrees (or, at least, doesn't object) to a particular build in the continuous 'committers' area being moved to the slightly more permanent 'downloads' area.  
+
To "declare a build" (sometimes called "promote a build") means, basically, a particular build in the continuous 'committers' area is moved to the more permanent 'downloads' area. Technically, any Orbit committer with a shell account can promote a build, but practically, the first choice for Orbit committers is just to ask on orbit-dev list for the Project Lead to do the promotion. The remaining instructions here are just for in case the PL is not available and someone else needs to do it.  
  
This is not an purely automatic process, by design, but someone (an Orbit committer, that is) must run a script which  
+
A build is declared or promoted by running a script on build.eclipse.org which  
copies a particular drop directory under 'committers' area to the drops directory of the download directory (with the same name). The other important, critical thing this script does is to "fix up" some of the links in the php files and in the HTTP version of the map file to point to the 'downloads' area version of a file, instead of the previous 'committers' area version of the file.  
+
copies a particular drop directory under 'committers' area to the drops directory of the download directory (with the same name). The other important thing this script does is to "fix up" some of the links in the php files and in some of the map files to point to the 'downloads' area version of a file, instead of the previous 'committers' area version of the file.  
  
After a build is mechanically declared, at least a day or two should be allowed to elapse just in case anyone was temporarily using a 'committers' build to test getting some particular Orbit build into their consuming project.  
+
Note: besides being a committer in Orbit, you need to have a shell account on eclipse.org, and if you do not have one already, then you do not want to get one just for this ... contact someone else to declare the build.
  
Here's an annotated example of running the script.  
+
After a build is declared, at least a day or two should be allowed to elapse before deleting the one in committers area just in case anyone was temporarily using a 'committers' build to test getting some particular Orbit build into their consuming project.  
  
 +
Below is an annotated example of running the promote script.
  
First, logon to the build.eclipse.org machine (with your normal committers ID, and a normal SSH shell).
+
==== How to run the promote script ====
Navigate to the 'committers' directory. You need to run the script from, exactly, this level in the directory.
+
  
 +
Update on 9/18/2011: As documented in {{bug|357523}} and {{bug|230061}} this procedure was changed. Instead of running "declare.sh" to move a build from one area of downloads server to another area of downloads server, the procedure is now to run a "promote.sh" script to promote a build from the build machine to to the download server.
  
        david_williams@build:~/downloads/tools/orbit> cd committers/
+
The promote script does not need a specific build to promote, but instead it just needs to be told which CruiseControl project to use (it will nearly always will be 'orbit-I') and the script will get the most recent build from that project and promote it. (In the rare case there has been more recent builds, not desired to promote, they would have to be removed (or moved) so the desired build is the most recent one, under .../orbit/committers/<projectname>/ but normally this is never an issue).  
        david_williams@build:~/downloads/tools/orbit/committers> ll
+
        total 44
+
        -rwxrwxr-x+  1 david_williams orbitadmin  1158 2007-04-25 20:21 cleanup.sh
+
        -rwxrwxr-x+  1 david_williams orbitadmin  528 2007-03-19 11:29 declare.sh
+
        -rwxrwxr-x+  1 david_williams orbitadmin  483 2007-03-14 02:49 declareUpdates.sh
+
        -rw-rwxr--+  1 david_williams orbitadmin  134 2007-02-07 22:49 dlconfig.txt
+
        drwxrwsr-x+ 14 david_williams orbitadmin  432 2007-05-23 22:10 drops
+
        -rw-rwxr--+  1 david_williams orbitadmin 11262 2007-05-01 01:35 index.php
+
        -rw-rwxr--+  1 david_williams orbitadmin  121 2007-01-16 14:52 phpinfo.php
+
        -rw-rwxr--+  1 david_williams orbitadmin  4493 2007-01-16 14:52 report.php
+
        -rw-rwxr--+  1 david_williams orbitadmin    4 2007-01-16 14:52 test.html
+
  
 +
Be sure to check that the latest, more recent build does not have "errors" in it (as indicated on the committers download page) as once promote, there is no longer an indication of error, or not, on the build ... it is assume we only promote builds that do not have errors ... but, so far, the promote script does not automatically check for this.
  
Get a listing of the directories under the drops directory. Typically, it is the last one in the list
+
To run the promote script, login to the build.eclipse.org machine (with your normal committers ID, and a normal SSH shell).  
that you will want to promote. (and you can use some copy/paste tricks to get the name, so the next step is easier).  
+
  
        david_williams@build:~/downloads/tools/orbit/committers> ll drops
+
Navigate to the '/shared/orbit/org.eclipse.orbit.releng.control/' directory.
        total 6
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-17 13:34 I200705171711
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-18 11:00 I200705181436
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-18 22:56 I200705190235
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-20 06:21 I200705200959
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-22 10:52 S200705221427
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-22 14:07 S200705221744
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-22 19:16 S200705222254
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 12:42 S200705231619
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 15:07 S200705231844
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 16:20 S200705231959
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 18:46 S200705232222
+
        drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 22:10 S200705240148
+
  
 +
From that directory, you'd run something similar to
  
Run the declare.sh script, using the directory name as the argument. Note, you need (only) the directory name under drops, the 'drops' part is assumed in the scripts.
+
  ./promote.sh -cdsczp orbit-I
  
        david_williams@build:~/downloads/tools/orbit/committers> ./declare.sh S200705240148
+
Note, running the script with no arguments, or -h will give brief usage help. But here's a wordier version.  
  
Here is a example of the type of ouput you would see during a successful run.  
+
<pre>
 +
 
 +
        Usage: promote.sh [-h] [-v] [-d] [-s] [-c] [-z] [-p projectname]
 +
 
 +
                -h      this help message
 +
                        [The brief help, that is].
 +
 
 +
                -v      verbose
 +
                        [Note: care is needed with 'verbose', it can provide so much output its hard to spot errors or warning.]
 +
 
 +
                -d      delete old builds
 +
                        [This deletes builds on the build machine (not downloads server) which are older than the one it is promoting.
 +
                        This is not technically required ... old builds will eventually be cleaned up automatically ... but,
 +
                        typically, there is no reason to leave them around, and it can help avoid confusion to remove old ones.
 +
                        The promote script always leaves in-place the build it promotes, so that for a while, there will be two versions of the build,
 +
                        one on build machine, one on downloads.]
 +
 
 +
                -s      send notification mail
 +
                        [This will automatically send a "Declaring Build" message to orbit-dev list,
 +
                        with handy link to the build on downloads server.]
 +
 
 +
                -c      copy the build from committers to downloads
 +
                        [Maybe surprisingly, a command option is required to actually be copied, that is promoted.
 +
                        Occasionally, one may want to use this promote script just to remove old builds (not actually do the promotion).
 +
                        Another use-case is that if -c is not specified, it will print message of what it will do, once -c is specified, which can serve as a
 +
                        sanity check changing download server ... omitting -c is similar to specifying --dry-run option on some commands.]
 +
 
 +
                -z      process artifacts (create pack200 gz jars, etc.) Remember, is long running
 +
                        [Before actually promoting the build, this option will invoke p2.process.artifacts, with
 +
                        the 'pack' attribute, to cause the build's repository
 +
                        to be updated so it also contains pack.gz versions of the bundle jars. It is done this way, since pack.gz files are not
 +
                        really useful until a repository is in place to use with http, so avoids having to run the lengthy process on every build.]
 +
 
 +
                -p <projectname>, where projectname is the CruiseControl project, such as orbit-I, orbit-M, etc. from which to get latest build.
 +
                        [Not currently many choices here, it will nearly always be orbit-I (may want to make that the default,
 +
                        someday, it is so infrequent to do anything else).]
 +
 
 +
</pre>
 +
 
 +
==== Example output from the promote script ====
 +
 
 +
Here is a example of the type of output you would see during a successful run.
 +
 
 +
<pre>
 +
$ ./promote.sh -dsczp orbit-I 2>&1 | tee ~promoteout.txt
 +
 +
        Promoting latest build from  ...
 +
 +
Drop directory: I20110919015729
 +
INFO: processing artifacts in code repo: /shared/orbit/committers/orbit-I/20110919015729/I20110919015729/repository
 +
BUILDFILE: process-artifacts.xml
 +
extraArgs: -DrepoDirLocation=/shared/orbit/committers/orbit-I/20110919015729/I20110919015729/repository
 +
dev: /shared/orbit/org.eclipse.orbit.releng.control/runAntRunner.sh
 +
devworkspace: /home/data/users/david_williams/workspace-runAntRunner
 +
devJRE: /shared/orbit/apps/ibm-java2-i386-50/jre/bin/java
 +
 +
 +
osgi> Buildfile: process-artifacts.xml
 +
 +
processArtifacts:
 +
      [echo] Remember, processing artifacts can take a long time, such as 10 minutes, to complete. Don't panic.
 +
BUILD SUCCESSFUL
 +
 +
BUILD SUCCESSFUL
 +
Total time: 9 minutes 18 seconds
 +
        declaring build I20110919015729
 +
                into /home/data/httpd/download.eclipse.org/tools/orbit/downloads/drops/
 +
                using the build from /shared/orbit/committers/orbit-I/20110919015729/I20110919015729
 +
 +
mail sent
 +
</pre>
 +
 
 +
==== Follow-up tasks ====
 +
 
 +
After running the script it should be "visible" on the HTTP downloads page nearly immediately. Be sure to test that links on web pages (and map files) are correct (that is, that they lead to "downloads" area, and not "committers" area).
 +
 
 +
==== Renaming and promoting a build ====
 +
 
 +
Often, such as for milestones or releases, it is desired to "rename" a build, such as from an I-build to an S-build.
 +
 
 +
It is possible to rename builds on the download server directly, but it is often useful to do so in the build servers committers area, and then promote that renamed build. In that case, it is recommended to first promote the I-build like usual, to have a "safe copy" on downloads" (if it has not been promoted, already). Then you can navigate to the committers area, on build machine, such as under .../committers/orbit-I/<datetimestampdirectory>, rename the build there, changing the one letter, such as
 +
 
 +
    ./renameBuild.sh I20100413151631 S20100413151631
 +
 
 +
Then, you can run the promote script again, still using orbit-I as the project argument (since still in the orbit-I hierarchy) and the promote script will happily find and promote the "S build".
 +
 
 +
===How to rename a build===
 +
 
 +
Often, when we get a milestone, or release, we just rename a build to have a more appropriate letter associated with it ('S' for Stable Milestone, 'R' for Recommended (or, Release), instead of 'I' for Integration).
 +
 
 +
To do this, we have a script, that normally is in the 'committer' directory, named renameBuild.sh.
 +
 
 +
Note: as with all our download machine scripts, this script is kept in CVS, in our project named ''org.eclipse.orbit.releng.downloadsites''. Deep in a directory named ''WebContent/downloadServer-tools-orbit/commonFiles/miscScripts/''.
 +
 
 +
In this example will assume an I build renamed to an S build, but similar would apply for creating an R build.
 +
 
 +
First, it is a good idea to "declare" the I-build, like usual, so you have a "spare" copy ... just in case ... since the ''renameBuild'' script does rename things ... not make a copy ... whereas the ''declare'' script makes a copy.
 +
 
 +
Next, copy down the renameBuild.sh script to the 'drops' directory (there's some assumptions in it, that assumes the directory to rename is in same directory as the script).
 +
 
 +
Then, simply run the script, with old-name and new-name. Something like
 +
 
 +
./renameBuild.sh I20090911225403 S20090911225403
 +
 
 +
After that, there's some logging information printed to console, so whole session would look something like the following:
 +
 
 +
Note: as of 9/18/2011, the download system was changed to 'committers' builds are kept on build.eclipse.org server, in /shared/orbit/committers until promoted. The concepts of running 'renameBuild' are exactly the same, though the example below is out-of-date in terms of which directories are being used during the rename.  
  
        S200705240148
+
david_williams@build:~> cd downloads/tools/orbit/committers/drops/
        declaring build S200705240148
+
david_williams@build:~/downloads/tools/orbit/committers/drops> pwd
          into /home/data/users/david_williams/downloads/tools/orbit/downloads/drops
+
/home/data/users/david_williams/downloads/tools/orbit/committers/drops
          using the build from /home/data/users/david_williams/downloads/tools/orbit/committers/drops
+
david_williams@build:~/downloads/tools/orbit/committers/drops> cp ../renameBuild.sh .
        david_williams@build:~/downloads/tools/orbit/committers>
+
david_williams@build:~/downloads/tools/orbit/committers/drops> ./renameBuild.sh I20090911225403 S20090911225403
 +
Renaming build I20090911225403 to S20090911225403
 +
changing ./S20090911225403/iplog-I20090911225403.html to ./S20090911225403/iplog-S20090911225403.html
 +
changing ./S20090911225403/orbitBundles-I20090911225403.p2.map to ./S20090911225403/orbitBundles-S20090911225403.p2.map
 +
changing ./S20090911225403/results-I20090911225403.xml to ./S20090911225403/results-S20090911225403.xml
 +
changing ./S20090911225403/checksum/orbit-I20090911225403.zip.md5 to ./S20090911225403/checksum/orbit-S20090911225403.zip.md5
 +
changing ./S20090911225403/checksum/orbit-I20090911225403.zip.md5antformat to ./S20090911225403/checksum/orbit-S20090911225403.zip.md5antformat
 +
changing ./S20090911225403/orbit-I20090911225403.psf to ./S20090911225403/orbit-S20090911225403.psf
 +
changing ./S20090911225403/orbit-I20090911225403.zip to ./S20090911225403/orbit-S20090911225403.zip
 +
changing ./S20090911225403/orbitBundles-I20090911225403.map to ./S20090911225403/orbitBundles-S20090911225403.map
  
After running the script, it can take anywhere from 5 to 30 minutes for all the internal cluster replication to take place and be "visible" on the HTTP downloads page.
+
Lastly, remove the 'renameBuild.sh' script from the 'drops' directory. (It can interfere with correct operation of the webpage which reads which builds are available, since it is a file, instead of a directory).
  
 
==Building Orbit In Your Workspace==
 
==Building Orbit In Your Workspace==
Line 188: Line 290:
  
 
* create project in your workspace to contain the builder (e.g. <code>orbit.builder</code>)
 
* create project in your workspace to contain the builder (e.g. <code>orbit.builder</code>)
 +
** File -> New -> Project
 +
** <code>orbit.builder</code>
 
* you can create a project in your workspace to contain the build results if you want (e.g. <code>orbit.buildDirectory</code>)
 
* you can create a project in your workspace to contain the build results if you want (e.g. <code>orbit.buildDirectory</code>)
* its also easier if you check out <code>org.eclipse.pde.build</code> into your workspace
+
** File -> New -> Project
* select the manifest for the org.eclipse.pde.build project and right-click and choose PDE Tools -> Create Ant Buildfile
+
** <code>orbit.buildDirectory</code>
* right click on the produced build.xml and choose Run -> As Ant File...
+
* check out <code>org.eclipse.pde.build</code> into your workspace
* de-select the default target and choose lib/pdebuild-ant.jar
+
** File -> Import -> Project from CVS
 +
** dev.eclipse.org, /cvsroot/eclipse, org.eclipse.pde.build
 +
** select the manifest for the org.eclipse.pde.build project and right-click and choose PDE Tools -> Create Ant Buildfile
 +
** right click on the produced build.xml and choose Run -> As Ant File...
 +
** de-select the default target and choose lib/pdebuild-ant.jar
 
* checkout the <code>org.eclipse.orbit.releng.tools</code> project from CVS
 
* checkout the <code>org.eclipse.orbit.releng.tools</code> project from CVS
* go to Windows -> Preferences -> Java -> Build Path -> Classpath Variables
+
** go to Windows -> Preferences -> Java -> Build Path -> Classpath Variables
* add ANT_HOME and point it to the <code>org.apache.ant</code> folder in your plugins/ dir for your Eclipse install
+
** add ANT_HOME and point it to the <code>org.apache.ant</code> folder in your plugins/ dir for your Eclipse install
 
* export the tools project into <code>tools.jar</code> and put it in the root of the <code>orbit.builder</code> project.
 
* export the tools project into <code>tools.jar</code> and put it in the root of the <code>orbit.builder</code> project.
 
* copy the <code>build.properties</code> file from <code>org.eclipse.pde.build/templates/headless-build/build.properties</code> to the root of your builder directory
 
* copy the <code>build.properties</code> file from <code>org.eclipse.pde.build/templates/headless-build/build.properties</code> to the root of your builder directory
Line 215: Line 323:
 
mapsRepo=:pserver:anonymous@dev.eclipse.org/cvsroot/tools
 
mapsRepo=:pserver:anonymous@dev.eclipse.org/cvsroot/tools
 
mapsRoot=org.eclipse.orbit/org.eclipse.orbit.releng
 
mapsRoot=org.eclipse.orbit/org.eclipse.orbit.releng
mapsCheckoutTag=set1
 
 
#skipFetch=true
 
#skipFetch=true
 
</pre>
 
</pre>
Line 305: Line 412:
 
** <code>org.eclipse.ant.core.antRunner</code> application
 
** <code>org.eclipse.ant.core.antRunner</code> application
 
* add the following program arguments
 
* add the following program arguments
** <code>-buildfile ${resource_loc:/org.eclipse.pde.build/scripts/build.xml} -Dbuilder=d:/eclipse/workspace/orbit.builder</code>
+
** <code>-buildfile ${resource_loc:/org.eclipse.pde.build/scripts/build.xml} -Dbuilder=${workspace_loc}/orbit.builder</code>
  
 
That's it. Then when you run your application, it should build Orbit and put the results in your <code>orbit.buildDirectory</code> project. Note that you may want to turn on <code>Window -> Preferences -> General -> Workspace -> Refresh Automatically</code> to save you from having to press F5 to refresh.
 
That's it. Then when you run your application, it should build Orbit and put the results in your <code>orbit.buildDirectory</code> project. Note that you may want to turn on <code>Window -> Preferences -> General -> Workspace -> Refresh Automatically</code> to save you from having to press F5 to refresh.
 +
 +
[[Category:Orbit]]

Latest revision as of 02:21, 25 January 2013

This page is the starting point for information on Orbit builds.

Orbit builds are a bit different from others, in that nothing is really "built" in the traditional sense of determining dependencies and compiling. The builds are more of a "copy and assemble" operation. One reason the builds are still very important, though, is that the resulting jars are "conditioned" for packing. These means that as others use the bundles, they are assured of getting the exact same bits, whether those bits come from a downloaded bundle, some assembled zip file, or some update site.

Orbit Builds for Consumers

The place to get bundles that are considered "released" is from http://download.eclipse.org/tools/orbit/downloads/

But, the preliminary versions are available from http://download.eclipse.org/tools/orbit/committers/

See Orbit/Promotion,_Release,_and_Retention_Policies

The bundles are made available in several forms;

as p2 repositories. The repository can be used in batch builds, or scripts that use p2 operations. Or, for developers, you can use the Orbit repository in your PDE target runtime, and specify which bundle(s) you want to take advantage of.
downloadable one-by-one. The bundles/jars can be downloaded one by one, and placed in dropins folder, for example, but this is not the recommended way. Care is needed to know if the bundle should be unzipped as a directory, or left in jar format, for example.

There are no features from Orbit, for consumers to use or refer to in their own features. The intent is that the bundles can be included in other project's features and redistributed by those other projects (after they submit appropriate CQs, of course). While this results in some duplication in the various projects zips or update sites, since all projects use exactly the same bundles from Orbit, there should be no duplication once installed. For this reason, it is important that for new development/releases, that all projects use the very latest Orbit deliveries so that these common bundles use exactly the version and qualifier.

There are a couple of methods to include bundles in consumer's features. One way is to include all the bundles in the "baseLocation" of PDE's build, and then they can be referred to in features and are included in packaged zips and update sites as needed (or determined) by PDE build. Another technique is to use the feature custom callback mechanism to copy the bundle to the directory being built or assembled.

For consumers to fetch Orbit bundles as part of their build process, and include in the "baseLocation", and ultimately their own features, the easiest way is to use the map files produced by the Orbit build process.


CVS-TEXT style map files [deprecated]. Orbit produces a map file, named directory.txt, containing CVS fetch entries which point directly to the versioned bundles in Orbit's repository. [This style of map file should be considered deprecated; we recommend the p2-style be used instead. In fact, as long as there is no "anonymous" access to Orbit's CVS repo, these maps are not usable, as is. For us to use our maps on build.eclipse.org, we use :local: instead of :pserver:anonymous:@dev.eclipse.org: (the replacement being done automatically during our build). Therefore, if anyone DID insist on using them, you'd have to do a similar replacement, either to ":local", if on build.eclipse.org, or to :ext:<committerid>:@dev.eclipse.org if you are committer on a machine besides build.eclipse.org.]
GET-HTTP style map files [deprecated]. Orbit produces a map file containing URL GET entries which point directly to the bundles in Orbit. These entries should replace the CVS GET entry currently in the consumer's map files with the new ones. For convenience, you can also just remove the entries from your map files and include the whole Orbit map file in your build, since the build is feature-diven the extra entries in the map file will be ignored. [This style of map file should be considered deprecated; we recommend the p2-style be used instead.]
P2 style map files. Orbit also map files in a "p2 style". In this case, the PDE build knows how to read the map entries and use p2 to fetch the bundles from the orbit build repository. To do this, the only change in build scripts, from other style map files, is to specify the 'transformedRepoLocation' property, say in the overall build.properties file such as below (where ${buildDirectory} is an already required location property. And ... not sure required ... but also might need <mkdir ${transformedRepoLocation}/> in scripts, to make sure that 'transfromedRepo' directory exists before PDE build needs it.
transformedRepoLocation=${buildDirectory}/transformedRepo

For more information about specifying bundles to be included in features, see Include Orbit bundles into a feature.

Orbit Builds for Orbit Committers

Principles and Concepts

In orbit, our goal is to produce bundles for consumption, and consumers can mix-and-match what ever they need in their own features. But, we'd still like to use the PDE build process, as much as possible, so some build-features are still needed just to drive the build process. These should have pretty neutral names, such as "set1", to emphasize there is no particular conceptual organization. At some point, we may decide not to continually rebuild the "old" set1, and put new bundles in set2.

Steps

There's probably several ways to do these steps, but I think of it just other Eclipse project builds I've seen, where projects are "released" to a build, via the platform's releng team's releng tool. In many of the following steps, if you are doing something for the first time, you can usually use what already exists as examples, and in most cases, this will suffice. Long term, we may detail (or point to) more of the details, if people need it. NOTE: the platform's releng tool does not currently work if there are more than one version of the same bundle in a map file.

  • Once your bundle is ready to put into a build, make sure you have that version (that is, branch) loaded in your workspace, as you want it released ... this means, usually, all the most recent code in that branch.
  • If not done already, be sure your bundle is included as one of the bundles ("plugins") in the feature.
    • Be sure the unpack attribute is set as you desire. Also, set the the download/install sizes ... a rough estimate is fine, say rounded up to within 10% of actually size. In the feature, leave the version as a "0.0.0" and it will later be computed from the data in the bundle itself.
  • The maps are contained in the project named 'org.eclipse.orbit.releng'. Be sure you have the HEAD version loaded in your workspace.
    • If not already done, be sure there is an entry in the map file that has your bundle info. Initially, you can use a "fake" cvs version number (or, just use "HEAD") as it will be corrected in the next step.
  • Use Team, Release to release the bundle to the build. This tags the project with a CVS tag, updates the map file, and commits the map file to that map files branch. If there's two versions of a bundle, you can not use the releng tool, so you need to "manually" tag your bundle project, and paste that value in the map entry, and then commit the map file to HEAD.
    • By convention, use CVS tags of the form vYYYYMMDDHHMM, where the date and time is in UTC time.
    • The feature that has your bundle must also be released, similar to the above step. This only has to be done whenever the feature actually changes, which is likely to be just once, when you first add the bundle to the feature.
  • Once released, the build system should notice this, and starts a new build. It will upload the results to the committers area. The build loop currently is set to 1 hour, so if you do not see an updated build on the committers area within, say, an hour and a half, please contact one of the Orbit release engineers.

Orbit Builds for Orbit Builds-masters

The build system uses CruiseControl to drive the build-loop.

The status is displayed via the web, and for those with the right "build-master" password, there is an agent that can control the builds. If either of these links are dead, it means the CruiseControl server is not running and needs to be restarted. With the introduction of CruiseControl 2.7.x, there is also a "dashboard" interface to CruiseControl.

This system was created by using the WTP build as a starting point, and then simplified and changed as needed. So, in some cases, there is likely old stuff left over from the WTP process that is not really needed in Orbit. Feel free to clean up as you find them.

An overview of the releng related project

All the build control scripts and templates are in CVS:

  • 'org.eclipse.orbit.releng.builder': this project contains the heart of the build process.
    • The 'components' directory codifies 'what' is built. As new build-feature sets are added, a new component (directory) needs to be added to build that feature.
    • The 'distribution' directory codifies 'how' something is built, uploaded, prepared for the download site, etc.
    • The 'scripts' directory holds various scripts used during the process.
    • The 'build.xml' file in the root of the project is the main entry point, and is called from the releng.control project.
  • 'org.eclipse.orbit.releng.control': this project holds the code that controls the build-loop, downloads fresh builders, etc.
  • 'org.eclipse.orbit.releng.downloadsites': this project holds the code that makes up the download site build pages, both on download server, and committers area of orbits build server.

How to update the base builder

The base builder is from the Eclipse Platform releng team. It is mostly a "stripped down" version of Eclipse, with a few extra tools and ant tasks. It is important, of course, since it provides the PDE build that we want to use. As the platform team makes forward progress, they periodically tag the base builder -- usually a few weeks after each milestone -- and that version of the base builder is used to build the next milestone. Orbit should stay current with the latest version of the base builder, to be sure we all stay well integrated -- in our case, one main need for this is to be sure we use the most recent qualifier versioning rules.

Conceptual principles

Like everything else, the base builder we use is specified in various property files and ant tasks. For our "production builds" we need to specify a tagged version of the base builder (not just "HEAD") so that our builds are reproducible. It is not often, if ever, we actually do reproduce a build ... but, that is simply a required software engineering practice.

We accomplish this in sort of an indirect way (for ease of use, mostly). Our own orbit builder specifies which version of the base builder we want to use, and our orbit builder is the one that is tagged and documented for each build. The specification of which orbit builder to use is in the HEAD version of build.cfg, but then that releng project is tagged for each build, so the information is retrievable from CVS after the fact.

Step by Step

  1. Find the latest tagged version of the base builder to use.
  2. In the 'org.eclipse.orbit.releng.builder' project, the build.properties file contains a property named 'eclipse.builder.version' and would be set to some value like M3_33, so the whole line would be eclipse.builder.version=M3_33 So, change it to be the tag desired, found above in the first step. For example eclipse.builder.version=M4_33
  3. Update the 'eclipse.base.version' property in the build.properties file to be the version of org.eclipse.pde.build that is in the base builder. This is used to help build the path to the build scripts.
  4. Commit and tag the builder project ('org.eclipse.orbit.releng.builder') using a conventional tag such as vYYYYMMDDHHMM (using UTC). Keep track of or swipe-copy the tag name, for the next step.
  5. In the HEAD of 'org.eclipse.orbit.releng' There is a file 'maps\build.cfg' that contains a 'builderVersion' property that should be set to the tag just created. So, for example, you would end up with builderVersion=v200701270624 Commit this version of the file to HEAD, and you're all set. The release to head should be detected and an autobuild started (within an hour or so) even if no other projects change.

How to declare or promote a build

To "declare a build" (sometimes called "promote a build") means, basically, a particular build in the continuous 'committers' area is moved to the more permanent 'downloads' area. Technically, any Orbit committer with a shell account can promote a build, but practically, the first choice for Orbit committers is just to ask on orbit-dev list for the Project Lead to do the promotion. The remaining instructions here are just for in case the PL is not available and someone else needs to do it.

A build is declared or promoted by running a script on build.eclipse.org which copies a particular drop directory under 'committers' area to the drops directory of the download directory (with the same name). The other important thing this script does is to "fix up" some of the links in the php files and in some of the map files to point to the 'downloads' area version of a file, instead of the previous 'committers' area version of the file.

Note: besides being a committer in Orbit, you need to have a shell account on eclipse.org, and if you do not have one already, then you do not want to get one just for this ... contact someone else to declare the build.

After a build is declared, at least a day or two should be allowed to elapse before deleting the one in committers area just in case anyone was temporarily using a 'committers' build to test getting some particular Orbit build into their consuming project.

Below is an annotated example of running the promote script.

How to run the promote script

Update on 9/18/2011: As documented in bug 357523 and bug 230061 this procedure was changed. Instead of running "declare.sh" to move a build from one area of downloads server to another area of downloads server, the procedure is now to run a "promote.sh" script to promote a build from the build machine to to the download server.

The promote script does not need a specific build to promote, but instead it just needs to be told which CruiseControl project to use (it will nearly always will be 'orbit-I') and the script will get the most recent build from that project and promote it. (In the rare case there has been more recent builds, not desired to promote, they would have to be removed (or moved) so the desired build is the most recent one, under .../orbit/committers/<projectname>/ but normally this is never an issue).

Be sure to check that the latest, more recent build does not have "errors" in it (as indicated on the committers download page) as once promote, there is no longer an indication of error, or not, on the build ... it is assume we only promote builds that do not have errors ... but, so far, the promote script does not automatically check for this.

To run the promote script, login to the build.eclipse.org machine (with your normal committers ID, and a normal SSH shell).

Navigate to the '/shared/orbit/org.eclipse.orbit.releng.control/' directory.

From that directory, you'd run something similar to

 ./promote.sh -cdsczp orbit-I

Note, running the script with no arguments, or -h will give brief usage help. But here's a wordier version.


        Usage: promote.sh [-h] [-v] [-d] [-s] [-c] [-z] [-p projectname] 

                -h      this help message 
                        [The brief help, that is]. 

                -v      verbose 
                        [Note: care is needed with 'verbose', it can provide so much output its hard to spot errors or warning.]

                -d      delete old builds 
                        [This deletes builds on the build machine (not downloads server) which are older than the one it is promoting. 
                        This is not technically required ... old builds will eventually be cleaned up automatically ... but, 
                        typically, there is no reason to leave them around, and it can help avoid confusion to remove old ones. 
                        The promote script always leaves in-place the build it promotes, so that for a while, there will be two versions of the build, 
                        one on build machine, one on downloads.]

                -s      send notification mail 
                        [This will automatically send a "Declaring Build" message to orbit-dev list, 
                        with handy link to the build on downloads server.]

                -c      copy the build from committers to downloads 
                        [Maybe surprisingly, a command option is required to actually be copied, that is promoted. 
                        Occasionally, one may want to use this promote script just to remove old builds (not actually do the promotion). 
                        Another use-case is that if -c is not specified, it will print message of what it will do, once -c is specified, which can serve as a 
                        sanity check changing download server ... omitting -c is similar to specifying --dry-run option on some commands.]

                -z      process artifacts (create pack200 gz jars, etc.) Remember, is long running 
                        [Before actually promoting the build, this option will invoke p2.process.artifacts, with 
                        the 'pack' attribute, to cause the build's repository 
                        to be updated so it also contains pack.gz versions of the bundle jars. It is done this way, since pack.gz files are not 
                        really useful until a repository is in place to use with http, so avoids having to run the lengthy process on every build.] 

                -p <projectname>, where projectname is the CruiseControl project, such as orbit-I, orbit-M, etc. from which to get latest build. 
                        [Not currently many choices here, it will nearly always be orbit-I (may want to make that the default, 
                        someday, it is so infrequent to do anything else).]

Example output from the promote script

Here is a example of the type of output you would see during a successful run.

 $ ./promote.sh -dsczp orbit-I 2>&1 | tee ~promoteout.txt
 
        Promoting latest build from  ...
 
 Drop directory: I20110919015729
 INFO: processing artifacts in code repo: /shared/orbit/committers/orbit-I/20110919015729/I20110919015729/repository
 BUILDFILE: process-artifacts.xml
 extraArgs: -DrepoDirLocation=/shared/orbit/committers/orbit-I/20110919015729/I20110919015729/repository
 dev: /shared/orbit/org.eclipse.orbit.releng.control/runAntRunner.sh
 devworkspace: /home/data/users/david_williams/workspace-runAntRunner
 devJRE: /shared/orbit/apps/ibm-java2-i386-50/jre/bin/java
 
 
 osgi> Buildfile: process-artifacts.xml
 
 processArtifacts:
       [echo] Remember, processing artifacts can take a long time, such as 10 minutes, to complete. Don't panic.
 BUILD SUCCESSFUL
 
 BUILD SUCCESSFUL
 Total time: 9 minutes 18 seconds
         declaring build I20110919015729 
                 into /home/data/httpd/download.eclipse.org/tools/orbit/downloads/drops/
                 using the build from /shared/orbit/committers/orbit-I/20110919015729/I20110919015729
 
 mail sent

Follow-up tasks

After running the script it should be "visible" on the HTTP downloads page nearly immediately. Be sure to test that links on web pages (and map files) are correct (that is, that they lead to "downloads" area, and not "committers" area).

Renaming and promoting a build

Often, such as for milestones or releases, it is desired to "rename" a build, such as from an I-build to an S-build.

It is possible to rename builds on the download server directly, but it is often useful to do so in the build servers committers area, and then promote that renamed build. In that case, it is recommended to first promote the I-build like usual, to have a "safe copy" on downloads" (if it has not been promoted, already). Then you can navigate to the committers area, on build machine, such as under .../committers/orbit-I/<datetimestampdirectory>, rename the build there, changing the one letter, such as

   ./renameBuild.sh I20100413151631 S20100413151631 

Then, you can run the promote script again, still using orbit-I as the project argument (since still in the orbit-I hierarchy) and the promote script will happily find and promote the "S build".

How to rename a build

Often, when we get a milestone, or release, we just rename a build to have a more appropriate letter associated with it ('S' for Stable Milestone, 'R' for Recommended (or, Release), instead of 'I' for Integration).

To do this, we have a script, that normally is in the 'committer' directory, named renameBuild.sh.

Note: as with all our download machine scripts, this script is kept in CVS, in our project named org.eclipse.orbit.releng.downloadsites. Deep in a directory named WebContent/downloadServer-tools-orbit/commonFiles/miscScripts/.

In this example will assume an I build renamed to an S build, but similar would apply for creating an R build.

First, it is a good idea to "declare" the I-build, like usual, so you have a "spare" copy ... just in case ... since the renameBuild script does rename things ... not make a copy ... whereas the declare script makes a copy.

Next, copy down the renameBuild.sh script to the 'drops' directory (there's some assumptions in it, that assumes the directory to rename is in same directory as the script).

Then, simply run the script, with old-name and new-name. Something like

./renameBuild.sh I20090911225403 S20090911225403

After that, there's some logging information printed to console, so whole session would look something like the following:

Note: as of 9/18/2011, the download system was changed to 'committers' builds are kept on build.eclipse.org server, in /shared/orbit/committers until promoted. The concepts of running 'renameBuild' are exactly the same, though the example below is out-of-date in terms of which directories are being used during the rename.

david_williams@build:~> cd downloads/tools/orbit/committers/drops/
david_williams@build:~/downloads/tools/orbit/committers/drops> pwd
/home/data/users/david_williams/downloads/tools/orbit/committers/drops
david_williams@build:~/downloads/tools/orbit/committers/drops> cp ../renameBuild.sh .
david_williams@build:~/downloads/tools/orbit/committers/drops> ./renameBuild.sh I20090911225403 S20090911225403
Renaming build I20090911225403 to S20090911225403
changing ./S20090911225403/iplog-I20090911225403.html to ./S20090911225403/iplog-S20090911225403.html
changing ./S20090911225403/orbitBundles-I20090911225403.p2.map to ./S20090911225403/orbitBundles-S20090911225403.p2.map
changing ./S20090911225403/results-I20090911225403.xml to ./S20090911225403/results-S20090911225403.xml
changing ./S20090911225403/checksum/orbit-I20090911225403.zip.md5 to ./S20090911225403/checksum/orbit-S20090911225403.zip.md5
changing ./S20090911225403/checksum/orbit-I20090911225403.zip.md5antformat to ./S20090911225403/checksum/orbit-S20090911225403.zip.md5antformat
changing ./S20090911225403/orbit-I20090911225403.psf to ./S20090911225403/orbit-S20090911225403.psf
changing ./S20090911225403/orbit-I20090911225403.zip to ./S20090911225403/orbit-S20090911225403.zip
changing ./S20090911225403/orbitBundles-I20090911225403.map to ./S20090911225403/orbitBundles-S20090911225403.map

Lastly, remove the 'renameBuild.sh' script from the 'drops' directory. (It can interfere with correct operation of the webpage which reads which builds are available, since it is a file, instead of a directory).

Building Orbit In Your Workspace

Here are some instructions on how to get an Orbit build running in your workspace.

  • create project in your workspace to contain the builder (e.g. orbit.builder)
    • File -> New -> Project
    • orbit.builder
  • you can create a project in your workspace to contain the build results if you want (e.g. orbit.buildDirectory)
    • File -> New -> Project
    • orbit.buildDirectory
  • check out org.eclipse.pde.build into your workspace
    • File -> Import -> Project from CVS
    • dev.eclipse.org, /cvsroot/eclipse, org.eclipse.pde.build
    • select the manifest for the org.eclipse.pde.build project and right-click and choose PDE Tools -> Create Ant Buildfile
    • right click on the produced build.xml and choose Run -> As Ant File...
    • de-select the default target and choose lib/pdebuild-ant.jar
  • checkout the org.eclipse.orbit.releng.tools project from CVS
    • go to Windows -> Preferences -> Java -> Build Path -> Classpath Variables
    • add ANT_HOME and point it to the org.apache.ant folder in your plugins/ dir for your Eclipse install
  • export the tools project into tools.jar and put it in the root of the orbit.builder project.
  • copy the build.properties file from org.eclipse.pde.build/templates/headless-build/build.properties to the root of your builder directory
  • modify the following properties
    • buildDirectory - this is the directory for your builder
    • base - this is the location of the bundles that you will build against. Note that even if your bundles don't have dependancies, you still need this to point to a valid existing directory. (can be empty though)
    • resolution.devMode - setting this property allows you to build multiple versions of singleton bundles (like source bundles)
    • skipMaps - indicates whether or not you want to fetch the map files from CVS. Note you will need to comment this key/value out to set it to false. (changing the value is not enough)
    • mapsRepo - the location of the repository where you will get the map files
    • mapsRoot - the path in the repository to the project containing the map files
    • mapsCheckoutTag - the CVS tag for the map file project
    • skipFetch - indicates whether or not we should check the bundles out of the repository. Note you will need to comment this key/value out to set it to false. (changing the value is not enough)
buildDirectory=D:/eclipse/workspace/orbit.buildDirectory
base=D:/eclipse
resolution.devMode=true
#skipMaps=true
mapsRepo=:pserver:anonymous@dev.eclipse.org/cvsroot/tools
mapsRoot=org.eclipse.orbit/org.eclipse.orbit.releng
#skipFetch=true
  • copy the customTargets.xml file from org.eclipse.pde.build/templates/headless-build/customTargets.xml to the root of your builder directory
    • make the following changes:
    • Add the following properties to the top of the file:
<property name="featureToBuild" value="org.eclipse.orbit.build.feature.set1"/>
<property name="sourceFeature" value="org.eclipse.pde.build.container.feature"/>
  • Remove the declaration of the "allElementsFile" property and import of the file and then replace the "allElements" target with:
<!--
<property name="allElementsFile" value="${builder}/allElements.xml"/>
<import file="${allElementsFile}" />
-->
<target name="allElements">
	<ant antfile="${genericTargets}" target="${target}">
		<property name="type" value="feature" />
		<property name="id" value="${featureToBuild}" />
	</ant>
	<ant antfile="${genericTargets}" target="${target}">
		<property name="type" value="feature" />
		<property name="id" value="${sourceFeature}" />
	</ant>
</target>
  • Add 2 new targets:
<target name="assemble.org.eclipse.orbit.build.feature.set1">
	<ant antfile="${assembleScriptName}" dir="${buildDirectory}">
		<property name="archiveName" value="full.zip"/>
	</ant>
</target>
<target name="assemble.org.eclipse.pde.build.container.feature">
	<ant antfile="${assembleScriptName}" dir="${buildDirectory}">
		<property name="archiveName" value="full.zip"/>
	</ant>
</target>
  • Replace the preFetch target:
<target name="preFetch">
	<echo message="Fetching the main feature in order to generate the corresponding source feature" />
	<taskdef name="eclipse.sourceFeatureGenerator"
		classname="org.eclipse.orbit.internal.releng.tools.SourceFeatureGenerator"
		classpath="${builder}/tools.jar" />
	<eclipse.fetch
		elements="feature@${featureToBuild}"
		buildDirectory="${buildDirectory}"
		directory="${buildDirectory}/directory.txt"
		configInfo="${configs}"
		baseLocation="${baseLocation}"
		recursiveGeneration="false"
		children="false"
	/>		
	<ant antfile="${buildDirectory}/fetch_${featureToBuild}.xml">
		<property name="featureOnly" value="true"/>
	</ant>
		
	<mkdir dir="${buildDirectory}/features/${sourceFeature}" />
	<eclipse.sourceFeatureGenerator
		inputFile="${buildDirectory}/features/org.eclipse.orbit.build.feature.set1/feature.xml"
		outputDir="${buildDirectory}/features" mapFile="${buildDirectory}/directory.txt" />
	<taskdef name="orbit.ipLog"
		classname="org.eclipse.orbit.internal.releng.tools.IPLogTask"
		classpath="${builder}/tools.jar" />
	<orbit.ipLog
		buildId="${buildLabel}"
		inputDir="${buildDirectory}/maps/org.eclipse.orbit/org.eclipse.orbit.releng/ip_logs"
		outputFile="${buildDirectory}/${buildLabel}/iplog.html" />
</target>
  • Remove the contents of the "postSetup" target:
<target name="postSetup">
</target>


  • create a new launch configuration
    • new Eclipse Application
    • org.eclipse.ant.core.antRunner application
  • add the following program arguments
    • -buildfile ${resource_loc:/org.eclipse.pde.build/scripts/build.xml} -Dbuilder=${workspace_loc}/orbit.builder

That's it. Then when you run your application, it should build Orbit and put the results in your orbit.buildDirectory project. Note that you may want to turn on Window -> Preferences -> General -> Workspace -> Refresh Automatically to save you from having to press F5 to refresh.