|
|
| (35 intermediate revisions by 7 users not shown) |
| Line 1: |
Line 1: |
| − | This page is the starting point for information on Orbit builds.
| + | {{warning|Note: The contents of this page has been migrated. Please see https://github.com/eclipse/orbit#readme for current information, or page history for historical versions. }} |
| − | | + | |
| − | Orbit builds are a bit different from others, in that nothing is really "built" in the traditional since of determining dependencies and compiling. The builds are more of a "copy and assemble" operation. One reason the builds are still very important, though, is that the resulting jars are "conditioned" for packing. These means that as others use the bundles, they are assured of getting the exact same bits, whether those bits come from a downloaded bundle, some assembled zip file, or some update site.
| + | |
| − | | + | |
| − | == Orbit Builds for Consumers ==
| + | |
| − | | + | |
| − | The place to get bundles that are considered "released" is from
| + | |
| − | http://download.eclipse.org/tools/orbit/downloads/
| + | |
| − | | + | |
| − | But, the preliminary versions are available from
| + | |
| − | http://download.eclipse.org/tools/orbit/committers/
| + | |
| − | | + | |
| − | Once committers are happy with the bundles produced, the bundles in the
| + | |
| − | committers area will be moved to the downloads area. But, naturally,
| + | |
| − | consumers are encouraged to use "early" versions from the committers area, and
| + | |
| − | be sure the bundles work as they expect.
| + | |
| − | | + | |
| − | The bundles are made available in two forms. First, all in one big zip file. Second, as individual bundles. The individual bundles are either a jar file, or a zip file. The jar files are ready to use as-is. That is, they are intended to be used in the jarred form. The zip files are bundles that are intended to be unzipped before use.
| + | |
| − | | + | |
| − | There are no features from Orbit, for consumers to use. The goal is that these bundles can be included in other project's features as needed. While this results in some duplication in projects zips or update sites, since all projects use exactly the same bundles from Orbit, there should be no duplication once installed.
| + | |
| − | | + | |
| − | There's been some discussion on the orbit-dev mailing list on methods to include bundles in consumers features. One way is to include all the bundles in the "baseLocation" of PDE's build, and then they can be referred to in features and are included in packaged zips and update sites as needed (or determined) by PDE build. Another technique is to use the feature custom callback mechanism to copy the bundle to the directory being built or assembled.
| + | |
| − | | + | |
| − | For consumers who wish to fetch Orbit bundles as part of their build process, the Orbit build produces a map file containing URL GET entries which point directly to the bundles in Orbit. These entries should replace the CVS GET entry currently in the consumer's map files with the new ones. For convenience, you can also just remove the entries from your map files and include the whole Orbit map file in your build. (since the build is feature-driven the extra entries in the map file will be ignored)
| + | |
| − | | + | |
| − | == Orbit Builds for Orbit Committers ==
| + | |
| − | | + | |
| − | === Principles and Concepts ===
| + | |
| − | | + | |
| − | In orbit, our goal is to produce bundles for consumption, and consumers can mix-and-match what ever they need
| + | |
| − | in their own features. But, we'd still like to use the PDE build process, as much as possible, so
| + | |
| − | some build-features are still needed just to drive the build process. These should have pretty neutral names,
| + | |
| − | such as "set1", to emphasize there is no particular conceptual organization. There are several reasons we can not
| + | |
| − | use just one big feature set:
| + | |
| − | | + | |
| − | #There can be bundles with identical symbolic names, but different versions (such as org.junit, versions 3.8.1 and 4.1.0.1). These can not be built in "one pass" of PDE builder. So, ideally, the first version of a bundle would go in "set 1", and subsequent versions would go in "set 2".
| + | |
| − | #Sometimes, PDE "thinks" there are dependencies between bundles. Well, there are dependencies, but they do not really matter at build time, since we are not really compiling anything. This may eventually be improved, see [https://bugs.eclipse.org/bugs/show_bug.cgi?id=169991 bug 169991] but in the meantime, we do need to group bundles that depend on one another.
| + | |
| − | #Lastly, we want separate build-features, since at some point, some of them would be ready to release, but not necessarily all. So, we can "release" build-feature by build-feature and still leave all the "old" features in place, in case they ever need to be re-done.
| + | |
| − | | + | |
| − | | + | |
| − | === Steps ===
| + | |
| − | | + | |
| − | There's probably several ways to do these steps, but I think of it just other Eclipse project builds I've seen, where projects are "released" to a build, via the platform's releng team's releng tool. In many of the following steps, if you are doing something for the first time, you can usually use what already exists as examples, and in most cases, this will suffice. Long term, we may detail (or point to) more of the details, if people need it.
| + | |
| − | | + | |
| − | * Once your bundle is ready to put into a build, make sure you have that version (that is, branch) loaded in your workspace, as you want it released ... this means, usually, all the most recent code in that branch.
| + | |
| − | | + | |
| − | * If not done already, be sure your bundle is included as one of the bundles ("plugins") in the feature.
| + | |
| − | | + | |
| − | ** Be sure the unpack attribute is set as you desire. Also, set the the download/install sizes ... a rough estimate is fine, say rounded up to within 10% of actually size. In the feature, leave the version as a "0.0.0" and it will later be computed from the data in the bundle itself.
| + | |
| − | | + | |
| − | * Be sure the map 'branch' is loaded in your workspace that matches the name of the feature, set1, set2, etc. The maps are contained in the project named 'org.eclipse.orbit.releng'.
| + | |
| − | | + | |
| − | ** If not already done, be sure there is an entry in the map file that has your bundle info. Initially, you can use a "fake" cvs version number (or, just use "HEAD") as it will be corrected in the next step.
| + | |
| − | | + | |
| − | * Use Team, Release to release the bundle to the build. This tags the project with a CVS tag, updates the map file, and commits the map file to that map files branch.
| + | |
| − | | + | |
| − | ** By convention, use CVS tags of the form vYYYYMMDDHHMM, where the date and time is in UTC time.
| + | |
| − | | + | |
| − | ** The feature that has your bundle must also be released, similar to the above step. This only has to be done whenever the feature actually changes, which is likely to be just once, when you first add the bundle to the feature.
| + | |
| − | | + | |
| − | * Once released, the build system should notice this, and starts a new build. It will upload the results to the [http://download.eclipse.org/tools/orbit/committers/ committers area]. The build loop currently is set to 1 hour, so if you do not see an updated build on the committers area within an hour and a half, please contact one of the Orbit release engineers.
| + | |
| − | | + | |
| − | == Orbit Builds for Orbit Builds-masters ==
| + | |
| − | | + | |
| − | The build system uses CruiseControl to drive the build-loop.
| + | |
| − | | + | |
| − | The [http://build.eclipse.org:9777/cruisecontrol/ status] is displayed via the web, and for those with the right "build-master" password, there is an [http://build.eclipse.org:9000/ agent] that can control the builds. If either of these links are dead, it means the CruiseControl server is not running and needs to be restarted.
| + | |
| − | | + | |
| − | This system was created by using the WTP build as a starting point, and then simplified and changed as needed. So, in some cases, there is likely old stuff left over from the WTP process that is not really needed in Orbit. Feel free to clean up as you find them.
| + | |
| − | | + | |
| − | === An overview of the releng related project ===
| + | |
| − | | + | |
| − | All the build control scripts and templates are in CVS:
| + | |
| − | | + | |
| − | * 'org.eclipse.orbit.releng.builder': this project contains the heart of the build process.
| + | |
| − | **The 'components' directory codifies 'what' is built. As new build-feature sets are added, a new component (directory) needs to be added to build that feature.
| + | |
| − | **The 'distribution' directory codifies 'how' something is built, uploaded, prepared for the download site, etc.
| + | |
| − | **The 'scripts' directory holds various scripts used during the process.
| + | |
| − | **The 'build.xml' file in the root of the project is the main entry point, and is called from the releng.control project.
| + | |
| − | * 'org.eclipse.orbit.releng.control': this project holds the code that controls the build-loop, downloads fresh builders, etc.
| + | |
| − | * 'org.eclipse.orbit.releng.dropsite': this project holds the code that makes up the download site build pages.
| + | |
| − | | + | |
| − | === How to update the base builder ===
| + | |
| − | | + | |
| − | The base builder is from the Eclipse Platform releng team. It is mostly a "stripped down" version of Eclipse, with a few extra tools and ant tasks. It is important, of course, since it provides the PDE build that we want to use. As the platform team makes forward progress, they periodically tag the base builder -- usually a few weeks after each milestone -- and that version of the base builder is used to build the next milestone. Orbit should stay current with the latest version of the base builder, to be sure we all stay well integrated -- in our case, one main need for this is to be sure we use the most recent qualifier versioning rules.
| + | |
| − | | + | |
| − | ==== Conceptual principles ====
| + | |
| − | | + | |
| − | Like everything else, the base builder we use is specified in various property files and ant tasks. For our "production builds" we need to specify a tagged version of the base builder (not just "HEAD") so that our builds are reproducible. It is not often, if ever, we actually do reproduce a build ... but, that is simply a required software engineering practice.
| + | |
| − | | + | |
| − | We accomplish this in sort of an indirect way (for ease of use, mostly). Our own orbit builder specifies which version of the base builder we want to use, and our orbit builder is the one that is tagged and documented for each build. The specification of which orbit builder to use is in the HEAD version of build.cfg, but then that releng project is tagged for each build, so the information is retrievable from CVS after the fact. [Note: in writing this up, I'd discovered this "principle" is not actually true yet ... I've somehow lost that in the translation (will track in [https://bugs.eclipse.org/bugs/show_bug.cgi?id=171904 bug 171904])].
| + | |
| − | | + | |
| − | ==== Step by Step ====
| + | |
| − | <ol>
| + | |
| − | <li>
| + | |
| − | Find the latest tagged version of the [http://wiki.eclipse.org/index.php/Platform-releng-basebuilder base builder] to use.
| + | |
| − | <li>
| + | |
| − | In the 'org.eclipse.orbit.releng.builder' project,
| + | |
| − | the build.properties file contains a property named
| + | |
| − | 'eclipse.builder.version'
| + | |
| − | and would be set to some value like M3_33, so the whole line would be
| + | |
| − | eclipse.builder.version=M3_33
| + | |
| − | So, change it to be the tag desired, found above in the first step. For example
| + | |
| − | eclipse.builder.version=M4_33
| + | |
| − | <li>
| + | |
| − | Commit and tag the builder project ('org.eclipse.orbit.releng.builder')
| + | |
| − | using a conventional tag such as vYYYYMMDDHHMM (using UTC).
| + | |
| − | Keep track of or swipe-copy the tag name, for the next step.
| + | |
| − | <li>
| + | |
| − | In the HEAD of
| + | |
| − | 'org.eclipse.orbit.releng'
| + | |
| − | There is a file
| + | |
| − | 'maps\build.cfg'
| + | |
| − | that contains a 'builderVersion' property
| + | |
| − | that should be set to the tag just created.
| + | |
| − | So, for example, you would end up with
| + | |
| − | builderVersion=v200701270624
| + | |
| − | Commit this version of the file to HEAD, and you're all set.
| + | |
| − | The release to head should be detected and an autobuild started (within an hour or so) even if no other projects change.
| + | |
| − | </ol>
| + | |
| − | | + | |
| − | === How to declare a build ===
| + | |
| − | | + | |
| − | To "Declare a build" means, basically, everyone agrees (or, at least, doesn't object) to a particular build in the continuous 'committers' area being moved to the slightly more permanent 'downloads' area.
| + | |
| − | | + | |
| − | This is not an purely automatic process, by design, but someone (an Orbit committer, that is) must run a script which
| + | |
| − | copies a particular drop directory under 'committers' area to the drops directory of the download directory (with the same name). The other important, critical thing this script does is to "fix up" some of the links in the php files and in the HTTP version of the map file to point to the 'downloads' area version of a file, instead of the previous 'committers' area version of the file.
| + | |
| − | | + | |
| − | After a build is mechanically declared, at least a day or two should be allowed to elapse just in case anyone was temporarily using a 'committers' build to test getting some particular Orbit build into their consuming project.
| + | |
| − | | + | |
| − | Here's an annotated example of running the script.
| + | |
| − | | + | |
| − | | + | |
| − | First, logon to the build.eclipse.org machine (with your normal committers ID, and a normal SSH shell).
| + | |
| − | Navigate to the 'committers' directory. You need to run the script from, exactly, this level in the directory.
| + | |
| − | | + | |
| − | | + | |
| − | david_williams@build:~/downloads/tools/orbit> cd committers/
| + | |
| − | david_williams@build:~/downloads/tools/orbit/committers> ll
| + | |
| − | total 44
| + | |
| − | -rwxrwxr-x+ 1 david_williams orbitadmin 1158 2007-04-25 20:21 cleanup.sh
| + | |
| − | -rwxrwxr-x+ 1 david_williams orbitadmin 528 2007-03-19 11:29 declare.sh
| + | |
| − | -rwxrwxr-x+ 1 david_williams orbitadmin 483 2007-03-14 02:49 declareUpdates.sh
| + | |
| − | -rw-rwxr--+ 1 david_williams orbitadmin 134 2007-02-07 22:49 dlconfig.txt
| + | |
| − | drwxrwsr-x+ 14 david_williams orbitadmin 432 2007-05-23 22:10 drops
| + | |
| − | -rw-rwxr--+ 1 david_williams orbitadmin 11262 2007-05-01 01:35 index.php
| + | |
| − | -rw-rwxr--+ 1 david_williams orbitadmin 121 2007-01-16 14:52 phpinfo.php
| + | |
| − | -rw-rwxr--+ 1 david_williams orbitadmin 4493 2007-01-16 14:52 report.php
| + | |
| − | -rw-rwxr--+ 1 david_williams orbitadmin 4 2007-01-16 14:52 test.html
| + | |
| − | | + | |
| − | | + | |
| − | Get a listing of the directories under the drops directory. Typically, it is the last one in the list
| + | |
| − | that you will want to promote. (and you can use some copy/paste tricks to get the name, so the next step is easier).
| + | |
| − | | + | |
| − | david_williams@build:~/downloads/tools/orbit/committers> ll drops
| + | |
| − | total 6
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-17 13:34 I200705171711
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-18 11:00 I200705181436
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-18 22:56 I200705190235
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-20 06:21 I200705200959
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-22 10:52 S200705221427
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-22 14:07 S200705221744
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-22 19:16 S200705222254
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 12:42 S200705231619
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 15:07 S200705231844
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 16:20 S200705231959
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 18:46 S200705232222
| + | |
| − | drwxrwsr-x+ 5 david_williams orbitbuild 536 2007-05-23 22:10 S200705240148
| + | |
| − | | + | |
| − | | + | |
| − | Run the declare.sh script, using the directory name as the argument. Note, you need (only) the directory name under drops, the 'drops' part is assumed in the scripts.
| + | |
| − | | + | |
| − | david_williams@build:~/downloads/tools/orbit/committers> ./declare.sh S200705240148
| + | |
| − | | + | |
| − | Here is a example of the type of ouput you would see during a successful run.
| + | |
| − | | + | |
| − | S200705240148
| + | |
| − | declaring build S200705240148
| + | |
| − | into /home/data/users/david_williams/downloads/tools/orbit/downloads/drops
| + | |
| − | using the build from /home/data/users/david_williams/downloads/tools/orbit/committers/drops
| + | |
| − | david_williams@build:~/downloads/tools/orbit/committers>
| + | |
| − | | + | |
| − | After running the script, it can take anywhere from 5 to 30 minutes for all the internal cluster replication to take place and be "visible" on the HTTP downloads page.
| + | |
| − | | + | |
| − | ==Building Orbit In Your Workspace==
| + | |
| − | | + | |
| − | Here are some instructions on how to get an Orbit build running in your workspace.
| + | |
| − | | + | |
| − | * create project in your workspace to contain the builder (e.g. <code>orbit.builder</code>)
| + | |
| − | * you can create a project in your workspace to contain the build results if you want (e.g. <code>orbit.buildDirectory</code>)
| + | |
| − | * its also easier if you check out <code>org.eclipse.pde.build</code> into your workspace
| + | |
| − | * checkout the <code>org.eclipse.orbit.releng.tools</code> project from CVS
| + | |
| − | * export the tools project into <code>tools.jar</code> and put it in the root of the <code>orbit.builder</code> project.
| + | |
| − | * copy the <code>build.properties</code> file from <code>org.eclipse.pde.build/templates/headless-build/build.properties</code> to the root of your builder directory
| + | |
| − | * modify the following properties
| + | |
| − | ** <strong>buildDirectory</strong> - this is the directory for your builder
| + | |
| − | ** <strong>base</strong> - this is the location of the bundles that you will build against. Note that even if your bundles don't have dependancies, you still need this to point to a valid existing directory. (can be empty though)
| + | |
| − | ** <strong>resolution.devMode</strong> - setting this property allows you to build multiple versions of singleton bundles (like source bundles)
| + | |
| − | ** <strong>skipMaps</strong> - indicates whether or not you want to fetch the map files from CVS. Note you will need to comment this key/value out to set it to false. (changing the value is not enough)
| + | |
| − | ** <strong>mapsRepo</strong> - the location of the repository where you will get the map files
| + | |
| − | ** <strong>mapsRoot</strong> - the path in the repository to the project containing the map files
| + | |
| − | ** <strong>mapsCheckoutTag</strong> - the CVS tag for the map file project
| + | |
| − | ** <strong>skipFetch</strong> - indicates whether or not we should check the bundles out of the repository. Note you will need to comment this key/value out to set it to false. (changing the value is not enough)
| + | |
| − | | + | |
| − | <pre>
| + | |
| − | buildDirectory=D:/eclipse/workspace/orbit.buildDirectory
| + | |
| − | base=D:/eclipse
| + | |
| − | resolution.devMode=true
| + | |
| − | #skipMaps=true
| + | |
| − | mapsRepo=:pserver:anonymous@dev.eclipse.org/cvsroot/tools
| + | |
| − | mapsRoot=org.eclipse.orbit/org.eclipse.orbit.releng
| + | |
| − | mapsCheckoutTag=set1
| + | |
| − | #skipFetch=true
| + | |
| − | </pre>
| + | |
| − | | + | |
| − | * copy the <code>customTargets.xml</code> file from <code>org.eclipse.pde.build/templates/headless-build/customTargets.xml</code> to the root of your builder directory
| + | |
| − | ** make the following changes:
| + | |
| − | ** Add the following properties to the top of the file:
| + | |
| − | | + | |
| − | <pre>
| + | |
| − | <property name="featureToBuild" value="org.eclipse.orbit.build.feature.set1"/>
| + | |
| − | <property name="sourceFeature" value="org.eclipse.pde.build.container.feature"/>
| + | |
| − | </pre>
| + | |
| − | | + | |
| − | * Remove the declaration of the "allElementsFile" property and import of the file and then replace the "allElements" target with:
| + | |
| − | <pre>
| + | |
| − | <!--
| + | |
| − | <property name="allElementsFile" value="${builder}/allElements.xml"/>
| + | |
| − | <import file="${allElementsFile}" />
| + | |
| − | -->
| + | |
| − | <target name="allElements">
| + | |
| − | <ant antfile="${genericTargets}" target="${target}">
| + | |
| − | <property name="type" value="feature" />
| + | |
| − | <property name="id" value="${featureToBuild}" />
| + | |
| − | </ant>
| + | |
| − | <ant antfile="${genericTargets}" target="${target}">
| + | |
| − | <property name="type" value="feature" />
| + | |
| − | <property name="id" value="${sourceFeature}" />
| + | |
| − | </ant>
| + | |
| − | </target>
| + | |
| − | </pre>
| + | |
| − | | + | |
| − | * Add 2 new targets:
| + | |
| − | <pre>
| + | |
| − | <target name="assemble.org.eclipse.orbit.build.feature.set1">
| + | |
| − | <ant antfile="${assembleScriptName}" dir="${buildDirectory}">
| + | |
| − | <property name="archiveName" value="full.zip"/>
| + | |
| − | </ant>
| + | |
| − | </target>
| + | |
| − | <target name="assemble.org.eclipse.pde.build.container.feature">
| + | |
| − | <ant antfile="${assembleScriptName}" dir="${buildDirectory}">
| + | |
| − | <property name="archiveName" value="full.zip"/>
| + | |
| − | </ant>
| + | |
| − | </target>
| + | |
| − | </pre>
| + | |
| − | | + | |
| − | * Replace the preFetch target:
| + | |
| − | <pre>
| + | |
| − | <target name="preFetch">
| + | |
| − | <echo message="Fetching the main feature in order to generate the corresponding source feature" />
| + | |
| − | <taskdef name="eclipse.sourceFeatureGenerator"
| + | |
| − | classname="org.eclipse.orbit.internal.releng.tools.SourceFeatureGenerator"
| + | |
| − | classpath="${builder}/tools.jar" />
| + | |
| − | <eclipse.fetch
| + | |
| − | elements="feature@${featureToBuild}"
| + | |
| − | buildDirectory="${buildDirectory}"
| + | |
| − | directory="${buildDirectory}/directory.txt"
| + | |
| − | configInfo="${configs}"
| + | |
| − | baseLocation="${baseLocation}"
| + | |
| − | recursiveGeneration="false"
| + | |
| − | children="false"
| + | |
| − | />
| + | |
| − | <ant antfile="${buildDirectory}/fetch_${featureToBuild}.xml">
| + | |
| − | <property name="featureOnly" value="true"/>
| + | |
| − | </ant>
| + | |
| − |
| + | |
| − | <mkdir dir="${buildDirectory}/features/${sourceFeature}" />
| + | |
| − | <eclipse.sourceFeatureGenerator
| + | |
| − | inputFile="${buildDirectory}/features/org.eclipse.orbit.build.feature.set1/feature.xml"
| + | |
| − | outputDir="${buildDirectory}/features" mapFile="${buildDirectory}/directory.txt" />
| + | |
| − | <taskdef name="orbit.ipLog"
| + | |
| − | classname="org.eclipse.orbit.internal.releng.tools.IPLogTask"
| + | |
| − | classpath="${builder}/tools.jar" />
| + | |
| − | <orbit.ipLog
| + | |
| − | buildId="${buildLabel}"
| + | |
| − | inputDir="${buildDirectory}/maps/org.eclipse.orbit/org.eclipse.orbit.releng/ip_logs"
| + | |
| − | outputFile="${buildDirectory}/${buildLabel}/iplog.html" />
| + | |
| − | </target>
| + | |
| − | </pre>
| + | |
| − | | + | |
| − | * Remove the contents of the "postSetup" target:
| + | |
| − | <pre>
| + | |
| − | <target name="postSetup">
| + | |
| − | </target>
| + | |
| − | </pre>
| + | |
| − | | + | |
| − | | + | |
| − | * create a new launch configuration
| + | |
| − | ** new Eclipse Application
| + | |
| − | ** <code>org.eclipse.ant.core.antRunner</code> application
| + | |
| − | * add the following program arguments
| + | |
| − | ** <code>-buildfile ${resource_loc:/org.eclipse.pde.build/scripts/build.xml} -Dbuilder=d:/eclipse/workspace/orbit.builder</code>
| + | |
| − | | + | |
| − | That's it. Then when you run your application, it should build Orbit and put the results in your <code>orbit.buildDirectory</code> project. Note that you may want to turn on <code>Window -> Preferences -> General -> Workspace -> Refresh Automatically</code> to save you from having to press F5 to refresh.
| + | |
