Jump to: navigation, search

Difference between revisions of "Orion/Releng Builds"

(New page: The Orion builds follow many of the same practices used by the Eclipse Platform Project. The builds are based on PDE/Build and run automatically on build.eclipse.or...)
 
(Repository cache)
(15 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
The Orion builds follow many of the same practices used by the [[Platform-releng|Eclipse Platform Project]]. The builds are based on [[PDE/Build]] and run automatically on build.eclipse.org via cron job.  
 
The Orion builds follow many of the same practices used by the [[Platform-releng|Eclipse Platform Project]]. The builds are based on [[PDE/Build]] and run automatically on build.eclipse.org via cron job.  
  
There are two types of builds: Nightly and Integration. Nightly ("N") builds run every night and are built with the latest on the HEAD stream in the repository. Integration ("I") builds run once a week and build specific versions of the code as tagged in the repository.  
+
There is only one kind of build. Integration builds run daily from tagged repository versions.  [http://help.eclipse.org/topic//org.eclipse.pde.doc.user/tasks/pde_fetch_phase.htm Map files] are used to specify where a project is located in the repository and what version of that project to use in the build. The map files are located in the [http://git.eclipse.org/c/e4/org.eclipse.orion.server.git/tree/releng/org.eclipse.orion.releng/maps/orion.map org.eclipse.orion.releng] project. The builder itself performs the tagging so all the latest changes in "master" stream are picked up by every build. No manual tagging by committers is required.
  
 
== Contributing changes to the build  ==
 
== Contributing changes to the build  ==
  
Contributing to a nightly build is easy, just commit your code to the Orion repository and it will be included in the nightly build.
+
Contributing to a build is easy, just commit your code to the master stream of the Orion repository and it will be included in the next integration build.  
 
+
Contributing to an Integration build involves tagging your changes and modifying the map files. [http://help.eclipse.org/topic//org.eclipse.pde.doc.user/tasks/pde_fetch_phase.htm Map files] are used to specify where a project is located in the repository and what version of that project to use in the build. The map files are located in the [http://git.eclipse.org/c/e4/org.eclipse.orion.server.git/tree/releng/org.eclipse.orion.releng/maps/orion.map org.eclipse.orion.releng] project.  
+
  
 
=== Setting up your workspace ===
 
=== Setting up your workspace ===
Line 23: Line 21:
 
  git push                                              #push local commits to the master
 
  git push                                              #push local commits to the master
  
*Tag the git repository and push the tag to the master<br>
+
== How the build works ==
 +
 
 +
=== Bootstrap ===
 +
 
 +
The build is kicked off by a "bootstrap" script that is not under version control. The only purpose of this script is to fetch the real build script and invoke the build. The script is located at:
 +
 
 +
/opt/buildhomes/e4Build/bootstrap.orion.sh
 +
 
 +
The build type is passed as an argument. -I means integration build (with tagging). -N is equivalent but does not perform any tagging (in both cases the contents of master are used to perform the build).
 +
 
 +
You can also pass an -email argument to cause the build notifications to only be sent to an individual. This is useful for test builds:
 +
 
 +
bootstrap.orion.sh -I -email bob_mackenzie@example.com
 +
 
 +
The bootstrap script is typically invoked by a cron entry in the e4Build user account. For example here is a cron entry to start a build every Wednesday at 8:55am:
 +
 
 +
55 08 * * Wed /opt/buildhomes/e4Build/bootstrap.orion.sh -I
 +
 
 +
Invoke "crontab -e" to edit the cron entries (using vi), or "crontab -l" to simply list them.
 +
 
 +
=== masterBuild.sh ===
 +
 
 +
The bootstrap fetches and invokes the main build script, called [http://git.eclipse.org/c/orion/org.eclipse.orion.server.git/tree/releng/org.eclipse.orion.releng/builder/scripts/masterBuild.sh masterBuild.sh]. This script is under version control, so you need to push any changes to this script into the master branch for them to take effect. If you scroll to the very bottom of this script you can see the steps it performs:
 +
 
 +
tagRepositories # perform auto-tagging of the Orion git repositories
 +
updateRelengProject # fetch latest from Orion releng project (our scripts and maps)
 +
updateBaseBuilder # fetch the PDE builder to use in the build
 +
setProperties # various initialization steps
 +
runBuild # invoke the PDE build
 +
runTests # invoke client and server tests
 +
publish # send results email
 +
 
 +
=== build log ===
 +
 
 +
The results of the build, including any failures from any step, can found found in the build log at:
 +
 
 +
/shared/eclipse/e4/orion/logs/current.log
 +
 
 +
This is the first place to check when the build fails. It is typically very long so it is often helpful to use <tt>tail</tt> to start near the end. This exact same log gets copied into the download directory so it is available for reference for any build we produce. For example:
 +
 
 +
/home/data/httpd/download.eclipse.org/orion/drops/I201201110855/buildLog-I20120111-0855.txt
 +
 
 +
=== Repository cache ===
 +
 
 +
For performance reasons, PDE build maintains a cache of fetched prerequisite bundles. This lives at /shared/eclipse/e4/orion/target/transformedRepos on the build machine. The build will pick up the newest version of each required bundle from that location. If you ever need to force the build to go back to an *older* version of a bundle, this cache needs to be cleared. Simply delete the transformedRepos directory prior to the build.
 +
 
 +
== Deploying builds to orion.eclipse.org or orionhub.org ==
  
git tag v20110127-1300
+
Builds are deployed using the script [http://git.eclipse.org/c/orion/org.eclipse.orion.server.git/tree/releng/org.eclipse.orion.releng/builder/scripts/deploy.sh deploy.sh]. This script should be copied to your home directory and run from there. Your home directory will contain a symlink to the downloads directory, so you can perform a deploy directly from the downloads area. The script takes a single argument which is the location of the zip containing the download. For our servers we want the Linux 64-bit build. It is useful to log the output of this script so it can be reviewed later. Example:
git push --tags
+
  
====Tag Format====
+
./deploy.sh -archive downloads/orion/drops/I201201102230/eclipse-orion-I20120110-2230-linux.gtk.x86_64.zip >> deploy.log
The tag specified for a project in the map file will be used as the version qualifier for that bundle (or feature) in the build.
+
It is important for this tag to follow a standard format so that the bundle version increments properly over time.
+
  
The standard format to use is '''vYYYYMMDD-HHMM''' (HH is in 24 hour form)
+
This script simply copies the build onto the deployment server, and invokes an upgrade script on that server - [http://git.eclipse.org/c/orion/org.eclipse.orion.server.git/tree/releng/org.eclipse.orion.releng/builder/scripts/upgrade.sh upgrade.sh]. The upgrade script shuts down the old server, moves it, unzips and configures the new build, and finally starts it.
  
Unlike CVS, in git tags apply to the entire repository. It is important to include the time in the tag instead of just the day because if someone else tags another project in the same repository, we don't want the original tag to get moved, which would change which changes make it into the build.
+
The entire deploy/upgrade process takes about 5 seconds when it runs smoothly. Occasionally there will be a communication error copying the new build onto the target machine. In this case simply re-running the script usually succeeds.
  
===Update the Map File===
+
=== Server configuration ===
Open the map file containing the entry for your project.  This is usually '''org.eclipse.orion.releng/maps/orion.map'''. 
+
  
*Find the entry for the project containing your changes and update it to match your new tag.
+
The [[Orion/Server_admin_guide#Server_configuration_file|server configuration file]] on orion.eclipse.org and orionhub.org is found in ~admin/current/orion.conf. This file gets copied into the server by the deployment script. So, any change made to this configuration file will take effect only on the next deployment.
As an example, the entry for org.elipse.orion.client.core looks like this:
+
plugin@org.eclipse.orion.client.core=GIT,tag=v20110127-1300,repo=dev.eclipse.org:/gitroot/e4/org.eclipse.orion.client.git,path=bundles/org.eclipse.orion.client.core
+
                                          ^^^^^^^^^^^^^^^^^^
+
* Commit your change to the git repo and push it back to the master
+
git add releng/org.eclipse.orion.releng/maps/orion.map  #in the server repo, add this change to be committed
+
git commit -m "bug ####"                                #commit the change
+
git push                                                #push the change to the master
+
  
=Advanced Tagging=
+
= FAQ =
For lots of good reasons, a project's tag in the map file should generally only updated if that project contains changes.
+
If a single person is going to tag multiple projects and does not know which of those projects actually contain changes, then some tooling is required.
+
  
There is currently no equivalent of the cvs releng tool used by the Eclipse Platform team to manage map files. (See [https://bugs.eclipse.org/bugs/show_bug.cgi?id=328745  bug 328745]).  There is a shell script written for e4, see the [[E4/Git#Submitting_for_a_build]] page for details.
 
  
 
[[Category:Orion|Releng Builds]]
 
[[Category:Orion|Releng Builds]]

Revision as of 13:56, 24 February 2012

The Orion builds follow many of the same practices used by the Eclipse Platform Project. The builds are based on PDE/Build and run automatically on build.eclipse.org via cron job.

There is only one kind of build. Integration builds run daily from tagged repository versions. Map files are used to specify where a project is located in the repository and what version of that project to use in the build. The map files are located in the org.eclipse.orion.releng project. The builder itself performs the tagging so all the latest changes in "master" stream are picked up by every build. No manual tagging by committers is required.

Contributing changes to the build

Contributing to a build is easy, just commit your code to the master stream of the Orion repository and it will be included in the next integration build.

Setting up your workspace

  1. See Orion/Getting the source for instructions on how to set up your workspace to work with Orion source code.
  2. Import 'org.eclipse.orion.releng' into your workspace. It is located in the org.eclipse.orion.server.git repository under the releng folder.

Contributing your changes

Once you have made code changes, you can contribute them to the Integration build as follows:

  • Commit your changes to the local repository and push them to the master
git commit -a -m "fix lots of bugs"                   #commits all changes
git push                                              #push local commits to the master

How the build works

Bootstrap

The build is kicked off by a "bootstrap" script that is not under version control. The only purpose of this script is to fetch the real build script and invoke the build. The script is located at:

/opt/buildhomes/e4Build/bootstrap.orion.sh 

The build type is passed as an argument. -I means integration build (with tagging). -N is equivalent but does not perform any tagging (in both cases the contents of master are used to perform the build).

You can also pass an -email argument to cause the build notifications to only be sent to an individual. This is useful for test builds:

bootstrap.orion.sh -I -email bob_mackenzie@example.com

The bootstrap script is typically invoked by a cron entry in the e4Build user account. For example here is a cron entry to start a build every Wednesday at 8:55am:

55 08 * * Wed /opt/buildhomes/e4Build/bootstrap.orion.sh -I

Invoke "crontab -e" to edit the cron entries (using vi), or "crontab -l" to simply list them.

masterBuild.sh

The bootstrap fetches and invokes the main build script, called masterBuild.sh. This script is under version control, so you need to push any changes to this script into the master branch for them to take effect. If you scroll to the very bottom of this script you can see the steps it performs:

tagRepositories # perform auto-tagging of the Orion git repositories
updateRelengProject # fetch latest from Orion releng project (our scripts and maps)
updateBaseBuilder # fetch the PDE builder to use in the build
setProperties # various initialization steps
runBuild # invoke the PDE build
runTests # invoke client and server tests
publish # send results email

build log

The results of the build, including any failures from any step, can found found in the build log at:

/shared/eclipse/e4/orion/logs/current.log

This is the first place to check when the build fails. It is typically very long so it is often helpful to use tail to start near the end. This exact same log gets copied into the download directory so it is available for reference for any build we produce. For example:

/home/data/httpd/download.eclipse.org/orion/drops/I201201110855/buildLog-I20120111-0855.txt

Repository cache

For performance reasons, PDE build maintains a cache of fetched prerequisite bundles. This lives at /shared/eclipse/e4/orion/target/transformedRepos on the build machine. The build will pick up the newest version of each required bundle from that location. If you ever need to force the build to go back to an *older* version of a bundle, this cache needs to be cleared. Simply delete the transformedRepos directory prior to the build.

Deploying builds to orion.eclipse.org or orionhub.org

Builds are deployed using the script deploy.sh. This script should be copied to your home directory and run from there. Your home directory will contain a symlink to the downloads directory, so you can perform a deploy directly from the downloads area. The script takes a single argument which is the location of the zip containing the download. For our servers we want the Linux 64-bit build. It is useful to log the output of this script so it can be reviewed later. Example:

./deploy.sh -archive downloads/orion/drops/I201201102230/eclipse-orion-I20120110-2230-linux.gtk.x86_64.zip >> deploy.log

This script simply copies the build onto the deployment server, and invokes an upgrade script on that server - upgrade.sh. The upgrade script shuts down the old server, moves it, unzips and configures the new build, and finally starts it.

The entire deploy/upgrade process takes about 5 seconds when it runs smoothly. Occasionally there will be a communication error copying the new build onto the target machine. In this case simply re-running the script usually succeeds.

Server configuration

The server configuration file on orion.eclipse.org and orionhub.org is found in ~admin/current/orion.conf. This file gets copied into the server by the deployment script. So, any change made to this configuration file will take effect only on the next deployment.

FAQ