Difference between revisions of "Platform-releng-basics"

From Eclipsepedia

Jump to: navigation, search
m (Common build failures)
m (Rsync)
 
(39 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
== Location of Git and CVS repos  ==
 +
 +
Git and CVS repos for releng related activity
 +
 +
ssh://userid@git.eclipse.org/gitroot/platform/eclipse.platform.git
 +
<br>branding plugins in platform/
 +
 +
ssh://userid@git.eclipse.org/gitroot/platform/eclipse.platform.releng.git
 +
<br>features in features/
 +
<br>branding plugins in bundles/
 +
<br>platform feature for 3.x stream builds in oldfeatures/
 +
 +
ssh://userid@git.eclipse.org/gitroot/platform/eclipse.platform.releng.maps.git
 +
<br>map files
 +
 +
ssh://userid@git.eclipse.org/gitroot/platform/eclipse.common.git
 +
<br>doc bundles in common/
 +
 +
The org.eclipse.releng.basebuilder and org.eclipse.releng.eclipsebuilder projects are still in CVS (/cvsroot/eclipse)
 +
They need to be migrated to Git by the end of 2012.  The basebuilder project should really be 1) In its own repo because of it's binary content or  2) Convert the build to use a product build from the repo of SDK bundles and consume the custom bundles separately or 3) Just extract a SDK and add custom bundles to it
 +
 +
The complete list of Git repos for the Eclipse and Equinox projects are here [[Platform-releng/Git_Workflows]]
 +
 
== Basic overview of platform builds  ==
 
== Basic overview of platform builds  ==
  
 
Integration builds are in crontab of build user on build machine  
 
Integration builds are in crontab of build user on build machine  
  
Integration builds run from the tags of org.eclipse.releng  
+
Integration builds run from the tags of org.eclipse.releng in the integration branch of org.eclipse.releng
<pre>0 8 * * 2 cd /builds; /home/users/releng/buildTools/eclipse37/runIBuild
+
<pre>0 8 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
 
</pre>  
 
</pre>  
Nightly builds run from HEAD of org.eclipse.releng  
+
Nightly builds run from mastter of org.eclipse.releng  
<pre>0 20 * * 1,2,3,5,0 cd /builds; /home/users/releng/buildTools/eclipse37/runNBuild-skipPerf
+
<pre>0 20 * * 1,2,3,5,0 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf
0 20 * * 4,6 cd /builds; /home/users/releng/buildTools/eclipse37/runNBuild-skipPerf
+
0 20 * * 4,6 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf
</pre>
+
3.5.x maintenance builds run from R3_6_maintenance branch of org.eclipse.releng
+
<pre>0 8 * * 3 cd /builds; /home/users/releng/buildTools/eclipse36x/runMBuild
+
 
</pre>  
 
</pre>  
 +
 
The machines used in the [http://wiki.eclipse.org/Platform-releng-faq#What_hardware_comprises_the_platform-releng_build_infrastructure.3F builds] and their locations are listed in the FAQ.  
 
The machines used in the [http://wiki.eclipse.org/Platform-releng-faq#What_hardware_comprises_the_platform-releng_build_infrastructure.3F builds] and their locations are listed in the FAQ.  
  
A script runs once a week to clean the test machines  
+
A script runs once a week to clean the test machines (Currently 6 PM, on Sunday)
<pre>0 18 * * 0 /home/users/releng/buildTools/scripts/buildblaster.sh
+
<pre>0 18 * * 0 /home/users/releng/buildTools/scripts/buildblaster.pl</pre>  
</pre>  
+
  
<br> 3.5.x test builds
+
<pre>clean git tagging repos on eclipsebuildserv
<pre>cd /builds; /home/users/releng/buildTools/eclipse35x/runMKimBuildTest
+
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37x
</pre>
+
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37
 
+
<br> 3.4.2 test builds
+
<pre>cd /builds;  /home/users/releng/buildTools/eclipse34x/runKimTestMBuild
+
</pre>
+
3.2.x test builds
+
<pre>cd /builds; /home/users/releng/buildTools/eclipse32x/runMTestBuild-skipPerf
+
 
</pre>
 
</pre>
 +
 +
== Test builds for older maintenance streams ==
 +
 +
<br>Eclipse 3.7.2+ test builds from R3_7_maintenance branch of org.eclipse.releng in Git (automatically tagged from R3_7_maintenance branch)
 +
<pre>20 9 * * 4 cd /builds; /home/users/releng/buildTools/eclipse37x/runKimMBuildtest</pre>
 +
 +
<br>Eclipse 3.6.2+ test builds from R3_6_maintenance branch of org.eclipse.releng in Git (not tagged automatically, have to update maps manually)
 +
<pre>26 16 * * 1 cd /builds; /home/users/releng/buildTools/eclipse36x/runMKimBuildTest</pre>
 +
 +
<br>Eclipse 3.6.2+ + Java7 from R3_6_maintenance_Java7 of org.eclipse.releng in Git (not tagged automatically, have to update maps manually)
 +
<pre>26 18 * * 1 cd /builds; /home/users/releng/buildTools/eclipse36x/runM362Java7</pre>
 +
 +
<br>Eclipse 3.5.x test builds from R3_5_maintenance branch of org.eclipse.releng in CVS
 +
<pre>cd /builds; /home/users/releng/buildTools/eclipse35x/runMKimBuildTest</pre>
 +
 +
<br> 3.4.2 test builds from R3_4_maintenance branch of org.eclipse.releng in CVS
 +
<pre>cd /builds;  /home/users/releng/buildTools/eclipse34x/runKimTestMBuild</pre>
 +
 +
<br> 3.2.x test builds from R3_2_maintenance branch of org.eclipse.releng in CVS
 +
<pre>cd /builds; /home/users/releng/buildTools/eclipse32x/runMTestBuild-skipPerf</pre>
  
 
==Build scripts==
 
==Build scripts==
Line 41: Line 75:
  
 
<pre>
 
<pre>
  /home/users/releng/buildTools/eclipse36/runIBuild
+
  /home/users/releng/buildTools/eclipse37/runIBuild
 
</pre>
 
</pre>
  
Line 49: Line 83:
 
cd /builds
 
cd /builds
  
command="/home/users/releng/buildTools/eclipse36/bootstrap.sh -notify platform-releng-dev@eclipse.org -buildDirectory /builds/I -tagMapFiles -compareMaps -sign -updateSite /builds/transfer/files/updates/3.6-I-builds I"
+
command="/home/users/releng/buildTools/eclipse38/bootstrap.sh -notify platform-releng-dev@eclipse.org -buildDirectory /builds/I -tagMapFiles -compareMaps -sign -updateSite /builds/transfer/files/updates/3.8-I-builds I"
  
 
/home/users/releng/buildTools/extraTools/monitor.sh $command
 
/home/users/releng/buildTools/extraTools/monitor.sh $command
  
 
</pre>
 
</pre>
 
You will notice that the integration build has a compareMaps flag passed to the build.  This paramater takes the current version of the map file project, and compares it to the version here in the the file "/home/users/releng/buildTools/eclipse36/mapTag.properties"
 
 
-bash-3.00$ cat /home/users/releng/buildTools/eclipse36/mapTag.properties
 
 
        lastMapTag=I20100429-1549
 
 
The bootstrap script that initiates 3.6 stream builds is located on the build machine here...
 
  
 
<pre>
 
<pre>
/home/users/releng/buildTools/eclipse36/bootstrap.sh
+
/home/users/releng/buildTools/eclipse38/bootstrap.sh
 
</pre>
 
</pre>
  
 
If you look search for "buildProjectTags" in this file, you'll see something like this
 
If you look search for "buildProjectTags" in this file, you'll see something like this
 
This indicates the tag of the the three releng projects (org.eclipse.releng.eclipsebuilder, org.eclipse.releng.basebuilder, eclipseInternalBuildTools) used to run the build.  If you need to run the build with a new version of the builder, update this tag.
 
This indicates the tag of the the three releng projects (org.eclipse.releng.eclipsebuilder, org.eclipse.releng.basebuilder, eclipseInternalBuildTools) used to run the build.  If you need to run the build with a new version of the builder, update this tag.
buildProjectTags=v20100430
+
buildProjectTags=v20120305
  
 
== Rsync ==
 
== Rsync ==
  
 
The build is rsynced from eclipsebuildserv to fullmoon.ottawa.ibm.com to download.eclipse.org.  Look in kmoir's crontab for the scripts.
 
The build is rsynced from eclipsebuildserv to fullmoon.ottawa.ibm.com to download.eclipse.org.  Look in kmoir's crontab for the scripts.
 +
 +
kmoir's crontab on eclipsebuildserv (will be deleted March 23)
 +
 +
<pre>
 +
#rsync
 +
10,20,30,45,50 * * * * cd /home/users/releng/buildTools/scripts; ./ottAll fullmoon.ottawa.ibm.com >> /dev/null
 +
 +
#nightly and integration builds
 +
0 8 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
 +
0 20 * * 1,2,3,5,0 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf
 +
#0 20 * * 5 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf
 +
0 20 * * 4,6 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild
 +
#0 20 * * 6 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild
 +
 +
#maintenance builds
 +
#0 8 * * 3 cd /builds; /home/users/releng/buildTools/eclipse37x/runMBuild
 +
 +
#test builds
 +
#3.5.x
 +
#10 17 * * 1 cd /builds; /home/users/releng/buildTools/eclipse35x/runMKimBuildTest
 +
 +
#milestone builds
 +
#0 8 * * 5 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
 +
#0 20 * * 0,2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
 +
#0 8,13,18 * * 1,3 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild-skipPerf
 +
#extra easter milestone builds
 +
#0 1 * * 4 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
 +
#0 8,13,18 * * 4 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild-skipPerf
 +
 +
#test builds
 +
50 16 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runKimNBuildTest
 +
#4 16 * * 4 cd /builds; /home/users/releng/buildTools/eclipse38/runKimNBuildTest42
 +
#0 4 * * 5 cd /builds;  /home/users/releng/buildTools/eclipse37/runIBuildJava7
 +
#16 17 * * 1 cd /builds;  /home/users/releng/buildTools/eclipse37x/runKimMBuildtest
 +
#34 11 * * 2 cd /builds;  /home/users/releng/buildTools/eclipse36/runKimIBuildTest
 +
#48 16 * * 3 cd /builds;  /home/users/releng/buildTools/eclipse36x/run36patchesTest
 +
#32 15 * * 3 cd /builds;  /home/users/releng/buildTools/eclipse36x/runMKimBuildTest
 +
#55 10 * * 3 cd /builds;  /home/users/releng/buildTools/eclipse36x/runMKimBuildTestIES361
 +
#47 11  * * 4 cd /builds; /home/users/releng/buildTools/eclipse35x/runMKimBuildTest
 +
#12 11 * * 4 cd /builds; /home/users/releng/buildTools/eclipse36x/runM362Java7
 +
 +
#clean test machines
 +
0 18 * * 0 /home/users/releng/buildTools/scripts/buildblaster.pl
 +
 +
#clean git tagging repos
 +
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37x
 +
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37
 +
#clean build machine of old builds
 +
 +
#perf tests
 +
#0 18 * * 1 /home/users/releng/buildTools/eclipse.perf/perf35.sh
 +
#0 18 * * 5 /home/users/releng/buildTools/eclipse.perf/perf36.sh
 +
#0 18 * * 5 /home/users/releng/buildTools/eclipse.perf/perf37.sh
 +
 +
##restart build
 +
#47 15 * * 3 cd /builds/perf37_201203071509/3.7_perf_37x/eclipsePerformanceBuildTools; sh command.txt
 +
#11 16 * * 3 cd /builds/N201203042000/org.eclipse.releng.eclipsebuilder; sh command.txt
 +
 +
#clean N-builds dir
 +
30 17 * * 3,4,6 rm -rf  /builds/transfer/files/updates/3.7-N-builds/*
 +
30 17 * * 3,4,6 rm -rf  /builds/transfer/files/updates/3.8-N-builds/*
 +
 +
#34x build
 +
#26 10 * * 3 cd /home/users/releng/buildTools/eclipse34x; ./runKimTestMBuild
 +
 +
30 6 * * *  ps -eo uid,pid,etime,cmd | egrep '^ *507' | grep rsh | grep epwin | egrep '([1-9]+-)?([0-9]{2}:?){3}' | awk '{print $2}'| xargs kill -9 > /dev/null
 +
 +
#21 9 * * 5  cd /builds; /home/users/releng/buildTools/eclipse34x/runKimTestMBuild
 +
 +
 +
#52 9 * * 4 cd /builds/I201203120800/org.eclipse.releng.eclipsebuilder; sh command.txt
 +
 +
# rebuild for PDE bug 2:00 Tuesday
 +
#0 14 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
 +
</pre>
 +
 +
kmoir's crontab on fullmoon
 +
 +
<pre>
 +
#rsync
 +
19,49 * * * * cd /home/users/kmoir/scripts; ./ottAll build.eclipse.org >> /dev/null
 +
 +
12,50 * * * * rsync -a -e "ssh -l kmoir" --delete --stats /home/www/eclipse/equinox/ kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/equinox/ --exclude devicekit --exclude sat >> /dev/null
 +
 +
15,45 * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.1-I-builds /home/www/eclipse/updates
 +
15,45 * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.2-I-builds /home/www/eclipse/updates
 +
20,50 * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.1 /home/www/eclipse/updates
 +
31,1  * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.2milestones /home/www/eclipse/updates
 +
 +
#clean n-builds directory
 +
18 0 * * 3,6 rm -rf /home/www/eclipse/updates/3.7-N-builds/*
 +
18 0 * * 3,6 rm -rf /home/www/eclipse/updates/3.8-N-builds/*
 +
 +
30 3  * * * /home/users/releng/buildTools/scripts/cleandrops.pl /home/www/eclipse/downloads/drops
 +
40 3  * * * /home/users/releng/buildTools/scripts/cleandrops.pl /home/www/eclipse/equinox/drops
 +
</pre>
  
 
== Common build failures  ==
 
== Common build failures  ==
Line 96: Line 218:
 
Since our build runs in an IBM lab, the http gets for orbit bundles from eclipse.org are usually redirected from download.eclipse.org to fullmoon.ottawa.ibm.com by the foundation. This redirection can be avoided by changing the url in the orbit.map from download.eclipse.org to www.eclipse.org/external. Sometimes it will take a day or so for the internal mirror to get the latest orbit build.  
 
Since our build runs in an IBM lab, the http gets for orbit bundles from eclipse.org are usually redirected from download.eclipse.org to fullmoon.ottawa.ibm.com by the foundation. This redirection can be avoided by changing the url in the orbit.map from download.eclipse.org to www.eclipse.org/external. Sometimes it will take a day or so for the internal mirror to get the latest orbit build.  
  
*Missing dependancies due to erroneous map file submission.&nbsp; Check that the map file refers to a version of a project that exists in the repo. If the version of a bundle in cvs doesn't match the one in the map files, ask team to resubmit and start a new build.  
+
*Missing dependencies due to erroneous map file submission.&nbsp; Check that the map file refers to a version of a project that exists in the repo. If the version of a bundle in Git doesn't match the one in the map files, ask team to resubmit and start a new build.  
*Build doesn't proceed - connent not updated etc.&nbsp; Check for stale cvs checkouts on eclipsebuildserv.&nbsp; ps -ef | grep cvs.&nbsp; If there is a cvs connection that's over an hour old, kill it and allow the build to proceed.
+
*Build doesn't proceed - connent not updated etc.&nbsp; Check for stale Git clone processes on eclipsebuildserv.&nbsp; ps -ef | grep git.&nbsp; If there is a git process that's over an hour old, kill it and allow the build to proceed.
 
*SDKs fail to provision due to missing dependancies. There are releng logs on each test results page of the build that provide logs of the from the call to the p2 director to provision the zips. The Equinox team is helpful in finding the cause of the problem. See this page for an example. http://download.eclipse.org/eclipse/downloads/drops/R-3.5-200906111540/buildlogs.php  
 
*SDKs fail to provision due to missing dependancies. There are releng logs on each test results page of the build that provide logs of the from the call to the p2 director to provision the zips. The Equinox team is helpful in finding the cause of the problem. See this page for an example. http://download.eclipse.org/eclipse/downloads/drops/R-3.5-200906111540/buildlogs.php  
*Missing dependancies due to errors in Manifest or missing dependancies in manifest. In the both cases, the error sent to the releng list will look something like http://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg09778.html  
+
*Missing dependencies due to errors in Manifest or missing dependencies in manifest. In the both cases, the error sent to the releng list will look something like http://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg09778.html  
*Dependancies expressed in manifest and feature don't match. This means that the p2 director cannot resolve the dependencies for the product it is trying to build. The resultant zips for that platform are 0 size. For instance, see bug [https://bugs.eclipse.org/bugs/show_bug.cgi?id=258489 | bug 258489]. Also, you can see the .log files that the directory built by following the ''Release Engineering build logs'' link off the ''Test results'' build page.
+
*Dependencies expressed in manifest and feature don't match. This means that the p2 director cannot resolve the dependencies for the product it is trying to build. The resultant zips for that platform are 0 size. For instance, see bug [https://bugs.eclipse.org/bugs/show_bug.cgi?id=258489 | bug 258489]. Also, you can see the .log files that the directory built by following the ''Release Engineering build logs'' link off the ''Test results'' build page.
*Message "Build failed, map files unchanged".  This means either one of two things.  One - a build ran and the map files were unchanged so it didn't need to run.  Two - you restarted a failed build but the map files didn't change. In the second case, you should update this /home/users/releng/buildTools/eclipse36/mapTag.properties to reflect the build id of the last successful build.  
+
*Message "Build failed, map files unchanged".  This means either one of two things.  One - a build ran and the map files were unchanged so it didn't need to run.  Two - you restarted a failed build but the map files didn't change. In the second case, you should update this /home/users/releng/buildTools/eclipse38/mapTag.properties to reflect the build id of the last successful build.  
 
*Message "A problem occured while invoking the director".  If there is a compile error in the build, the bundle with the compile error will not be published to the repository.  Thus when the director is invoked to build zips, the director operation will fail.   
 
*Message "A problem occured while invoking the director".  If there is a compile error in the build, the bundle with the compile error will not be published to the repository.  Thus when the director is invoked to build zips, the director operation will fail.   
*The director logs, mirror logs etc. are located on the test results pages, under release engineering logs. [http://download.eclipse.org/eclipse/downloads/drops/R-3.6-201006080911/buildlogs.php Example of 3.6 release engineering logs. ]
+
*The director logs, mirror logs etc. are located on the test results pages, under release engineering logs. [http://download.eclipse.org/eclipse/downloads/drops/R-3.6-201006080911/buildlogs.php Example of 3.6 release engineering logs. ] The location on the build machine filesystem is
*All build tasks are invoked via the userid kmoir
+
/builds/transfer/files/master/download/drops/<buildId>/buildlogs.  If the build has failed invoking the director, the director logs may not be copied to this location yet.  In this case, there will be director logs in /builds/<buildId>/org.eclipse.releng.basebuilder/configuration directory.
 +
*All build tasks are invoked via the userid kmoir on the internal build machine, test machines and eclipse.org.
  
 
<br>
 
<br>
Line 176: Line 299:
 
</pre>
 
</pre>
  
It's better to start the build from cron if you need the tests to proceed without having to retain your ssh session.
+
It's better to start the build from cron if you need the tests to proceed without having to retain your ssh session. (And/or, use 'screen' so work continues on host, even if connection lost.)
 +
 
 +
== Common tasks during a milestone  ==
 +
 
 +
'''Move to next Orbit milestone build'''
 +
 
 +
See [https://bugs.eclipse.org/bugs/show_bug.cgi?id=373958#c2 bug 373958 comment 2]
 +
 
 +
This is a very simple change this milestone. I just replaced the old Orbit buildId with the new one.
 +
 
 +
[http://git.eclipse.org/c/platform/eclipse.platform.releng.maps.git/commit/?id=55ec07487a54e6c4ba7392e88c917823e283d76b git commit to update to Orbit M6 build]
 +
 
 +
Usually there if there are new Orbit bundles released during a milestone that a team needs to consume, we run test builds before milestone week to ensure that the new Orbit bundles are the right shape etc. For example, see bug  [https://bugs.eclipse.org/bugs/show_bug.cgi?id=368174 bug 368174]
 +
 
 +
'''Move to latest EMF and ECF contributions'''
 +
 
 +
::[TODO: Kim, can you elaborate? correct?]
 +
 
 +
Update maps for pre-req'd version.  For the 4.2 build we consume EMF from http://download.eclipse.org/modeling/emf/emf/updates/2.8milestones/base/.  This repo has to have the Juno EMF contribution for that milestone.  Kenn Hussey is the one that we contact for this.
 +
 
 +
In the 3.8 build we do that same thing, consume new versions of ECF.  Not very often these days.  Scott Lewis is the contact for this and Ian Bull should know about the need to consume a new version. Again, similar to EMF, we consume a version of ECF at +0, where they are higher up on the release train stack.  So they usually provide a special repo for us to consume.
 +
 
 +
Update build.properties to point to right location (needed for JavaDoc builds to work right)
 +
 
 +
 
 +
'''Test milestone bundles in org.eclipse.releng.basebuilder'''
 +
 
 +
Release the relevant subset of bundles to org.eclipse.releng.basebuilder and run a test build to ensure the milestone build can build Eclipse.  Update the builder to the milestone build after it's released, run test build.
 +
 
 +
'''Update b3aggrcon file for aggregation'''
 +
 
 +
This includes both ep and equinox b3aggrcon files. Must do after milestone is "final". Often helps to update with "near milestone" results a few days or week early, to give others an "early look" at what's coming.
 +
 
 +
'''Update composite artifacts to add new milestone repository'''
 +
 +
Every milestone, once final, update the platform's composite sites, such as update compositeContent.jar and compositeArtifacts.jar at
 +
 
 +
<pre><nowiki>/home/data/httpd/download.eclipse.org/eclipse/updates/4.2milestones</nowiki></pre>
 +
 
 +
[TODO: Kim, how/when do the "subdirectory repos" get copied there? (i.e. built there? or copied from build location? or mirrored from build location? Is there a script? ]
 +
 
 +
I just copy the milestone child repo /home/data/httpd/download.eclipse.org/eclipse/updates/4.2-I-builds to 4.2milestones and update the compositeContent.jar and compositeArtifacts.jar to point to the new repo. 
 +
 
 +
[TODO: Similar for weekly integration builds?]
 +
 
 +
For weekly integration builds, the build just adds a new child repo automatically. Once a milestone, I go and clean out the integration build repos.
 +
 
 +
== Other common tasks  ==
 +
 
 +
Move to a newer version of an Orbit bundle
 +
 
 +
#Update orbit.map with pointer to new location
 +
#Update appropriate platformOptions.txt file in org.eclipse.platform.doc.isv or jdtOptions.txt in org.eclipse.jdt.doc.isv to reflect the new name of the bundle as appropriate. 
 +
#Modify the appropriate build.properties in the feature that generates the source bundles for that Orbit bundle.  Source bundles for Orbit bundles should not be generated because they are contributed in binary form.  For instance, if you look at this build.properties for the Eclipse 3.8 SDK feature, you can see that the orbit bundles are all listed as "unpack=true". 
 +
 
 +
<pre>
 +
###############################################################################
 +
#  Copyright (c) 2000, 2009 IBM Corporation and others.
 +
#  All rights reserved. This program and the accompanying materials
 +
#  are made available under the terms of the Eclipse Public License v1.0
 +
#  which accompanies this distribution, and is available at
 +
#  http://www.eclipse.org/legal/epl-v10.html
 +
#
 +
#  Contributors:
 +
#    IBM Corporation - initial API and implementation
 +
###############################################################################
 +
bin.includes=eclipse_update_120.jpg,feature.xml,feature.properties
 +
 
 +
generate.feature@org.eclipse.platform.source=org.eclipse.platform,feature@org.eclipse.rcp.source,feature@org.eclipse.equinox.p2.user.ui.source;optional="true",plugin@org.eclipse.platform.doc.isv;unpack="false",\
 +
  plugin@org.apache.ant.source;version=1.8.2.qualifier;unpack="false",\
 +
  plugin@com.jcraft.jsch.source;version=0.1.44.qualifier;unpack="false",\
 +
  exclude@org.eclipse.platform.doc.user
 +
 
 +
generate.feature@org.eclipse.jdt.source=org.eclipse.jdt, plugin@org.eclipse.jdt.doc.isv;unpack="false",\
 +
plugin@org.junit.source;version=3.8.2.qualifier;unpack="false",\
 +
plugin@org.junit.source;version=4.8.2.qualifier;unpack="false",\
 +
plugin@org.hamcrest.core.source;version=1.1.0.qualifier;unpack="false",\
 +
exclude@org.eclipse.jdt.doc.user
 +
generate.feature@org.eclipse.pde.source=org.eclipse.pde,plugin@org.objectweb.asm.source;version=3.3.1.qualifier;unpack="false",\exclude@org.eclipse.pde.doc.user
 +
generate.feature@org.eclipse.cvs.source=org.eclipse.cvs
 +
generate.feature@org.eclipse.help.source=org.eclipse.help,\
 +
  plugin@javax.servlet.source;version=3.0.0.qualifier;unpack="false",\
 +
  plugin@javax.servlet.jsp.source;version=2.2.0.qualifier;unpack="false",\
 +
  plugin@org.apache.jasper.glassfish.source;version=2.2.2.qualifier;unpack="false",\
 +
  plugin@com.sun.el.source;version=2.2.0.qualifier;unpack="false",\
 +
  plugin@org.apache.commons.logging.source;version=1.0.4.qualifier;unpack="false",\
 +
  plugin@org.apache.lucene.source;version=2.9.1.qualifier;unpack="false",\
 +
  plugin@org.apache.lucene.analysis.source;version=2.9.1.qualifier;unpack="false",\
 +
  plugin@org.apache.lucene.core.source;version=2.9.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.jetty.continuation.source;version=8.1.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.jetty.http.source;version=8.1.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.jetty.io.source;version=8.1.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.jetty.security.source;version=8.1.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.jetty.server.source;version=8.1.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.jetty.servlet.source;version=8.1.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.jetty.util.source;version=8.1.1.qualifier;unpack="false",\
 +
  plugin@org.eclipse.javax.el.source;version=2.2.0.qualifier;unpack="false"
 +
 
 +
generatedVersionLength=45
 +
</pre>
 +
 
 +
Bug [https://bugs.eclipse.org/bugs/show_bug.cgi?id=367794 bug 367794]  is an example of a similar change, where the version of ECF was updated.
 +
 
 +
 
 +
[[Category:Eclipse_Platform_Releng| ]]

Latest revision as of 17:08, 21 March 2012

Contents

[edit] Location of Git and CVS repos

Git and CVS repos for releng related activity

ssh://userid@git.eclipse.org/gitroot/platform/eclipse.platform.git
branding plugins in platform/

ssh://userid@git.eclipse.org/gitroot/platform/eclipse.platform.releng.git
features in features/
branding plugins in bundles/
platform feature for 3.x stream builds in oldfeatures/

ssh://userid@git.eclipse.org/gitroot/platform/eclipse.platform.releng.maps.git
map files

ssh://userid@git.eclipse.org/gitroot/platform/eclipse.common.git
doc bundles in common/

The org.eclipse.releng.basebuilder and org.eclipse.releng.eclipsebuilder projects are still in CVS (/cvsroot/eclipse) They need to be migrated to Git by the end of 2012. The basebuilder project should really be 1) In its own repo because of it's binary content or 2) Convert the build to use a product build from the repo of SDK bundles and consume the custom bundles separately or 3) Just extract a SDK and add custom bundles to it

The complete list of Git repos for the Eclipse and Equinox projects are here Platform-releng/Git_Workflows

[edit] Basic overview of platform builds

Integration builds are in crontab of build user on build machine

Integration builds run from the tags of org.eclipse.releng in the integration branch of org.eclipse.releng

0 8 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild

Nightly builds run from mastter of org.eclipse.releng

0 20 * * 1,2,3,5,0 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf
0 20 * * 4,6 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf

The machines used in the builds and their locations are listed in the FAQ.

A script runs once a week to clean the test machines (Currently 6 PM, on Sunday)

0 18 * * 0 /home/users/releng/buildTools/scripts/buildblaster.pl
clean git tagging repos on eclipsebuildserv
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37x
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37

[edit] Test builds for older maintenance streams


Eclipse 3.7.2+ test builds from R3_7_maintenance branch of org.eclipse.releng in Git (automatically tagged from R3_7_maintenance branch)

20 9 * * 4 cd /builds; /home/users/releng/buildTools/eclipse37x/runKimMBuildtest


Eclipse 3.6.2+ test builds from R3_6_maintenance branch of org.eclipse.releng in Git (not tagged automatically, have to update maps manually)

26 16 * * 1 cd /builds; /home/users/releng/buildTools/eclipse36x/runMKimBuildTest


Eclipse 3.6.2+ + Java7 from R3_6_maintenance_Java7 of org.eclipse.releng in Git (not tagged automatically, have to update maps manually)

26 18 * * 1 cd /builds; /home/users/releng/buildTools/eclipse36x/runM362Java7


Eclipse 3.5.x test builds from R3_5_maintenance branch of org.eclipse.releng in CVS

cd /builds; /home/users/releng/buildTools/eclipse35x/runMKimBuildTest


3.4.2 test builds from R3_4_maintenance branch of org.eclipse.releng in CVS

cd /builds;  /home/users/releng/buildTools/eclipse34x/runKimTestMBuild


3.2.x test builds from R3_2_maintenance branch of org.eclipse.releng in CVS

cd /builds; /home/users/releng/buildTools/eclipse32x/runMTestBuild-skipPerf

[edit] Build scripts

Here is an overview of the build scripts used in the build.

http://wiki.eclipse.org/Platform-releng-faq#Is_the_Eclipse_platform_build_process_completely_automated.3F

As mentioned above, the builds are initiated by cron.

The integration build is run as follows

 /home/users/releng/buildTools/eclipse37/runIBuild
# !/bin/sh

cd /builds

command="/home/users/releng/buildTools/eclipse38/bootstrap.sh -notify platform-releng-dev@eclipse.org -buildDirectory /builds/I -tagMapFiles -compareMaps -sign -updateSite /builds/transfer/files/updates/3.8-I-builds I"

/home/users/releng/buildTools/extraTools/monitor.sh $command

/home/users/releng/buildTools/eclipse38/bootstrap.sh

If you look search for "buildProjectTags" in this file, you'll see something like this This indicates the tag of the the three releng projects (org.eclipse.releng.eclipsebuilder, org.eclipse.releng.basebuilder, eclipseInternalBuildTools) used to run the build. If you need to run the build with a new version of the builder, update this tag. buildProjectTags=v20120305

[edit] Rsync

The build is rsynced from eclipsebuildserv to fullmoon.ottawa.ibm.com to download.eclipse.org. Look in kmoir's crontab for the scripts.

kmoir's crontab on eclipsebuildserv (will be deleted March 23)

#rsync
10,20,30,45,50 * * * * cd /home/users/releng/buildTools/scripts; ./ottAll fullmoon.ottawa.ibm.com >> /dev/null

#nightly and integration builds
0 8 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
0 20 * * 1,2,3,5,0 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf
#0 20 * * 5 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild-skipPerf
0 20 * * 4,6 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild
#0 20 * * 6 cd /builds; /home/users/releng/buildTools/eclipse38/runNBuild

#maintenance builds
#0 8 * * 3 cd /builds; /home/users/releng/buildTools/eclipse37x/runMBuild

#test builds
#3.5.x
#10 17 * * 1 cd /builds; /home/users/releng/buildTools/eclipse35x/runMKimBuildTest

#milestone builds
#0 8 * * 5 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
#0 20 * * 0,2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
#0 8,13,18 * * 1,3 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild-skipPerf
#extra easter milestone builds
#0 1 * * 4 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild
#0 8,13,18 * * 4 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild-skipPerf

#test builds
50 16 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runKimNBuildTest
#4 16 * * 4 cd /builds; /home/users/releng/buildTools/eclipse38/runKimNBuildTest42
#0 4 * * 5 cd /builds;  /home/users/releng/buildTools/eclipse37/runIBuildJava7
#16 17 * * 1 cd /builds;  /home/users/releng/buildTools/eclipse37x/runKimMBuildtest
#34 11 * * 2 cd /builds;  /home/users/releng/buildTools/eclipse36/runKimIBuildTest
#48 16 * * 3 cd /builds;  /home/users/releng/buildTools/eclipse36x/run36patchesTest
#32 15 * * 3 cd /builds;  /home/users/releng/buildTools/eclipse36x/runMKimBuildTest
#55 10 * * 3 cd /builds;  /home/users/releng/buildTools/eclipse36x/runMKimBuildTestIES361
#47 11  * * 4 cd /builds; /home/users/releng/buildTools/eclipse35x/runMKimBuildTest
#12 11 * * 4 cd /builds; /home/users/releng/buildTools/eclipse36x/runM362Java7

#clean test machines
0 18 * * 0 /home/users/releng/buildTools/scripts/buildblaster.pl

#clean git tagging repos
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37x
0 18 * * 0 /home/users/releng/buildTools/scripts/cleandrops.pl /builds/eclipse/sdk37
#clean build machine of old builds

#perf tests
#0 18 * * 1 /home/users/releng/buildTools/eclipse.perf/perf35.sh
#0 18 * * 5 /home/users/releng/buildTools/eclipse.perf/perf36.sh
#0 18 * * 5 /home/users/releng/buildTools/eclipse.perf/perf37.sh

##restart build
#47 15 * * 3 cd /builds/perf37_201203071509/3.7_perf_37x/eclipsePerformanceBuildTools; sh command.txt
#11 16 * * 3 cd /builds/N201203042000/org.eclipse.releng.eclipsebuilder; sh command.txt

#clean N-builds dir
30 17 * * 3,4,6 rm -rf  /builds/transfer/files/updates/3.7-N-builds/*
30 17 * * 3,4,6 rm -rf  /builds/transfer/files/updates/3.8-N-builds/*

#34x build
#26 10 * * 3 cd /home/users/releng/buildTools/eclipse34x; ./runKimTestMBuild

30 6 * * *  ps -eo uid,pid,etime,cmd | egrep '^ *507' | grep rsh | grep epwin | egrep '([1-9]+-)?([0-9]{2}:?){3}' | awk '{print $2}'| xargs kill -9 > /dev/null

#21 9 * * 5  cd /builds; /home/users/releng/buildTools/eclipse34x/runKimTestMBuild


#52 9 * * 4 cd /builds/I201203120800/org.eclipse.releng.eclipsebuilder; sh command.txt

# rebuild for PDE bug 2:00 Tuesday
#0 14 * * 2 cd /builds; /home/users/releng/buildTools/eclipse38/runIBuild

kmoir's crontab on fullmoon

#rsync
19,49 * * * * cd /home/users/kmoir/scripts; ./ottAll build.eclipse.org >> /dev/null

12,50 * * * * rsync -a -e "ssh -l kmoir" --delete --stats /home/www/eclipse/equinox/ kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/equinox/ --exclude devicekit --exclude sat >> /dev/null

15,45 * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.1-I-builds /home/www/eclipse/updates
15,45 * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.2-I-builds /home/www/eclipse/updates
20,50 * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.1 /home/www/eclipse/updates
31,1  * * * *  rsync -ar -e "ssh -l kmoir" --delete kmoir@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/updates/4.2milestones /home/www/eclipse/updates

#clean n-builds directory
18 0 * * 3,6 rm -rf /home/www/eclipse/updates/3.7-N-builds/*
18 0 * * 3,6 rm -rf /home/www/eclipse/updates/3.8-N-builds/*

30 3  * * * /home/users/releng/buildTools/scripts/cleandrops.pl /home/www/eclipse/downloads/drops
40 3  * * * /home/users/releng/buildTools/scripts/cleandrops.pl /home/www/eclipse/equinox/drops

[edit] Common build failures

  • Fail to fetch bundles from Orbit - symptom - java net url timeout in build failure message. Start a new build -> www.eclipse.org timeouts are intermittent.
  • "Error occurred while transforming repository" usually means that the bundle couldn't be fetched from Orbit. For example
Build M20090825-1330 (Timestamp: 200908251330):  The following error occurred while executing this line:
/builds/M200908251330/org.eclipse.releng.eclipsebuilder/buildAll.xml:166: The following error occurred while executing this line:
/builds/M200908251330/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.5.1.R35x_20090721/scripts/build.xml:78: The following error occurred while executing this line:
/builds/M200908251330/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/master/customTargets.xml:18: The following error occurred while executing this line:
/builds/M200908251330/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/master/allElements.xml:16: The following error occurred while executing this line:
/builds/M200908251330/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.5.1.R35x_20090721/scripts/genericTargets.xml:59: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_master.xml:11: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_master.xml:73: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_org.eclipse.sdk.xml:11: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_org.eclipse.sdk.xml:41: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_org.eclipse.platform.xml:11: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_org.eclipse.platform.xml:904: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_org.eclipse.equinox.p2.user.ui.xml:10: The following error occurred while executing this line:
/builds/M200908251330/src/fetch_org.eclipse.equinox.p2.user.ui.xml:304: Error occurred while transforming repository.

Since our build runs in an IBM lab, the http gets for orbit bundles from eclipse.org are usually redirected from download.eclipse.org to fullmoon.ottawa.ibm.com by the foundation. This redirection can be avoided by changing the url in the orbit.map from download.eclipse.org to www.eclipse.org/external. Sometimes it will take a day or so for the internal mirror to get the latest orbit build.

  • Missing dependencies due to erroneous map file submission.  Check that the map file refers to a version of a project that exists in the repo. If the version of a bundle in Git doesn't match the one in the map files, ask team to resubmit and start a new build.
  • Build doesn't proceed - connent not updated etc.  Check for stale Git clone processes on eclipsebuildserv.  ps -ef | grep git.  If there is a git process that's over an hour old, kill it and allow the build to proceed.
  • SDKs fail to provision due to missing dependancies. There are releng logs on each test results page of the build that provide logs of the from the call to the p2 director to provision the zips. The Equinox team is helpful in finding the cause of the problem. See this page for an example. http://download.eclipse.org/eclipse/downloads/drops/R-3.5-200906111540/buildlogs.php
  • Missing dependencies due to errors in Manifest or missing dependencies in manifest. In the both cases, the error sent to the releng list will look something like http://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg09778.html
  • Dependencies expressed in manifest and feature don't match. This means that the p2 director cannot resolve the dependencies for the product it is trying to build. The resultant zips for that platform are 0 size. For instance, see bug | bug 258489. Also, you can see the .log files that the directory built by following the Release Engineering build logs link off the Test results build page.
  • Message "Build failed, map files unchanged". This means either one of two things. One - a build ran and the map files were unchanged so it didn't need to run. Two - you restarted a failed build but the map files didn't change. In the second case, you should update this /home/users/releng/buildTools/eclipse38/mapTag.properties to reflect the build id of the last successful build.
  • Message "A problem occured while invoking the director". If there is a compile error in the build, the bundle with the compile error will not be published to the repository. Thus when the director is invoked to build zips, the director operation will fail.
  • The director logs, mirror logs etc. are located on the test results pages, under release engineering logs. Example of 3.6 release engineering logs. The location on the build machine filesystem is

/builds/transfer/files/master/download/drops/<buildId>/buildlogs. If the build has failed invoking the director, the director logs may not be copied to this location yet. In this case, there will be director logs in /builds/<buildId>/org.eclipse.releng.basebuilder/configuration directory.

  • All build tasks are invoked via the userid kmoir on the internal build machine, test machines and eclipse.org.


[edit] Missing test results

  • The test results page of the build isn't updated until all the test results have completed. To see what machines are currently being used for tests look the the following directory. There will be marker files that correspond to the machine operating system and build number.
-bash-3.00$ ls /home/users/releng/buildTools/markers/*.marker
eclipse-macosx-M20090824-0800.marker
eclipse-rhelws5-6.0-M20090824-0800.marker
eclipse-rhelws5-M20090824-0800.marker
eclipse-rhelws5-perf-M20090824-0800.marker
eclipse-sled10-perf-M20090824-0800.marker
eclipse-win32xp-6.0-M20090824-0800.marker
eclipse-win32xp-M20090824-0800.marker
eclipse-win32xp-perf-M20090824-0800.marker

If you cat the marker files you can see the hostname of the machine that corresponds to the marker file

-bash-3.00$ cat /home/users/releng/buildTools/markers/*.marker
0=lb6mac
0=ejlnx2
0=ejlnx1
0=eplnx2
0=eplnx1
0=ejwin2
0=ejwin1
0=epwin2

The windows machines run tests via rsh. The linux and mac machines run tests via ssh. The windows machines need to be rebooted every so often. The old build artifacts are cleaned automatically by cron jobs.

  • The windows tests machines crash every so often. The event logs don't indicate that anything is wrong with the build. They are on the KVM next to the Eclipse rack in lab, on number 5 and 6. They can just be rebooted and logged into as the build user. I usually reboot these machines once a week so this doesn't happen.

[edit] Restarting the build

Each build has a corresponding directory on eclipsebuildserv in /builds. For instance /builds/I201004291549/. If you cd to the org.eclipse.releng.eclipsebuilder directory, there is a script called buildAll.xml. The main target looks something like this

<target name="main" depends="init">
                <antcall target="buildEclipseSourceDrops" />
                <antcall target="buildMasterFeature" />
                <parallel failonany="true">
                        <sequential>
                                <antcall target="updatePackProperties" />
                                <antcall target="signMasterFeature" />
                        </sequential>
                        <sequential>
                                <antcall target="buildSdkTestFeature" />
                                <ant antfile="${eclipse.build.configs}/../helper.xml" target="verifyCompile" />
                        </sequential>
                </parallel>
                <parallel failonany="true">
                        <sequential>
                                <antcall target="packageEclipseDistributables" />
                                <ant antfile="${equinox.build.configs}/equinox.prov/run.xml" />
                                <antcall target="packageRepos" />
                                <antcall target="packageEquinoxDistributables" />

                                <antcall target="apiTooling" />
                                <antcall target="publishEclipse" />
                                <antcall target="publishEquinox" />
                        </sequential>
                        <antcall target="testEclipse" />
                </parallel>
                <antcall target="publishRSS" /> 

If the build fails, you can restart it by commenting out the sections of the build that have already completed in the main target. Then cd to /builds/I201004291549/org.eclipse.releng.eclipsebuilder, for example, and run sh command.txt. This will allow the build to continue. Alternatively, you can add a cron entry to restart the build. For example,

36 20 * * 3 cd /builds/I201004291549/org.eclipse.releng.eclipsebuilder; sh command.txt

It's better to start the build from cron if you need the tests to proceed without having to retain your ssh session. (And/or, use 'screen' so work continues on host, even if connection lost.)

[edit] Common tasks during a milestone

Move to next Orbit milestone build

See bug 373958 comment 2

This is a very simple change this milestone. I just replaced the old Orbit buildId with the new one.

git commit to update to Orbit M6 build

Usually there if there are new Orbit bundles released during a milestone that a team needs to consume, we run test builds before milestone week to ensure that the new Orbit bundles are the right shape etc. For example, see bug bug 368174

Move to latest EMF and ECF contributions

[TODO: Kim, can you elaborate? correct?]

Update maps for pre-req'd version. For the 4.2 build we consume EMF from http://download.eclipse.org/modeling/emf/emf/updates/2.8milestones/base/. This repo has to have the Juno EMF contribution for that milestone. Kenn Hussey is the one that we contact for this.

In the 3.8 build we do that same thing, consume new versions of ECF. Not very often these days. Scott Lewis is the contact for this and Ian Bull should know about the need to consume a new version. Again, similar to EMF, we consume a version of ECF at +0, where they are higher up on the release train stack. So they usually provide a special repo for us to consume.

Update build.properties to point to right location (needed for JavaDoc builds to work right)


Test milestone bundles in org.eclipse.releng.basebuilder

Release the relevant subset of bundles to org.eclipse.releng.basebuilder and run a test build to ensure the milestone build can build Eclipse. Update the builder to the milestone build after it's released, run test build.

Update b3aggrcon file for aggregation

This includes both ep and equinox b3aggrcon files. Must do after milestone is "final". Often helps to update with "near milestone" results a few days or week early, to give others an "early look" at what's coming.

Update composite artifacts to add new milestone repository

Every milestone, once final, update the platform's composite sites, such as update compositeContent.jar and compositeArtifacts.jar at

/home/data/httpd/download.eclipse.org/eclipse/updates/4.2milestones

[TODO: Kim, how/when do the "subdirectory repos" get copied there? (i.e. built there? or copied from build location? or mirrored from build location? Is there a script? ]

I just copy the milestone child repo /home/data/httpd/download.eclipse.org/eclipse/updates/4.2-I-builds to 4.2milestones and update the compositeContent.jar and compositeArtifacts.jar to point to the new repo.

[TODO: Similar for weekly integration builds?]

For weekly integration builds, the build just adds a new child repo automatically. Once a milestone, I go and clean out the integration build repos.

[edit] Other common tasks

Move to a newer version of an Orbit bundle

  1. Update orbit.map with pointer to new location
  2. Update appropriate platformOptions.txt file in org.eclipse.platform.doc.isv or jdtOptions.txt in org.eclipse.jdt.doc.isv to reflect the new name of the bundle as appropriate.
  3. Modify the appropriate build.properties in the feature that generates the source bundles for that Orbit bundle. Source bundles for Orbit bundles should not be generated because they are contributed in binary form. For instance, if you look at this build.properties for the Eclipse 3.8 SDK feature, you can see that the orbit bundles are all listed as "unpack=true".
###############################################################################
#  Copyright (c) 2000, 2009 IBM Corporation and others.
#  All rights reserved. This program and the accompanying materials
#  are made available under the terms of the Eclipse Public License v1.0
#  which accompanies this distribution, and is available at
#  http://www.eclipse.org/legal/epl-v10.html
# 
#  Contributors:
#     IBM Corporation - initial API and implementation
###############################################################################
bin.includes=eclipse_update_120.jpg,feature.xml,feature.properties

generate.feature@org.eclipse.platform.source=org.eclipse.platform,feature@org.eclipse.rcp.source,feature@org.eclipse.equinox.p2.user.ui.source;optional="true",plugin@org.eclipse.platform.doc.isv;unpack="false",\
  plugin@org.apache.ant.source;version=1.8.2.qualifier;unpack="false",\
  plugin@com.jcraft.jsch.source;version=0.1.44.qualifier;unpack="false",\
  exclude@org.eclipse.platform.doc.user
   
generate.feature@org.eclipse.jdt.source=org.eclipse.jdt, plugin@org.eclipse.jdt.doc.isv;unpack="false",\
plugin@org.junit.source;version=3.8.2.qualifier;unpack="false",\
plugin@org.junit.source;version=4.8.2.qualifier;unpack="false",\
plugin@org.hamcrest.core.source;version=1.1.0.qualifier;unpack="false",\
exclude@org.eclipse.jdt.doc.user
generate.feature@org.eclipse.pde.source=org.eclipse.pde,plugin@org.objectweb.asm.source;version=3.3.1.qualifier;unpack="false",\exclude@org.eclipse.pde.doc.user
generate.feature@org.eclipse.cvs.source=org.eclipse.cvs
generate.feature@org.eclipse.help.source=org.eclipse.help,\
  plugin@javax.servlet.source;version=3.0.0.qualifier;unpack="false",\
  plugin@javax.servlet.jsp.source;version=2.2.0.qualifier;unpack="false",\
  plugin@org.apache.jasper.glassfish.source;version=2.2.2.qualifier;unpack="false",\
  plugin@com.sun.el.source;version=2.2.0.qualifier;unpack="false",\
  plugin@org.apache.commons.logging.source;version=1.0.4.qualifier;unpack="false",\
  plugin@org.apache.lucene.source;version=2.9.1.qualifier;unpack="false",\
  plugin@org.apache.lucene.analysis.source;version=2.9.1.qualifier;unpack="false",\
  plugin@org.apache.lucene.core.source;version=2.9.1.qualifier;unpack="false",\
  plugin@org.eclipse.jetty.continuation.source;version=8.1.1.qualifier;unpack="false",\
  plugin@org.eclipse.jetty.http.source;version=8.1.1.qualifier;unpack="false",\
  plugin@org.eclipse.jetty.io.source;version=8.1.1.qualifier;unpack="false",\
  plugin@org.eclipse.jetty.security.source;version=8.1.1.qualifier;unpack="false",\
  plugin@org.eclipse.jetty.server.source;version=8.1.1.qualifier;unpack="false",\
  plugin@org.eclipse.jetty.servlet.source;version=8.1.1.qualifier;unpack="false",\
  plugin@org.eclipse.jetty.util.source;version=8.1.1.qualifier;unpack="false",\
  plugin@org.eclipse.javax.el.source;version=2.2.0.qualifier;unpack="false"

generatedVersionLength=45

Bug bug 367794 is an example of a similar change, where the version of ECF was updated.