Difference between revisions of "EMF Facet/Releng/How to Use"

Latest revision as of 05:16, 30 May 2012

The EMF Facet project is built using Buckminster, with this releng project.

EMF Facet uses the same build system as MoDisco. So, for more explanation about how the build process works, see also MoDisco Releng : How it Works.

Step by step

• Install B3 using the update site http://download.eclipse.org/modeling/emft/b3/updates-3.7/
• Checkout the project org.eclipse.emf.facet.releng.tools
• Launch a 'runtime' Eclipse to enable 'org.eclipse.emf.facet.releng.tools' (target platform = 'Running platform')
• Checkout the project: :extssh:<committerId>@dev.eclipse.org:/cvsroot/callisto
• Checkout the project 'org.eclipse.indigo.build' from org.eclipse.emf.facet.releng.buckminster
• Right click on the file /org.eclipse.emf.facet.releng.buckminster Juno/emffacet.rmap and press on 'Update Rmap ...'
• Update orbit URL in /org.eclipse.emf.facet.releng.buckminster Juno/emffacet.rmap
  • Orbit's URL can be found at http://download.eclipse.org/tools/orbit/downloads/
• Launch the build : https://hudson.eclipse.org/hudson/job/emffacet-nightly-0.2.0/build?delay=0sec and set the following parameter
  • BUILD_TYPE=S
  • BUILD_ALIAS=0.2.0RC2
  • SIGN_UPDATE_SITE=true
• Wait the end of the build
• Log in to build.eclipse.org:22 (SSH)
• cd /opt/public/modeling/emft/facet/
• ./manualPromote.sh
• update the file /org.eclipse.juno.build/emft-emffacet.b3aggrcon (change the update site URL)
• Open /org.eclipse.juno.build/simrel.b3aggr
• Wait until all jobs are finished
• Right click on the first node and press 'Validate aggregation'
• Open the error log view to track the validation
• Commit the file /org.eclipse.juno.build/emft-emffacet.b3aggrcon

What is built?

Two features are built:

• org.eclipse.emf.facet.all.feature
• org.eclipse.emf.facet.tests.site.feature

They are referenced as root dependencies in buckminster.cspec.

These features must include all other features that must be built, either directly or indirectly.

The releng project contains a Buckminster rmap, which specifies how to get the plug-ins and features that are to be built. For each plug-in or feature, the map defines its location on a version control system (CVS, SVN).

The rmap also specifies the update sites from which to retrieve the binary dependencies from the dependent projects.

Updating the rmap

The rmap reflects the dependencies on the other projects. So, it must be updated at least before each milestone, to make sure that the code that is built works with the latest versions of all the dependencies.

The easiest way to update the dependencies is to copy them from the aggregation build model, which all projects must fill in with the correct reference to their update site(s) and feature(s).

This model is in a project with the release name on the Eclipse CVS:

• "org.eclipse.<releaseName>.build" on dev.eclipse.org:/cvsroot/callisto, where releaseName is helios or indigo or juno etc. depending on the version being built.

There is a tool that copies most of the information automatically in this plug-in:

• https://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk/releng/org.eclipse.emf.facet.releng.tools (see the README.TXT)

Warning: don't forget to update the aggregation model project before copying information from it

A few dependencies are not in the aggregation build model though, so they must be updated manually:

• Orbit : take the latest version from : http://download.eclipse.org/tools/orbit/downloads/
• SWTBot : it is not part of the simultaneous release (its update site has not changed since Helios anyway)

Finding dependencies

If you don't know where a particular plug-in or feature is located, the easiest way to find it is to search the Eclipse download area using "find" on build.eclipse.org. For example, to look for "org.eclipse.emf.compare" :

find /home/data/httpd/download.eclipse.org/modeling/emf/ -name 'org.eclipse.emf.compare_*'

How to start a build?

Automatically

45 2-23/3 * * *

• Nightly builds are run every 3 hours everyday, if the EMF Facet SVN changed since the last build.

• Integration builds are started manually, usually to prepare a milestone or release

Manually

• Only committers in the EMF Facet project, and members of the Callisto-dev group can launch EMF Facet build jobs from Hudson.
• Nightly build : go to https://hudson.eclipse.org/hudson/job/emffacet-nightly/
• Integration build : go to https://hudson.eclipse.org/hudson/job/emffacet-integration/

Then:

• In Hudson, click on Build Now, change the build parameters as needed (see #Build parameters), and click on Build.
• You can then click on the job name in the Build History section in the left column, and then on Console Output, to follow build progress in real time.

How to publish a build?

Automatically

Successful Nightly and Integration builds are automatically published to download.eclipse.org. For example, a nightly build for version 0.1.0, created on April 21 2011 at 04:13 would be published to:

• the download drops: http://download.eclipse.org/facet/downloads/drops/0.1.0/N201104210413
• the nightly update site: http://download.eclipse.org/facet/updates/nightly

These builds can then be seen and downloaded from http://www.eclipse.org/modeling/emft/facet/downloads/, where additional information is available (test results, build log), or installed from the update site: http://download.eclipse.org/facet/updates/nightly/

The publication is done by a script running as a cronjob under user nbros:

*/5 * * * * /opt/public/modeling/emft/facet/cronPromoteMonitor.sh

cronPromoteMonitor.sh runs cronPromote.sh which does the actual publication, and sends a mail to team members if the publication failed.

To re-run the publication of the last nightly build, log in to build.eclipse.org, and do:

cd /opt/public/modeling/emft/facet/
touch promoteSignalN && ./cronPromote.sh

With a script

First, if not already done, you need to retrieve all the necessary scripts and resources:

• create a directory (in your home directory for example) that will contain the scripts : mkdir emffacetBuildScripts
• get the scripts that will fetch all needed scripts: svn export http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk/releng/org.eclipse.emf.facet.releng.buckminster/serverConfiguration/updateServerSideScripts.sh
• make it executable: chmod +x updateServerSideScripts.sh
• launch it: ./updateServerSideScripts.sh

Then, you can launch the publishing script:

• run ./manualPromote.sh and fill in the parameters

Manually

Stable, Maintenance and Release builds are not automatically published. They should be first tested internally before publishing. For example, to publish the 0.1.0M6 milestone build:

• First, fetch the build archive to test (from Hudson, or using wget or scp for example):

wget https://hudson.eclipse.org/hudson/job/emffacet-integration/lastSuccessfulBuild/artifact/S201103151256.zip

or, if https access doesn't work, a NFS access can be used (from build.eclipse.org). For example:

cp /shared/jobs/emffacet-nightly/lastSuccessful/archive/S201103151256.zip .

• Then, test the build locally
• Then, publish it:

ssh <commiterid>@build.eclipse.org
wget https://hudson.eclipse.org/hudson/job/emffacet-integration/lastSuccessfulBuild/artifact/S201103151256.zip
unzip S201103151256.zip -d /home/data/httpd/download.eclipse.org/facet/downloads/drops/0.1.0/

Add a new update site with the new build to the composite:

• unzip the content (the update site zip that was in the first zip) in a new folder with the release number:

unzip EMFFacet-Update-0.1.1RC3.zip -d /home/data/httpd/download.eclipse.org/facet/updates/milestones/0.1/SR1_RC3

• update both the compositeContent.xml and compositeArtifacts.xml files of the update site (that are located in the parent of the folder to which you extracted the update site) to add a reference to your newly added update site
  • set the value of p2.timestamp to the result of "date +%s000"
  • increase the "size" attribute of the children element
  • add a "child" element inside the "children" element with a "location" set to the name of the folder (e.g "SR1_RC3")
• You can enable download stats on the repository by running /opt/public/modeling/emft/facet/addDownloadStats.sh on the update site
• Check that the new build appears on http://www.eclipse.org/modeling/emft/facet/downloads/.
• Builds can be hidden from this page before a release by modifying downloads-scripts.php in www/modeling/emft/facet/downloads/ on :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse
• Update the archive site with the new build:
  • unzip S201109070430.zip -d /home/data/httpd/archive.eclipse.org/facet/downloads/drops/0.1.1/
  • update /home/data/httpd/archive.eclipse.org/facet/downloads/index.html with a link to the newly added update zip
• Tag the build

Simultaneous Release

If the build must be part of the simultaneous release, you must also:

• Use the B3 Aggregator (or a text editor if the modification is trivial) to update the build model for the aggregator (after checking it out from CVS).
• Open juno.b3aggr in the B3 Aggregator editor, right-click on the first root node and select Validate Aggregation. Check in the error log that there are no errors.

Release

There are no "release builds". The release is a release candidate that is promoted to an actual release by following these steps:

• copy the last release candidate's update site (e.g downloads/facet/updates/milestones/0.1/SR2_RC4) to a release update site (e.g downloads/facet/updates/release/0.1.2/)
• fix the "p2.mirrorsURL" in the release update site's artifacts.jar, so that it matches the update site's location. For example:

<property name="p2.mirrorsURL" value="http://www.eclipse.org/downloads/download.php?file=/facet/updates/release/0.1.2&amp;format=xml&amp;protocol=http"/>

• copy the last release candidate drop (e.g downloads/facet/downloads/drops/0.1.2/S201202140309) to a release drop (e.g downloads/facet/downloads/drops/0.1.2/R201202140309).
• rename the update site in the drop so that it doesn't have a RCx suffix. E.g: R201202140309/EMFFacet-Update-0.1.2.zip

The release process for the simultaneous release is a bit more involved. The process is usually documented in a document on the wiki named "<release> final daze". E.g http://wiki.eclipse.org/Indigo/Final_Daze. It consists of:

• when creating the release update site, do not add the release update site to its parent composite yet: this must be done on the release day (when announced on the cross-project-issues-dev mailing list), in order to let mirroring complete before users try to download the bits
• hide the drops from the download page, by modifying the PHP script for the download page:

:dev.eclipse.org:/cvsroot/org.eclipse/www/modeling/emft/facet/downloads/downloads-scripts.php

• uncomment the line that contains the comment "XXX hide release until release day", and set the release folder to hide
• check after a few minutes that the release does not appear on http://www.eclipse.org/modeling/emft/facet/downloads/
• delete all the milestone (S20*) builds from the drops, and delete all the previous milestone update sites, in order to minimize the amount of data that mirrors have to copy
• ask the webmaster to update the Eclipse help infocenter with the new help plug-ins (e.g on Bug 348355 - Will soon need an Indigo Info Center)

Then on the release day — once the announce is made on the cross-project-issues-dev mailing list — do the following:

• show the release drop by re-commenting the line you uncommented previously in the download page PHP script.
• add the release update site to its parent composite
• update the documentation to add a reference to the new release update site

Build parameters

Hudson builds expect these parameters:

• BUILDTYPE : the kind of build, represented by a code letter (see this page):
  • N: Nightly
  • I: Integration
  • M: Maintenance (NOT milestone)
  • S: Stable (for Milestones and Release Candidate builds)
  • R: Release
• BUILD_ALIAS : the name of the release (0.8.0, 0.9.0M1, etc.); leave it blank for nightly and integration builds
• SIGN_UPDATE_SITE : whether to sign the update site (can take up to an hour on the Eclipse build server); mandatory for all builds except nightlies
• VERSION : the version being built. Dictates in which folder the build will be placed under the download "drops" folder.

Tagging

Releases should have a tag like R0_1_0, and milestones should have a tag like S0_1_0M7.

Tag names
Since the tag name is used as a bundle qualifier by PDE Build, it should not contain periods. For example, use R0_1_0 instead of v0.1.0

Checking

Check bundles

Check that each bundle contains an about.html file:

for f in $( ls *.jar ); do unzip -t $f | grep -q about.html || echo $f; done

or for plug-ins in your workspace:

find $workspaceRoot -name 'build.properties' | while read i; do grep -q about.html "$i" || echo "$i"; done

Check licenses

find $workspaceRoot -type f -name 'license.html' | while read i; do md5 $i; done

Check Manifest.MF

find $workspaceRoot -name 'MANIFEST.MF' | while read i; do grep -q Bundle-Vendor "$i" || echo "$i"; done
find $workspaceRoot -name 'feature.xml' | while read i; do grep -q provider-name "$i" || echo "$i"; done

Check p2.mirrorsURL

Each milestone and release update site should have a correct "p2.mirrorsURL" defined in its artifacts.jar (see Equinox/p2/p2.mirrorsURL).

To quickly check this on all update sites under the current directory, execute the following command:

 find . -type f -name 'artifacts.jar' -or -name 'artifacts.xml' | while read i; do 
   echo "$i"
   if [[ "$i" =~ .*/artifacts.jar ]]; then echo "  "$(unzip -p "$i"|grep "p2.mirrorsURL")
   else echo "  "$(cat "$i"|grep "p2.mirrorsURL"); fi
 done

Check simrel reports

for url in \
   "http://build.eclipse.org/juno/simrel/reports/layoutCheck.txt" \
   "http://build.eclipse.org/juno/simrel/reports/verifydiroutput/unsigned.txt" \
   "http://build.eclipse.org/juno/simrel/reports/versionPatternCheck.txt" \
   "http://build.eclipse.org/juno/simrel/reports/breedata.txt" \
   "http://build.eclipse.org/juno/simrel/reports/pack200data.txt" \
   "http://build.eclipse.org/juno/simrel/reports/nonUniqueVersions.txt"
do
  echo "checking $url"
  curl -s -S "$url" | egrep 'modisco|facet'
done

url="http://build.eclipse.org/juno/simrel/reports/licenseConsistency.html"
echo "checking $url"
curl -s -S $url > licenseConsistency
n=$(cat licenseConsistency | grep -n "Features with matching" | sed 's/:.*//')
cat licenseConsistency | head -n $n | egrep 'modisco|facet'

url="http://build.eclipse.org/juno/simrel/reports/featureNames.html"
echo "checking $url"
curl -s -S $url > featureNames
n=$(cat featureNames | grep -n "Probably correct names" | sed 's/:.*//')
cat featureNames | head -n $n | egrep 'modisco|facet'

url="http://build.eclipse.org/juno/simrel/reports/bundleNames.html"
echo "checking $url"
curl -s -S $url > bundleNames
n=$(cat bundleNames | grep -n "Probably correct bundle name" | sed 's/:.*//')
cat bundleNames | head -n $n | egrep 'modisco|facet'

url="http://build.eclipse.org/juno/simrel/reports/providerNames.html"
echo "checking $url"
curl -s -S $url > providerNames
n=$(cat providerNames | grep -n "Probably using correct provider name" | sed 's/:.*//')
cat providerNames | head -n $n | egrep 'modisco|facet'

url="http://build.eclipse.org/juno/simrel/reports/copyrights.html"
echo "checking $url"
curl -s -S $url > copyrights
n=$(cat copyrights | grep -n "Features with copyrights that are probably ok" | sed 's/:.*//')
cat copyrights | head -n $n | egrep 'modisco|facet'

url="http://build.eclipse.org/juno/simrel/reports/esdata.txt"
echo "checking $url"
curl -s -S $url > esdata
n=$(cat esdata | grep -n "Bundles without an Eclipse-SourceReference" | sed 's/:.*//')
cat esdata | tail -n +$n | egrep 'modisco|facet'