Jump to: navigation, search

Difference between revisions of "EMF Facet/Releng/How to Use"

Line 4: Line 4:
  
 
== What is built? ==
 
== What is built? ==
<code>org.eclipse.emf.facet.all</code> is the feature that is built (defined in buckminster.cspec).
+
Two features are built:
This feature must include all other features that must be built, either directly or indirectly.
+
* org.eclipse.emf.facet.all.feature
 +
* org.eclipse.emf.facet.tests.site.feature
  
The releng project contains a Buckminster rmap, which specifies how to get the plugins and features that are to be built. For each plugin or feature, the map defines its location on a version control system (CVS, SVN), and the version that must be used.
+
They are referenced as root dependencies in <code>buckminster.cspec</code>.
  
The rmap also specifies the update sites from which to retrieve the dependencies.
+
These features must include all other features that must be built, either directly or indirectly.
 +
 
 +
The releng project contains a Buckminster rmap, which specifies how to get the plug-ins and features that are to be built. For each plug-in or feature, the map defines its location on a version control system (CVS, SVN).
 +
 
 +
The rmap also specifies the update sites from which to retrieve the binary dependencies from the dependent projects.
  
 
== How to start a build? ==
 
== How to start a build? ==
Line 17: Line 22:
 
45 2-23/3 * * *
 
45 2-23/3 * * *
  
* Nightly builds are run every 3 hours everyday, if the [https://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk/plugins EMF Facet SVN] changed since the last build.
+
* Nightly builds are run every 3 hours everyday, if the [https://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk EMF Facet SVN] changed since the last build.
  
* Integration builds are started manually
+
* Integration builds are started manually, usually to prepare a milestone or release
  
 
=== Manually ===
 
=== Manually ===
Line 34: Line 39:
 
=== Automatically ===
 
=== Automatically ===
 
Successful '''N'''ightly and '''I'''ntegration builds are automatically published to download.eclipse.org. For example, a nightly build for version 0.1.0, created on April 21 2011 at 04:13 would be published to:
 
Successful '''N'''ightly and '''I'''ntegration builds are automatically published to download.eclipse.org. For example, a nightly build for version 0.1.0, created on April 21 2011 at 04:13 would be published to:
http://download.eclipse.org/facet/downloads/drops/0.1.0/N201104210413
+
* the download drops: http://download.eclipse.org/facet/downloads/drops/0.1.0/N201104210413
These builds can then be seen and downloaded from http://www.eclipse.org/modeling/emft/facet/downloads/, where additional information is available (test results, build log).
+
* the nightly update site: http://download.eclipse.org/facet/updates/nightly
 +
These builds can then be seen and downloaded from http://www.eclipse.org/modeling/emft/facet/downloads/, where additional information is available (test results, build log), or installed from the update site: http://download.eclipse.org/facet/updates/nightly/
  
 
=== Manually ===
 
=== Manually ===
Line 41: Line 47:
 
* First, fetch the build archive to test (from Hudson, or using <code>wget</code> or <code>scp</code> for example):
 
* First, fetch the build archive to test (from Hudson, or using <code>wget</code> or <code>scp</code> for example):
 
  wget https://hudson.eclipse.org/hudson/job/emffacet-integration/lastSuccessfulBuild/artifact/S201103151256.zip
 
  wget https://hudson.eclipse.org/hudson/job/emffacet-integration/lastSuccessfulBuild/artifact/S201103151256.zip
 +
or, if https access doesn't work, a NFS access can be used (from build.eclipse.org). For example:
 +
cp /shared/jobs/emffacet-nightly/lastSuccessful/archive/S201103151256.zip .
 
* Then, test the build locally
 
* Then, test the build locally
 
* Then, publish it:
 
* Then, publish it:
Line 47: Line 55:
 
  unzip S201103151256.zip -d /home/data/httpd/download.eclipse.org/facet/downloads/drops/0.1.0/
 
  unzip S201103151256.zip -d /home/data/httpd/download.eclipse.org/facet/downloads/drops/0.1.0/
 
Add a new update site with the new build to the composite:
 
Add a new update site with the new build to the composite:
* unzip the content in a new folder with the release number:
+
* unzip the content (the update site zip that was in the first zip) in a new folder with the release number:
 
  unzip EMFFacet-Update-0.1.1RC3.zip -d /home/data/httpd/download.eclipse.org/facet/updates/milestones/0.1/SR1_RC3
 
  unzip EMFFacet-Update-0.1.1RC3.zip -d /home/data/httpd/download.eclipse.org/facet/updates/milestones/0.1/SR1_RC3
* update <code>compositeContent.xml</code> and <code>compositeArtifacts.xml</code> to add a reference to your newly added update site
+
* update both the <code>compositeContent.xml</code> and <code>compositeArtifacts.xml</code> files of the update site (that are located in the parent of the folder to which you extracted the update site) to add a reference to your newly added update site
 
** set the value of p2.timestamp to the result of "<code>date +%s000</code>"
 
** set the value of p2.timestamp to the result of "<code>date +%s000</code>"
* You can enable download stats on the repository by running [http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk/releng/org.eclipse.emf.facet.releng.buckminster/serverConfiguration/addDownloadStats.sh addDownloadStats.sh] on the update site
+
** increase the "size" attribute of the children element
 +
** add a "child" element inside the "children" element with a "location" set to the name of the folder (e.g "SR1_RC3")
 +
* You can enable download stats on the repository by running [http://dev.eclipse.org/svnroot/modeling/org.eclipse.emft.facet/trunk/releng/org.eclipse.emf.facet.releng.buckminster/serverConfiguration/addDownloadStats.sh /opt/public/modeling/emft/facet/addDownloadStats.sh] on the update site
 
* Check that the new build appears on [http://www.eclipse.org/modeling/emft/facet/downloads/ http://www.eclipse.org/modeling/emft/facet/downloads/].
 
* Check that the new build appears on [http://www.eclipse.org/modeling/emft/facet/downloads/ http://www.eclipse.org/modeling/emft/facet/downloads/].
 
* Builds can be hidden from this page before a release by modifying <code>downloads-scripts.php</code> in <code>www/modeling/emft/facet/downloads/</code> on <code>:pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse</code>
 
* Builds can be hidden from this page before a release by modifying <code>downloads-scripts.php</code> in <code>www/modeling/emft/facet/downloads/</code> on <code>:pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse</code>
Line 61: Line 71:
 
=== Simultaneous Release ===
 
=== Simultaneous Release ===
 
If the build must be part of the simultaneous release, you must also:
 
If the build must be part of the simultaneous release, you must also:
* Use the [[Eclipse_b3/aggregator/manual|B3 Aggregator]] (or a text editor if the modification is trivial) to [[Indigo/Contributing_to_Indigo_Build|update the build model for the aggregator]] ([http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.indigo.build/?root=Callisto after checking it out from CVS]).
+
* Use the [[Eclipse_b3/aggregator/manual|B3 Aggregator]] (or a text editor if the modification is trivial) to [[Juno/Contributing_to_Juno_Build|update the build model for the aggregator]] ([http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.juno.build/?root=Callisto after checking it out from CVS]).
  
 
== Build parameters ==
 
== Build parameters ==
Line 71: Line 81:
 
** '''S''': Stable (for Milestones and Release Candidate builds)
 
** '''S''': Stable (for Milestones and Release Candidate builds)
 
** '''R''': Release
 
** '''R''': Release
* BUILD_ALIAS : the name of the release (0.8.0, 0.9.0M1, etc.); blank for nightly and integration builds
+
* BUILD_ALIAS : the name of the release (0.8.0, 0.9.0M1, etc.); leave it blank for nightly and integration builds
* SIGN_UPDATE_SITE : whether to sign the update site (takes about an hour on the Eclipse build server); mandatory for releases
+
* SIGN_UPDATE_SITE : whether to sign the update site (can take up to an hour on the Eclipse build server); mandatory for all builds except nightlies
 
* VERSION : the version being built. Dictates in which folder the build will be placed under the download "drops" folder.
 
* VERSION : the version being built. Dictates in which folder the build will be placed under the download "drops" folder.
* PROJRELENGROOT : The path on the SVN (trunk, branch or tag) from which to take the version of the releng project that must be used for this build.
 
  
 
== Tagging ==
 
== Tagging ==
Line 80: Line 89:
 
{{warning|Tag names|Since the tag name is used as a bundle qualifier by PDE Build, it should not contain periods. For example, use '''R0_1_0''' instead of <s>v0.1.0</s>}}
 
{{warning|Tag names|Since the tag name is used as a bundle qualifier by PDE Build, it should not contain periods. For example, use '''R0_1_0''' instead of <s>v0.1.0</s>}}
  
[[Category:Releng]]
+
== Checking ==
  
== Checking bundles ==
+
=== Check bundles ===
 
Check that each bundle contains an about.html file:
 
Check that each bundle contains an about.html file:
 
  for f in $( ls *.jar ); do unzip -t $f | grep -q about.html || echo $f; done
 
  for f in $( ls *.jar ); do unzip -t $f | grep -q about.html || echo $f; done
 
or for plug-ins in your workspace:
 
or for plug-ins in your workspace:
 
  find $workspaceRoot -name 'build.properties' | while read i; do grep -q about.html "$i" || echo "$i"; done
 
  find $workspaceRoot -name 'build.properties' | while read i; do grep -q about.html "$i" || echo "$i"; done
 +
 +
=== Check licenses ===
 +
find $workspaceRoot -type f -name 'license.html' | while read i; do md5 $i; done
 +
 +
=== Check Manifest.MF ===
 +
find $workspaceRoot -name 'MANIFEST.MF' | while read i; do grep -q Bundle-Vendor "$i" || echo "$i"; done
 +
find $workspaceRoot -name 'feature.xml' | while read i; do grep -q provider-name "$i" || echo "$i"; done
 +
 +
=== Check simrel reports ===
 +
for url in \
 +
    "http://build.eclipse.org/juno/simrel/reports/layoutCheck.txt" \
 +
    "http://build.eclipse.org/juno/simrel/reports/verifydiroutput/unsigned.txt" \
 +
    "http://build.eclipse.org/juno/simrel/reports/versionPatternCheck.txt" \
 +
    "http://build.eclipse.org/juno/simrel/reports/breedata.txt" \
 +
    "http://build.eclipse.org/juno/simrel/reports/pack200data.txt" \
 +
    "http://build.eclipse.org/juno/simrel/reports/nonUniqueVersions.txt"
 +
do
 +
  echo "checking $url"
 +
  curl -s -S "$url" | egrep 'modisco|facet'
 +
done
 +
 +
url="http://build.eclipse.org/juno/simrel/reports/licenseConsistency.html"
 +
echo "checking $url"
 +
curl -s -S $url > licenseConsistency
 +
n=$(cat licenseConsistency | grep -n "Features with matching" | sed 's/:.*//')
 +
cat licenseConsistency | head -n $n | egrep 'modisco|facet'
 +
 +
url="http://build.eclipse.org/juno/simrel/reports/featureNames.html"
 +
echo "checking $url"
 +
curl -s -S $url > featureNames
 +
n=$(cat featureNames | grep -n "Probably correct names" | sed 's/:.*//')
 +
cat featureNames | head -n $n | egrep 'modisco|facet'
 +
 +
url="http://build.eclipse.org/juno/simrel/reports/bundleNames.html"
 +
echo "checking $url"
 +
curl -s -S $url > bundleNames
 +
n=$(cat bundleNames | grep -n "Probably correct bundle name" | sed 's/:.*//')
 +
cat bundleNames | head -n $n | egrep 'modisco|facet'
 +
 +
url="http://build.eclipse.org/juno/simrel/reports/providerNames.html"
 +
echo "checking $url"
 +
curl -s -S $url > providerNames
 +
n=$(cat providerNames | grep -n "Probably using correct provider name" | sed 's/:.*//')
 +
cat providerNames | head -n $n | egrep 'modisco|facet'
 +
 +
url="http://build.eclipse.org/juno/simrel/reports/copyrights.html"
 +
echo "checking $url"
 +
curl -s -S $url > copyrights
 +
n=$(cat copyrights | grep -n "Features with copyrights that are probably ok" | sed 's/:.*//')
 +
cat copyrights | head -n $n | egrep 'modisco|facet'
 +
 +
url="http://build.eclipse.org/juno/simrel/reports/esdata.txt"
 +
echo "checking $url"
 +
curl -s -S $url > esdata
 +
n=$(cat esdata | grep -n "Bundles without an Eclipse-SourceReference" | sed 's/:.*//')
 +
cat esdata | tail -n +$n | egrep 'modisco|facet'
 +
 +
 +
[[Category:Releng]]

Revision as of 11:35, 21 December 2011

The EMF Facet project is built using Buckminster, with this releng project.

EMF Facet uses the same build system as MoDisco. So, for more explanation about how the build process works, see also MoDisco Releng : How it Works.

What is built?

Two features are built:

  • org.eclipse.emf.facet.all.feature
  • org.eclipse.emf.facet.tests.site.feature

They are referenced as root dependencies in buckminster.cspec.

These features must include all other features that must be built, either directly or indirectly.

The releng project contains a Buckminster rmap, which specifies how to get the plug-ins and features that are to be built. For each plug-in or feature, the map defines its location on a version control system (CVS, SVN).

The rmap also specifies the update sites from which to retrieve the binary dependencies from the dependent projects.

How to start a build?

Automatically

45 2-23/3 * * *

  • Nightly builds are run every 3 hours everyday, if the EMF Facet SVN changed since the last build.
  • Integration builds are started manually, usually to prepare a milestone or release

Manually

Then:

  • In Hudson, click on Build Now, change the build parameters as needed (see #Build parameters), and click on Build.
  • You can then click on the job name in the Build History section in the left column, and then on Console Output, to follow build progress in real time.

How to publish a build?

Automatically

Successful Nightly and Integration builds are automatically published to download.eclipse.org. For example, a nightly build for version 0.1.0, created on April 21 2011 at 04:13 would be published to:

These builds can then be seen and downloaded from http://www.eclipse.org/modeling/emft/facet/downloads/, where additional information is available (test results, build log), or installed from the update site: http://download.eclipse.org/facet/updates/nightly/

Manually

Stable, Maintenance and Release builds are not automatically published. They should be first tested internally before publishing. For example, to publish the 0.1.0M6 milestone build:

  • First, fetch the build archive to test (from Hudson, or using wget or scp for example):
wget https://hudson.eclipse.org/hudson/job/emffacet-integration/lastSuccessfulBuild/artifact/S201103151256.zip

or, if https access doesn't work, a NFS access can be used (from build.eclipse.org). For example:

cp /shared/jobs/emffacet-nightly/lastSuccessful/archive/S201103151256.zip .
  • Then, test the build locally
  • Then, publish it:
ssh <commiterid>@build.eclipse.org
wget https://hudson.eclipse.org/hudson/job/emffacet-integration/lastSuccessfulBuild/artifact/S201103151256.zip
unzip S201103151256.zip -d /home/data/httpd/download.eclipse.org/facet/downloads/drops/0.1.0/

Add a new update site with the new build to the composite:

  • unzip the content (the update site zip that was in the first zip) in a new folder with the release number:
unzip EMFFacet-Update-0.1.1RC3.zip -d /home/data/httpd/download.eclipse.org/facet/updates/milestones/0.1/SR1_RC3
  • update both the compositeContent.xml and compositeArtifacts.xml files of the update site (that are located in the parent of the folder to which you extracted the update site) to add a reference to your newly added update site
    • set the value of p2.timestamp to the result of "date +%s000"
    • increase the "size" attribute of the children element
    • add a "child" element inside the "children" element with a "location" set to the name of the folder (e.g "SR1_RC3")
  • You can enable download stats on the repository by running /opt/public/modeling/emft/facet/addDownloadStats.sh on the update site
  • Check that the new build appears on http://www.eclipse.org/modeling/emft/facet/downloads/.
  • Builds can be hidden from this page before a release by modifying downloads-scripts.php in www/modeling/emft/facet/downloads/ on :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse
  • Update the archive site with the new build:
    • unzip S201109070430.zip -d /home/data/httpd/archive.eclipse.org/facet/downloads/drops/0.1.1/
    • update /home/data/httpd/archive.eclipse.org/facet/downloads/index.html with a link to the newly added update zip
  • Tag the build

Simultaneous Release

If the build must be part of the simultaneous release, you must also:

Build parameters

Hudson builds expect these parameters:

  • BUILDTYPE : the kind of build, represented by a code letter (see this page):
    • N: Nightly
    • I: Integration
    • M: Maintenance (NOT milestone)
    • S: Stable (for Milestones and Release Candidate builds)
    • R: Release
  • BUILD_ALIAS : the name of the release (0.8.0, 0.9.0M1, etc.); leave it blank for nightly and integration builds
  • SIGN_UPDATE_SITE : whether to sign the update site (can take up to an hour on the Eclipse build server); mandatory for all builds except nightlies
  • VERSION : the version being built. Dictates in which folder the build will be placed under the download "drops" folder.

Tagging

Releases should have a tag like R0_1_0, and milestones should have a tag like S0_1_0M7.

Warning2.png
Tag names
Since the tag name is used as a bundle qualifier by PDE Build, it should not contain periods. For example, use R0_1_0 instead of v0.1.0


Checking

Check bundles

Check that each bundle contains an about.html file:

for f in $( ls *.jar ); do unzip -t $f | grep -q about.html || echo $f; done

or for plug-ins in your workspace:

find $workspaceRoot -name 'build.properties' | while read i; do grep -q about.html "$i" || echo "$i"; done

Check licenses

find $workspaceRoot -type f -name 'license.html' | while read i; do md5 $i; done

Check Manifest.MF

find $workspaceRoot -name 'MANIFEST.MF' | while read i; do grep -q Bundle-Vendor "$i" || echo "$i"; done
find $workspaceRoot -name 'feature.xml' | while read i; do grep -q provider-name "$i" || echo "$i"; done

Check simrel reports

for url in \
   "http://build.eclipse.org/juno/simrel/reports/layoutCheck.txt" \
   "http://build.eclipse.org/juno/simrel/reports/verifydiroutput/unsigned.txt" \
   "http://build.eclipse.org/juno/simrel/reports/versionPatternCheck.txt" \
   "http://build.eclipse.org/juno/simrel/reports/breedata.txt" \
   "http://build.eclipse.org/juno/simrel/reports/pack200data.txt" \
   "http://build.eclipse.org/juno/simrel/reports/nonUniqueVersions.txt"
do
  echo "checking $url"
  curl -s -S "$url" | egrep 'modisco|facet'
done
url="http://build.eclipse.org/juno/simrel/reports/licenseConsistency.html"
echo "checking $url"
curl -s -S $url > licenseConsistency
n=$(cat licenseConsistency | grep -n "Features with matching" | sed 's/:.*//')
cat licenseConsistency | head -n $n | egrep 'modisco|facet'
url="http://build.eclipse.org/juno/simrel/reports/featureNames.html"
echo "checking $url"
curl -s -S $url > featureNames
n=$(cat featureNames | grep -n "Probably correct names" | sed 's/:.*//')
cat featureNames | head -n $n | egrep 'modisco|facet'
url="http://build.eclipse.org/juno/simrel/reports/bundleNames.html"
echo "checking $url"
curl -s -S $url > bundleNames
n=$(cat bundleNames | grep -n "Probably correct bundle name" | sed 's/:.*//')
cat bundleNames | head -n $n | egrep 'modisco|facet'
url="http://build.eclipse.org/juno/simrel/reports/providerNames.html"
echo "checking $url"
curl -s -S $url > providerNames
n=$(cat providerNames | grep -n "Probably using correct provider name" | sed 's/:.*//')
cat providerNames | head -n $n | egrep 'modisco|facet'
url="http://build.eclipse.org/juno/simrel/reports/copyrights.html"
echo "checking $url"
curl -s -S $url > copyrights
n=$(cat copyrights | grep -n "Features with copyrights that are probably ok" | sed 's/:.*//')
cat copyrights | head -n $n | egrep 'modisco|facet'
url="http://build.eclipse.org/juno/simrel/reports/esdata.txt"
echo "checking $url"
curl -s -S $url > esdata
n=$(cat esdata | grep -n "Bundles without an Eclipse-SourceReference" | sed 's/:.*//')
cat esdata | tail -n +$n | egrep 'modisco|facet'