Jump to: navigation, search

Difference between revisions of "Common Build Procedures, Tips + Tricks"

(Suppressing Compiler Warnings)
(Suppressing Compiler Warnings)
Line 400: Line 400:
 
You can either fix your code to remove these warnings, or suppress the warnings themselves. See [http://dev.eclipse.org/viewcvs/indextech.cgi/*checkout*/org.eclipse.emft/releng/jet/buildAll.xml?rev=HEAD&content-type=text/xml /org.eclipse.emft/releng/jet/buildAll.xml], for an example of how to suppress these warnings.
 
You can either fix your code to remove these warnings, or suppress the warnings themselves. See [http://dev.eclipse.org/viewcvs/indextech.cgi/*checkout*/org.eclipse.emft/releng/jet/buildAll.xml?rev=HEAD&content-type=text/xml /org.eclipse.emft/releng/jet/buildAll.xml], for an example of how to suppress these warnings.
  
  <echo message="Set compilerArgs = '-enableJavadoc -encoding ISO-8859-1 -warn:-serial,-nls,-unchecked'"/>
+
  <echo message="Set compilerArgs = '-enableJavadoc -encoding ISO-8859-1 -warn:-serial'"/>
  <property name="compilerArg" value="-enableJavadoc -encoding ISO-8859-1 -warn:-serial,-nls,-unchecked" />
+
  <property name="compilerArg" value="-enableJavadoc -encoding ISO-8859-1 -warn:-serial" />
  
 
For a complete list of supported warning flags in 3.1/3.2, go here: [http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_api_compile.htm http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_api_compile.htm] (see Warning Options).
 
For a complete list of supported warning flags in 3.1/3.2, go here: [http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_api_compile.htm http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_api_compile.htm] (see Warning Options).

Revision as of 13:15, 21 April 2006

Introduction

This document is directed to the EMFT technology project committers and was created help them to set up and work on their projects. It describes the existing procedures that all EMFT subprojects are supposed to adopt, describing, for example, how the files should be organized, what the build files are, and the Integration and Milestone release process.

This is a live document! We will be enriching it as questions are raised.

Recent changes

Directory Structure <example>

We will ask you to organize your files as we've done for EMF and UML2. Basically you will need to create the following directory structure for each subdirectory you own under the EMFT module:

emft/[subproject]/
  plugins
  doc
  tests
  examples
  1. plugins:
    • the plugins that provide the function
    • the features that include these plugins + the branding plugins for these features
    • the SDK feature (to generate a bundle that includes source + the plugins)
  2. doc:
    • the doc plugins (with the build scripts)
    • the features that include these plugins + the branding plugins for these features
  3. tests:
    • the test plugins (with the Eclipse test framework artifacts - such as this one)
    • the features that include these plugins + the branding plugins for these features
  4. examples:
    • any plugin you want to use as example
    • the features that include these plugins + the branding plugins for these features

The directories containing features and fragments must be suffixed by -feature and -fragment respectively.

If you want source plugins and features, you will have to create them.

Plugin and Feature Files

The contents of your features and plugins directories should mimic what is available in the EMF and UML2 projects. Although this section tries to summarize the important points, "learning by example" is the recommended approach.

General Recommendations

  • Each feature and plugin directory should be also an Eclipse project, containing all the necessary files such as, for example, .project.
  • The .project must only refer to builders available to the open-source community. Also, since Eclipse 3M2, it should not refer to other plugin projects. <example>
  • We use CVS to backup the files, so...
    • You can, and probably should, use the $Id$ CVS tag. For more details and other tags read the CVS documentation
    • Add a .cvsignore file to keep unnecessary files (such as the output directory) out of CVS. <example>
  • Don't add unnecessary files to your plugins and features. If you use a "non-Eclipse standard" file, please ensure that it has a purpose and that that purpose is clear.
  • All plugins should provide the following files:
  • The build.properties file is extremely important and must be accurate to allow PDE to build your plugins and features. Please review the EMF build.properties files for plugins with code, documentation and branding plugins, and features.

Plugin with Code <example>

  • The EMF artifacts and model specifications (XML schema, for example) should be located in a models directory in the root of the plugin. Don't forget to include the genmodel to allow users to regenerate your code.
  • DO commit all generated code. This makes it easier to run your plugin from the workspace.
  • The source files should be placed in "source folders" and each "source folder" should have a different "output folder".
  • The plugins with code should be deployed as jarred plugins. In order to tip the PDE build to jar a plugin, you need to set the MANIFEST.MF's Bundle-ClassPath attribute to . and use . in the build.properties file. As an example, look at the ecore's manifest and build.properties mentally replacing ecore.jar by ..

Right now you are probably asking yourself why the ecore files are declaring a jar instead of using .. To make a long story short, this is necessary for EMF to bootstrap itself during development (required when you use EMF to develop EMF). When developing, we can easily zip the contents of the output folder into the appropriate jar and restart Eclipse. Our build "fixes" the files before the plugins are packaged. Btw, to jar your plugin, you will also need to add unpack="false" to the plugin reference in your feature. <example>

  • Almost all files should have a copyright. This is what is used for the Java files in EMF (please check the other types of files in the example):
/**
 * <copyright>
 *
 * Copyright (c) 2005 IBM Corporation and others.
 * All rights reserved.   This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 * 
 * Contributors: 
 *   IBM - Initial API and implementation
 *
 * </copyright>
 */
  • The META-INF/MANIFEST.MF must end with an empty line. Also, don't forget to add the following line to the manifest to ensure the plugin classes are properly initialized:
Eclipse-LazyStart: true
  • The .classpath files should only contain references to the JDK and PDE container and the definition of the source and output folders.
  • In order to ensure that your plugin.properties files can be translated, you will need to add the following "directives" to them. All strings after a given directive are supposed to conform to its rules:
NLS_MESSAGEFORMAT_ALL Each string is assumed to be processed by the MessageFormat class (single quote must be coded as 2 consecutive single quotes '').
NLS_MESSAGEFORMAT_NONE All strings are assumed to NOT be processed by the MessageFormat class (single quote must be coded as 1 single quote ').
NLS_MESSAGEFORMAT_VAR Strings which contain replacement variables are processed by the MessageFormat class (single quote must be coded as 2 consecutive single quotes ''). Strings which do NOT contain replacement variables are NOT processed by the MessageFormat class (single quote must be coded as 1 single quote ').
  • Add a schema for each extension point you define in your plugin. <example>

Documentation Plugin <example>

  • While the PDE build is able to automatically create the Ant build script for the plugins with source code, you will need to hard code and maintain the script for the documentation plugins. The easiest way of creating the script (called build.xml) is:
    1. After adding all the files to the plugin directory and configuring the build.properties file, right click the plugin.xml file and select PDE Tools->Create Ant Build File.
    2. Open the build.xml file and delete the tasks in the gather.bin.parts; target.
    3. In the build.jars, use the zip ant tasks to zip up the content of the plugin - this is required since Eclipse expects the documentation files to be in an archive file.
    4. Open the build.properties editor and select the "Custom Build" check box at the top of the page
    5. Every time you start code a new release, you will need to manually update this build script. See the "Release Process" section for more details on new releases.

Javadoc <example>


  • Note that unless you have images, references, or tutorials at the time you're creating these files, you'll have to toss a placeholder into them so they won't be pruned (ignored) when extracting from CVS. A simple (blank) .cvsignore file will do nicely.

Branding Plugin <example>

  • The "branding plugin" is the plugin that provides specific files to describe a feature - check the example to see what these files are.
  • It is recommended that the branding plugin be given the same name and id as the feature it describes.
  • The "branding plugin" may also contain source, so you don't need to create an additional plugin just to hold the branding files.

Features <example>

  • The PDE build process is based on features, so all your plugins must be referenced by a feature.
  • A feature directory can contain the definition of other features and plugins. As you can see in the example, we use this to define "derived" elements, like EMF's SDK feature, source feature and source plugin.
  • A feature's build.properties file can specify that the source plugin and feature should be generated. In EMF, this is done in the SDK feature's build.properties.

Qualifiers (1.0.0.qualifier)

  • To add qualifiers to features and plugins, you must have 4 pieces set up correctly.
1. Plugins <example>
  • Append .qualifier as the 4th field of the plugin version in every MANIFEST.MF file.
2. Features <example>
  • Append .qualifier as the 4th field of the feature version in every feature.xml file.
  • Ensure that if you have comments before the <feature> tag, they do NOT include the word "feature". If they do, 1.0.0.qualifier will not be replaced by PDE with the build's timestamp/id. This is documented in bug 129868.
  • Change the versions in all plugins included in feature.xml files to 0.0.0. PDE will replace 0.0.0 by the appropriate version during the build.
3. Doc
  • Ensure that org.eclipse.[subproject].doc/build.xml uses ${pluginVersion}.${forceContextQualifier} instead of ${pluginVersion} or a hard-coded version like 1.0.0. You may need to add a new property. <example>
<property name="pluginVersion" value="1.0.0"/>
  • Add these lines to the end of the gather.bin.parts target in org.eclipse.[subproject].doc/build.xml (line break added in path attribute). <example>
<eclipse.versionReplacer
  path="${destination.temp.folder}/org.eclipse.[subproject].doc_\
    ${pluginVersion}.${forceContextQualifier}"
  version="${pluginVersion}.${forceContextQualifier}"/>
4. Releng
  • Add the following lines to the create.label.properties target of releng/[subproject]/buildAll.xml. <example>
<property name="forceContextQualifier" value="v${timestamp}"/>
<echo file="${buildDirectory}/label.properties" append="true" >
 forceContextQualifier=${forceContextQualifier}
</echo>
  • OPTIONAL: If you would like your features' versions to be suffixed with a dash followed by a random string of letters and numbers, you must also add a line to your build.properties files. Why would you want this? It is apparently calculated from the CVS tags to ensure that the version for the features will change if the source changes in the build.
5. Releng
  • Add generateFeatureVersionSuffix=true to all build.properties files in every releng/[subproject]/builder/ directory. <example>
  • If available, add generateFeatureVersionSuffix=true to any build.properties.template files in releng/[subproject]/templateFiles/. <example>

Tip: due to property file parser limitations in Eclipse, it is recommended that you always leave an empty line at the end of .properties (and .properties.template) files.

Source Plugin & Feature <example>

  • To create a source plugin and feature, you must have 5 pieces set up correctly.
1. org.eclipse.[subproject]-feature/feature.xml <example>
This will define which *src.zip files will be created - one per plugin include in the feature. If you want source for your examples, do the same in your [subproject].examples-feature.
2. org.eclipse.[subproject]-feature/sourceTemplateFeature/* <example>
3. org.eclipse.[subproject]-feature/sourceTemplatePlugin/* <example>
These define the contents that will go into your source feature(s) and plugin(s).
4. org.eclipse.[subproject]-feature/org.eclipse.[subproject].sdk/feature.xml <example>
<includes
  id="org.eclipse.[subproject].source"
  version="1.0.0"
  match="compatible"/>
This connects your SDK to your .source plugins/features so that they will be included in the SDK.
5. org.eclipse.[subproject].*/build.properties (for ALL applicable plugins) <example>
# NLS_MESSAGEFORMAT_VAR
source.''[Bundle-ClassPath]'' = src/
output.''[Bundle-ClassPath]'' = bin/

src.includes = about.html
bin.includes = plugin.xml,\
               plugin.properties,\
               icons/,\
               ''[Bundle-ClassPath]'',\
               META-INF/

Note that [Bundle-ClassPath] is the jar name defined in the matching MANIFEST.MF, such as:

Bundle-ClassPath: eodmeditor.jar

These define what gets packaged in the *src.zip files. You should NOT use src.includes = src/ unless you want your source plugin to include BOTH a *src.zip (containing *.java source files) and the unpacked *.java source files themselves.

Customizing Zip Bundles

In the event that you have plugins or features that PDE will not normally bundle together, such as .ui plugins in the runtime zip, you will need to adjust the way PDE behaves in order to ensure all your plugins/features appear in your zips as you need them. If you need a custom zip, like EMF has for Models or Standalone, this is also how you can accomplish this.

There's two ways to customize what gets put into a zip bundle by PDE: the correct way (a) and the shortcut (b)

Option A

The first method involves creating a feature which sets up the included features/plugins that have to be in there, as with .sdk features in the EMFT subprojects (ocl, query, validation, transaction). See details in CVS. An example of this is org.eclipse.emf-feature/org.eclipse.emf.sdk. It is nested (rather than being its own org.eclipse.emf.[subproject].[bundlename]-feature) for cosmetic reasons (it looks in the file system).

When creating a new org.eclipse.emf.[subproject]-feature/org.eclipse.emf.[subproject].[bundlename] (or org.eclipse.emf.[subproject].[bundlename]-feature), you will need to ensure it's properly connected to the build harness in 3 ways:

1. First, you'll need a folder under your builder/ directory for the feature build, such as:
org.eclipse.emft/releng/validation/builder/sdk/

Into this folder must go a customTargets.xml and a build.properties file. Then make sure that customTargets.xml refers to the feature correctly, as in customTargets.xml. (Search for "sdk" on lines 9, 20, 187.)

2. Then, the buildAll.xml script must be told how to build the new zip, eg, buildAll.xml (line 139):
<target name="buildAll" depends="init">
  <ant antfile="build.xml" target="main">
    <property name="component" value="builder/sdk"/>
  </ant>
  ...
</target>
3. And finally, you need to add the custom feature to the mapfile & mapfile template, since it won't be generated by PDE into the mapfile automagically. Examples: org.eclipse.emft/releng/validation/maps/validation.map
org.eclipse.emft/releng/validation/templateFiles/validation.map.template

Add an entry such as (line break added for clarity):
feature@org.eclipse.emf.validation.sdk=@cvsTag@,@cvsRoot1@,,org.eclipse.emft/validation/plugins/ org.eclipse.emf.validation-feature/org.eclipse.emf.validation.sdk

Option B

The second method is faster (and, arguably, more hackish). Instead of one feature per zip, as above, you can have custom instructions/rules in the buildAll.xml script which allow you to copy extra files that would normally be excluded from the zip to clean up missing content. These instructions are kept in one place (ie., only one ant script), so maintenance is easier, but this solution should only be used to add files to existing bundles, not to create new, custom ones.

For example, there's a validation.ocl plugin which must be included in the validation runtime, but since the validation-feature makes no mention of it, it's excluded. So, to work around this, once the SDK is built, copy the validaton.ocl plugin & feature from the SDK zip to the runtime zip after its assembly. This is done in the org.eclipse.emft/releng/validation/buildAll.xml script:

<target name="buildAll" depends="init">
  <ant antfile="build.xml" target="main">
    <property name="component" value="builder/sdk" />
  </ant>
  
  <ant antfile="build.xml" target="main">
    <property name="component" value="builder/runtime" />
  </ant>
  ...
  
  <!-- add .ocl feature + plugin from SDK to runtime -->
  <zip update="true"
    destfile="${buildDirectory}/${buildLabel}/emft-${subprojectName}-runtime-${buildAlias}.zip">
    <zipfileset 
      src="${buildDirectory}/${buildLabel}/emft-${subprojectName}-SDK-${buildAlias}.zip">
      <include name="**/org.eclipse.emf.${subprojectName}.ocl*"/>
      <include name="**/org.eclipse.emf.${subprojectName}.ocl*/*"/>
    </zipfileset>
  </zip>
  ...
</target>

Standalone Zip Bundles

Building a Standalone Zip allows you to provide code for use outside Eclipse. This is not very well documented (yet), but to get a feel for how to build a Standalone bundle, have a look at the following EMF examples:

Build

org.eclipse.emf.releng.build/zips/standalone/
org.eclipse.emf.releng.build/buildAll.xml
org.eclipse.emf.releng.build/zips/build.xml
  (line 14, target "standalone")

Test

org.eclipse.emf.releng.build/testManifest.xml
org.eclipse.emf.releng.build/templateFiles/testManifest.xml.template
org.eclipse.emf.releng.build/tests/configs/local/customTest.xml

Build and Promotion

  • We use the Eclipse build process which relies on PDE. See this article if you are interested in the details. It also covers the JUnit automated tests that can be run during the build.
  • Promotion is the process of uploading the files generated by the build to Eclipse.org, together with the release notes, Javadoc for online browsing, what's new content, and others.
  • If you want to associate CVS artifact changes with a bugzilla, start the commit comment with the bugzilla number between square brackets. The comment need only describe what has changed in the file, given that the problem should be fully described in the bugzilla.
[12345] Description of what's changed in the file

The build process will check these comments to generate the release notes and other documents.

Build Server Directory Structure

Here's a quick snapshot of the directory structure used on the emft build server, so that you can reproduce it elsewhere.

The emft build server (emf.torolab) is running Debian 3.0 (Woody), kernel 2.4.25, with dual 2G AMD Opteron processors, 3G RAM, 1.5G swap, and 33G HD.

Path Purpose/Description
/var/www/emft/ internal web paths (build.php)
/var/www/technology/emft/ external web path mirrored to download.eclipse (downloads, etc.)
/var/www/technology/emft/build.options.txt file used by both build.php and downloads/index.php to display only those builds specified
/var/www/technology/emft/downloads/index.php where to download project zips
/var/www/technology/emft/[subproject]/downloads/ symlink to /home/www-data/build/emft/[subproject]/downloads/ (so that the downloads page can find the zips)
/home/www-data/build/emft/scripts/ build path to where shell scripts are initiated by the web (the apache1.3 user is www-data)
/home/www-data/build/emft/requests/ build path to where build.php's data (URLs and log of requests for builds) is stored
/home/www-data/build/emft/[subproject]/downloads/ where required .zips are downloaded prior to building (eg., Eclipse SDK, EMF SDK)
/home/www-data/build/emft/[subproject]/downloads/drops/ where [subproject] is built (all versions here, including any NLS zip, once available (if any))
/home/www-data/build/emft/[subproject]/downloads/drops/1.0.0/ where [subproject] v1.0.0 is built
/home/www-data/build/emft/[subproject]/downloads/drops/1.0.0/ [type][yyyymmddHHMM]/ a specific build's folder, including zips, test results, docs
/opt/ibm-java2-1.4 -> /opt/ibm-java2-ws-sdk-pj9xia32142-20050609 default build JDK / used for JUnit tests
/opt/sun-java2-1.4 -> /opt/j2sdk1.4.2_03 optional build JDK / used for EMF JDK tests
/opt/sun-java2-5.0 -> /opt/jdk1.5.0_03 optional build JDK / used for EMF JDK tests
/opt/ibm-java2-1.3 -> /opt/IBMJava2-131 used to test EMF 2.0.x ONLY
/opt/apache-ant-1.6.1 (or newer) build & utility scripts
/usr/bin/php4 (4.1.2 or newer) both Apache module and CLI (command line interface) modules required

Testing & Diagnosing Builds

To determine what's wrong with your build, make sure you're building in debug mode. eg., [1] or [2]. Then set the following parameters:


 Build Type: N (Nightly)
 Tag Build: No
 Run Tests: JUnit
 debug java home: /opt/ibm-java2-ws-sdk-pj9xia32142-20050609
 debug noclean: Y

Then, you can use the following shell commands:

# shortcut to builds folder
alias odd=' cd /home/www-data/build/emft/ocl/downloads/drops/1.0.0'

# shortcut to LATEST build folder
alias lastfolder='cd `find . -maxdepth 1 -type d -not -name \
  "." -exec date -r {} +%s\ {} \; | sort | tail -1 | \
  sed -e "s/[0-9]\+\ \.\///g"`'

alias oddl='odd; lastfolder'

# check build log
oddl; tail -60 buildlog.txt

# check JUnit test console log
oddl; cd testing/*/testing/; tail linux.gtk_consolelog.txt

# check JUnit test eclipse workspace error log
oddl; cd testing/*/testing/eclipse/workspace/.metadata/; less .log

Suppressing Compiler Warnings

After 3.2M6, PDE will now report compilation warnings in its JUnit test results page.

You can either fix your code to remove these warnings, or suppress the warnings themselves. See /org.eclipse.emft/releng/jet/buildAll.xml, for an example of how to suppress these warnings.

<echo message="Set compilerArgs = '-enableJavadoc -encoding ISO-8859-1 -warn:-serial'"/>
<property name="compilerArg" value="-enableJavadoc -encoding ISO-8859-1 -warn:-serial" />

For a complete list of supported warning flags in 3.1/3.2, go here: http://help.eclipse.org/help31/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_api_compile.htm (see Warning Options).

You can also bring up that same help page in Eclipse's built-in help (JDT Plug-in Developer Guide > Programmer's Guide > JDT Core > Compiling Java code) for the 3.2 version, which adds a couple other options to the list.

Build Problems & Solutions

In no particular order, here are some lessons learned from setting up the various EMFT builds:

Error Message Fix
eclipse/test.assembly/eclipse/ plugins/${org.eclipse.test} not found.

Add the "org.eclipse.test" and "org.eclipse.ant.optional.junit" plugins to each test feature. The task creates one property for each test plugin (the property name is the plugin name minus the version). Only plugins referenced by features are downloaded, and, if a plugin has to be downloaded, the map file has to describe how to do it. Thus this information must appear in two places: tests/customTargets.xml and *.tests-feature/feature.xml. Since "org.eclipse.test" is not listed on the tests feature, it is not being downloaded - regardless of it being listed in the map file.

org.eclipse.emft.ocl.releng/ builder/sdk/customTargets.xml -> eclipse/assemble.org.eclipse.emf.ocl.sdk.xml -> eclipse/tmp/eclipse/plugins/ org.eclipse.emf.ocl.doc_1.0.0.200602231606 not found

This could be a number of problems, but the most likely is that org.eclipse.emft/ocl/doc/org.eclipse.emf.ocl.doc/build.xml needs to have the correct value for pluginVersion set or is missing this property entirely.

<property name="pluginVersion" value="1.0.0"/>

If that doesn't solve the problem, run a debug build with the -noclean flag checked to see what's in the eclipse/tmp/eclipse/plugins/ folder. For the case of the above build, the folder created was called org.eclipse.emf.ocl.doc_${pluginVersion}.200602231617 instead of org.eclipse.emf.ocl.doc_1.0.0.200602231617, since the property was not defined.

java.io.FileNotFoundException: testing/target/eclipse/ plugins/org.eclipse.emf.ocl.tests_1.0.0/ (Plugin is a jar, not a folder)

If your test plugin is compiled as a jar, not a folder, you need to fix your feature.xml. Remove the attribute unpack="false" from the org.eclipse.emf.*.tests plugin:
<plugin id="org.eclipse.emf.ocl.tests" download-size="0" install-size="0" unpack="false" version="1.0.0"/>

java.io.FileNotFoundException: testing/target/eclipse/ plugins/org.eclipse.emf.ocl.tests_1.0.0/ test.xml (No such file or directory)

Add a "test.xml" script to each test plugin, in tests/org.eclipse.emf.*/test.xml.

If this file already exists, verify in releng/ [subproject]/ builder/ tests/ scripts/ test.xml that target runtests knows whether your tests plugin is jarred or not. Comment/uncomment out test.xml script accordingly.

java.io.FileNotFoundException: eclipse/fetch_org.eclipse.emft.ocl.sdk.xml (No such file or directory) Either the feature doesn't exist (and must be added), or the feature is misreferenced. Make sure that you do not have any scripts referring to, say, plugins or features called org.eclipse.emft.ocl.sdk, but rather org.eclipse.emf.ocl.sdk. The org.eclipse.emft namespace is only for the CVS module; everywhere else, you should use org.eclipse.emf when referencing plugins and features.
JUnits tests not appearing on testResults.php page

Add an entry for the missing tests into org.eclipse.emft/releng/[subproject]/ templateFiles/testManifest.xml.template and org.eclipse.emft/releng/[subproject]/ testManifest.xml, eg:


<logFile 
  name="org.eclipse.emf.query.ocl.tests_linux.gtk.xml">
<effectedFile id="SDK"></effectedFile>
<effectedFile id="projRUN"></effectedFile>
</logFile>
Failure to download and unzip one of the SDKs that your project depends on

You probably have a typo in the URL that you entered into the "New URLs" field. Try the following to get the correct URL every time:

  1. Surf to your dependency's downloads page on eclipse.org.
  2. Navigate to the build that you need.
  3. Click the download link for the all-encompassing SDK ZIP.
  4. When the "blah now downloading, if it doesn't start immediately, click here" message appears, right-click on the "click here" link and select "Copy Link Location" (or the equivalent action in your browser or choice).
  5. You now have in your cut buffer the exact link to the file that you need.

Depending on your configuration, you may have an extra navigation to a mirror site in the above sequence.

Release Process

Like EMF, all the EMFT subprojects are supposed to follow these release process:

  • One integration (I) build every Thursday morning (EST time). This means that by Wednesday night, your code in CVS must be "compile error free".
  • After releasing the final build of a given version, you need to remember to upgrade the version of the plugins you change. In other words, if a plugin is not changed then its version remains the same. After upgrading the version of a plugin, you need to ensure that the version of the appropriate features and branding plugins are also upgraded, and that the feature is referring to the new versions of the plugins. Also, if you change a plugin with source code, you may need to upgrade the appropriate documentation plugin (and documentation feature and branding plugin) if the build is adding Javadoc to it.

Promoting A Build

Adding new builds or even new subprojects is relatively simple. The hands-on manual approach is to use scp to copy you new build's folder from where it was built, eg., /home/www-data/build/emft/transaction/downloads/drops/1.0.0/I200601111814 to its destination on download1.eclipse.org, eg., /home/data/httpd/download.eclipse.org/technology/emft/transaction/downloads/drops/1.0.0/I200601111814. From there, /home/data/httpd/download.eclipse.org/technology/emft/downloads/index.php will find & display it automatically.

You can modify the way the downloads page appears in two simple ways by editing build.options.txt, which is used to both create builds and display them for downloading:

  • you can add/remove branches from display, by adding entries under the [Branch] entry. Display entries are in the form -=1.0.0, whereas build instruction entries are in the form 1.0.1=R1_0_maintenance.
  • you can reorder the way build types are displayed by resequencing the values under the [BuildType] entry. The remaining entries are used when building.

The above process ONLY adds the build to the downloads page; it does not fix permissions or group ownership of files, nor update any map files, news files, release notes, or other meta information. As such, the full process for promoting a build is rather lengthy, and thus has been simplified into a script, promoteToEclipse.sh and its accompanying property file, promoteToEclipse.properties.

The simplest way to learn how to use this script is to first run it without options to see the inline help, or to read the help within the script itself. From there, running it is straightforward, but certain SSH permisions must be set in order for things to work as expected.

To make CVS happy when promoting a build, ensure the existence of a .cvspass file in your home directory:

touch ~/.cvspass

As an example, here's how an EMFT Transaction build is promoted:

cd /home/www-data/build/emft/scripts; 
./promoteToEclipse.sh -sub transaction -buildID I200601111814 -Q \
  -announce 2>&1 | tee ~/promo_`date +%Y%m%d_%H%M%S`.txt

The -Q argument suppresses noisy CVS checkout messages and -announce posts an announcement of the build availability to the EMFT newsgroup.

Generating Release Notes

Generating release notes is easy - promoteToEclipse.sh does it for you! However, there are a few simple steps to follow to ensure the generated content is accurate & non-duplicative. Here's the general process for closing a bug and using Bugzilla & CVS to provide the information required to generate release notes.

  • Working in Eclipse, fix your code. Test is as much as possible before committing it to CVS.
  • Commit your change(s) to CVS, using the bug's number in square brackets to begin your commit comment, eg:
[127509] javadocs now available
  • Note that you can include multiple bugs (each number on its own line) and comments are not required. The information that will appear in the release notes comes from the bug's Summary field, not your CVS comments.
  • Browse to the bug's webpage, and assign it the status Assigned. This tells the release notes generator that this is a bug which should be included in the release notes this week. Your bugzilla comment could be something like:
Committed to CVS
  • Produce a build, and promote it. Unless you use the -nodocs flag, release notes will be generated automatically and will appear in /cvsroot/org.eclipse/www/emft/news/.
  • You can also use the -since flag to manually identify the start date & time used to generate the list of bugs. If omitted, the start will be the timestamp of the previous build as identified in /cvsroot/org.eclipse/www/emft/news/news.xml.
  • Verify that the contents of your project's release notes are correct. Depending on the time you closed the bug relative to the time the build was done, you may find duplicate bugs missing bugs. This can be corrected by editing the appropriate release-notes-[subproject].xml file.
  • In order to prevent bugs from being added to the release notes over and over, every week you should ensure any Assigned bugs which are available in a released build are marked Fixed.
Fixed in 1.0.0 M5a

Newsgroup Announcement Setup

Setup for posting automatically to the newsgroup to announce new builds is easy.

1. Copy newsgroup-post.txt into your ~/.ssh folder.

build_id@build_server:~ $ mkdir -p ~/.ssh; cp /home/nickb/newsgroup-post/newsgroup-post.txt ~/.ssh

You are now set up to post automatically to the newsgroup while promoting a build. If you want to post as a specific person, rather than the default name & email, there are two additional steps.

2. Edit org.eclipse.emft/releng/[subproject]/promoteToEclipse.[subproject].properties <example>, placing the correct name and email address for the person whose name will appear on newsgroup posts. This step - and the next one - is ONLY required if the name & email are not the same as the default value recorded here.
newsgroupPublisherEmail="John Q. Committer <john_q_emft_committer_guy@eclipse.org>"
3. Copy org.eclipse.emft/releng/[subproject]/promoteToEclipse.[subproject].properties <example> into /home/www-data/build/emft/scripts.

To announce a build that's already been built, you can use the -announceonly flag to skip all other steps and ONLY post to the newsgroup. Otherwise, adding -announce when promoting will add the extra step of posting a quick note to the newsgroup.

SSH Key Setup

You must have ssh access to your build server, download1.eclipse.org, and dev.eclipse.org. If you do not have access, please the administrators of these servers to gain access. Once you have ssh access, you need to set up limited ssh key access between your build server and the eclipse.org download and cvs servers. This will allow you to run the promote script without being prompted for a password periodically.

To grant yourself ssh access from your build server to eclipse.org, ssh to the build server normally.

Under windows, you can use cygwin or PuTTY. Under Linux, open a shell and type `ssh`.

If you don't already have an ssh key pair created, you will need to generate one.

build_id@build_server:~ $ mkdir -p ~/.ssh; chmod 700 ~/.ssh; cd ~/.ssh; ssh-keygen -b 2048 -t rsa

When prompted for a passphrase, hit enter to skip. Having a passphrase, while more secure than having no passphrase, prevents automated login.

Save your public key in ~/.ssh/id_rsa.pub (or some other file).

Connect to the following systems, replacing build_id and eclipse_id with your actual system login for that machine (which MUST be the same). Replace build_server and FQDN with the actual values for your build server's hostname and fully-qualified domain name.

  • build_id@build_server
  • build_id@FQDN
  • build_id@localhost
  • eclipse_id@download1.eclipse.org
  • eclipse_id@dev.eclipse.org

Get yourself added to the "www" group (or whatever is the user group that can update the website) on your build machine. This normally must be done by an administrator (root) of that machine.

All of those should work without being prompted for a password. If you are prompted, you need to set an ssh key in your ~/.ssh/authorized_keys file for each user & hostname pair to which you want to connect. The first time you try each of these connections, you should be be prompted to add the host's fingerprint to your ~/.ssh/known_hosts file. This is also required.

To copy your local public key to the remote machine, you will need to scp (secure copy) the file. From the build server, copy the file to the remote server, then ssh to that server combine the new key with the any existing keys:

build_id@build_server:~/.ssh $ scp id_rsa.pub eclipse_id@dev.eclipse.org:~/.ssh/eclipse_id.temp
build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
(you will still be prompted for a password when connecting)
eclipse_id@dev.eclipse.org:~ $ cd ~/.ssh
eclipse_id@dev.eclipse.org:~/.ssh $ touch ~/.ssh/authorized_keys
eclipse_id@dev.eclipse.org:~/.ssh $ cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys_old; # backup
eclipse_id@dev.eclipse.org:~/.ssh $ cat ~/.ssh/eclipse_id.temp ~/.ssh/authorized_keys_old > ~/.ssh/authorized_keys; # prepend new key into old file; save as new file
eclipse_id@dev.eclipse.org:~/.ssh $ rm -fr ~/.ssh/eclipse_id.temp ~/.ssh/authorized_keys_old
eclipse_id@dev.eclipse.org:~/.ssh $ exit
build_id@build_server:~/.ssh $ ssh eclipse_id@dev.eclipse.org
(this time you should get in using your key instead of needing a password)

Once you have ~/.ssh/authorized_keys and ~/.ssh/known_hosts updated with your public key and server fingerprint(s) (respectively), you should be able to connect to these systems without being prompted, and the promote script should be good to go unassisted.

If you require further assistance setting up key access, try here: http://cfm.gs.washington.edu/security/ssh/client-pkauth/.

Promotion Problems (CVS Errors, SCP Errors, Artifacts Not Created...)

Please check your promote logs for errors or transient glitches, like CVS disconnects or SCP problems. Sometimes one problem can manifest as several problems, as in the case of this EMFT Validation build's UM jar creation, which threw four warnings that something had gone wrong, all starting from a transient CVS checkout problem (bandwidth overload or connection reset).

[promote] [9] [13:53:18] Create & promote Update Manager jars:
[promote] Running buildUpdate.sh:
[umj-co] [1] Checking out org.eclipse.releng.basebuilder from dev 
  using HEAD
cvs checkout: failed to open /home/vramaswamy/.cvspass for reading: 
  No such file or directory
cvs [checkout aborted]: recv() from server dev.eclipse.org: 
  Connection reset by peer
The java class is not found: org/eclipse/core/launcher/Main
features/*.jar: No such file or directory
plugins/*.jar: No such file or directory
[umj] [7d] [13:54:47] Comparing local and remote folders MD5 sums 
  to ensure SCP completeness...
[compare] Compare md5sums...
[compare] MD5s do not match! Compare failed.
[umj] [13:54:52] ERROR! Script exiting from compareFolders.sh
[promote] [9] [13:54:52] Create & promote Update Manager jars done.

In this case, the problems were:

  • CVS disconnected while checking out required builder code
  • Build failed looking for Eclipse Main, so no build occurred
  • Copying files from temp to /var/www/technology/emft/updates/ failed as they were not created
  • MD5 check failed as nothing was copied to download1.eclipse.org

In this case, the solution was to:

  1. Verify that nothing had been produced on either the build server or download1.eclipse.org.
  2. Confirm that site*.xml had not been changed in CVS, and rollback any changes for this build's ID (if any).
  3. Re-run the promote using the -jarsonly flag in order to bypass all the steps that completed successfully and ONLY do the UM jar creation step.

Maintenance Processes

By following a few simple rules, you can ensure that you never run out of memory or hard disc space when building, and that your neighbours on the server are similarly unaffected.

  • Release builds should be kept at all times.
  • Stable builds should be kept until they are considered stale, eg., M1 need not be kept once M4 or M5 is available. This is subject to your discretion or customer needs.
  • Interim builds should be purged after every Milestone build so that you never have more than a few weeks of Interim builds listed.
  • Nightly builds should be scrubbed every week or so in order to save space and keep your build server's downloads page clean.

Removing CVS Tags

To remove CVS tags, you can run the following commands (from any machine that can connect to dev.eclipse.org via ssh):

cvs -d [eclipse_id]@dev.eclipse.org:/cvsroot/technology -q rtag -d [build_tag] org.eclipse.emft/[subproject];
cvs -d [eclipse_id]@dev.eclipse.org:/cvsroot/technology -q rtag -d [build_tag] org.eclipse.emft/releng/[subproject];

where:

  • eclipse_id is your cvs username for dev.eclipse.org
  • build_timestamp is "build_" plus the timestamp of the build for which you want to remove its tag from CVS. For example, for a milestone build 1.0.0M6a (S200604131352) (RC1, use build_200604131352.

Note that in order to remove tags from within org.eclipse.emft/releng you must be a member of the emft-releng group on dev.eclipse.org, or have similar write access via ACL.

Removing Old Builds

The simplest way to remove old builds is simply to delete them, both from the build server and from the public download1.eclipse.org server. Doing so will also (eventually) cause them to be removed from the eclipse.org mirrors.

Delete From Build Server

To delete an old build, ssh to your build server, then run, for example:

cd /home/www-data/build/emft/[subproject]/downloads/drops/1.0.0;
sudo -u [web_user] rm -fr N200604152353;

where:

  • web_user is the id on your build server which runs the Apache webserver processes, eg., www-data or apache

Delete From Production Server (download.eclipse.org)

When deleting from dev.eclipse.org (or download1.eclipse.org), you do not need to sudo to the web user:

cd /home/data/httpd/download.eclipse.org/technology/emft/[subproject]/downloads/drops/1.0.0;
rm -fr N200604152353;

Removing Update Manager Jars

Update Manager jars should be purged from build and production servers with the same frequencies as above.

Remove Jar Files

To get a list of jars (ordered by modification time) that are older than a certain age (eg., 14 days), run:

 cd /home/data/httpd/download.eclipse.org/technology/emft/updates;
 find . -maxdepth 2 -type f -name "*eodm*" -mtime +14 \
   -exec date -r {} +%s\ {} \; | sort -r | sed -e "s/[0-9]\+\ \.\///g";

Then, if you are satisfied with that list, delete them like this:

 for f in \
   `find . -maxdepth 2 -type f -name "*eodm*" -mtime +14 \
     -exec date -r {} +%s\ {} \; | sort -r | sed -e "s/[0-9]\+\ \.\///g"`; do \
       echo "Delete: "$f; rm -fr $f; \
 done

Remove Entries From site*.xml

In addition to deleting jars, you must also remove the entries from site-interim.xml. This file must be edited in CVS then checked out to the local filesystem to cause the removed builds to disappear.

If you do not update the file in CVS, but only edit it locally, the next UM jar build will recreate the removed entries since it extracts the latest version of that file from CVS before adding to it. To update the file in CVS and push it to eclipse.org, ssh to your build server or to any machine that can ssh to dev.eclipse.org, then:

cd /tmp;
cvs -d [eclipse_id]@dev.eclipse.org:/cvsroot/org.eclipse \
  -q co -d updates www/emft/updates;
cd /tmp/updates;
# edit file: use dd to delete lines, :wq to save & quit
vi site-interim.xml; 
cvs -q ci -m "remove old builds"; 
scp site-interim.xml \
  [eclipse_id]@dev.eclipse.org:/home/data/httpd/download.eclipse.org/technology/emft/updates
cd /tmp; rm -fr /tmp/updates; 

Of course, you can also edit the file in Eclipse or another editor, commit those changes, then check out the latest files and scp them to download.eclipse.org as shown above.

Build.php Dependency URL Cleanup

To remove old dependency URLs shown on your build.php page, edit the following file on your build server:

 sudo -u [web_user] vi /home/www-data/build/emft/requests/dependencies.urls.txt

Note that the most recent entries are listed at the bottom of the file, so delete from the top down (starting on line 3). If you delete too many, you can always re-add them using the web UI of build.php.



(Thanks to http://diberri.dyndns.org/html2wiki.html for assisting in migrating this document.)