Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Modeling Project Releng/Building

< Modeling Project Releng
Revision as of 19:28, 29 August 2007 by Codeslave.ca.ibm.com (Talk | contribs) (New page: ==Building Zips & Jars== * We use the Eclipse build process which relies on PDE. See this [http://www.eclipse.org/articles/Article-PDE-Automation/automation.html article] if you are inter...)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Building Zips & Jars

  • We use the Eclipse build process which relies on PDE. See this article if you are interested in the details. It also covers the JUnit automated tests that can be run during the build.
  • You can use PDE to configure what will appear in your zips.


Incubation Status

In order to comply with Eclipse.org's Incubation Rules, EMFT components must adhere to these 3 rules.

For example, here are the changes required to make EMF Compare compliant: Search CVS results

1. All downloadable zip files for builds and milestones must include the word incubation in the filename. For example, emft-compare-SDK-incubation-N200708031343.zip. The jar files in the download zip file are NOT required (bug 178944) to contain the word incubation in the filename.
To change your build to provide -incubation in the zip names, search your .releng project for ".zip" and fix all files accordingly. Releng Example
  • Building: customTargets.xml (all versions of this file)
  • Testing: relengbuildgtk.sh, testManifest.xml, testManifest.xml.template, testing.properties, readme.html
  • Promoting: promoteToEclipse.*.properties
2. All Bundle-Names must include the world incubation. Note that Bundle-SymbolicNames should not include incubation because the Bundle-SymbolicName is a technical namespace, not a user namespace. For example, Bundle-Name: Foo Plug-in (Incubation).
Update all your component's MANIFEST.MF and/or plugin.properties files, including tests, examples, and doc (which file to edit depends on where "Bundle-Name" strings are located). MANIFEST.MF Example
Bundle-Name: %pluginName (Incubation) [1]
3. The names for update manager features must include the word incubation. For example, EMF Compare Documentation (Incubation).
Update all your component's feature.xml and/or feature.properties files, including tests, examples, doc, and SDK features. Add (Incubation) to both the feature label and the feature description. Feature.* Example
<feature
     id="org.eclipse.emf.compare.sdk"
     label="EMF Compare SDK (Incubation)"
     version="0.7.0.qualifier"
     provider-name="Eclipse.org">
  <description>
     EMF Compare SDK includes runtime, source, and documentation (Incubation)
  </description> [2]

When a project exits incubation, above changes should be reversed to remove the incubation identifiers from features, plugins, and zip file names.

Customizing Zip Bundles

In the event that you have plugins or features that PDE will not normally bundle together, such as .ui plugins in the runtime zip, you will need to adjust the way PDE behaves in order to ensure all your plugins/features appear in your zips as you need them. If you need a custom zip, like EMF has for Models or Standalone, this is also how you can accomplish this.

There's two ways to customize what gets put into a zip bundle by PDE: the correct way (a) and the shortcut (b)

Option A

The first method involves creating a feature which sets up the included features/plugins that have to be in there, as with .sdk features in the EMFT subprojects (ocl, query, validation, transaction). See details in CVS. An example of this is org.eclipse.emf-feature/org.eclipse.emf.sdk. It is nested (rather than being its own org.eclipse.emf.[subproject].[bundlename]-feature) for cosmetic reasons (it looks in the file system).

When creating a new org.eclipse.emf.[subproject]-feature/org.eclipse.emf.[subproject].[bundlename] (or org.eclipse.emf.[subproject].[bundlename]-feature), you will need to ensure it's properly connected to the build harness in 3 ways:

1. First, you'll need a folder under your builder/ directory for the feature build, such as:
org.eclipse.emft/releng/validation/builder/sdk/

Into this folder must go a customTargets.xml and a build.properties file. Then make sure that customTargets.xml refers to the feature correctly, as in customTargets.xml. (Search for "sdk" on lines 9, 20, 187.)

2. Then, the buildAll.xml script must be told how to build the new zip, eg, buildAll.xml (line 139):
<target name="buildAll" depends="init">
  <ant antfile="build.xml" target="main">
    <property name="component" value="builder/sdk"/>
  </ant>
  ...
</target>
3. And finally, you need to add the custom feature to the mapfile & mapfile template, since it won't be generated by PDE into the mapfile automagically. Examples: org.eclipse.emft/releng/validation/maps/validation.map
org.eclipse.emft/releng/validation/templateFiles/validation.map.template

Add an entry such as (line break added for clarity):
feature@org.eclipse.emf.validation.sdk=@cvsTag@,@cvsRoot1@,,org.eclipse.emft/validation/plugins/ org.eclipse.emf.validation-feature/org.eclipse.emf.validation.sdk

Option B (or adding an optional dependency)

The second method is faster (and, arguably, more hackish). This is also a valid method for adding an optional dependency, like for example adding optional support for OCL.

Instead of one feature per zip, as above, you can have custom instructions/rules in the buildAll.xml script which allow you to copy extra files that would normally be excluded from the zip to clean up missing content. These instructions are kept in one place (ie., only one ant script), so maintenance is easier, but this solution should only be used to add files to existing bundles, not to create new, custom ones.

For example, there's a validation.ocl plugin which must be included in the validation runtime, but since the validation-feature makes no mention of it, it's excluded. So, to work around this, once the SDK is built, copy the validaton.ocl plugin & feature from the SDK zip to the runtime zip after its assembly. This is done in the org.eclipse.emft/releng/validation/buildAll.xml script:

<target name="buildAll" depends="init">
  <ant antfile="build.xml" target="main">
    <property name="component" value="builder/sdk" />
  </ant>
  
  <ant antfile="build.xml" target="main">
    <property name="component" value="builder/runtime" />
  </ant>
  ...
  
  <!-- add .ocl feature + plugin from SDK to runtime -->
  <zip update="true"
    destfile="${buildDirectory}/${buildLabel}/emft-${subprojectName}-runtime-${buildAlias}.zip">
    <zipfileset 
      src="${buildDirectory}/${buildLabel}/emft-${subprojectName}-SDK-${buildAlias}.zip">
      <include name="**/org.eclipse.emf.${subprojectName}.ocl*"/>
      <include name="**/org.eclipse.emf.${subprojectName}.ocl*/*"/>
    </zipfileset>
  </zip>
  ...
</target>

Bear in mind that you must ensure that everything you need to compile is available. You will need to edit all the customTargets.xml files that are used to build code which depends on the new reqiurement. See target "postSetup".

<ant target="getDependency" antfile="${getDeps.xml}">
   <property name="url" value="${oclURL}"/>
   <property name="file" value="${oclFile}"/>
   <property name="isUnpackedFile" value="${buildDirectory}/plugins/org.eclipse.emf.ocl/plugin.xml"/>
</ant>

If your tests require addtional plugins in order to compile or run, be sure to add them to your test.xml file. See target "setup".

Using Third Party Jars

You want to use third party jars during your build, but don't want to ship them due to licensing or other reasons? Here's how:

<target name="postFetch">
  <replace dir="${buildDirectory}/plugins" value="${timestamp}" token="@buildid@">
    <include name="**/about.mappings" />
  </replace>

  <echo message="Copy hsqldb.jar into plugins/org.eclipse.emf.cdo.jdbc.hsqldb/lib ..."/>
  <copy 
    todir="${buildDirectory}/plugins/org.eclipse.emf.cdo.jdbc.hsqldb/lib" 
    file="${buildDirectory}/../../../../../../../downloads/hsqldb.jar" failonerror="true"/>

  <echo message="Copy mysql-connector-java-3.1.12-bin.jar into plugins/org.eclipse.emf.cdo.jdbc.mysql/lib ..."/>
  <copy 
    todir="${buildDirectory}/plugins/org.eclipse.emf.cdo.jdbc.mysql/lib" 
    file="${buildDirectory}/../../../../../../../downloads/mysql-connector-java-3.1.12-bin.jar" failonerror="true"/>
</target>
<echo message="Repack zip w/ exclusion filter"/>
<zip destfile="${buildDirectory}/${buildLabel}/emft-${subprojectName}-SDK-${buildAlias}.zip_" update="true">
  <zipfileset src="${buildDirectory}/${buildLabel}/emft-${subprojectName}-SDK-${buildAlias}.zip">
    <exclude name="**/lib/hsqldb.jar"/>
    <exclude name="**/lib/mysql-connector-java-3.1.12-bin.jar"/>
  </zipfileset>
</zip>

<echo message="Remove orig. zip and rename new one"/>
<delete file="${buildDirectory}/${buildLabel}/emft-${subprojectName}-SDK-${buildAlias}.zip"/>
<move file="${buildDirectory}/${buildLabel}/emft-${subprojectName}-SDK-${buildAlias}.zip_" 
  tofile="${buildDirectory}/${buildLabel}/emft-${subprojectName}-SDK-${buildAlias}.zip"/>

Standalone Zip Bundles

Building a Standalone Zip allows you to provide code for use outside Eclipse. This is not very well documented (yet), but to get a feel for how to build a Standalone bundle, have a look at the following EMF examples:

Build

org.eclipse.emf.releng.build/zips/standalone/
org.eclipse.emf.releng.build/buildAll.xml
org.eclipse.emf.releng.build/zips/build.xml
  (line 14, target "standalone")

Test

org.eclipse.emf.releng.build/testManifest.xml
org.eclipse.emf.releng.build/templateFiles/testManifest.xml.template
org.eclipse.emf.releng.build/tests/configs/local/customTest.xml


Build Server Directory Structure

Here's a quick snapshot of the directory structure used on the emft build server, so that you can reproduce it elsewhere.

The emft build server (emf.torolab) is running Debian 3.0 (Woody), kernel 2.4.25, with dual 2G AMD Opteron processors, 3G RAM, 1.5G swap, and 33G HD.

Path Purpose/Description
/var/www/emft/ internal web paths (build.php)
/var/www/technology/emft/ external web path mirrored to download.eclipse (downloads, etc.)
/var/www/technology/emft/build.options.txt file used by both build.php and downloads/index.php to display only those builds specified
/var/www/technology/emft/downloads/index.php where to download project zips
/var/www/technology/emft/[subproject]/downloads/ symlink to /home/www-data/build/emft/[subproject]/downloads/ (so that the downloads page can find the zips)
/home/www-data/build/emft/scripts/ build path to where shell scripts are initiated by the web (the apache1.3 user is www-data)
/home/www-data/build/emft/requests/ build path to where build.php's data (URLs and log of requests for builds) is stored
/home/www-data/build/emft/[subproject]/downloads/ where required .zips are downloaded prior to building (eg., Eclipse SDK, EMF SDK)
/home/www-data/build/emft/[subproject]/downloads/drops/ where [subproject] is built (all versions here, including any NLS zip, once available (if any))
/home/www-data/build/emft/[subproject]/downloads/drops/1.0.0/ where [subproject] v1.0.0 is built
/home/www-data/build/emft/[subproject]/downloads/drops/1.0.0/ [type][yyyymmddHHMM]/ a specific build's folder, including zips, test results, docs
/opt/ibm-java2-1.4 -> /opt/ibm-java2-ws-sdk-pj9xia32142-20050609 default build JDK / used for JUnit tests
/opt/sun-java2-1.4 -> /opt/j2sdk1.4.2_03 optional build JDK / used for EMF JDK tests
/opt/sun-java2-5.0 -> /opt/jdk1.5.0_03 optional build JDK / used for EMF JDK tests
/opt/ibm-java2-1.3 -> /opt/IBMJava2-131 used to test EMF 2.0.x ONLY
/opt/apache-ant-1.6.1 (or newer) build & utility scripts
/usr/bin/php4 (4.1.2 or newer) both Apache module and CLI (command line interface) modules required


Testing & Diagnosing Builds

To determine what's wrong with your build, make sure you're building in debug mode. eg., [3] or [4]. Then set the following parameters:

 Build Type: N (Nightly)
 Tag Build: No
 Run Tests: JUnit
 debug java home: /opt/ibm-java2-ws-sdk-pj9xia32142-20050609
 debug noclean: Y

Then, you can use the following shell commands:

# shortcut to builds folder
alias odd=' cd /home/www-data/build/emft/ocl/downloads/drops/1.0.0'

# shortcut to LATEST build folder
alias lastfolder='cd `find . -maxdepth 1 -type d -not -name \
  "." -exec date -r {} +%s\ {} \; | sort | tail -1 | \
  sed -e "s/[0-9]\+\ \.\///g"`'

alias oddl='odd; lastfolder'

# check build log
oddl; tail -60 buildlog.txt

# check JUnit test console log
oddl; cd testing/*/testing/; tail linux.gtk_consolelog.txt

# check JUnit test eclipse workspace error log
oddl; cd testing/*/testing/eclipse/workspace/.metadata/; less .log


Suppressing Compiler Warnings

After 3.2M6, PDE will now report compilation warnings in its JUnit test results page.

You can either fix your code to remove these warnings, or suppress the warnings themselves. See /org.eclipse.emft/releng/jet/buildAll.xml, for an example of how to suppress these warnings.

<echo message="Set compilerArgs = '-enableJavadoc -encoding ISO-8859-1 -warn:-serial'"/>
<property name="compilerArg" value="-enableJavadoc -encoding ISO-8859-1 -warn:-serial" />

For a complete list of supported warning flags in 3.1 - 3.3, go here: http://help.eclipse.org/help33/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_api_compile.htm (see Warning Options).

You can also bring up that same help page in Eclipse's built-in help (JDT Plug-in Developer Guide > Programmer's Guide > JDT Core > Compiling Java code) for the 3.2 version, which adds a couple other options to the list.

Heterogeneous Compiler Source/Target Levels

If your project has plug-ins that have different requirements for the compiler source and target compliance level, the PDE's build.properties file can be used to specify these on a plug-in by plug-in basis. For a plug-in that supports J2SE 1.4 and above, just add:

javacSource = 1.4
javacTarget = 1.4

in that plug-in's build.properties. For a plug-in that uses J2SE 5.0 language capabilities, just add:

javacSource = 1.5
javacTarget = 1.5

and similarly for 6.0, 7.0, ... For an example, see the validation.ocl plug-in, which differs from the other validation plug-ins in requiring J2SE 5.0.

You may already have an EMFT releng system that forces 1.4 source and target in the buildAll.xml file. Simply remove that and the javacSource=1.4 entries from the build.properties files in each builder/* folder. For example, see in the Query component's releng:

Build Problems & Solutions

In no particular order, here are some lessons learned from setting up the various EMFT builds:

Error Message Fix
eclipse/test.assembly/eclipse/
plugins/${org.eclipse.test} not found.

Add the "org.eclipse.test" and "org.eclipse.ant.optional.junit" plugins to each test feature. The task creates one property for each test plugin (the property name is the plugin name minus the version). Only plugins referenced by features are downloaded, and, if a plugin has to be downloaded, the map file has to describe how to do it. Thus this information must appear in two places: tests/customTargets.xml and *.tests-feature/feature.xml. Since "org.eclipse.test" is not listed on the tests feature, it is not being downloaded - regardless of it being listed in the map file.

If you have added this to your mapfile and features, and you're still having problems, verify your mapfile has the right tag for org.eclipse.test.

If you have checked all these conditions and still get an error, make sure your test feature specifies version="0.0.0" for the org.eclipse.test plugin so that PDE can calculate the version for you. This problem manifests with releng.basebuilder versions after M4_33, as discussed in bugs 190834 and 192231.

Additionally, good hygiene suggests that all your features should have the correct builder/nature in their .project files. This will ensure that when you're manipulating your features in Eclipse, PDE will report problems (eg., typos) as they happen rather than during a build.

<buildSpec>
  <buildCommand>
    <name>org.eclipse.pde.FeatureBuilder</name>
    <arguments/>
  </buildCommand>
</buildSpec>
<natures>
  <nature>org.eclipse.pde.FeatureNature</nature>
</natures>
SomeBundleOrClass cannot be resolved (compilation failed).

Check your feature.xml files. PDE will build features in the order they're listed. See also bug 140745.

[exec]   -cp  org.eclipse.equinox.launcher.Main
                    ...
[exec] Unrecognized option: -ws
[exec] Could not create the Java virtual machine.

Check relengbuildgtk.sh, and make sure the $cp or $cpAndMain variable is being set correctly. Also, make sure that if you're building with Eclipse 3.2 you use org.eclipse.core.launcher.Main instead of Eclipse 3.3's org.eclipse.equinox.launcher.Main. See also Running Eclipse and PDE JUnit Testing.

org.eclipse.emft.ocl.releng/ builder/sdk/customTargets.xml -> eclipse/assemble.org.eclipse.emf.ocl.sdk.xml -> eclipse/tmp/eclipse/plugins/ org.eclipse.emf.ocl.doc_1.0.0.200602231606 not found

This could be a number of problems, but the most likely is that org.eclipse.emft/ocl/doc/org.eclipse.emf.ocl.doc/build.xml needs to have the correct value for pluginVersion set or is missing this property entirely.

<property name="pluginVersion" value="1.0.0"/>

If that doesn't solve the problem, run a debug build with the -noclean flag checked to see what's in the eclipse/tmp/eclipse/plugins/ folder. For the case of the above build, the folder created was called org.eclipse.emf.ocl.doc_${pluginVersion}.200602231617 instead of org.eclipse.emf.ocl.doc_1.0.0.200602231617, since the property was not defined.

java.io.FileNotFoundException: testing/target/eclipse/ plugins/org.eclipse.emf.ocl.tests_1.0.0/ (Plugin is a jar, not a folder)

If your test plugin is compiled as a jar, not a folder, you need to fix your feature.xml. Remove the attribute unpack="false" from the org.eclipse.emf.*.tests plugin:
<plugin id="org.eclipse.emf.ocl.tests" download-size="0" install-size="0" unpack="false" version="1.0.0"/>

java.io.FileNotFoundException: testing/target/eclipse/ plugins/org.eclipse.emf.ocl.tests_1.0.0/ test.xml (No such file or directory)

Add a "test.xml" script to each test plugin, in tests/org.eclipse.emf.*/test.xml.

If this file already exists, verify in releng/ [subproject]/ builder/ tests/ scripts/ test.xml that target runtests knows whether your tests plugin is jarred or not. Comment/uncomment out test.xml script accordingly.

java.io.FileNotFoundException: eclipse/fetch_org.eclipse.emft.ocl.sdk.xml (No such file or directory) Either the feature doesn't exist (and must be added), or the feature is misreferenced. Make sure that you do not have any scripts referring to, say, plugins or features called org.eclipse.emft.ocl.sdk, but rather org.eclipse.emf.ocl.sdk. The org.eclipse.emft namespace is only for the CVS module; everywhere else, you should use org.eclipse.emf when referencing plugins and features.
JUnits tests not appearing on testResults.php page

Add an entry for the missing tests into org.eclipse.emft/releng/[subproject]/ templateFiles/testManifest.xml.template and org.eclipse.emft/releng/[subproject]/ testManifest.xml, eg:

<logFile 
  name="org.eclipse.emf.query.ocl.tests_linux.gtk.xml">
<effectedFile id="SDK"></effectedFile>
<effectedFile id="projRUN"></effectedFile>
</logFile>
Failure to download and unzip one of the SDKs that your project depends on

You probably have a typo in the URL that you entered into the "New URLs" field. Try the following to get the correct URL every time:

  1. Surf to your dependency's downloads page on eclipse.org.
  2. Navigate to the build that you need.
  3. Click the download link for the all-encompassing SDK ZIP.
  4. When the "blah now downloading, if it doesn't start immediately, click here" message appears, right-click on the "click here" link and select "Copy Link Location" (or the equivalent action in your browser or choice).
  5. You now have in your cut buffer the exact link to the file that you need.

Depending on your configuration, you may have an extra navigation to a mirror site in the above sequence.

Build is ok, everything compiles, but plugins are missing from the SDK!
Build is ok, everything compiles, but SDK includes BOTH unpacked source files and packed src.zip files.

Verify that you have included your branding plugin (eg., org.eclipse.net4j) in your branding feature (eg., org.eclipse.net4j-feature/feature.xml), even if it contains NO source and that you've set that plugin to be unpack="false":

<plugin
  id="org.eclipse.net4j"
  download-size="0"
  install-size="0"
  version="0.0.0"
  unpack="false"/>
[185001] Plugin jars contain nested jars instead of class files.

See Bundle-ClassPath With Dot.

External jar files (like log4j) are not included in the plugin when building and zipping the plugins

Ensure that the Bundle-ClassPath in the MANIFEST.MF contains the dot (.) for example:

Bundle-ClassPath: .,commons-logging.jar,log4j-1.2.8.jar
publish.xml fails to generate build artefacts!

This has been deprecated in the latest org.eclipse.releng.basebuilder. For the solution, see details in bug 165698.

Need to update your build to EMF 2.3 and J2SDK 5.0.

That's a pretty big subject. See EMF 2.3 Adoption for details.

java.lang.NoClassDefFoundError: org/eclipse/core/launcher/Main

You need to upgrade to the Equinox Launcher. See Automated PDE JUnit Testing With Eclipse for details on how to fix this problem.

Back to the top