Difference between revisions of "Data Tools Platform:Release Engineering"

From Eclipsepedia

Jump to: navigation, search
(Still to Do)
 
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
←[[Data Tools Platform Project | Back to DTP Main Page]]
+
{{Back To|name=DTP Main Page|href=Data Tools Platform Project}}
  
 
'''This information is preliminary.  While this system is available within CVS, it is not currently used to generate the DTP builds.'''
 
'''This information is preliminary.  While this system is available within CVS, it is not currently used to generate the DTP builds.'''
Line 8: Line 8:
 
This system was derived from the system used by WTP and STP (and probably others).  The system performs the following tasks:
 
This system was derived from the system used by WTP and STP (and probably others).  The system performs the following tasks:
 
*Component builds
 
*Component builds
*:builds plugins and features
+
**builds plugins and features
*:generates install ZIP
+
**generates install ZIP
*:generates update site JARs
+
**generates update site JARs
 
*Component testing
 
*Component testing
*:executes unit tests
+
**executes unit tests
*:generates results files
+
**generates results files
 
*Update site generation
 
*Update site generation
 
*:[TBD]
 
*:[TBD]
 
*Download site generation
 
*Download site generation
*:generates html for download site
+
**generates html for download site
*:incorporates build settings
+
**incorporates build settings
*:incorporates build results (e.g. warnings/errors)
+
**incorporates build results (e.g. warnings/errors)
*:incorporates test results
+
**incorporates test results
 
*Uploading build information
 
*Uploading build information
*:publishes generated files to DTP web site
+
**publishes generated files to DTP web site
  
 
==Design==
 
==Design==
Line 89: Line 89:
  
 
The breakdown of target types is as follows:
 
The breakdown of target types is as follows:
:*'''build'''
+
:;build
:*:invokes the [http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm PDE builder] for each component in the distribution.
+
::invokes the [http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm PDE builder] for each component in the distribution.
:*'''site'''
+
:;site
:*:generates the download site files for the build.  (verify: does this also generate the update site)
+
::generates the download site files for the build.  (verify: does this also generate the update site)
:*'''api'''
+
:;api
:*:TODO: document this
+
::TODO: document this
:*'''tests'''
+
:;tests
:*:invokes tests and consolidates results
+
::invokes tests and consolidates results
:*'''upload'''
+
:;upload
:*:uploads generated build zips and site files to the web site
+
::uploads generated build zips and site files to the web site
:*'''whatisfixed'''
+
:;whatisfixed
:*:TODO: document this
+
::TODO: document this
  
 
The following distributions and targets are defined for DTP:
 
The following distributions and targets are defined for DTP:
:*'''dtp'''
+
:;dtp
:*:the main DTP distribution
+
::the main DTP distribution
:**build
+
::;build
:**:builds component zips
+
:::builds component zips
:**site
+
::;site
:**:builds download site files (TODO: should update the html template files to fit in with the new-look Eclipse web site)
+
:::builds download site files (TODO: should update the html template files to fit in with the new-look Eclipse web site)
:**tests
+
::;tests
:**:TODO: needs to be updated for DTP
+
:::TODO: needs to be updated for DTP
:**upload
+
::;upload
:**:TODO: needs to be updated for DTP
+
:::TODO: needs to be updated for DTP
  
 
====scripts folder====
 
====scripts folder====
Line 127: Line 127:
 
#:The [http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm PDE builder] is invoked through a Java task, meaning properties are not inherited.  Need to make sure any [http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm PDE builder] specific properties are passed through as vmargs.
 
#:The [http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm PDE builder] is invoked through a Java task, meaning properties are not inherited.  Need to make sure any [http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm PDE builder] specific properties are passed through as vmargs.
 
#Notify tasks in ''scripts/monitoring'' need to be updated
 
#Notify tasks in ''scripts/monitoring'' need to be updated
 +
#Update html templates so they are compatible with [[Phoenix]].
  
 
==DTP Component Breakdown==
 
==DTP Component Breakdown==
Line 132: Line 133:
  
 
The following components are defined:
 
The following components are defined:
*'''connectivity-core'''
+
;'''connectivity-core'''
*:core connectivity plugins: connection profile framework, driver definition framework, data source explorer view
+
:core connectivity plugins: connection profile framework, driver definition framework, data source explorer view
*'''connectivity-jdt'''
+
;'''connectivity-jdt'''
*:provides JDT classpath container extension for driver definitions
+
:provides JDT classpath container extension for driver definitions
*'''connectivity-oda'''
+
;'''connectivity-oda'''
*:base ODA framework
+
:base ODA framework
*'''sqldevtools'''
+
;'''sqldevtools'''
*:SQL development tools framework
+
:SQL development tools framework
*'''enablement-db'''
+
;'''enablement-db'''
*:vendor specific plugins for use with ''sqldevtools''
+
:vendor specific plugins for use with ''sqldevtools''
*'''enablement-oda'''
+
;'''enablement-oda'''
*:specialized plugins for use with ''connectivity-oda''
+
:specialized plugins for use with ''connectivity-oda''
*'''dtp-all'''
+
;'''dtp-all'''
*:all DTP plugins, with the exception of source and ISV documentation
+
:all DTP plugins, with the exception of source and ISV documentation
*'''dtp-all-sdk'''
+
;'''dtp-all-sdk'''
*:all DTP plugins including source and ISV documentation
+
:all DTP plugins including source and ISV documentation
  
 
==Executing a Build==
 
==Executing a Build==
 
===Building a Distribution===
 
===Building a Distribution===
 
A distribution build may be exectuted through either of the following build files: (note, these may be invoked using a normal Ant install; i.e. no special tasks or additional libraries are required)
 
A distribution build may be exectuted through either of the following build files: (note, these may be invoked using a normal Ant install; i.e. no special tasks or additional libraries are required)
*org.eclipse.datatools.releng.builder/cruise/build.xml
+
*''org.eclipse.datatools.releng.builder/cruise/build.xml''
 
*:This is the build file used with CruiseControl.  Executing the build through this file will fetch the org.eclipse.datatools.releng and org.eclipse.datatools.releng.builder projects, the former will be fetched using the tag specified by the ''mapVersionTag'' property, while the latter will be fetched using the tag specified by the ''dtpBuilderVersion'' property located in the fetched org.eclipse.datatools.releng/maps/build.cfg file.  The build is then executed through the fetched org.eclipse.datatools.releng.builder/build.xml file.
 
*:This is the build file used with CruiseControl.  Executing the build through this file will fetch the org.eclipse.datatools.releng and org.eclipse.datatools.releng.builder projects, the former will be fetched using the tag specified by the ''mapVersionTag'' property, while the latter will be fetched using the tag specified by the ''dtpBuilderVersion'' property located in the fetched org.eclipse.datatools.releng/maps/build.cfg file.  The build is then executed through the fetched org.eclipse.datatools.releng.builder/build.xml file.
*org.eclipse.datatools.releng.builder/build.xml
+
*''org.eclipse.datatools.releng.builder/build.xml''
 
*:This is the main build file for the DTP, containing targets for building a distribution's build, site, api, tests, upload and whatisfixed targets.  There is no default target for this build file, so you must specify one or more on the command line.
 
*:This is the main build file for the DTP, containing targets for building a distribution's build, site, api, tests, upload and whatisfixed targets.  There is no default target for this build file, so you must specify one or more on the command line.
 
The following properties must be defined when invoking either of the above targets:
 
The following properties must be defined when invoking either of the above targets:
Line 171: Line 172:
 
;lpgJavaRuntimeJarFileLocation
 
;lpgJavaRuntimeJarFileLocation
 
:Location of lpgjavaruntime.jar file.  This is required for building SQL tools.  This file is copied into org.eclipse.datatools.lpg/lib prior to building and deleted prior to packaging.
 
:Location of lpgjavaruntime.jar file.  This is required for building SQL tools.  This file is copied into org.eclipse.datatools.lpg/lib prior to building and deleted prior to packaging.
 +
;build.trial
 +
:Set to true when testing the build system.
 
;build.clean
 
;build.clean
 
:Set to true to remove any existing build directory (only for cruise/build.xml)
 
:Set to true to remove any existing build directory (only for cruise/build.xml)
Line 176: Line 179:
 
===Building a Component===
 
===Building a Component===
 
[TBD]
 
[TBD]
 +
 +
[[Category:Data Tools Platform]]

Latest revision as of 23:10, 26 August 2007

Back to DTP Main Page


This information is preliminary. While this system is available within CVS, it is not currently used to generate the DTP builds.

This page details the build system utilised by DTP.

Contents

[edit] Overview

This system was derived from the system used by WTP and STP (and probably others). The system performs the following tasks:

  • Component builds
    • builds plugins and features
    • generates install ZIP
    • generates update site JARs
  • Component testing
    • executes unit tests
    • generates results files
  • Update site generation
    [TBD]
  • Download site generation
    • generates html for download site
    • incorporates build settings
    • incorporates build results (e.g. warnings/errors)
    • incorporates test results
  • Uploading build information
    • publishes generated files to DTP web site

[edit] Design

The build system is designed to organized as follows:

distribution
a group of one or more components
component
synonymous with feature; the input to the builder/assembler/packager

The system is comprised of custom Ant scripts which are used to initialize build properties, download dependencies, invoke the PDE builder for individual components, generate site files.

The head-less PDE builder is used to generate plugin JARs and ZIPs. The PDE builder executes a build using information in the feature and plugin manifests. This helps ensure the plugin build environment is similar to its execution environment.

The system is composed of two projects within CVS:

[edit] org.eclipse.datatools.releng

This project contains the following files:

*.map files
used by the PDE builder to locate and fetch source for the build. These files are used to specify the following information:
  • plugin/fragment/bundle/feature ID
  • plugin/fragment/bundle/feature location in CVS (project and path)
  • tag to fetch
dependencies.properties
specifies names and download locations for dependencies (e.g. Eclipse platform, EMF, GEF)
build.cfg
specifies which tag of the org.eclipse.datatools.releng.builder should be used to execute the build

[edit] org.eclipse.datatools.releng.builder

This project contains the scripts used to execute the build. It is divided into the following sections (by folder):

  • components
  • cruise
  • distribution
  • scripts

At the root is the build.xml file that drives the build for a particular distribution (i.e. the distribution must be specified when this file is invoked; e.g. dtp).

[edit] components folder

The components folder contains a folder for each component that will be built. These folders may be used directly with PDE builder as the buildLocation for the build.

A component is synonymous with a feature. The feature specified by the component is used to drive the build. See the PDE builder documentation for more information.

This system deviates from the general PDE build integration by the following:

  • allElements.xml is incorporated directly within customTargets.xml
    The allElements and assemble.element.id[.config.spec] targets are defined directly within customTargets.xml.
  • most of the targets for customTargets.xml are imported from scripts/build/defaultCustomTargets.xml
    scripts/build/defaultCustomTargets.xml contains targets that are common to all component builds (e.g. fetching dependencies). (Note, individual components may override any targets imported from defaultCustomTargets.xml.)
  • dependency.xml
    This file is used to specify the dependencies required by the component. The dependencies.properties file in org.eclipse.datatools.releng must contain an entry for each of the group IDs specified in this file.

[edit] cruise folder

The cruise folder contains scripts for driving the main build from CruiseControl. The build.xml file within this folder fetches the org.eclipse.datatools.releng and org.eclipse.datatools.releng.builder projects, then invokes the main build file in the root of org.eclipse.datatools.releng.builder.

TODO: Some files in this folder need to be updated for use with DTP; specifically the following files:

  • cc.sh
  • config.xml
  • cruise.properties

[edit] distribution folder

Contains distribution specific build files. Each folder in this directory uses the following naming convention:

<distribution name>.<target name>
e.g. dtp.build, dtp.site, etc.

The scripts within these folders are pretty specialized (although there may be an opportunity to refactor some of these scripts so common targets can be shared across distributions).

The breakdown of target types is as follows:

build
invokes the PDE builder for each component in the distribution.
site
generates the download site files for the build. (verify: does this also generate the update site)
api
TODO: document this
tests
invokes tests and consolidates results
upload
uploads generated build zips and site files to the web site
whatisfixed
TODO: document this

The following distributions and targets are defined for DTP:

dtp
the main DTP distribution
build
builds component zips
site
builds download site files (TODO: should update the html template files to fit in with the new-look Eclipse web site)
tests
TODO: needs to be updated for DTP
upload
TODO: needs to be updated for DTP

[edit] scripts folder

Contains common scripts available for use from distribution and component build scripts. For example, scripts/dependency contains scripts for downloading dependencies required by a particular component; scripts/build/build.xml contains targets for executing the PDE builder for a particular component.

[edit] Still to Do

  1. Update ODA plugins for use with PDE builder
    The ODA plugins are currently built using custom build scripts. Need to make sure the resources used by the PDE builder are consistent with the custom scripts.
  2. Integrate generation of ISV documentation
  3. Add documentation plugins (user and ISV) to respective featurs
  4. Integrate unit tests
  5. Fix upload target (verify/update settings in upload.properties)
  6. Ensure all relevant properties are passed to the PDE builder
    The PDE builder is invoked through a Java task, meaning properties are not inherited. Need to make sure any PDE builder specific properties are passed through as vmargs.
  7. Notify tasks in scripts/monitoring need to be updated
  8. Update html templates so they are compatible with Phoenix.

[edit] DTP Component Breakdown

Note, this is preliminary and does not reflect the current Europa release strategy.

The following components are defined:

connectivity-core
core connectivity plugins: connection profile framework, driver definition framework, data source explorer view
connectivity-jdt
provides JDT classpath container extension for driver definitions
connectivity-oda
base ODA framework
sqldevtools
SQL development tools framework
enablement-db
vendor specific plugins for use with sqldevtools
enablement-oda
specialized plugins for use with connectivity-oda
dtp-all
all DTP plugins, with the exception of source and ISV documentation
dtp-all-sdk
all DTP plugins including source and ISV documentation

[edit] Executing a Build

[edit] Building a Distribution

A distribution build may be exectuted through either of the following build files: (note, these may be invoked using a normal Ant install; i.e. no special tasks or additional libraries are required)

  • org.eclipse.datatools.releng.builder/cruise/build.xml
    This is the build file used with CruiseControl. Executing the build through this file will fetch the org.eclipse.datatools.releng and org.eclipse.datatools.releng.builder projects, the former will be fetched using the tag specified by the mapVersionTag property, while the latter will be fetched using the tag specified by the dtpBuilderVersion property located in the fetched org.eclipse.datatools.releng/maps/build.cfg file. The build is then executed through the fetched org.eclipse.datatools.releng.builder/build.xml file.
  • org.eclipse.datatools.releng.builder/build.xml
    This is the main build file for the DTP, containing targets for building a distribution's build, site, api, tests, upload and whatisfixed targets. There is no default target for this build file, so you must specify one or more on the command line.

The following properties must be defined when invoking either of the above targets:

basearch
The architecture (e.g. x86)
baseos
The operating system (e.g. win32)
basews
The windowing system (e.g. win32)
build.home
The working directory for the build (e.g. c:\tmp\dtp\build)
env.J2SE14
The build classpath for plugins built against J2SE-1.4 execution environment. Currently, this must include rt.jar, jsse.jar and jce.jar (e.g. c:\j2sdk1.4.2\jre\lib\rt.jar;c:\j2sdk1.4.2\jre\lib\jsse.jar;c:\j2sdk1.4.2\jre\lib\jce.jar)
env.J2SE15
Similar to J2SE15, but applies to J2SE-1.5 execution environment.
lpgJavaRuntimeJarFileLocation
Location of lpgjavaruntime.jar file. This is required for building SQL tools. This file is copied into org.eclipse.datatools.lpg/lib prior to building and deleted prior to packaging.
build.trial
Set to true when testing the build system.
build.clean
Set to true to remove any existing build directory (only for cruise/build.xml)

[edit] Building a Component

[TBD]