Jump to: navigation, search

Difference between revisions of "Making Builds Reproducible"

(Making Builds Reproducible)
(Background)
Line 3: Line 3:
 
==Background==
 
==Background==
  
In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following:
+
In order to build non-SDK projects offline -- a requirement of our buildsystem
 +
for security purposes -- we must do the following:
  
# check out releng plugin
+
# Generate a source drop
# add a fetch target  
+
## check out releng plugin
 +
## add a fetch target
 +
## run the fetch target with the Eclipse antrunner
 +
## tar up the results
 +
# Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main
  
In an ideal world, all it would take to build a project on top of the SDK would be something like this:
+
==Common Pain Points==
  
# assume SDK is available
+
The last step -- the actual building with PDE -- is often different between
# check out project releng plugin
+
projects. Some common variable differences:
# ideally there would be no custom build.xml which is essentially a shell for PDE build
+
## (if things need to be done that are special, can they not
+
# java -cp $SDK/startup.jar org.eclipse.core.launcher.Main -application org.eclipse.ant.core.antRunner -Dcomponent=<whatever> -DbaseLocation=$SDK -DskipFetch=true
+
  
In practice, we run into common gotchas such as:
+
* javacFailOnError - why isn't this always set?
 +
* mapsLocal
 +
* pde.build.scripts
 +
* skipFetch
 +
* buildLabel
 +
* buildAlias
 +
* baseExists
 +
* eclipse.builder.fetch
 +
* baseLocation vs. base.location
 +
* pde.builder.path
 +
* mapVersionTag
 +
* buildDirectory
  
* customTargets.xml attempts to fetch dependencies with no flags to override this behaviour
+
* a build.xml that is essentially a shell for PDE's build.xml - why not just use
** ex. getBaseBuilder is in postFetch unconditionally - I have to patch this out
+
PDE build.xml directly?
* inconsistent flags to avoid fetching
+
* ant build files that are shells around the "standard" (essentially empty)
** ie. we would like there to be
+
build.xml that set obscure variables that are necessary but non-standard
 +
* build scripts (bash, DOS) that set obscure variables that are necessary but
 +
non-standard
 +
* build scripts that assume relative/absolute paths
 +
* customTargets.xml attempts to fetch dependencies with no flags to override
 +
this behaviour
 +
** ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
 +
* build scripts that assume certain usernames/passwords -- as said in
 +
[[Europa_Build_Workshop_Report#Eclipse_Build_Best_Practices Eclipse Build Best
 +
Practices]], a common cause of this is the lack of separation between fetching,
 +
building, and publishing
 +
 
 +
==Solutions to these common gotchas==
 +
* separate fetching of your project's source code from fetching basebuilder/SDK
 +
and dependencies
 +
* wrap fetching of basebuilder/SDK and dependencies with flags that can be
 +
overridden (ie. dontFetch)
 +
* use consistent flags/variables
 +
** derive values from other variables (ie. pde.build.scripts/path from
 +
baseLocation
 +
* do not assume use of basebuilder as it is ''almost always'' perfectly fine to use the
 +
SDK -- especially for I- or M-buildsand releases -- and we do this :)
 +
* do not hard-code usernames, paths, etc. that aren't either universal or at the
 +
very least overridable
 +
*
 +
 
 +
==An Ideal Future==
 +
 
 +
In an ideal world, all it would take to build a project on top of the SDK would
 +
be something like this:
 +
 
 +
# assume SDK is available and have a standard way of passing its path during the
 +
build
 +
# download source drop from project -- no generated files (including build.xmls), no binary code
 +
# run the PDE build.xml with consistent options
 +
## I envision us using:  baseLocation, dontUnzip, dontFetch -- everything else
 +
should be derived from these values

Revision as of 11:58, 15 September 2006

Making Builds Reproducible

Background

In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following:

  1. Generate a source drop
    1. check out releng plugin
    2. add a fetch target
    3. run the fetch target with the Eclipse antrunner
    4. tar up the results
  2. Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main

Common Pain Points

The last step -- the actual building with PDE -- is often different between projects. Some common variable differences:

  • javacFailOnError - why isn't this always set?
  • mapsLocal
  • pde.build.scripts
  • skipFetch
  • buildLabel
  • buildAlias
  • baseExists
  • eclipse.builder.fetch
  • baseLocation vs. base.location
  • pde.builder.path
  • mapVersionTag
  • buildDirectory
  • a build.xml that is essentially a shell for PDE's build.xml - why not just use

PDE build.xml directly?

  • ant build files that are shells around the "standard" (essentially empty)

build.xml that set obscure variables that are necessary but non-standard

  • build scripts (bash, DOS) that set obscure variables that are necessary but

non-standard

  • build scripts that assume relative/absolute paths
  • customTargets.xml attempts to fetch dependencies with no flags to override

this behaviour

    • ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
  • build scripts that assume certain usernames/passwords -- as said in

[[Europa_Build_Workshop_Report#Eclipse_Build_Best_Practices Eclipse Build Best Practices]], a common cause of this is the lack of separation between fetching, building, and publishing

Solutions to these common gotchas

  • separate fetching of your project's source code from fetching basebuilder/SDK

and dependencies

  • wrap fetching of basebuilder/SDK and dependencies with flags that can be

overridden (ie. dontFetch)

  • use consistent flags/variables
    • derive values from other variables (ie. pde.build.scripts/path from

baseLocation

  • do not assume use of basebuilder as it is almost always perfectly fine to use the

SDK -- especially for I- or M-buildsand releases -- and we do this :)

  • do not hard-code usernames, paths, etc. that aren't either universal or at the

very least overridable

An Ideal Future

In an ideal world, all it would take to build a project on top of the SDK would be something like this:

  1. assume SDK is available and have a standard way of passing its path during the

build

  1. download source drop from project -- no generated files (including build.xmls), no binary code
  2. run the PDE build.xml with consistent options
    1. I envision us using: baseLocation, dontUnzip, dontFetch -- everything else

should be derived from these values