Difference between revisions of "Making Builds Reproducible"

From Eclipsepedia

Jump to: navigation, search
(Background)
(An Ideal Future)
 
(3 intermediate revisions by one user not shown)
Line 31: Line 31:
 
* buildDirectory
 
* buildDirectory
  
* a build.xml that is essentially a shell for PDE's build.xml - why not just use
+
* a build.xml that is essentially a shell for PDE's build.xml - why not just use PDE build.xml directly?
PDE build.xml directly?
+
* ant build files that are shells around the "standard" (essentially empty) build.xml that set obscure variables that are necessary but non-standard
* ant build files that are shells around the "standard" (essentially empty)
+
* build scripts (bash, DOS) that set obscure variables that are necessary but non-standard
build.xml that set obscure variables that are necessary but non-standard
+
* build scripts (bash, DOS) that set obscure variables that are necessary but
+
non-standard
+
 
* build scripts that assume relative/absolute paths
 
* build scripts that assume relative/absolute paths
* customTargets.xml attempts to fetch dependencies with no flags to override
+
* customTargets.xml attempts to fetch dependencies with no flags to override this behaviour
this behaviour
+
 
** ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
 
** ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
* build scripts that assume certain usernames/passwords -- as said in
+
* build scripts that assume certain usernames/passwords -- as mentioned in [[Europa_Build_Workshop_Report#Eclipse_Build_Best_Practices|Eclipse Build Best Practices]], a common cause of this is the lack of separation between fetching, building, and publishing
[[Europa_Build_Workshop_Report#Eclipse_Build_Best_Practices Eclipse Build Best
+
Practices]], a common cause of this is the lack of separation between fetching,
+
building, and publishing
+
  
 
==Solutions to these common gotchas==
 
==Solutions to these common gotchas==
* separate fetching of your project's source code from fetching basebuilder/SDK
+
* separate fetching of your project's source code from fetching basebuilder/SDK and dependencies
and dependencies
+
* wrap fetching of basebuilder/SDK and dependencies with flags that can be overridden (ie. dontFetch)
* wrap fetching of basebuilder/SDK and dependencies with flags that can be
+
overridden (ie. dontFetch)
+
 
* use consistent flags/variables
 
* use consistent flags/variables
** derive values from other variables (ie. pde.build.scripts/path from
+
** derive values from other variables (ie. pde.build.scripts/path from baseLocation
baseLocation
+
* do not assume use of basebuilder as it is ''almost always'' perfectly fine to use the SDK -- especially for I- or M-buildsand releases -- and we do this :)
* do not assume use of basebuilder as it is ''almost always'' perfectly fine to use the
+
* do not hard-code usernames, paths, etc. that aren't either universal or at the very least overridable
SDK -- especially for I- or M-buildsand releases -- and we do this :)
+
* do not hard-code usernames, paths, etc. that aren't either universal or at the
+
very least overridable
+
*
+
  
 
==An Ideal Future==
 
==An Ideal Future==
Line 65: Line 52:
 
be something like this:
 
be something like this:
  
# assume SDK is available and have a standard way of passing its path during the
+
# assume SDK is available and have a standard way of passing its path during the build
build
+
 
# download source drop from project -- no generated files (including build.xmls), no binary code
 
# download source drop from project -- no generated files (including build.xmls), no binary code
 
# run the PDE build.xml with consistent options
 
# run the PDE build.xml with consistent options
## I envision us using:  baseLocation, dontUnzip, dontFetch -- everything else
+
## I envision us using:  baseLocation, dontUnzip, dontFetch -- everything else should be derived from these values
should be derived from these values
+

Latest revision as of 12:01, 15 September 2006

Contents

[edit] Making Builds Reproducible

[edit] Background

In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following:

  1. Generate a source drop
    1. check out releng plugin
    2. add a fetch target
    3. run the fetch target with the Eclipse antrunner
    4. tar up the results
  2. Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main

[edit] Common Pain Points

The last step -- the actual building with PDE -- is often different between projects. Some common variable differences:

  • javacFailOnError - why isn't this always set?
  • mapsLocal
  • pde.build.scripts
  • skipFetch
  • buildLabel
  • buildAlias
  • baseExists
  • eclipse.builder.fetch
  • baseLocation vs. base.location
  • pde.builder.path
  • mapVersionTag
  • buildDirectory
  • a build.xml that is essentially a shell for PDE's build.xml - why not just use PDE build.xml directly?
  • ant build files that are shells around the "standard" (essentially empty) build.xml that set obscure variables that are necessary but non-standard
  • build scripts (bash, DOS) that set obscure variables that are necessary but non-standard
  • build scripts that assume relative/absolute paths
  • customTargets.xml attempts to fetch dependencies with no flags to override this behaviour
    • ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
  • build scripts that assume certain usernames/passwords -- as mentioned in Eclipse Build Best Practices, a common cause of this is the lack of separation between fetching, building, and publishing

[edit] Solutions to these common gotchas

  • separate fetching of your project's source code from fetching basebuilder/SDK and dependencies
  • wrap fetching of basebuilder/SDK and dependencies with flags that can be overridden (ie. dontFetch)
  • use consistent flags/variables
    • derive values from other variables (ie. pde.build.scripts/path from baseLocation
  • do not assume use of basebuilder as it is almost always perfectly fine to use the SDK -- especially for I- or M-buildsand releases -- and we do this :)
  • do not hard-code usernames, paths, etc. that aren't either universal or at the very least overridable

[edit] An Ideal Future

In an ideal world, all it would take to build a project on top of the SDK would be something like this:

  1. assume SDK is available and have a standard way of passing its path during the build
  2. download source drop from project -- no generated files (including build.xmls), no binary code
  3. run the PDE build.xml with consistent options
    1. I envision us using: baseLocation, dontUnzip, dontFetch -- everything else should be derived from these values