Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Making Builds Reproducible"
(→Making Builds Reproducible) |
(→Background) |
||
Line 3: | Line 3: | ||
==Background== | ==Background== | ||
− | In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following: | + | In order to build non-SDK projects offline -- a requirement of our buildsystem |
+ | for security purposes -- we must do the following: | ||
− | # check out releng plugin | + | # Generate a source drop |
− | # add a fetch target | + | ## check out releng plugin |
+ | ## add a fetch target | ||
+ | ## run the fetch target with the Eclipse antrunner | ||
+ | ## tar up the results | ||
+ | # Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main | ||
− | + | ==Common Pain Points== | |
− | + | The last step -- the actual building with PDE -- is often different between | |
− | + | projects. Some common variable differences: | |
− | + | ||
− | + | ||
− | + | ||
− | + | * javacFailOnError - why isn't this always set? | |
+ | * mapsLocal | ||
+ | * pde.build.scripts | ||
+ | * skipFetch | ||
+ | * buildLabel | ||
+ | * buildAlias | ||
+ | * baseExists | ||
+ | * eclipse.builder.fetch | ||
+ | * baseLocation vs. base.location | ||
+ | * pde.builder.path | ||
+ | * mapVersionTag | ||
+ | * buildDirectory | ||
− | * customTargets.xml attempts to fetch dependencies with no flags to override this behaviour | + | * a build.xml that is essentially a shell for PDE's build.xml - why not just use |
− | ** ex. getBaseBuilder is in postFetch unconditionally - | + | PDE build.xml directly? |
− | * | + | * ant build files that are shells around the "standard" (essentially empty) |
− | ** ie. we would | + | build.xml that set obscure variables that are necessary but non-standard |
+ | * build scripts (bash, DOS) that set obscure variables that are necessary but | ||
+ | non-standard | ||
+ | * build scripts that assume relative/absolute paths | ||
+ | * customTargets.xml attempts to fetch dependencies with no flags to override | ||
+ | this behaviour | ||
+ | ** ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out | ||
+ | * build scripts that assume certain usernames/passwords -- as said in | ||
+ | [[Europa_Build_Workshop_Report#Eclipse_Build_Best_Practices Eclipse Build Best | ||
+ | Practices]], a common cause of this is the lack of separation between fetching, | ||
+ | building, and publishing | ||
+ | |||
+ | ==Solutions to these common gotchas== | ||
+ | * separate fetching of your project's source code from fetching basebuilder/SDK | ||
+ | and dependencies | ||
+ | * wrap fetching of basebuilder/SDK and dependencies with flags that can be | ||
+ | overridden (ie. dontFetch) | ||
+ | * use consistent flags/variables | ||
+ | ** derive values from other variables (ie. pde.build.scripts/path from | ||
+ | baseLocation | ||
+ | * do not assume use of basebuilder as it is ''almost always'' perfectly fine to use the | ||
+ | SDK -- especially for I- or M-buildsand releases -- and we do this :) | ||
+ | * do not hard-code usernames, paths, etc. that aren't either universal or at the | ||
+ | very least overridable | ||
+ | * | ||
+ | |||
+ | ==An Ideal Future== | ||
+ | |||
+ | In an ideal world, all it would take to build a project on top of the SDK would | ||
+ | be something like this: | ||
+ | |||
+ | # assume SDK is available and have a standard way of passing its path during the | ||
+ | build | ||
+ | # download source drop from project -- no generated files (including build.xmls), no binary code | ||
+ | # run the PDE build.xml with consistent options | ||
+ | ## I envision us using: baseLocation, dontUnzip, dontFetch -- everything else | ||
+ | should be derived from these values |
Revision as of 11:58, 15 September 2006
Contents
Making Builds Reproducible
Background
In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following:
- Generate a source drop
- check out releng plugin
- add a fetch target
- run the fetch target with the Eclipse antrunner
- tar up the results
- Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main
Common Pain Points
The last step -- the actual building with PDE -- is often different between projects. Some common variable differences:
- javacFailOnError - why isn't this always set?
- mapsLocal
- pde.build.scripts
- skipFetch
- buildLabel
- buildAlias
- baseExists
- eclipse.builder.fetch
- baseLocation vs. base.location
- pde.builder.path
- mapVersionTag
- buildDirectory
- a build.xml that is essentially a shell for PDE's build.xml - why not just use
PDE build.xml directly?
- ant build files that are shells around the "standard" (essentially empty)
build.xml that set obscure variables that are necessary but non-standard
- build scripts (bash, DOS) that set obscure variables that are necessary but
non-standard
- build scripts that assume relative/absolute paths
- customTargets.xml attempts to fetch dependencies with no flags to override
this behaviour
- ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
- build scripts that assume certain usernames/passwords -- as said in
[[Europa_Build_Workshop_Report#Eclipse_Build_Best_Practices Eclipse Build Best Practices]], a common cause of this is the lack of separation between fetching, building, and publishing
Solutions to these common gotchas
- separate fetching of your project's source code from fetching basebuilder/SDK
and dependencies
- wrap fetching of basebuilder/SDK and dependencies with flags that can be
overridden (ie. dontFetch)
- use consistent flags/variables
- derive values from other variables (ie. pde.build.scripts/path from
baseLocation
- do not assume use of basebuilder as it is almost always perfectly fine to use the
SDK -- especially for I- or M-buildsand releases -- and we do this :)
- do not hard-code usernames, paths, etc. that aren't either universal or at the
very least overridable
An Ideal Future
In an ideal world, all it would take to build a project on top of the SDK would be something like this:
- assume SDK is available and have a standard way of passing its path during the
build
- download source drop from project -- no generated files (including build.xmls), no binary code
- run the PDE build.xml with consistent options
- I envision us using: baseLocation, dontUnzip, dontFetch -- everything else
should be derived from these values