Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Making Builds Reproducible"
(→Making Builds Reproducible) |
(→An Ideal Future) |
||
| (4 intermediate revisions by the same user not shown) | |||
| Line 3: | Line 3: | ||
==Background== | ==Background== | ||
| − | In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following: | + | In order to build non-SDK projects offline -- a requirement of our buildsystem |
| + | for security purposes -- we must do the following: | ||
| − | # check out releng plugin | + | # Generate a source drop |
| − | # add a fetch target | + | ## check out releng plugin |
| + | ## add a fetch target | ||
| + | ## run the fetch target with the Eclipse antrunner | ||
| + | ## tar up the results | ||
| + | # Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main | ||
| − | + | ==Common Pain Points== | |
| − | + | The last step -- the actual building with PDE -- is often different between | |
| − | + | projects. Some common variable differences: | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | * javacFailOnError - why isn't this always set? | |
| + | * mapsLocal | ||
| + | * pde.build.scripts | ||
| + | * skipFetch | ||
| + | * buildLabel | ||
| + | * buildAlias | ||
| + | * baseExists | ||
| + | * eclipse.builder.fetch | ||
| + | * baseLocation vs. base.location | ||
| + | * pde.builder.path | ||
| + | * mapVersionTag | ||
| + | * buildDirectory | ||
| + | * a build.xml that is essentially a shell for PDE's build.xml - why not just use PDE build.xml directly? | ||
| + | * ant build files that are shells around the "standard" (essentially empty) build.xml that set obscure variables that are necessary but non-standard | ||
| + | * build scripts (bash, DOS) that set obscure variables that are necessary but non-standard | ||
| + | * build scripts that assume relative/absolute paths | ||
* customTargets.xml attempts to fetch dependencies with no flags to override this behaviour | * customTargets.xml attempts to fetch dependencies with no flags to override this behaviour | ||
| − | ** ex. getBaseBuilder is in postFetch unconditionally - | + | ** ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out |
| − | * | + | * build scripts that assume certain usernames/passwords -- as mentioned in [[Europa_Build_Workshop_Report#Eclipse_Build_Best_Practices|Eclipse Build Best Practices]], a common cause of this is the lack of separation between fetching, building, and publishing |
| − | ** ie. we would | + | |
| + | ==Solutions to these common gotchas== | ||
| + | * separate fetching of your project's source code from fetching basebuilder/SDK and dependencies | ||
| + | * wrap fetching of basebuilder/SDK and dependencies with flags that can be overridden (ie. dontFetch) | ||
| + | * use consistent flags/variables | ||
| + | ** derive values from other variables (ie. pde.build.scripts/path from baseLocation | ||
| + | * do not assume use of basebuilder as it is ''almost always'' perfectly fine to use the SDK -- especially for I- or M-buildsand releases -- and we do this :) | ||
| + | * do not hard-code usernames, paths, etc. that aren't either universal or at the very least overridable | ||
| + | |||
| + | ==An Ideal Future== | ||
| + | |||
| + | In an ideal world, all it would take to build a project on top of the SDK would | ||
| + | be something like this: | ||
| + | |||
| + | # assume SDK is available and have a standard way of passing its path during the build | ||
| + | # download source drop from project -- no generated files (including build.xmls), no binary code | ||
| + | # run the PDE build.xml with consistent options | ||
| + | ## I envision us using: baseLocation, dontUnzip, dontFetch -- everything else should be derived from these values | ||
Latest revision as of 12:01, 15 September 2006
Contents
Making Builds Reproducible
Background
In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following:
- Generate a source drop
- check out releng plugin
- add a fetch target
- run the fetch target with the Eclipse antrunner
- tar up the results
- Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main
Common Pain Points
The last step -- the actual building with PDE -- is often different between projects. Some common variable differences:
- javacFailOnError - why isn't this always set?
- mapsLocal
- pde.build.scripts
- skipFetch
- buildLabel
- buildAlias
- baseExists
- eclipse.builder.fetch
- baseLocation vs. base.location
- pde.builder.path
- mapVersionTag
- buildDirectory
- a build.xml that is essentially a shell for PDE's build.xml - why not just use PDE build.xml directly?
- ant build files that are shells around the "standard" (essentially empty) build.xml that set obscure variables that are necessary but non-standard
- build scripts (bash, DOS) that set obscure variables that are necessary but non-standard
- build scripts that assume relative/absolute paths
- customTargets.xml attempts to fetch dependencies with no flags to override this behaviour
- ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
- build scripts that assume certain usernames/passwords -- as mentioned in Eclipse Build Best Practices, a common cause of this is the lack of separation between fetching, building, and publishing
Solutions to these common gotchas
- separate fetching of your project's source code from fetching basebuilder/SDK and dependencies
- wrap fetching of basebuilder/SDK and dependencies with flags that can be overridden (ie. dontFetch)
- use consistent flags/variables
- derive values from other variables (ie. pde.build.scripts/path from baseLocation
- do not assume use of basebuilder as it is almost always perfectly fine to use the SDK -- especially for I- or M-buildsand releases -- and we do this :)
- do not hard-code usernames, paths, etc. that aren't either universal or at the very least overridable
An Ideal Future
In an ideal world, all it would take to build a project on top of the SDK would be something like this:
- assume SDK is available and have a standard way of passing its path during the build
- download source drop from project -- no generated files (including build.xmls), no binary code
- run the PDE build.xml with consistent options
- I envision us using: baseLocation, dontUnzip, dontFetch -- everything else should be derived from these values