Jump to: navigation, search

Difference between revisions of "Build Workshop 2: Build Harder/Report"

(Project Info Data)
m (Other considerations)
 
(7 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
Igor Fedorenko, Nick Boldt, Bjorn Freeman-Benson, Denis Roy, Kim Moir (morning), Pascal Rapicault (morning), DJ Houghton (morning)
 
Igor Fedorenko, Nick Boldt, Bjorn Freeman-Benson, Denis Roy, Kim Moir (morning), Pascal Rapicault (morning), DJ Houghton (morning)
 
=== Pain points ===
 
=== Pain points ===
* Build takes 2 weeks to setup
+
* Build and server can take up to 2 weeks to set up
* Steep learning curve
+
* Steep learning curve (server config, .releng project, ant/bash scripting, etc.)
* Enforcing eclipse.org rules
+
* Enforcing eclipse.org rules (eg., [[Ganymede#Must_Do|Ganymede Must Dos]])
* No one wants to build
+
* No one wants to build, but everyone NEEDS a build
 +
 
 
=== Existing build approaches ===
 
=== Existing build approaches ===
 
* Set of scripts that run headless eclipse with basebuilder. Builds are scheduled from WebUI but actual work is done by cron job. WebUI and the cron job communicate via tmp file.
 
* Set of scripts that run headless eclipse with basebuilder. Builds are scheduled from WebUI but actual work is done by cron job. WebUI and the cron job communicate via tmp file.
Line 13: Line 14:
 
=== Other considerations ===
 
=== Other considerations ===
 
* Ability to build/test projects against multiple versions of target platform. Not many projects use that, ones that do would need to setup separate build for each target platform.
 
* Ability to build/test projects against multiple versions of target platform. Not many projects use that, ones that do would need to setup separate build for each target platform.
* Ability to reproduce builds (needs to be tagging, but there is more to this). Some exploiters rebuild eclipse.org projects from sources, so builds should be reproducible outside of eclipse.org too.  
+
* Ability to reproduce builds (needs to be tagging, but there is more to this). Some member companies (particularly those in the [[Linux_Distributions_Project|Linux]] world) rebuild eclipse.org projects from sources, so builds should be reproducible outside of eclipse.org too.
 +
* Ability to generate source builds for both legal considerations and for rebuild needs
 +
 
 
=== Discussed topics ===
 
=== Discussed topics ===
* Common build infrastructure for smaller projects. Configuration will be done via project portal. Committers will be able to trigger builds via http://build.eclipse.org/cbi/.
+
* Common build infrastructure for smaller projects. Configuration will be done via project portal. Committers will be able to trigger builds via '''<nowiki>http://build.eclipse.org/cbi/</nowiki>'''.
 
* As a way to promote best practices, there will be only one way of doing builds in CBI. Projects that want/need to deviate, will have to fork CBI or setup their own build from scratch.
 
* As a way to promote best practices, there will be only one way of doing builds in CBI. Projects that want/need to deviate, will have to fork CBI or setup their own build from scratch.
 
* One build or one build per top-level project. Use build queue as a way to measure build server utilization.
 
* One build or one build per top-level project. Use build queue as a way to measure build server utilization.
 
* Bigger projects will likely need dedicated build person/team.  
 
* Bigger projects will likely need dedicated build person/team.  
* Building Eclipse SDK is outside of CBI scope.
+
* Building Eclipse SDK -- or any product/project with platform specifics -- is outside of CBI scope. This is just for all-Java, simple projects.
 
* Separate virtual server for each project will require too much hardware resources, but individual projects can request virtual build server.
 
* Separate virtual server for each project will require too much hardware resources, but individual projects can request virtual build server.
* Cruisecontrol or similar tool to execute builds on as needed bases or by request. For CVS, watch changes to map file(s) instead of entire source tree. Not a problem for SVN.
+
* Cruisecontrol or similar tool to execute builds on as needed bases or by request. For CVS, watch changes to map file(s) instead of entire source tree. Not a problem for SVN, though build system currently isn't tested w/ SVN repo.
==Project Info Data==
+
 
We're trying to move the configuration information from static files into project info meta-data entered in the portal. So far we've moved these parameters which are all values under the "build" meta-data item:
+
===See Also===
 +
 
 +
* [http://dev.eclipse.org/blogs/releng/2008/07/02/build-workshop-2-build-harder/ .releng Blog]
 +
* [http://cruisecontrol.sourceforge.net/ Cruise Control]
 +
* [http://www.eclipse.org/buckminster/ Buckminster Project]
 +
* [http://www.eclipse.org/dash/ Dash Project]
 +
* Maven Integration for Eclipse [http://maven.apache.org/eclipse-plugin.html @ Apache] & [http://www.eclipse.org/m2e/ @ Eclipse]
 +
 
 +
==Build Configuration Data==
 +
 
 +
We're trying to move the configuration information from static files (php arrays) into project info meta-data entered in the portal. So far we've moved these parameters which are all values under the "build" meta-data item:
 +
 
 +
* '''ProjRelengRoot''' e.g. :pserver:anonymous@dev.eclipse.org:/cvsroot/tools -- defines CVSROOT for source & releng checkouts
 +
* '''BranchAndJDK''' e.g. 3.4.0=HEAD,/opt/public/stp/apps/IBMJava2-ppc-142 -or- 3.4.0=HEAD,/opt/public/common/ibm-java2-ppc-50 -- defines version, CVS branch, and path to java
 +
* '''regex''' e.g. I200.*/eclipse-SDK-|[SR]-.*200.*/eclipse-SDK- (optional) regular expression to be used to collect the most recent matching dependencies for running a build -- this is used by the web UI to preselect the "latest" available dependencies or by the crontab scripts to build using the latest dependencies. In time this may be replaced by the use of an RSS feed aggregator/parser, so that builds will always run with the latest deps.
 +
* '''Mapfile_Rule_Default''' e.g. 0 or 1 == "Use Map" or "Generate Map" -- this defines whether or not to use a static mapfile in CVS, or to use a mapfile template to generate a map using either HEAD or the specified branch tag (eg., for an 0.8.1 build, R0_8_maintenance)
 +
* '''EmailDefault''' e.g. nickboldt@gmail.com,anthonyh@ca.ibm.com -- email address(es) to contact when build is done or a problem occurs
  
* ProjRelengRoot e.g. :pserver:anonymous@dev.eclipse.org:/cvsroot/tools
+
The semantics of these parameters will eventually be documented, once the system is ready for general use. This is by no means a complete list -- some items may be rendered obsolete (eg., replacing EmailDefault with a defacto $project-dev@eclipse.org or $project-releng@eclipse.org mailing list) and others may be added. '''BranchAndJDK''' ought to be refactored into a CSV format.
* BranchAndJDK e.g. 3.4.0=HEAD,/opt/public/stp/apps/IBMJava2-ppc-142 -or- 3.4.0=HEAD,/opt/public/common/ibm-java2-ppc-50
+
* regex e.g. I200.*/eclipse-SDK-|[SR]-.*200.*/eclipse-SDK- (optional) regular expression to be used to collect the most recent matching dependencies for running a build
+
* Mapfile_Rule_Default e.g. 0; 0: "Use Map, No Tagging=use-false" or 1:"Generate Map, No Tagging=gen-false"
+
* EmailDefault e.g. nickboldt@gmail.com,anthonyh@ca.ibm.com
+

Latest revision as of 13:13, 3 July 2008

Meeting notes

Attendees

Igor Fedorenko, Nick Boldt, Bjorn Freeman-Benson, Denis Roy, Kim Moir (morning), Pascal Rapicault (morning), DJ Houghton (morning)

Pain points

  • Build and server can take up to 2 weeks to set up
  • Steep learning curve (server config, .releng project, ant/bash scripting, etc.)
  • Enforcing eclipse.org rules (eg., Ganymede Must Dos)
  • No one wants to build, but everyone NEEDS a build

Existing build approaches

  • Set of scripts that run headless eclipse with basebuilder. Builds are scheduled from WebUI but actual work is done by cron job. WebUI and the cron job communicate via tmp file.
  • Maven
  • Scripts + Buckminster to collect dependencies

Other considerations

  • Ability to build/test projects against multiple versions of target platform. Not many projects use that, ones that do would need to setup separate build for each target platform.
  • Ability to reproduce builds (needs to be tagging, but there is more to this). Some member companies (particularly those in the Linux world) rebuild eclipse.org projects from sources, so builds should be reproducible outside of eclipse.org too.
  • Ability to generate source builds for both legal considerations and for rebuild needs

Discussed topics

  • Common build infrastructure for smaller projects. Configuration will be done via project portal. Committers will be able to trigger builds via http://build.eclipse.org/cbi/.
  • As a way to promote best practices, there will be only one way of doing builds in CBI. Projects that want/need to deviate, will have to fork CBI or setup their own build from scratch.
  • One build or one build per top-level project. Use build queue as a way to measure build server utilization.
  • Bigger projects will likely need dedicated build person/team.
  • Building Eclipse SDK -- or any product/project with platform specifics -- is outside of CBI scope. This is just for all-Java, simple projects.
  • Separate virtual server for each project will require too much hardware resources, but individual projects can request virtual build server.
  • Cruisecontrol or similar tool to execute builds on as needed bases or by request. For CVS, watch changes to map file(s) instead of entire source tree. Not a problem for SVN, though build system currently isn't tested w/ SVN repo.

See Also

Build Configuration Data

We're trying to move the configuration information from static files (php arrays) into project info meta-data entered in the portal. So far we've moved these parameters which are all values under the "build" meta-data item:

  • ProjRelengRoot e.g. :pserver:anonymous@dev.eclipse.org:/cvsroot/tools -- defines CVSROOT for source & releng checkouts
  • BranchAndJDK e.g. 3.4.0=HEAD,/opt/public/stp/apps/IBMJava2-ppc-142 -or- 3.4.0=HEAD,/opt/public/common/ibm-java2-ppc-50 -- defines version, CVS branch, and path to java
  • regex e.g. I200.*/eclipse-SDK-|[SR]-.*200.*/eclipse-SDK- (optional) regular expression to be used to collect the most recent matching dependencies for running a build -- this is used by the web UI to preselect the "latest" available dependencies or by the crontab scripts to build using the latest dependencies. In time this may be replaced by the use of an RSS feed aggregator/parser, so that builds will always run with the latest deps.
  • Mapfile_Rule_Default e.g. 0 or 1 == "Use Map" or "Generate Map" -- this defines whether or not to use a static mapfile in CVS, or to use a mapfile template to generate a map using either HEAD or the specified branch tag (eg., for an 0.8.1 build, R0_8_maintenance)
  • EmailDefault e.g. nickboldt@gmail.com,anthonyh@ca.ibm.com -- email address(es) to contact when build is done or a problem occurs

The semantics of these parameters will eventually be documented, once the system is ready for general use. This is by no means a complete list -- some items may be rendered obsolete (eg., replacing EmailDefault with a defacto $project-dev@eclipse.org or $project-releng@eclipse.org mailing list) and others may be added. BranchAndJDK ought to be refactored into a CSV format.