Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Common Build Infrastructure/Athena Progress Report
Contents
- 1 Recent Changes
- 2 2010-04-15
- 3 2010-03-28
- 4 2010-02-01
- 5 2009-11-27
- 6 2009-10-13
- 7 2009-09-01
- 8 2009-08-07
- 9 2009-07-23
- 10 2009-07-15
- 11 2009-07-10
- 12 2009-06-05
- 13 2009-05-26
- 14 2009-05-18
- 15 2009-05-09
- 16 2009-04-22
- 17 2009-03-30
- 18 2009-03-15
- 19 2009-03-05
- 20 2009-03-01
- 21 2009-02-26
- 22 2009-02-22
- 23 2009-02-16
- 24 2009-01-16
- 25 2008-11-26
- 26 2008-11-19
- 27 2008-11-06
- 28 2008-10-28
Recent Changes
- Bugs changed: this week, this month
- RSS feed: weekly changes
2010-04-15
- Ease of use / documentation
- bug 309389 New, simpler method for bootstrapping a build: Common Build Infrastructure/Getting Started/Bootstrapping. Now, like with Tycho/Maven, you need only check out the project from the repo, and run it.
2010-03-28
- Ease of use
- build.xml is now simpler. Sample project template updated. You can now use the same build.xml script to run a build in Eclipse or in Hudson from an Ant-based job. Note that the page Build In Hudson/Ant Script is no longer current. See also bug 304800.
- You can now run
buildExtra.xml#extraPackaging
viabuild.steps=buildExtra
rather than requiringbuild.steps=buildZips
.
- You can now run
- Publishing support will improve thanks to bug 306854... anyone want to contribute to writing a Hudson job, please feel free!
- Packaging Support
- bug 306300 Athena removes .jar files and only contains pack 200 files in update site - new default setting (keep both artifact types) is
removeUnpackedJars=false
, but can revert to old behaviour (and smaller update site) withremoveUnpackedJars=true
- bug 306300 Athena removes .jar files and only contains pack 200 files in update site - new default setting (keep both artifact types) is
- bug 307016 SDK and Runtime zips produced with buildZips target are missing {notice,epl-v10}.html files; if root files exist they will now be copied into the SDK, Runtime, Examples zips; ${allZip} will also include this so it can be copied from there too (custom packaging in buildExtra.xml#extraPackaging task)
- Testing Support
- bug 296352 Can't connect to vnc server - fixed using Xnvc option in Hudson job and improvements to testLocal task
- Publishing Support
- bug 302170 Work around Hudson's missing lastS*Build folders - promote.xml will now recurse into Hudson job tree looking for correct build to publish
- Publishing support will improve thanks to bug 306854... anyone want to contribute to writing a Hudson job, please feel free!
- Bug Fixes
- bug 304800 Temporary regression caused by adopting new build.xml script with too-aggressive cleanup default
- bug 307016 SDK and Runtime zips produced with buildZips target are missing {notice,epl-v10}.html files; if root files exist they will now be copied into the SDK, Runtime, Examples zips; ${allZip} will also include this so it can be copied from there too (custom packaging in buildExtra.xml#extraPackaging task)
- Documentation & Branding
- EclipseCon 2010 Presentation, "Dash Athena Exposed" is available here.
- Sample project template & generic build.xml updated.
- bug 272723 Logo design contest for Athena under way: vote early, vote often!
2010-02-01
There are now 63 Athena jobs on build.eclipse.org, building 41 projects using 6 job templates (Bash/Ant, CVS/SVN, Nightly/Integration).
- Infrastructure Changes
- bug 287013 - Support generation of Helios .build files - see Common Build Infrastructure/Publishing/Helios
- bug 272991 - Support user-defined site.xml for p2 metadata generation in Update zip
- Cross-Platform / Ease of Use
- bug 294927 - Remove hard coding of /tmp directory
- getCompilerResults.sh is now getCompilerResults.xml, so that this information can be generated on Windows now too. No more perl and egrep, just loadfile, replaceregex, and linecontainsregexp.
- Bug Fixes
- bug 296326 - subprojectName should == projectName if projectid only contains 1 part (foo, not foo.bar)
- Documentation
- Getting Started - Build In Eclipse updated w/ more .releng project example links
- Testing/VMArgs How to configure VM args and define a custom library.xml for your tests
2009-11-27
- Infrastructure Changes
- bug 295394 - Headless UI tests are broken on build.eclipse.org! See workaround here.
- Cross-Platform / Ease of Use
- bug 282987 Can now define Eclipse platform as a p2 repo + required features/plugins. Can also use platform, SDK, or RCP zip, as before. See also Defining Binary Dependencies.
- bug 292589 Support building from Ant in Hudson (instead of via Bash script)
- bug 295670 Support running JUnit tests run from Ant script instead of Bash - tests can now be run on Linux and Windows (and to a lesser extent on Mac OSX) using new
testLocal
build step
- bug 295670 Support running JUnit tests run from Ant script instead of Bash - tests can now be run on Linux and Windows (and to a lesser extent on Mac OSX) using new
- Bug Fixes
- bug 292486 Allow builds to fail if unit tests fail using
failBuildIfTestFailuresOrErrors=true
- bug 292486 Allow builds to fail if unit tests fail using
- bug 294678 Categories don't show up with IBM 1.6 JDK - implemented workaround so a different JDK can be used for p2 operations than for compilation of sources: use
PACK200_JAVA_HOME=/path/to/different/JDK
- bug 294678 Categories don't show up with IBM 1.6 JDK - implemented workaround so a different JDK can be used for p2 operations than for compilation of sources: use
- bug 295773 Non-incubating projects no longer need to set "incubation=" in their build.properties
- bug 292235 Included pre-compiled binary features/plugins are now included by default using PDE's runPackager property; can disable this behaviour with
packageFeaturesIncludedBinaries=false
- bug 292235 Included pre-compiled binary features/plugins are now included by default using PDE's runPackager property; can disable this behaviour with
- Documentation
- Testing - Different ways to run or re-run tests, including as a secondary process after a build (so they can be run on a different platform or by a different user)
- Defining Binary Dependencies - All four types of repos are now currently supported: remote zip, remote site, local zip, local site.
- p2 does not natively support remote repo zips; to workaround this the zip is fetched and the URL is rewritten from
http://path/to/foo.zip
tojar:file:/tmp/path/foo.zip!/
- Build In Hudson - Ant Job, Ant Script - new, simpler way to run builds in Hudson using pure Ant
- Build In Hudson - Bash Job - updated & refactored
2009-10-13
- Infrastructure changes
- There are now 43 Athena jobs on build.eclipse.org! Of those, 30 are green, 1 is yellow, and 6 have not yet been enabled. These jobs represent 29 different projects' builds! 6 of them use SVN sources instead of CVS.
- bug 257074 comment 12 build.eclipse.org now has SVN 1.6.5 on it; if your map files no longer work (your build complains it can't resolve plugin or features sources) then read this.
- Bug fixes
- bug 291446 Provide hook for extra actions after fetching code from repo and before compiling it (e.g. code generation, parser generators, etc.)
- bug 275529 Athena is now a full Project rather than a Component! Now if we could just get someone to design a logo... Do I need to offer up prizes?
- Documentation
- Tips for Building on Windows - community contributed! Thanks to Nicolas Bros!
- New category created for User-contributed build stories: Category:Athena Common Build Users
- Bugzilla updated to move
product=Dash&component=Common+Builder
toproduct=Dash+Athena
; all old bugs moved too.
2009-09-01
- Bug fixes
- bug 284593 improved documentation and sample code for setting up a crontab script for publishing your project's bits.
- bug 285359 Add option to ignore existing test.xml file if it exists.
- bug 287240 Add more entries ${JAVA14_HOME}/lib/*.jar entries to J2SE-1.4 to support a wider audience
- bug 284968 SDK zips do not include sources - they do now!
- In the past, to install extra Orbit bundles into the base Eclipse platform before building, you needed a buildExtra.xml (and/or testExtra.xml), with a task like this:
<target name="getDependencies"> <mkdir dir="${buildDirectory}/../eclipse/plugins/" /> <get dest="${buildDirectory}/../eclipse/plugins/org.apache.xalan_2.7.1.v200905122109.jar" src="http://download.eclipse.org/tools/orbit/downloads/drops/R20090825191606/bundles/org.apache.xalan_2.7.1.v200905122109.jar" /> </target>
- Now you can now install plugins (not just features) into the target platform prior to building. For example, to install orbit jars for which there are no features, you can add the following into your build.properties file. Note that the update site MUST BE A p2 REPO, not simply a "Classic" Update Site.
repositoryURLs=http://download.eclipse.org/tools/orbit/downloads/drops/R20090825191606/updateSite pluginIDsToInstall=org.apache.xml.resolver+javax.xml+org.apache.xml.serializer+org.apache.xerces+org.apache.xalan
- If you want to install features and their included plugins, simply use the
featreIDsToInstall
property:
- If you want to install features and their included plugins, simply use the
repositoryURLs=http://download.eclipse.org/releases/galileo/ featureIDsToInstall=org.eclipse.emf+org.eclipse.gef
2009-08-07
- Ease of use / inter-build dependencies
- bug 285519 support static URLs for "latest build" zips in Hudson so builds can depend on each other's output -- set the
- bug 284959 hudson shell script should be completely generic
When launching your build via shell script in Hudson, use this instead of your own custom shell script. You'll get N-SNAPSHOT.zip instead of N200908071234.zip so that others can more easily depend on your latest stable build.
export SNAPSHOT="true" . /opt/public/cbi/build/org.eclipse.dash.common.releng/hudson/run.sh
- New builds
- Modeling now has 8 builds
- PDE Visualization
2009-07-23
- Ease of Use
- bug 283776 create dir2svnmap task to generate a map file from checked out SVN sources (use with build.steps=dir2svnmap in build.properties)
- bug 284055 Build seems not to unpack all dependencies - see FAQ
- bug 284516 make optional pre-build dependency unpacking and post-build packaging more optional
- Better testing support
- bug 284331 Support test plugins as jars, not just folders
- Better documentation / error handling
- New Getting Started FAQ
- bug 284509 warn users of possible linux + SVN problems when checking out
- New builds
- EMF Query, Validation and Transaction now have a 3-part build
- Ajax Tools Framework has been rebooted with its first nightly build - tests coming soon.
2009-07-15
- Improve signing recovery w/ better log output and better trapping for errors; also, using Ant-Contrib we can avoid the stack overflow issue w/ tasks that call themselves indefinitely.
- bug 280642 Improve recovery from infinitely long signing queue
- bug 254205 Signing should fail more gracefully
- Also updated GEF releng example to build from Eclipse 3.5.0 w/ latest map sources, individual source plugins, and these properties in build.properties:
flattenDependencies=true parallelCompilation=true generateFeatureVersionSuffix=true individualSourceBundles=true
2009-07-10
- Some minor bug fixes:
- bug 280747 testPluginsToRun property doesn't ignore whitespace
- bug 282593 Signing should not be default-enabled for non-eclipse.org builds
- bug 282910 start.sh help says one can pass projRelengName but the arg is not parsed
- Several non-Eclipse.org build examples are available, including maps and generated psfs (using map2psf build.step):
- Discussion starting re: different ways to build - using RCP vs. SDK, using default-specified SDKs for user's platform (rather than needing to be explicit in dependencyURLs)
2009-06-05
- bug 272991 Documented generating custom update site categories and hiding unwanted features.
- bug 279056 Added support for excluding some jars from packing or signing; solved issue w/ VE's "corrupt" uninstallable jars.
- bug 278687 Added a new optional build.step for generating Project Set Files (.psf) from PDE .map files.
2009-05-26
- Documented p2 repo and SDK zip input options.
- buildExtra.xml and testExtra.xml are now optional; by default all zips will be unpacked, all repos+features will be installed w/o explicitly needing to direct the build how and where
- bug 252774 improve support for CVS/SVN repo dumps so building from local sources is easier/smarter (to ease local setup when repo tree is non-standard): testing complete; released to HEAD.
2009-05-18
- Started Hudson Management Documentation. Includes HOWTO for restarting server to install updates.
- Started Jar Signing Documentation. Includes code sample.
- SWTBot builds added to Hudson
- bug 276574 generate default test.xml if not present in test plugin (to ease migration to Athena builder - less boilerplate crap you have to write/maintain)
- bug 252774 improve support for CVS/SVN repo dumps so building from local sources is easier/smarter (to ease local setup when repo tree is non-standard)
- bug 273518 Support building against p2 repos using both Eclipse 3.4 and new 3.5 versions of p2.director (work in progress in
bug272774-bug273518
branch)
- Began testing Athena for use with projects outside the org.eclipse namespace.
2009-05-09
- Builds now using Athena in Hudson include: GEF, Linux Tools, Visual Editor (VE), Voice Tools (VTP), and Nebula: https://build.eclipse.org/hudson/view/Athena%20CBI/
- Wiki docs refactored to use Category:Athena Common Build category.
- Other closed bugs since 2009-04-22:
- bug 273299 index.php should link to http://www.eclipse.org/downloads/download.php?file=/path/to/files/ to use mirrors
- bug 253277 Local build should warn/fail gracefully in the absence of specific tools
- bug 266374 design process for publishing builds from Hudson to download.eclipse.org
- bug 273302 Create crontab'able script for rsync/copy, fix perms, and unpack update zip
- bug 251926 support setting feature/plugin qualifiers to buildId (instead of map tags) when running nightly builds
- bug 272965 support packaging for namespaces other than org.eclipse.*
2009-04-22
- The 'buildZips' step is now optional; you can omit it entirely and still run your tests from the generated update site. This will also verify that your update site can be installed into Eclipse SDK + your runtime requirements using the p2 director (bug 272403).
- As with PDE SVN support and ant4eclipse, Athena can now fetch ant-contrib from Sourceforge if it's not already installed (bug 253277).
- Working on a way to input a custom site.xml so that your own categories will be used in the generated p2 repo / update site (bug 272991).
- Anonymous access to Hudson on build.eclipse.org is now available! Log in using your committerid for more control. To create a new job, you must be in the callisto-dev group (or ask someone who's in the group to create the job and add you to its ACL).
- Nebula and VE are now building using the Athena system (bug 237588, bug 270849).
- Other closed bugs since 2009-03-30:
- bug 272404 build.step "cleanup" should purge Master + All zips
- bug 252401 Make unpacking doc.isv optional
- bug 252030 standalone buildserver-in-a-box (vmware/vbox/qemu/kvm)
- bug 252049 Separate exploded SDK and sources to build
- bug 256212 verify default & extra packaging steps are producing correct zips
- bug 271145 Document build types
- bug 271211 optional lib folder in project.releng should be optional
- bug 271322 Need another bootclasspath location
- bug 271524 run (not runEclipse) task can't find ant4eclipse
- bug 271549 Should we be able to run tests w/ an archived p2 repo instead of an SDK zip?
- bug 271661 if map contains svn ref and pde-svn plugin not found in basebuilder, spit out a smarter error message
- bug 271849 Document requirement to have version variable in script launching build
- bug 272906 Features with fragments don't build properly
- bug 252035 Provide a p2 repo after the master feature is built
- bug 256192 build folder's index.php contains broken links
- bug 264579 define a better workflow for build steps
2009-03-30
- Had a great week at EclipseCon '09 talking to people about Athena, particularly during Hands-On: Using the new Common Builder for Push-Button PDE Builds. Slides (supporting material & exercise instructions) + exercises are posted here, which include sample properties files for building on Mac, Linux, and Windows. Two build projects are included along with one simple PDE feature build as an appetizer before the main course. :)
- Due to popular demand, we'll be adding support for simple product builds (bug 271186) using the Athena system. Anyone want to help beta-test it?
2009-03-15
- progress made on building locally in Eclipse on Mac OS X 10.5 and on Windows XP. Tests do not yet run on these platforms due to shell script and Xvfb/Xvnc requirements.
- prototyping work done using ant4eclipse to fetch org.eclipse.test and org.eclipse.ant.optional.junit from a .psf file so they need not be included in project's map file
2009-03-05
- progress made on building from local sources. It's a hack for now, but it's workable (bug 252774)
- working on a virtual server image on which Hudson and Eclipse can be run for testing
2009-03-01
- Signing in Hudson now works (bug 264633)
- verify build can run in Hudson via shell script
- verify build can run in Eclipse via build.xml (bug 253269), or commandline via start.sh, including signing
- refactor relative paths to absolute so build can be configured to run in more ways
- verify alternate path options: /tools/gef/d/d/$v/$bid/ vs. in /$bid/; /opt/public/cbi/build vs. /tmp/build
- verify cleanup step actually works for eclipse/ and testing/ folders, locally and in Hudson
- bugzilla cleanup (previously completed items are now closed); currently 27 closed Athena bugs, 39 open
2009-02-26
- Getting Started Guide updated with details about running in Eclipse, running commandline, and running in Hudson.
- signing works when run from local build (headless, in Eclipse) and Hudson build
- build directory path is more configurable than ever, with (hopefully) no more relative paths
2009-02-22
Numerous items in progress this week thanks to help from Andrew O and Bjorn F-B, including:
- test running build system on Mac OS X (problems with bootclasspath & JVM still to be solved)
- Getting Started Guide now includes information about running in eclipse (bug 252041
- run build in Eclipse / convert start.sh to build.xml (bug 252028, bug 253269)
- it seems that the new branch won't run in Hudson, so this is still in progress
- investigated SVN tagging and SVN change tracking in Hudson; the former does not seem to be working, but the latter is
- reported Hudson problems w/ CVS checkouts and use of Xvnc for UI testing (bug 265750, bug 265751)
2009-02-16
- Defined a better way to control flow of build steps, so ant targets "runWithoutTest" and "runWithoutBuildTest" are now obsolete (bug 264579)
- Improved startup process for build. Migrated several chunks of bash script to Ant (bug 253269)
- Started working with Hudson on build.eclipse.org for controlling and administering builds' logs and artifacts (bug 251945)
- Installed several new plugins and got server updated to latest version (bug 264778)
- However, Hudson cannot currently sign jars (bug 251945)
- See also next week's workshop plans
2009-01-16
Due to other commitments, not much is new.
- Plan doc updated with target milestones
- Linuxtools is now running scheduled builds using Dash Athena CBI via crontab
- Building with PPC Eclipse, GEF's automated JUnit tests PASS. See bug 253114 for followup problems on another server.
2008-11-26
- Build System Test Cases document started to capture tests, invocations, and problems
- Getting Started Guide drafted
- Tests (for GEF project build) are working with some success locally, but fail with "Java Result: 13" when run on build.eclipse.org
java.lang.UnsatisfiedLinkError: no swt-gtk-3448 or swt-gtk in swt.library.path, java.library.path or the jar file
- More refactoring and simplification (less shell, more Ant)
- System now uses ant-contrib for looping and conditional processing
- bug 256379 opened to optionally clean cvs exports out of the log (using <cvs reallyquiet="true">)
2008-11-19
2008-11-06
- SVN support implemented in setup script (to get svn-pde-plugin into the cached copy of basebuilder), startup script (to get a .releng project from svn repo or cvs repo), and generated fetch scripts (map files can specify cvs and svn entries interchangeably) (bug 251923).
- Signing now works for GEF (bug 253286).
- clean up customTargets.xml (for both ALL and Tests builders) (bug 251879).
- Support for archived update sites and SDK zips as input to build :: in progress (bug 252423).
- Testing code refactored and almost working for GEF.
2008-10-28
- Server config simplified on build.eclipse.org; documented and scripted for reproduction on localhost. Testing through cygwin on Windows TBD.
- basebuilder.releng from both 3.4 (tag: RC2_34) and 3.5M2 (R35_M2) checked out on build.eclipse.org; builds work equally.
- GEF build on build.eclipse.org and localhost has succeeded in producing GEF "Master" zip, as well as SDK, runtime, and examples.
- Signing fails, but can be skipped by running an N build.
- Test assembly fails; test run fails because tests are not yet built. builder/tests/customTargets.xml needs to be updated/fixed.
- Creation of Update site, p2 metadata, site digest, etc. still TBD.
- Web UI to run builds has been decoupled from the build system in case we decide to use Hudson or Cruise Control instead. Paths in web UI still broken (expect /gef/ but have /cbi/gef; downloads page shows builds on download.eclipse.org, not local builds on build.eclipse.org)
--
- To set up a build, run this script.
- For example:
cd /opt/public/cbi/build/org.eclipse.dash.common.releng_HEAD; \ cvs up -Pd; cd tools/scripts;./start.sh -projectid tools.gef -version 3.4.0 -buildType N \ -projRelengRoot ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology' \ -projRelengPath 'org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng' \ -basebuilderBranch R35_M2 -javaHome /opt/public/common/ibm-java2-142 \ -URL http://download.eclipse.org/eclipse/downloads/drops/R-3.4-200806172000/eclipse-SDK-3.4-linux-gtk.tar.gz \ 2>&1 | tee /opt/public/cbi/logs/buildlog_`date +%H%M%S`.txt
- To run locally, see examples in start.sh.