Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Platform-releng-faq-obsolete"

m
Line 225: Line 225:
  
 
<h4>Where can I learn how to use the packager?</h4>
 
<h4>Where can I learn how to use the packager?</h4>
http://dev.eclipse.org/viewcvs/index.cgi/pde-build-home/doc/packager/doc.html?rev=1.3
+
http://wiki.eclipse.org/index.php/Platform-releng-packager
 
+
  
 
<h4>How can I run the update manager from a command line to mirror a remote site?</h4>  
 
<h4>How can I run the update manager from a command line to mirror a remote site?</h4>  

Revision as of 13:55, 7 June 2006

Contents

Eclipse Platform Release Engineering FAQ

If you would like additional content added to this FAQ, please send a note to the platform releng developers mailing list

What hardware comprises the platform-releng build infrastructure?

The eclipse platform build infrastructure consists of

  1. 2 linux build machines (2GHz, 1GB and PowerMac G5, 2GB )
  2. 2 junit test machines, one windows (2GHz, 1 GB) one linux (2.66 GHz, 1.2 GB )
  3. 4 performance test machines - one set of slower machines (one windows, one linux 2GHz, 512 MB) and one set of fast machines (one windows, one linux 3GHz, 2GB)
  4. 1 linux cvs test server(500MHz, 500MB)
  5. 1 database server with apache derby installed to store performance test results (2.5GHz, 500MB)


Is the Eclipse platform build process completely automated?

Yes, the Eclipse build process starts with a cron job on our main build machine that initiates a shell script that checks out the builder projects.

  • org.eclipse.releng.eclipsebuilder -> scripts to build each type of drop
  • org.eclipse.releng.basebuilder -> a subset of eclipse plugins required to run the build.
  • we also use several small cvs projects on an internal server that store login credentials, publishing information and jdks

  • Fetch, build and assemble scripts are generated automatically the org.eclipse.pde.build plugin in org.eclipse.releng.basebuilder. Fetch scripts specify the version of code to retrieve from the repository. Build scripts are also generated automatically to compile all java code, determine compile order and manage dependancies. Assembly scripts are generated to assemble the compiled code into it's final form, tar.gz, zip etc. We also use custom build scripts that you can see in org.eclipse.releng.eclipsebuilder.

    After the build drops are built, the automated junit and performance testing begins. Tests occur over ssh for Linux machines, and rsh for Windows machines. Each component team contributes it's own tests. Once the tests are completed,the results are copied back to the build machine. Also, the images for the performance tests are generated.


    What's the difference between an integration and nightly build?

    With integration builds, we specify a version of the plugin to retrieve in the map files. (org.eclipse.releng/maps). With nightly builds, we retrieve the plugins specified in the map files from the HEAD stream. The types of builds we run are described here http://download.eclipse.org/eclipse/downloads/build_types.html.


    How do I run a build using the scripts in org.eclipse.releng.eclipsebuilder ?

    This document provides an overview of how to build eclipse components using the releng scripts.


    How do I automate my builds with PDE Build?

    This document describes how to automate builds using PDE Build.


    How do I contribute a JUnit Plugin-in Test to the build?

    Contributing JUnit Plug-in Tests

    How do I contribute an example plugin to the build?

    Once you have your plugin ready, install the org.eclipse.releng.tools plug-in, and use the "Release" action in the context menu of the test plug-in project to update and commit the map file.

    Send a request to platform-releng-dev@eclipse.org to add the test plugins to the build process. Include the following information:

  • the plug-in ids
  • The Eclipse release engineering team will update the org.eclipse.sdk.examples-feature to include the new tests. They will also update org.eclipse.releng.eclipsebuilder/all/publishing/templateFiles/testManifest.xml to indicate the zip that should get a red X if there are compile errors associated with that plugin on the build page.

    I would like to recompile eclipse on a platform that is not currently supported. What is the best approach to this? How can I ensure that the community can use my work?

    The best approach is use the source build drop and modify the scripts to support that you are interested in. Then open a bug https://bugs.eclipse.org with product platform and component releng attaching the patches to the scripts that were required to build the new drop. If you are interested in contributing a port to eclipse, here is the procedure.


    How long does the build take to complete?

    It takes two hours for all the drops to be produced. Junit (four hours) and performance tests (eight hours) run in parallel after the drops have been built. It takes a further two hours for the performance results to be generated. We will reduce the amount of time that the build takes in the 3.2 timeframe by implementing faster hardware and more parallel ant tasks.


    When is the next build?

    Please refer to the build schedule http://www.eclipse.org/eclipse/platform-releng/buildSchedule.html


    When is the next milestone?

    Please refer to the Eclipse Platform Project Plan http://www.eclipse.org/eclipse/development/eclipse_project_plan_3_2.html


    I noticed that you promoted an integration build to a milestone build. The integration build had test failures but the milestone builds doesn't show any. Why is this?

    See https://bugs.eclipse.org/bugs/show_bug.cgi?id=134413

    We have a number of tests that intermittently fail. The reasons are

  • issues with the tests themselves
  • tests subject to timing issues
  • tests that fail intermittently due to various conditions
  • The component teams are always trying to fix their tests but unfortunately they are still some issues. When we promote a build to a milestone, we rerun the tests that failed. Many pass on the second time because the tests initially failed due to a timing issue or intermittent condition. Or a team will have a broken test that doesn't warrant a rebuild for a milestone. In that case, the releng team sprinkles pixie dust over the build page to erase the red Xes, but leaves the appropriate build failures intact.

    I'd like to run a build with the version of the code that was used for build X? How do I know what versions of code in the repository were used?

    There are several ways to approach this issue.

    • Look at the directory.txt linked from the top of every build page. The directory.txt concatenates all the map files used in a build into a single file. Please note that every nightly build runs from HEAD. Therefore it is impossible to replicate a nightly build.
    • In addition, with every maintenance and integration build, the org.eclipse.releng project, which contains the map files for each project, is tagged with a version corresponding to that build. For instance, the version I20051018-0932 corresponds to the integration build of the same name. Please note that the builder projects, (org.eclipse.releng.eclipsebuilder and org.eclipse.releng.basebuilder) are also tagged during an integration build with the corresponding build name, but are not reflected in the directory.txt because they are not included in the eclipse distribution itself.
    • Finally, after each release the releng team tags all projects from the map files used in the release. So, all the projects used to construct version 3.1.1 are tagged R_3_1_1.


    I'm looking for an older build but can't find them on the download page?

    Check out archive site the http://archive.eclipse.org/eclipse/downloads for vintage builds.



    I'm a platform committer and have made major changes today. I'd like to run a test build to ensure that my changes don't break the build. Can I request a test build from a web page?

    No, you cannot request a eclipse test build from a web page. Our build takes too long and much of the time the builds machines are busy. Please send an email to the releng team requesting a test build and explaining the changes you have made. If you would like an expediated test build, please provide chocolate.


    How do I set up performance tests?

    Refer to the performance tests how-to.


    How do I run the performance tests from the zips provided on the download page?

    Check out this https://bugs.eclipse.org/bugs/show_bug.cgi?id=91229#c3 bug


    What do I need to do to use org.eclipse.releng.basebuilder from HEAD now that the Xerces plugin has been removed?

    If you use the org.eclipse.releng.basebuilder from HEAD in your builds and you use TestVersionTracker to create a test.properties file (search for "TestVersionTracker" in your customTargets.xml build scripts), please read on.

    Some older versions of the TestVersionTracker class required xerces on the classpath. We have removed the xerces jars from the HEAD stream only of org.eclipse.releng.basebuilder.

    For any team broken by this, a <generateTestProperties> custom Ant task has been created which does the same. It is available in the org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools plug-in from HEAD. To use all you will need to do is the following replacement:

    Replace text such as this (taken from GEF):


    <property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter"/><br><javac verbose="true" failonerror="true" srcdir="${builderDirectory}/tools" destdir="${builderDirectory}/tools" 
      classpath="${eclipse.home}/plugins/org.apache.xerces_4.0.13/xercesImpl.jar:${eclipse.home}/plugins/org.apache.xerces_4.0.13/xmlParserAPIs.jar"/><br><java classname="TestVersionTracker" >
    
    <arg line="${workingDirectory}/eclipse/features/org.eclipse.gef.test_3.1.0/feature.xml
    ${buildDirectory} ${workingDirectory}/gef-testing/test.properties" /> 
    <classpath>
    <pathelement path="${eclipse.home}/plugins/org.apache.xerces_4.0.13/xercesImpl.jar:${eclipse.home}/plugins/org.apache.xerces_4.0.13/xmlParserAPIs.jar:${builderDirectory}/tools" />
    </classpath><br></java>

    with this:

    <generateTestProperties
    buildDirectory="${buildDirectory}"
    
    featureId="org.eclipse.sdk.tests"
    outputFile="${workingDirectory}/eclipse-testing/test.properties"
    />


    How do I update from 3.1 to 3.1.1 or 3.1.2 using update manager?

    Refer to this document http://www.eclipse.org/eclipse/platform-releng/updatesfor3.1.1.html


    How can I use update manager to move to the latest integration build?

    If you have at least 3.2M5 installed, you can update between integration builds by adding the following as a remote update site.

    http://download.eclipse.org/eclipse/testUpdates

    This site provides the updates jars for integration builds as of April 12, 2006. Stale builds will be removed occasionally.

    Why should I package plugins and features as jars?

    See the Core team's document Running Eclipse from JARs http://www.eclipse.org/eclipse/platform-core/documents/3.1/run_from_jars.html

    Why should I use qualifiers?

    See the Core team's document Recommendation to version plug-ins http://www.eclipse.org/equinox/documents/plugin-versioning.html

    How do I incorporate qualifiers into the build process?

    Update your plugins and features to use qualifiers as described in the previous FAQ entry.

    Additional steps that the platform releng team uses to incorporate qualifiers into the build process

    In our bootstrap script that kicks off the build, versionQualifier set to an empty string by default

    #versionQualifier - qualifier to use in plug-ins and features where specified
    versionQualifier=""

    For nightly builds
    if [ "$buildType" = "N" ]
    then

           versionQualifier="-DforceContextQualifier=$buildId"

    fi

    For nightly builds, we set the qualifier to the buildId. For integration builds, pde build sets it to the tag in the map file for plugins, and generates feature qualifiers.

    The versionQualifier property is passed to the antRunner in our bootstrap script

    $antRunner -buildfile all.xml $mail $testBuild $compareMaps -DbuildingOSGi=true -DmapVersionTag=$mapVersionTag -DbuildDirectory=$buildDirectory -DpostingDirectory=$postingDirectory -Dbootclasspath=$bootclasspath -DbuildType=$buildType -D$buildType=true -DbuildId=$buildId -Dbuildid=$buildId -DbuildLabel=$buildLabel -Dtimestamp=$timestamp -DmapCvsRoot=:ext:someuser@dev.eclipse.org:/cvsroot/eclipse -Dzipargs=-y $skipPerf $skipTest $tag $tagMaps -DjavacDebugInfo=true -DjavacSource=1.3 -DjavacTarget=1.2 -DcompilerArg="-enableJavadoc -encoding ISO-8859-1" $versionQualifier -DJ2SE-1.5=$bootclasspath_15 -listener org.eclipse.releng.build.listeners.EclipseBuildListener

    Where can I learn how to use the packager?

    http://wiki.eclipse.org/index.php/Platform-releng-packager

    How can I run the update manager from a command line to mirror a remote site?

    See "Running update manager from command line" in Eclipse Help for the lastest information. I used this to generate the update site for the 3.2 milestones. It copies the appropriate versions of features and associated plugins you specify and generates the site.xml. For instance, if you wanted to create an update site for the platform.

    /home/kmoir/IBMJava2-142/jre/bin/java -cp startup.jar org.eclipse.core.launcher.Main -application org.eclipse.update.core.standaloneUpdate -command mirror -from http://eclipsebuildserv.ottawa.ibm.com/testupdates -to /home/kmoir/update -featureId org.eclipse.platform -version 3.2.0.v20060525a-_OZ2g6CPhlzUdGe -ignoreMissingPlugins true

    I specify ignoreMissingPlugins as true because we have several plugins in our platform feature which are not contributed as part of the build. For instance, the org.eclipse.swt.solaris.gtk.x86 fragment is built by the a team that has access to the hardware.

    Running update manager from the command line will not copy packed jars if they exist. To generate the packed jars in the newly mirrored update site from the conditioned jars, run the following command with a java 1.5 vm

    /home/kmoir/jdk1.5.0_06/jre/bin/java -jar startup.jar -application org.eclipse.update.core.siteOptimizer -jarProcessor -outputDir /home/kmoir/update -processAll -pack /home/kmoir/update

    Then generate the digest

    /home/kmoir/IBMJava2-142/jre/bin/java -jar /home/kmoir/eclipse3.2rc6/eclipse/startup.jar -application org.eclipse.update.core.siteOptimizer -digestBuilder -digestOutputDir=/home/kmoir/update -siteXML=/home/kmoir/update/site.xml

    Ensure that your site.xml specifies that it uses pack200 and the specifies the location of the digest...for instance

    <site pack200="true" digestURL="http://download.eclipse.org/testUpdates/" mirrorsURL="http://www.eclipse.org/downloads/download.php?file=/eclipse/testUpdates&format=xml">

    I would like a plaform releng committer to comment on a bug assigned to another component. What mailbox do I add to the cc list in bugzilla?

    Please cc the generic mail alias platform-releng-inbox@eclipse.org. Please don't just cc your favourite release engineer using their individual email address. They might be on a beach somewhere enjoying a sunny vacation and therefore unable to comment.

    Eclipsecon 2006 presentation

    http://www.eclipsecon.org/2006/Sub.do?id=254

    Best releng practices from our Eclipsecon 2005 poster

    1. Automate the build process from end-to-end and automate early.
    2. Use PDE Build in Eclipse to generate build scripts.
    3. Automate JUnit testing and performance monitoring.
    4. Automate build notifications (email)
    5. Use gentle humiliation to encourage developers to contribute carefully. (Clown nose technique).
    6. Ensure stability in the build process by thorough testing of builder changes.
    7. Schedule builds at regular intervals and enforce rebuild policies.
    8. Use map files in a central cvs repository.
    9. Build on Linux.
    10. Have fun!



    Useful reference documents

    Build and Test Automation for plug-ins and features http://eclipse.org/articles/Article-PDE-Automation/automation.html

    Eclipse's culture of shipping http://www.artima.com/lejava/articles/eclipse_culture.html

    The Eclipse Way: Proccesses that Adapt http://eclipsecon.org/2005/presentations/econ2005-eclipse-way.pdf

    Who are the platform-releng committers?

    The platform releng team consists of Sonia Dimitrov and Kim Moir.

    As the holiday season approaches, what gifts are appropriate for your favourite release engineer?

    We enjoy fine chocolate, Shaan, and apple juice.


    What does the platform-releng team do when they're not running the build farm and wrangling bugs?

    We herd cats. s/eds/ibm/g

    http://video.google.com/videoplay?docid=4057591681481453187

    Back to the top