Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Platform-releng/Platform Build Automated"

(Other tasks of the automated build)
(Primary Method)
(6 intermediate revisions by 2 users not shown)
Line 9: Line 9:
 
==== Primary Method ====  
 
==== Primary Method ====  
  
Typically, all a committer has to do is commit and push to the branch that code belongs too, and the automated build will fetch the most recent version of that branch, build it, and tag it. [tagging is currently enabled, but not yet "turned on", as of 2/11/2013].
+
Typically, all a committer has to do is commit and push to the branch that code belongs too, and the automated build will fetch the most recent version of that branch, tag it (with buildId, such as I20130709-0800), and then build it. Note: only I- and M-builds are tagged, not N-builds.  
  
But which branch? The branches that the automated build pulls from, for each repository, are listed in '''repositories.txt''', as before with PDE based builds. But, that is now in a different location. It is in the aggregator repository {{Git|platform|eclipse.platform.releng.aggregator.git}}, in a directory named "streams". For the master (currently Kepler) version, see  
+
But which branch? The branches that the automated build pulls from, for each repository, are listed in '''repositories.txt'''. That file is in the "streams" directory of the aggregator repository. For the master (currently Luna) version, see  
  
  http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/streams/repositories.txt  
+
  http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/streams/repositories.txt
 +
 
 +
For Kepler maintenance, see
 +
 
 +
http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/plain/streams/repositories.txt?h=R4_3_maintenance
  
 
==== Secondary ("Freeze") Method ====
 
==== Secondary ("Freeze") Method ====
  
There are times, such as when nearing a milestone or a release, the project's committers may decide to "freeze" a contribution at a certain point, while they continue to make changes in the branch. At those times, the branch name should be replaced with a specific tag for that repository. You can not, as before "comment out" the repository line. In the PDE case, we essentially still used the maps files, so if repository commented out, we'd still use what was in the map files. In the current system, we still need to know the version to build, since we still need to clone/reset the repository to make sure it is "clean". [Review: Is this paragraph accurate? If so, seems we could enhance in future, and simply use last commit of the submodule to the aggregator itself, assuming the correct version had been built or committed there.]
+
There are times, such as when nearing a milestone or a release, the project's committers may decide to "freeze" a contribution at a certain point, while they continue to make changes in the branch. At those times, the branch name in 'repositories.txt' should be replaced with a specific tag for that repository.
  
 
==== Nightlies (master) versus Integration Builds ====  
 
==== Nightlies (master) versus Integration Builds ====  
  
It is not implemented yet, but eventually, you will have the ability to name an 'integration' branch in repositories.txt, and that will be used for I-builds, but N-builds will always force a pull/reset to 'master' to have an early look at code/fixes. [Initially, nightlies will be "just like" integration builds, except not signed and qualifiers based on current date, not date of last change.]
+
If desired, committers have the ability to name an 'integration' branch in repositories.txt, and that will be used for I-builds, but N-builds will always force a pull/reset to 'master' to have an early look at code/fixes.
  
 
=== Other tasks of the automated build ===  
 
=== Other tasks of the automated build ===  
  
The automated build also creates summary pages of the build, and starts jobs on Hudson to do the Unit testing and then creates summaries of test results, as before. Much of this is controlled by scripts and cron jobs that are in a special directory called "production", and some in resources in "eclipse.platform.releng.tychoeclipsebuilder" which conceptually corresponds to the previously used "eclipse.platform.releng.eclipsebuilder".  
+
The automated build also creates summary pages of the build, and starts jobs on Hudson to do the Unit testing and then creates summaries of test results, as before. Much of this is controlled by scripts and cron jobs that are in a special directory called "production", and some in resources in "eclipse.platform.releng.tychoeclipsebuilder" which conceptually corresponds to the previously used "eclipse.platform.releng.eclipsebuilder". Some of this may be "mavenized" eventually, but for now, bash scripts and ant tasks.
 +
 
 +
=== Routine release engineering tasks for builds ===
 +
 
 +
There are some routine tasks that release engineers have to do, with help of committers, to keep the build "current" (and these usually effect "local, casual builds" as well as production builds). These routine tasks are described in the sections that follow.  
  
 
==== JDT Compiler ====
 
==== JDT Compiler ====
  
Typically, the JDT compiler is updated after each milestone (and then at end, the RC4 version is created with RC3 version of compiler). This is done partially to make sure our code is generated with the best possible compiler, but also to help test our compiler, on our own code.  
+
Typically, the JDT compiler is updated after each milestone and then at end, the RC4 (final release) version is created with RC3 version of compiler (if not in an earlier RC). This is done partially to make sure our code is generated with the best possible compiler, but also to help test our compiler, on our own code.  
  
 
The version of the compiler used is controlled by two elements in the eclipse-platform-parent. Current examples are as follows:  
 
The version of the compiler used is controlled by two elements in the eclipse-platform-parent. Current examples are as follows:  
Line 42: Line 50:
 
# Place the jdt core jar in <code>/shared/eclipse/jdtstaging/</code>
 
# Place the jdt core jar in <code>/shared/eclipse/jdtstaging/</code>
 
# Update the pom.xml file in that directory to refer to the correct version of the bundle (artifact).   
 
# Update the pom.xml file in that directory to refer to the correct version of the bundle (artifact).   
# Run the Hudson job named <code>deployJDTCompiler</code>. That is a parameterized Hudson job which requires the version of the JDT core bundle and the location where the bundle and pom.xml file are located. (Default value should be correct for location, but version has to be update for each new version).  
+
# Run the Hudson job named <code>deployJDTCompiler</code>. That is a parameterized Hudson job which requires the version of the JDT core bundle and the location where the bundle and pom.xml file are located. (Default value should be correct for location, but version has to be update for each new version). [Note: this is a Hudson job that is set up with special security tokens, to make it harder for someone to accidentally (or, maliciously) put something bad in the maven repository, since it is used by ohters.]
  
 
Remember to watch the comparator logs after a compiler update, as some bundles may have to be manually "touched" to force a qualifier change, or else new byte codes, if any, will not be picked up for distribution.  
 
Remember to watch the comparator logs after a compiler update, as some bundles may have to be manually "touched" to force a qualifier change, or else new byte codes, if any, will not be picked up for distribution.  
  
  
==== API Descriptor and JavaDoc Generation ====
+
==== API Description and JavaDoc Generation ====
 
+
Our builds generate ".api_descriptors" and also "JavaDoc" both of which, depend on using a "current" version of Eclipse so the proper functionality is used to do this generation. For example, if PDE ever improves the function provided or the details of .api_descriptors, then the version of Eclipse we use to "run" this part of the build has to be updated. Also, typically, as we near a milestone or especially a release, we'd want to update to "current version" just in case some change was forgotten or overlooked (Similar to how we update the JDT Compiler every milestone).
+
 
+
A. The instance of Eclipse used, as a "helper" for these tasks, is created by the Tycho eclipse-run plugin and makes use of what ever repository is defined in the eclipse-platform-parent. See the <eclipserun-repo> element in the global properties section, and update it, any time there is new API descriptor function or in the few functions used to help create documentation.
+
  
B. Remember, in such cases, where the "build mechanisms" change, some bundles may have to be manually "touched" for them to change qualifiers, and hence pick up the new changes (such as new .api_descriptors) rather than an older version (with same qualifier) being picked by comparator to include in distribution. If someone forgets, though, these "differences" should show up in comparator logs.  
+
The automated platform build generates some extra content.  API descriptions (.api_description files at the root of a bundle) provide information on API restrictions for use with [[PDE/API Tools|PDE API Tools]].  JavaDoc provides documentation for API packages. To build this extra content, the automated build must depend on an existing version of Eclipse.
  
 +
The version of Eclipse used for generation will lag behind the current release.  This means any changes to the generation code will not be reflected in the build until the build script is updated (similar to how we update the JDT compiler every milestone).
  
 +
To update:
 +
# The instance of Eclipse used, as a "helper" for these tasks, is created by the Tycho eclipse-run plugin and makes use of whatever repository is defined in the eclipse-platform-parent. See the <eclipserun-repo> element in the global properties section, and update it, any time there is new API descriptor function or in the few functions used to help create documentation.
 +
# Remember, in such cases, where the "build mechanisms" change, some bundles may have to be manually "touched" for them to change qualifiers, and hence pick up the new changes (such as new .api_descriptors) rather than an older version (with same qualifier) being picked by comparator to include in distribution. If someone forgets, though, these "differences" should show up in comparator logs.
  
 
[[Category:Eclipse_Platform_Releng| ]]
 
[[Category:Eclipse_Platform_Releng| ]]

Revision as of 17:48, 9 July 2013

Purpose

This page is a small addition to Platform-releng/Platform_Build. Be sure you have read that first and are familiar with the concepts (and preferably have done "local builds" as described there).

The CBI-based automated builds for, Platform Releng, essentially automates the steps described in Platform-releng/Platform_Build and does a few additional things meant to be convenient for Platform committers, who are used to the procedures in the PDE based builds. This page gives the briefest overview, and describes what's different. [Note: the scripts/code used to do these things might well someday make their way to some "common tools" repository, but most are "platform specific" and not meant as "API" or to be re-used as-is ... but, naturally, if others find them useful as a starting point are free to make a copy, etc., under EPL constraints.]

Contributing to a build

Primary Method

Typically, all a committer has to do is commit and push to the branch that code belongs too, and the automated build will fetch the most recent version of that branch, tag it (with buildId, such as I20130709-0800), and then build it. Note: only I- and M-builds are tagged, not N-builds.

But which branch? The branches that the automated build pulls from, for each repository, are listed in repositories.txt. That file is in the "streams" directory of the aggregator repository. For the master (currently Luna) version, see

http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/streams/repositories.txt

For Kepler maintenance, see

http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/plain/streams/repositories.txt?h=R4_3_maintenance

Secondary ("Freeze") Method

There are times, such as when nearing a milestone or a release, the project's committers may decide to "freeze" a contribution at a certain point, while they continue to make changes in the branch. At those times, the branch name in 'repositories.txt' should be replaced with a specific tag for that repository.

Nightlies (master) versus Integration Builds

If desired, committers have the ability to name an 'integration' branch in repositories.txt, and that will be used for I-builds, but N-builds will always force a pull/reset to 'master' to have an early look at code/fixes.

Other tasks of the automated build

The automated build also creates summary pages of the build, and starts jobs on Hudson to do the Unit testing and then creates summaries of test results, as before. Much of this is controlled by scripts and cron jobs that are in a special directory called "production", and some in resources in "eclipse.platform.releng.tychoeclipsebuilder" which conceptually corresponds to the previously used "eclipse.platform.releng.eclipsebuilder". Some of this may be "mavenized" eventually, but for now, bash scripts and ant tasks.

Routine release engineering tasks for builds

There are some routine tasks that release engineers have to do, with help of committers, to keep the build "current" (and these usually effect "local, casual builds" as well as production builds). These routine tasks are described in the sections that follow.

JDT Compiler

Typically, the JDT compiler is updated after each milestone and then at end, the RC4 (final release) version is created with RC3 version of compiler (if not in an earlier RC). This is done partially to make sure our code is generated with the best possible compiler, but also to help test our compiler, on our own code.

The version of the compiler used is controlled by two elements in the eclipse-platform-parent. Current examples are as follows:

     <cbi-jdt-repo.url>https://repo.eclipse.org/content/repositories/eclipse-staging/</cbi-jdt-repo.url>
     <cbi-jdt-version>3.9.0.v20130604-1421</cbi-jdt-version>

The URL is pretty much constant, and is a special "maven repository" where we can put unreleased versions of Eclipse bundles (and remove them later, as desired). The method of adding a bundle to that maven repo is outlined as follows:

  1. Place the jdt core jar in /shared/eclipse/jdtstaging/
  2. Update the pom.xml file in that directory to refer to the correct version of the bundle (artifact).
  3. Run the Hudson job named deployJDTCompiler. That is a parameterized Hudson job which requires the version of the JDT core bundle and the location where the bundle and pom.xml file are located. (Default value should be correct for location, but version has to be update for each new version). [Note: this is a Hudson job that is set up with special security tokens, to make it harder for someone to accidentally (or, maliciously) put something bad in the maven repository, since it is used by ohters.]

Remember to watch the comparator logs after a compiler update, as some bundles may have to be manually "touched" to force a qualifier change, or else new byte codes, if any, will not be picked up for distribution.


API Description and JavaDoc Generation

The automated platform build generates some extra content. API descriptions (.api_description files at the root of a bundle) provide information on API restrictions for use with PDE API Tools. JavaDoc provides documentation for API packages. To build this extra content, the automated build must depend on an existing version of Eclipse.

The version of Eclipse used for generation will lag behind the current release. This means any changes to the generation code will not be reflected in the build until the build script is updated (similar to how we update the JDT compiler every milestone).

To update:

  1. The instance of Eclipse used, as a "helper" for these tasks, is created by the Tycho eclipse-run plugin and makes use of whatever repository is defined in the eclipse-platform-parent. See the <eclipserun-repo> element in the global properties section, and update it, any time there is new API descriptor function or in the few functions used to help create documentation.
  2. Remember, in such cases, where the "build mechanisms" change, some bundles may have to be manually "touched" for them to change qualifiers, and hence pick up the new changes (such as new .api_descriptors) rather than an older version (with same qualifier) being picked by comparator to include in distribution. If someone forgets, though, these "differences" should show up in comparator logs.

Back to the top