Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Platform-releng/Platform Build Automated"

(Primary Method)
(Primary Method)
Line 11: Line 11:
 
Typically, all a committer has to do is commit and push to the branch that code belongs too, and the automated build will fetch the most recent version of that branch, tag it (with buildId, such as I20130709-0800), and then build it.  
 
Typically, all a committer has to do is commit and push to the branch that code belongs too, and the automated build will fetch the most recent version of that branch, tag it (with buildId, such as I20130709-0800), and then build it.  
  
But which branch? The branches that the automated build pulls from, for each repository, are listed in '''repositories.txt'''. That file is in the "streams" directory of the aggregator repository. For the master (currently Luna) version, see  
+
But which branch? The branches that the automated build pulls from, for each repository, are listed in '''repositories.txt'''. That file is in the "streams" directory of the aggregator repository. For the master version, see  
  
  http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/streams/repositories.txt
+
  https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/cje-production/streams/repositories_master.txt
  
 
==== Secondary ("Freeze") Method ====
 
==== Secondary ("Freeze") Method ====

Revision as of 04:17, 27 November 2020

Purpose

This page is a small addition to Platform-releng/Platform_Build. Be sure you have read that first and are familiar with the concepts (and preferably have done "local builds" as described there).

The CBI-based automated builds for, Platform Releng, essentially automates the steps described in Platform-releng/Platform_Build and does a few additional things meant to be convenient for Platform committers, who are used to the procedures in the PDE based builds. This page gives the briefest overview, and describes what's different. [Note: the scripts/code used to do these things might well someday make their way to some "common tools" repository, but most are "platform specific" and not meant as "API" or to be re-used as-is ... but, naturally, if others find them useful as a starting point are free to make a copy, etc., under EPL constraints.]

Contributing to a build

Primary Method

Typically, all a committer has to do is commit and push to the branch that code belongs too, and the automated build will fetch the most recent version of that branch, tag it (with buildId, such as I20130709-0800), and then build it.

But which branch? The branches that the automated build pulls from, for each repository, are listed in repositories.txt. That file is in the "streams" directory of the aggregator repository. For the master version, see

https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/cje-production/streams/repositories_master.txt

Secondary ("Freeze") Method

There are times, such as when nearing a milestone or a release, the project's committers may decide to "freeze" a contribution at a certain point, while they continue to make changes in the branch. At those times, the branch name in 'repositories.txt' should be replaced with a specific tag for that repository.

Nightlies (master) versus Integration Builds

If desired, committers have the ability to name an 'integration' branch in repositories.txt, and that will be used for I-builds, but N-builds will always force a pull/reset to 'master' to have an early look at code/fixes.

Tip on making sure qualifiers increment correctly

See Tycho/Reproducible_Version_Qualifiers for Tycho documentation on this topic.

Bundles

Normally, for I-builds, the qualifier of bundles is computed automatically, from the most recent commit to the source tree of that bundle. One exception to this rule is if the only the POM file changes. POM changes are ignored in computing the qualifier. Most of the time, this is what's desired (that is, the POM change doesn't really effect what'd built) ... but, many times, a change in the POM file will effect what is built, in which case the bundle has to be "touched" to force the qualifier to increase. Our project's convention is to use a file named 'forceQualifierUpdate.txt' at the root of bundle. In the event the qualifier needs to be "forced" to increment, it is customary to added a line of text to that file, such as the bug number that lead to it needing to be touched, and committing that to Git. (The file is not otherwise "exported" with the bundle binary or source).

Features

Features are similar to bundles, described above, except it's not only the feature that is considered, but is also all the bundles it contains. It's qualifier is basically determined by the most recent commit of that whole collection. (The same "exclusion of POM files) applies to features too.).

One well known exception is "third party jars" such as from Orbit. Since they don't have a "commit" like the rest of the source tree, they won't cause a qualifier to increase, even if it is a new or different version that is added to a feature. In these cases, the "forceQualifierUpdate.txt" is the mechanism to use to get it to update, if 3rd party bundles change.

There is another, more problematic case that's been mentioned in several places, such as bug 436039 and bug 416619. In short, the qualifier is determined by "commit date" ... nothing like "push date". So it's fairly easy, especially when we do I-build every day, for a feature (or other bundle) to have "the latest commit date" if some other bundle was committed, but not pushed right away. That is, the feature qualifier would have already been increment for "most recent" bundle ... and the new "push" doesn't look any newer, since it's commit is older than what's already being used.

Therefore, especially during critical periods, such as "right before a milestone or release", if you change a bundle, you will want to strongly consider proactively "touching" the feature (or, features, if contained in several) to make sure they are correctly incremented, instead of depending on the automatic computation. Ironically, this problem is worse, when we do frequent I-builds (since that serves as the basis for the comparator "swapping out" items whose qualifier has not changed) since then it is more likely "the most recent" commit has already incremented a feature's qualifier in that repo -- and an older "commit", even though more recently pushed, will not be "picked up" in already qualified feature.

Trigger builds manually

The builds can be triggered manually or scheduled to run at a specified time. The build jobs are available at releng JIPP instance. We use I-build for integration builds. These jobs can be directly triggered after logging in or can be scheduled by modifying the configuration of the specific job.

Syntax of cron consists of 5 fields separated by TAB or whitespace as below:

   MINUTE HOUR DOM MONTH DOW
   MINUTE	Minutes within the hour (0–59)
   HOUR		The hour of the day (0–23)
   DOM		The day of the month (1–31)
   MONTH	The month (1–12)
   DOW		The day of the week (0–7) where 0 and 7 are Sunday, 1 is Monday, 2 is Tuesday, 3 is Wednesday, 4 is Thursday, 5 is Friday and 6 is Saturday

Other tasks of the automated build

The automated build also creates summary pages of the build, and starts jobs on Hudson to do the Unit testing and then creates summaries of test results, as before. Much of this is controlled by scripts and cron jobs that are in a special directory called "production", and some in resources in "eclipse.platform.releng.tychoeclipsebuilder" which conceptually corresponds to the previously used "eclipse.platform.releng.eclipsebuilder". Some of this may be "mavenized" eventually, but for now, bash scripts and ant tasks.

Routine release engineering tasks for builds

There are some routine tasks that release engineers have to do, with help of committers, to keep the build "current" (and these usually effect "local, casual builds" as well as production builds). These routine tasks are described in the sections that follow.

Update JDT Compiler Version

Typically, the JDT compiler is updated after each milestone and then at end, the RC4 (final release) version is created with RC3 version of compiler (if not in an earlier RC). This is done partially to make sure our code is generated with the best possible compiler, but also to help test our compiler, on our own code.

The version of the compiler used is controlled by two elements in the eclipse-platform-parent. Current examples are as follows:

     <cbi-jdt-repo.url>https://repo.eclipse.org/content/repositories/eclipse-staging/</cbi-jdt-repo.url>
     <cbi-jdt-version>3.9.0.v20130604-1421</cbi-jdt-version>

The URL is pretty much constant, and is a special "maven repository" where we can put unreleased versions of Eclipse bundles (and remove them later, as desired). The method of adding a bundle to that maven repo is outlined as follows:

  1. Place the org.eclipse.jdt.core_(version).jar in /shared/eclipse/jdtstaging/
  2. Update the pom.xml file in that directory to refer to the correct version of the bundle (artifact).
  3. Run the Hudson job named deployJDTCompiler. That is a parameterized Hudson job which requires the version of the JDT core bundle and the location where the bundle and pom.xml file are located. (Default value should be correct for location, but version has to be update for each new version). [Note: this is a Hudson job that is set up with special security tokens, to make it harder for someone to accidentally (or, maliciously) put something bad in the maven repository, since it is used by others.]

Remember to watch the comparator logs after a compiler update, as some bundles may have to be manually "touched" to force a qualifier change, or else new byte codes, if any, will not be picked up for distribution.

NOTE: We typically will leave a "milestone compiler" in the staging repo for only 2 or 3 milestones, and then remove it, without announcement. So if anyone else is "following along", and doing frequent updates of the compiler, don't forget you too will need to update every milestone or two.

Once a compiler is part of an official release, it will be moved to Eclipse Foundations maven repository at

https://repo.eclipse.org/content/repositories/releases

(And, by "compiler", is meant the org.eclipse.jdt.core_(version).jar OSGi bundle, which, of course, can be used as a stand-alone compiler (see Eclipse Help for details. Thus far, we've not had any requests to put a mavenized version of ecj.jar there, which is the same thing ... just a little smaller since doesn't contain all the function that is in org.eclipse.jdt.core_(version).jar).

Compiler versions association to releases

The following table maps the released version of JDT core compiler to released version of Eclipse SDK:

 3.9.0.v20121208-144805    Juno SR2 (4.2.2)
 3.9.0.v20130604-1421      Kepler SR0 (4.3)
 3.9.2.v20140114-1555      Kepler SR2 (4.3.2)
 3.10.0.v20140604-1726     Luna SR0 (4.4)
 3.10.0.v20140902-0626     Luna SR1 (4.4.1)
 3.10.2.v20150120-1634     Luna SR2 (4.4.2)
 3.11.0.v20150602-1242     Mars (4.5)
 3.11.1.v20150902-1521     Mars.1 (4.5.1)
 3.11.2.v20160128-0629     Mars.2 (4.5.2)
 3.12.0.v20160516-2131     Neon (4.6)
 3.12.1.v20160829-0950     Neon.1 (4.6.1)
 3.12.2.v20161117-1814     Neon.2 (4.6.2)
 3.12.3.v20170228-1205     Neon.3 (4.6.3)
 3.13.0.v20170516-1929     Oxygen (4.7)

During Mars, we also learned that org.eclipse.jdt.compiler.apt should also be updated, to keep "in sync" with core compiler (prior to that, we just used what ever shipped with Tycho release).

The following table maps the released version of APT to the released version of Eclipse SDK:

 1.2.0.v20150514-0146      Mars (4.5)
 1.2.0.v20150514-0146      Mars.1 (4.5.1)
 1.2.0.v20150514-0146      Mars.2 (4.5.2)
 1.2.100.v20160418-1457    Neon (4.6)
 1.2.100.v20160418-1457    Neon.1 (4.6.1)
 1.2.100.v20160418-1457    Neon.2 (4.6.2)
 1.2.100.v20160418-1457    Neon.3 (4.6.3)
 1.3.0.v20170502-0408      Oxygen (4.7)

Update Eclipse version for eclipserun-plugin (API Description and JavaDoc Generation)

The automated platform build generates some extra content. API descriptions (.api_description files at the root of a bundle) provide information on API restrictions for use with PDE API Tools. JavaDoc provides documentation for API packages. To build this extra content, the automated build must depend on an existing version of Eclipse as used by the tycho-eclipserun-plugin.

The version of Eclipse used will lag behind the current release. This means any changes to the generation code will not be reflected in the build until the build script is updated (similar to how we update the JDT compiler every milestone).

To update:

  1. The instance of Eclipse used, as a "helper" for these tasks, is created by the Tycho eclipse-run plugin and makes use of whatever repository is defined in the eclipse-platform-parent. See the <eclipserun-repo> element in the global properties section, and update it, any time there is new API descriptor function or in the few functions used to help create documentation.
  2. Remember, in such cases, where the "build mechanisms" change, some bundles may have to be manually "touched" for them to change qualifiers, and hence pick up the new changes (such as new .api_descriptors) rather than an older version (with same qualifier) being picked by comparator to include in distribution. If someone forgets, though, these "differences" should show up in comparator logs.

Build Notes

Build notes are used to provide information that is relevant to a specific build or builds over a relatively short period of time (otherwise, the notes should go into the "help" or "readme" files). For example, perhaps there is a known problem with a known workaround, or similar.

If the issue is expected to last several builds, committers can include build notes for their component, by adding a file to the root of a bundle's source tree. The file should be named according to the pattern: 'buildnotes_<component-name>.html'. The automated publishing scripts then gathers these files together and puts them in a directory named "buildnotes" at the top of the drop directory. Note, these buildnotes*.html files do not need to be (and probably should not be) "exported" to the plugin, they are gathered directly from the git clone area on the build machine. Also note, unlike in the past, the files must end with ".html" (not ".htm"). Also unlike the distant past, no "*.css" files are gathered. If some custom display is needed, committers should work with releng to have added to our standard templates or wrappers, for consistency. The build notes can and should contain ordinary HTML tags though, such as paragraphs (<p>) or links (<a>).

If an issue is discovered after a build is published, then someone with suitable permissions and access (such as, releng) can copy the HTML build notes in place in the drop (build) directory. The directory 'buildnotes' will have to be created first, if it does not already exist.

Other releng tasks

See also Platform-releng/How_to_do_miscellaneous_releng_tasks for other, miscellaneous releng tasks.

Back to the top