Skip to main content
Jump to: navigation, search

Difference between revisions of "CBI"

(37 intermediate revisions by 4 users not shown)
Line 1: Line 1:
The Eclipse Common Build Infrastructure (CBI) is an initiative combining infrastructure, technologies and practices for building Eclipse Software.
+
The Eclipse Common Build Infrastructure (CBI) is an initiative combining infrastructure, services, technologies and best practices for building, testing and delivering software at the Eclipse Foundation.
  
==What is CBI==
+
= Goals =
 +
Primary goals of the CBI initiative are:
 +
* Make it easy for anyone to contribute to Eclipse projects.
 +
* Make it easy for projects to follow open governance and transparency best practices.
 +
* Provide a set of industry-grade services (or integration with some widely adopted 3rd party services) for building and distributing Eclipse projects.
  
The core of CBI today is Maven with the Tycho plugins. The Tycho plugins teach Maven how to build (and consume) Eclipse plugins and OSGi bundles. This enables building Eclipse projects with "maven clean install" just as one would build other Maven projects.
+
= Asking for Help =
  
Common services such as the Jar signing facility, MacOS signing facility, and Windows signing facility are also included with CBI. Other tools and services may be included in the future as the need arises.
+
* Need help actually building your code: ask your project mentors, or ask on the Common Build mailing list (cbi-dev). There are no dumb questions.
 +
* Subscribe to cbi-dev here: https://dev.eclipse.org/mailman/listinfo/cbi-dev
  
Over time mature templates and common pom.xml files will be provided that set common values finely honed with experience.
+
= Service Level Agreement (SLA) =
  
One might go so far as to include Git, Jenkins, the build slaves, and Nexus (aka. the artifact repository & server side of Maven) as part of CBI since they are also common and crucial to builds.
+
Most CBI services are ''Tier 2 - Best Effort'', which means they are expected to be available at all times, and rapid restoration can be expected in the event of an outage. Eclipse Strategic Members can contact Webmaster in certain cases of off-hours support.
  
Gerrit, Bugzilla, and the Downloads site are closely related. Some might consider them part of CBI as well.
+
Please see [[IT_SLA]] for more information on the Eclipse Foundation IT Services SLA.
  
==Initiative Goals==
+
= Provided Services =
Primary goals are:
+
* Make it really easy to contribute Eclipse projects
+
** Make it really easy to copy & modify source
+
** Make it really easy to build
+
** Make it really easy to test
+
** Make it really easy to post a change for review
+
** Make it really easy to sign software
+
  
Secondary goals are:
+
== CI Environment (Jenkins) ==
* Get all Eclipse projects building their software on Eclipse Foundation hardware.
+
* Make it easy for people to build custom Eclipse distributions.
+
  
==Preferred Build Technologies==
+
We provide dedicated Jenkins instances to projects. See also [[Jenkins]].
  
===Jenkins===
+
Jenkins is a continuous integration (CI) server used on Eclipse Foundation servers for Eclipse projects as part of the [[CBI|Common Build Infrastructure (CBI)]]. Jenkins instances are maintained by the Eclipse Webmasters/Release Engineers. List of Jenkins Instances Per Project (JIPP):
  
* [http://ci.eclipse.org The list of Jenkins instances at Eclipse]
+
* https://ci.eclipse.org/ (standalone / Kubernetes cluster)
* [[Jenkins]]
+
  
===Maven===
+
NOTE: JIPP instances are being migrated from a standalone implementation to a Kubernetes cluster implementation.
  
Maven 3.0 drives the builds. Projects are expected to provide standard Maven 3.0 POM files for their builds. The builds should be built in such a way that they can be run on the local workstation, or on the Eclipse build server. Note that builds can only be signed on the Eclipse build server.
+
=== Requesting a JIPP instance ===
  
* [[Maven/Parent POM|Parent Maven POM]] for Eclipse projects;
+
Please [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=CI-Jenkins&short_desc=JIPP%20Request%20for%20project%20XXX file a bug] against Eclipse Foundation > Community > CI-Jenkins to request your project's own instance. Please ensure your project lead can +1 the request. Please specify if you wish to grant write access to your download area and/or code repositories.
* [[Minerva#Signing|Signing Builds]] using Maven; and
+
* [[Maven|Maven repository support at Eclipse]].
+
  
===Tycho===
+
{{important|Security issues| There may be security issues related to using the Gerrit plugin, more especially when allowing the CI system to write directly to the code repos and downloads area. If you request plugins other than those installed by default, webmasters may not be able to help troubleshoot any issues that you may encounter with your instance.}}
  
Tycho is focused on a Maven-centric, manifest-first approach to building Eclipse plug-ins, features, update sites, RCP applications and OSGi bundles.
+
=== What's provided? ===
  
Helpful links:
+
Each Eclipse Project has access to one Jenkins instance (JIPP), including the following:
  
* [[Tycho|Tycho project]] information, including [[Tycho/Demo Projects|demo projects]]; and
+
* (1) Jenkins instance, with (1) Resources Pack (see below)
* [http://waynebeaton.wordpress.com/2010/09/23/building-woolsey-with-maven-and-tycho/ Building Woolsey with Maven and Tycho]
+
** Membership-sponsored projects may allocate more resources (see below)
* [[Tycho/Reference_Card|Reference Card]]
+
* Digital signing Service: Java JAR, Java Cryptography Extensions, Windows Portable Executable with Microsoft Authenticode,  macOS application bundles.
* [[Tycho/Packaging_Types|Packaging Types]]
+
* Packaging service: Apple Disk Image (.dmg), Linux Flatpak
 +
* Disk space: Ephemeral for builds, permanent for release builds.
 +
* (1) 1vCPU/3.75G/30G Linux Virtual Server (if needed) (courtesy of Microsoft Azure). Projects sponsored by Strategic Members can engage with the Foundation to get out of spec Virtual Server.
 +
* Access to worldwide download mirrors
  
=== p2 Repo checks ===
+
=== Additional Resource Packs ===
  
It's highly recommended that any Eclipse.org project runs frequently, and maybe even systematically, the [[CBI/p2repoAnalyzers/Repo Reports|p2 repo analyzer]] to make sure it conforms to some requirements of being a nice citizen in the Eclipse.org world.
+
Each Eclipse Project has access to one Resources Pack for building. For some projects, that may not be enough. Projects sponsored by [https://www.eclipse.org/membership/exploreMembership.php Eclipse Membership] (via Project Lead) have additional Packs, based on [https://www.eclipse.org/membership/become_a_member/membershipTypes.php membership level]. These Packs can be allocated to projects. See the [[Jenkins#About_resource_packs_and_quotas|Jenkins wiki page]] to see how packs translate to Jenkins builds.
  
===Nexus===
+
* Some resources are only available to [https://www.eclipse.org/membership/become_a_member/membershipTypes.php Enterprise and Strategic members].
 +
* Enterprise and Strategic members can engage with the Foundation to acquire additional Packs.
  
[[Services/Nexus]]
+
{| class="wikitable" style="float: left; width:48%"
 +
|+ style="text-align: center;caption-side:top; color:#e76700;font-size:150%"|''Resource Pack''
 +
|-
 +
! scope="row"| Agent type
 +
| style="text-align: center" | Linux (containerized, no root)
 +
|-
 +
! scope="row"| vCPU
 +
| style="text-align: center" | 2 (burst to 4)
 +
|-
 +
! scope="row"| RAM
 +
| style="text-align: center" | 8GiB
 +
|-
 +
! scope="row"| Disk
 +
| style="text-align: center" | 50GB
 +
|-
 +
! scope="row"| External slave support
 +
| style="text-align: center" | Yes
 +
|}
  
===Signing tool===
 
  
* [http://git.eclipse.org/c/cbi/org.eclipse.cbi.git/tree/maven-plugins/README.md Maven plugins for signing artifacts]
+
{| class="wikitable" style="float: right; width:48%"
* [[IT_Infrastructure_Doc#Sign_my_plugins.2FZIP_files.3F|On demand signing tool]]
+
|+ style="text-align: center;caption-side:top; color:#5E97F6;font-size:150%"|''Dedicated Agent''
 +
|-
 +
! scope="row"| Agent type
 +
| style="text-align: center" | Linux/Windows/macOS (VMs)
 +
|-
 +
! scope="row"| vCPU
 +
| style="text-align: center" | 4
 +
|-
 +
! scope="row"| RAM
 +
| style="text-align: center" | 8GiB
 +
|-
 +
! scope="row"| Disk
 +
| style="text-align: center" | 100GB
 +
|}
  
== Deliverables ==
+
==== Resource Packs Included in Membership ====
  
Additionally to recommendation and infrastructure, the CBI also produces pieces of software that are meant to be commonly used by all Eclipse.org projects.
+
Eclipse Foundation Member Organizations have access to Resource Packs based on their [https://www.eclipse.org/membership/become_a_member/membershipTypes.php membership level].
  
===CBI License bundle===
+
{| class="wikitable"
 +
|+ style="text-align: center;caption-side:top; color:#239F60;font-size:150%"|''Membership Level and Resource Packs''
 +
|-
 +
! scope="col" style="background-color: #fff; border-top: 1px solid white;border-left: 1px solid white;border-bottom: 1px solid white"|
 +
! scope="col"| Not a member
 +
! scope="col" colspan="2"| Solution
 +
! scope="col"| Enterprise
 +
! scope="col" colspan="3"| Strategic
 +
|-
 +
! scope="row" style="background-color: #fff; border-top: 1px solid white;border-left: 1px solid white"|
 +
| style="text-align: center" | —
 +
| style="text-align: center" | [$0, $15k)
 +
| style="text-align: center" | [$15k, $20k]
 +
| style="text-align: center" | [$50k, $100k]
 +
| style="text-align: center" | [$25k, $50k)
 +
| style="text-align: center" | [$50k, $100k)
 +
| style="text-align: center" | [$100k, $500k]
 +
|-
 +
! scope="row"| Resource packs
 +
| style="text-align: center" | 1
 +
| style="text-align: center" | 1
 +
| style="text-align: center" | 2
 +
| style="text-align: center" | 4
 +
| style="text-align: center" | 3
 +
| style="text-align: center" | 5
 +
| style="text-align: center" | 10
 +
|-
 +
! scope="row" | Dedicated Agents
 +
| style="text-align: center" | N/A
 +
| style="text-align: center" | N/A
 +
| style="text-align: center" | N/A
 +
| style="text-align: center" | 0
 +
| style="text-align: center" | 0
 +
| style="text-align: center" | 0
 +
| style="text-align: center" | 2
 +
|}
  
We offer a P2 repository containing the org.eclipse.license bundle which is located at:
+
{{important|Windows and macOS agents|Note that Windows and macOS agents (shared and headless - not suitable to run UI tests) can be added to your Jenkins instance. In this case, each shared headless agent count for 1 resource packs. These agents have common dependencies like JDK, Maven, etc installed. If your build requires specific/special dependencies you might need to setup your own external build agent instead.}}
  
    http://download.eclipse.org/cbi/updates/license/
+
==== Assigning Resource Packs to a Project ====
  
This URL is a composite P2 repo containing the license bundle.
+
Resource Packs are assigned by Member organizations of the Eclipse Foundation to Eclipse Projects they sponsor. Packs are assigned as a whole to a single project (i.e., can’t split Packs across multiple projects). A member can assign several packs to a single project.
  
 +
{{note|Important| When asking for packs for your project, please ensure that project leads and your organization representatives are copied to the bugzilla ticket. We require approval from project leads but assume immediate approval from organization representatives. We strongly advise you seek authorization internally to your organization before opening such a request though. Should conflictual requests arise, the organization representatives will be asked to actively arbitrate.}}
  
If you are using Tycho you can add the p2 repo to the <repositories> section of your pom.xml file. Something similar to this:
+
To assign a pack to a project, please [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=CI-Jenkins&short_desc=Assign%20resource%20pack%20to%20project%20NAME_OF_THE_PROJECT file a Bug].
  
<source lang="xml">
+
''By default, resource packs are assigned build agents. In some cases, it may be required to scale up the Jenkins master. In such case, we can allocate resource packs to the master instance.''
    <repository>
+
      <id>license-feature</id>
+
      <url>http://download.eclipse.org/cbi/updates/license/</url>
+
      <layout>p2</layout>
+
    </repository>
+
</source>
+
  
In any particular feature which you need the license you can use the usual feature.xml section:
+
==== Sponsored Projects ====
  
<source lang="xml">
+
We maintain a public [https://docs.google.com/spreadsheets/d/e/2PACX-1vQGhAH2b-BdoW5MAgmCG8NLfAjZMe7fIgsYNtUMREmIeAXmAPjDkZOx1OHaz0-INS51TGN41C3zcbI8/pubhtml?gid=832654757&single=true list of sponsored projects]. We also maintain a list of the [https://docs.google.com/spreadsheets/d/e/2PACX-1vQGhAH2b-BdoW5MAgmCG8NLfAjZMe7fIgsYNtUMREmIeAXmAPjDkZOx1OHaz0-INS51TGN41C3zcbI8/pubhtml?gid=746314272&single=true Member Organizations benefits]. On this last spreadsheet, you can also check how many Resource Packs each organization have left for project sponsoring.
<?xml version="1.0" encoding="UTF-8"?>
+
<feature
+
      id="org.eclipse.help"
+
      label="%featureName"
+
      version="2.0.0.qualifier"
+
      provider-name="%providerName"
+
      plugin="org.eclipse.help.base"
+
      license-feature="org.eclipse.license"
+
      license-feature-version="1.0.0.qualifier"/>
+
....
+
</source>
+
  
===Signing tool===
+
== CI Environment (Third-party) ==
  
* [https://git.eclipse.org/c/cbi/org.eclipse.cbi.git/tree/maven-plugins/README.md Maven plugins for signing artifacts]
+
If you host your Eclipse project git repository at Github, we also support some third-party services:
* [[IT_Infrastructure_Doc#Sign_my_plugins.2FZIP_files.3F|On demand signing tool]]
+
* ''TravisCI''. [https://travis-ci.com/ TravisCI] is free to use for open source projects, but you should be aware that it limits to 5 the number of executors per organizations. It means that only 5 builds can run concurrently in any organization. If your repositories are part of the [https://github.com/eclipse/ Eclipse] organization, you may face long delays between the trigger and the actually start of the job. If it's your case, and would like to stick to TravisCI for building, we suggest you ask to move to a [https://wiki.eclipse.org/GitHub#Project_dedicated_organization dedicated organization].
 +
* ''CircleCI''. https://circleci.com/
 +
* ''Azure Pipelines''. https://azure.microsoft.com/en-us/services/devops/pipelines/
  
=== p2 repo checks ===
+
If an environment is not listed in the unsupported list, you can ask whether it can be supported by [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=CI-Jenkins opening a bug].
  
A set of "tests" which create reports or can be ran as unit tests that check to correctness of p2 repositories. That is partially just "correctness" in general (such as, that jars are signed, etc.) but more so that repositories conform to the requirements of the Eclipse Simultaneous release (such as, that jars have correct "Provider names", licenses, etc.). For more information, see See [[CBI/p2repoAnalyzers/Repo Reports]].
+
{{important|Third party build services (e.g., Travis) limitations|The Eclipse Foundation's IT Team is responsible for the credentials (OSSRH, SSH and GPG keys...) it creates for projects. Keeping track of where credentials are disseminated is crucial for security reasons. As long as we have no automation in place to set/update/remove credentials on 3rd party build services, we don't put any credential on such services. For instance, it means that you can't use them to publish to Maven Central, or to upload build results to download.eclipse.org.}}
  
=== p2 repo aggregator ===
+
== GitHub and Bot Accounts ==  
  
A tool to combine several p2 repositories. Among other things, it makes sure they all have consistent constraints (that is, can be "installed together") unlike a raw p2 mirror task. For more information see [[CBI/aggregator/manual]].
+
The Eclipse Foundation can create a bot user that can be used from your Jenkins (JIPP) instance to build branches and pull requests, push, tag, and comment on the repo.
  
==Resources==
+
We don't share the credentials of those bots for external usage (e.g., to use it with a private, behind firewall Jenkins instance, or with third-party services). Those bots have the same set of permissions as a committer and it would be against the Eclipse Development Process to give those permissions to anybody. Likewise, we don't grant any elevated permissions on repositories or organizations to any other bot account.
* [https://dev.eclipse.org/mailman/listinfo/cbi-dev cbi-dev]
+
* See your [[CBI/FAQ|Frequently Asked Question list]]
+
  
=Eclipse CBI (JIPP)=
+
=== Webhooks ===
Projects hosted at Eclipse.org can request one hosted instance of CBI. Details are provided on the [[Jenkins]] page. There's a [https://ci.eclipse.org/ list of projects] currently building with CBI available.
+
  
== Tools (and locations) ==
+
However, we can setup webhooks for you. We create webhooks that can ''only'' receive the following events:
Build tools like JDK, Maven, Ant and Gradle are already configured in every Jenkins instance.  
+
  
* JDK
+
* Branch or tag creation (Branch or tag created)
** jdk10-latest (/shared/common/java/oracle/jdk-10_x64-latest)
+
* Branch or tag deletion (Branch or tag deleted)
** jdk9-latest (/shared/common/jdk-9_x64-latest)
+
* Pull requests (Pull request opened, closed, reopened, edited, assigned, unassigned, review requested, review request removed, labeled, unlabeled, synchronized, ready for review, locked, or unlocked)
** jdk1.8.0-latest (/shared/common/jdk1.8.0_x64-latest)  
+
* Pushes (Git push to a repository)
** jdk1.7.0-latest (/shared/common/jdk1.7.0-latest)
+
** jdk1.6.0-latest (/shared/common/jdk1.6.0-latest)
+
** jdk1.5.0-latest (/shared/common/jdk1.5.0-latest)  
+
  
* Maven
+
See the [https://developer.github.com/webhooks/ GitHub developer documentation] for more information.
** apache-maven-latest (/shared/common/apache-maven-latest)
+
** apache-maven-3.0.5 (/shared/common/apache-maven-3.0.5)
+
  
* Ant
+
=== Personal Access Token ===
** apache-ant-1.9.6 (/shared/common/apache-ant-1.9.6)
+
  
* Gradle
+
We can also create personal access token for the bot account. We create tokens with ''only'' the following scopes:
** gradle-latest (/shared/common/gradle-latest)
+
** gradle-3.1 (/shared/common/gradle-3.1)
+
  
More generally, all tools listed on http://build.eclipse.org/common/ are available from '''/shared/common/'''.
+
* repo:status
 +
* repo_deployment
 +
* write:discussion
  
If you need tools that are not general purpose installed, project members can install them in your project's home directory, for example ~/buildtools. [https://dev.eclipse.org/mhonarc/lists/cbi-dev/msg01869.html See email on cbi-dev]
+
See the [https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps GitHub developer documentation about scopes and OAuth] for more information.
  
== Default plugins ==
+
=== Getting admin permissions on project's GitHub repositories ===
  
The following plugins are installed by default. Additional plugins can be installed on request.
+
Project leads may require admin level access to their project's GitHub repositories. If that is the case, please [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Community&component=GitHub open a new bug] with your request. To be eligible to such permission level, your project must be in the mature phase and the request shall be highly motivated, so please tell us why you need that access (this information will be handy when sorting out the more general solution).
  
<div style="column-count:4;-moz-column-count:4;-webkit-column-count:4">
+
== Docker Hub ==
* ace-editor
+
* analysis-core
+
* ant
+
* antisamy-markup-formatter
+
* authentication-tokens
+
* bouncycastle-api
+
* branch-api
+
* build-timeout
+
* cloudbees-folder
+
* conditional-buildstep
+
* credentials
+
* credentials-binding
+
* dashboard-view
+
* disk-usage
+
* display-url-api
+
* docker-commons
+
* docker-workflow
+
* durable-task
+
* external-monitor-job
+
* extra-columns
+
* find-bugs
+
* gerrit-trigger
+
* git
+
* git-client
+
* git-parameter
+
* git-server
+
* gradle
+
* greenballs
+
* handlebars
+
* icon-shim
+
* javadoc
+
* jobConfigHistory
+
* jquery
+
* jquery-detached
+
* junit
+
* ldap
+
* mailer
+
* matrix-auth
+
* matrix-project
+
* maven-plugin
+
* momentjs
+
* pam-auth
+
* parameterized-trigger
+
* pipeline-build-step
+
* pipeline-graph-analysis
+
* pipeline-input-step
+
* pipeline-milestone-step
+
* pipeline-model-api
+
* pipeline-model-declarative-agent
+
* pipeline-model-definition
+
* pipeline-model-extensions
+
* pipeline-rest-api
+
* pipeline-stage-step
+
* pipeline-stage-tags-metadata
+
* pipeline-stage-view
+
* plain-credentials
+
* promoted-builds
+
* rebuild
+
* resource-disposer
+
* scm-api
+
* script-security
+
* sonar
+
* ssh-credentials
+
* ssh-slaves
+
* structs
+
* timestamper
+
* token-macro
+
* windows-slaves
+
* workflow-aggregator
+
* workflow-api
+
* workflow-basic-steps
+
* workflow-cps
+
* workflow-cps-global-lib
+
* workflow-durable-task-step
+
* workflow-job
+
* workflow-multibranch
+
* workflow-scm-step
+
* workflow-step-api
+
* workflow-support
+
* ws-cleanup
+
* xvnc
+
</div>
+
  
== Setup for specific plugins ==
+
The Eclipse Foundation owns the [https://hub.docker.com/u/eclipse eclipse] organization and a couple of other project specific organizations at https://hub.docker.com. You can ask to get a repository being created on one of these organizations. We will set permissions so that committers have write access to this repo (you will need to share your Docker Hub ID with us).
  
=== GitHub Pull Request Builder Plugin ===
+
You can also ask us to create a project specific organization. The organization name needs to follow the pattern ''eclipse<projectname>''.
  
The [https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin GitHub Pull Request Builder Plugin] (GHPRB) allows to build/test pull requests and provide immediate feedback in the pull request on GitHub.
+
Note that we don't grant admin permissions on any Eclipse Foundation owned organization. We recognize that this means that you will have to go through us for all new repo creation, but we can't grant organizations-wide admin permission assigned to committers for security reasons.
  
'''To set this up, please open a Bugzilla issue against the CI-Jenkins component (Product: Community) and request this feature.'''
+
==Nexus/Maven repository==
  
Here are some details about what happens during the setup process:
+
See [[Services/Nexus]].
* The GHPRB plugin is installed in the JIPP.
+
* Webmaster creates a GitHub bot user and adds it to the respective team on GitHub.
+
* The credentials of the GitHub bot user are added to the JIPP (with user name and password, because SSH keys are not recommended/supported by the plugin).
+
* The GHPRB plugin's main config is set up.
+
  
Once the ticket is resolved you should be able to configure and use the GHPRB plugin in your jobs.
+
==Signing tool==
  
Instructions how to set up GHPRB plugin in jobs can be found here:
+
* [https://git.eclipse.org/c/cbi/org.eclipse.cbi.git/tree/maven-plugins/README.md Maven plugins for signing artifacts]
https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md
+
* [[IT_Infrastructure_Doc#Sign_my_plugins.2FZIP_files.3F|On demand signing tool]]
  
'''Please note: ''' Currently we don't recommend to use the 'Use github hooks for build triggering' option. Instead, with this option turned off, Jenkins is polling GitHub instead. Which should work just fine in most cases.
+
==Eclipse Platform / plugins specific tooling==
  
More info can be found in the GitHub readme:
+
=== CBI license bundle ===
https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md
+
  
=== Jenkins Pipeline (aka configuration in code) ===
+
We offer a P2 repository containing the `org.eclipse.license` bundle which is located at:
  
An example how Eclipse plugins can be build with Tycho using a Jenkins pipeline can be found here (Thanks to Mickael Istria!):<br/>
+
    http://download.eclipse.org/cbi/updates/license/
https://github.com/eclipse/aCute/blob/master/Jenkinsfile
+
  
More info about Jenkins Pipeline can be found here:<br/>
+
This URL is a composite P2 repo containing the license bundle.
https://jenkins.io/doc/book/pipeline/<br/>
+
https://jenkins.io/doc/book/pipeline/shared-libraries/
+
  
=== Gerrit Trigger Plugin ===
+
If you use Tycho you can add the p2 repo to the <repositories> section of your pom.xml file. Something similar to this:
  
You may use the [https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger Jenkins Gerrit Trigger Plugin] in order to run a Jenkins job to verify each new patchset uploaded to Gerrit for code review. Jenkins (named "CI Bot") will then also vote on these changes using the "Verify" voting category.
+
<source lang="xml">
 +
    <repository>
 +
      <id>license-feature</id>
 +
      <url>http://download.eclipse.org/cbi/updates/license/</url>
 +
      <layout>p2</layout>
 +
    </repository>
 +
</source>
  
[[Image:Jgit.gerrit-reviewer.png|450px]]
+
In any particular feature which you need the license you can use the usual feature.xml section:
  
[[Image:Jgit.gerrit-vote.png|450px]]
+
<source lang="xml">
 +
<?xml version="1.0" encoding="UTF-8"?>
 +
<feature
 +
      id="org.eclipse.help"
 +
      label="%featureName"
 +
      version="2.0.0.qualifier"
 +
      provider-name="%providerName"
 +
      plugin="org.eclipse.help.base"
 +
      license-feature="org.eclipse.license"
 +
      license-feature-version="1.0.0.qualifier"/>
 +
....
 +
</source>
  
Below, the configuration sections for the Git plugin and the Gerrit trigger plugin of the verification job used by the EGit project may serve as an example.
+
=== p2 repo checks ===
  
====General configuration settings====
+
A set of "tests" which create reports or can be ran as unit tests that check to correctness of p2 repositories. That is partially just "correctness" in general (such as, that jars are signed, etc.) but more so that repositories conform to the requirements of the Eclipse Simultaneous release (such as, that jars have correct "Provider names", licenses, etc.). For more information, see See [[CBI/p2repoAnalyzers/Repo Reports]].
  
# Check ''This project is parameterized''. Click the ''Add'' button and select ''String Parameter''. Set the parameter ''Name'' to '''GERRIT_REFSPEC''' and ''Default Value'' to '''refs/heads/master'''.
+
=== p2 repo aggregator ===
  
[[Image:Gerrit-refspec-param.png|600px]]
+
A tool to combine several p2 repositories. Among other things, it makes sure they all have consistent constraints (that is, can be "installed together") unlike a raw p2 mirror task. For more information see [[CBI/aggregator]].
  
====Configuration of Source Code Management====
+
= Resources =
 
+
* [https://dev.eclipse.org/mailman/listinfo/cbi-dev cbi-dev]
# Under ''Source Code Management'' select Git.
+
* See our [[CBI/FAQ|Frequently Asked Question list]]
 
+
# Under ''Repositories'', click on ''Advanced'' and change the '''Refspec''' to '''${GERRIT_REFSPEC}'''.
+
 
+
# Under ''Additional Behaviours'', add '''Strategy for choosing what to build''' and select '''Gerrit Trigger''' as a strategy.
+
 
+
[[Image:Jgit.gerrit-git-config.png|600px]]
+
 
+
Note that the section '''Branches to build''' won't be used and may be deleted.
+
 
+
====Configuration of Build Triggers ====
+
 
+
# Under ''Build Triggers'', select '''Gerrit event'''.
+
 
+
[[Image:Jgit.gerrit-gerrit-config.png|600px]]
+
 
+
# Under ''Trigger on'', click on ''Add'' and select at least '''Patchset Created'''. This will configure the job to run on each new patchset. You can also add additional trigger, like '''Comment Added Contains Regular Expression'''. In the example below, a build will be triggered for the latest patch set if the comment is exactly '''CI Bot, run a build please'''.
+
 
+
[[Image:gerrit-trigger-events.png|600px]]
+
 
+
# Finally, configure at least one ''Gerrit Project''. The pattern is the name of project (i.e. if your repository is <code>git.eclipse.org/<xx>/<yy>.git</code>, then fill the pattern <code>xx/yy</code>). The ''Branches'' section is the list of branch to listen for events as configured above. Generally, you want one, named '''master''' to build patches submitted for the master branche, or <code>**</code> to build patches submitted to each and every branches. Set the type to '''Path'''.
+
 
+
[[Image:gerrit-trigger-project.png|600px]]
+
 
+
====Configuration of the build action====
+
 
+
Under "Build" click the "Add a build step" button, and select the appropriate action. The actual action depends on what you want Hudson to do. A typical example, for projects build with Maven is to select "Invoke Maven 3" and set "Maven 3" to "apache-maven-latest" and "Goals" to "clean verify".
+
 
+
 
+
===Tutorials, News, and other resources===
+
* [http://www.vogella.com/articles/EclipseTycho/article.html Tycho tutorial by Lars Vogel]
+
* [http://www.fosslc.org/drupal/content/tycho-good-bad-and-ugly Video discussing JBoss tools use of Tycho]
+
* [http://wiki.eclipse.org/CBI/Workshops Workshops being developed]
+
* [http://www.vogella.com/blog/2012/10/08/building-eclipse-sdk-locally-with-maven/ Building Eclipse SDK locally with Maven]
+
* [http://mickaelistria.wordpress.com/2012/10/08/sonar-at-eclipse-org/ Sonar at Eclipse.org !]
+
* [http://youtu.be/KJUfLvXiTSw Tycho and CBI Adoption: Feedback from the trenches]
+
* [http://www.bsiag.com/scout/?p=678 Eclipse Scout builds with CBI]
+
  
  
 
[[Category:CBI]] [[Category:Releng]] [[Category:Jenkins]]
 
[[Category:CBI]] [[Category:Releng]] [[Category:Jenkins]]

Revision as of 11:42, 20 August 2020

The Eclipse Common Build Infrastructure (CBI) is an initiative combining infrastructure, services, technologies and best practices for building, testing and delivering software at the Eclipse Foundation.

Goals

Primary goals of the CBI initiative are:

  • Make it easy for anyone to contribute to Eclipse projects.
  • Make it easy for projects to follow open governance and transparency best practices.
  • Provide a set of industry-grade services (or integration with some widely adopted 3rd party services) for building and distributing Eclipse projects.

Asking for Help

Service Level Agreement (SLA)

Most CBI services are Tier 2 - Best Effort, which means they are expected to be available at all times, and rapid restoration can be expected in the event of an outage. Eclipse Strategic Members can contact Webmaster in certain cases of off-hours support.

Please see IT_SLA for more information on the Eclipse Foundation IT Services SLA.

Provided Services

CI Environment (Jenkins)

We provide dedicated Jenkins instances to projects. See also Jenkins.

Jenkins is a continuous integration (CI) server used on Eclipse Foundation servers for Eclipse projects as part of the Common Build Infrastructure (CBI). Jenkins instances are maintained by the Eclipse Webmasters/Release Engineers. List of Jenkins Instances Per Project (JIPP):

NOTE: JIPP instances are being migrated from a standalone implementation to a Kubernetes cluster implementation.

Requesting a JIPP instance

Please file a bug against Eclipse Foundation > Community > CI-Jenkins to request your project's own instance. Please ensure your project lead can +1 the request. Please specify if you wish to grant write access to your download area and/or code repositories.

Important.png
Security issues
There may be security issues related to using the Gerrit plugin, more especially when allowing the CI system to write directly to the code repos and downloads area. If you request plugins other than those installed by default, webmasters may not be able to help troubleshoot any issues that you may encounter with your instance.


What's provided?

Each Eclipse Project has access to one Jenkins instance (JIPP), including the following:

  • (1) Jenkins instance, with (1) Resources Pack (see below)
    • Membership-sponsored projects may allocate more resources (see below)
  • Digital signing Service: Java JAR, Java Cryptography Extensions, Windows Portable Executable with Microsoft Authenticode, macOS application bundles.
  • Packaging service: Apple Disk Image (.dmg), Linux Flatpak
  • Disk space: Ephemeral for builds, permanent for release builds.
  • (1) 1vCPU/3.75G/30G Linux Virtual Server (if needed) (courtesy of Microsoft Azure). Projects sponsored by Strategic Members can engage with the Foundation to get out of spec Virtual Server.
  • Access to worldwide download mirrors

Additional Resource Packs

Each Eclipse Project has access to one Resources Pack for building. For some projects, that may not be enough. Projects sponsored by Eclipse Membership (via Project Lead) have additional Packs, based on membership level. These Packs can be allocated to projects. See the Jenkins wiki page to see how packs translate to Jenkins builds.

  • Some resources are only available to Enterprise and Strategic members.
  • Enterprise and Strategic members can engage with the Foundation to acquire additional Packs.
Resource Pack
Agent type Linux (containerized, no root)
vCPU 2 (burst to 4)
RAM 8GiB
Disk 50GB
External slave support Yes


Dedicated Agent
Agent type Linux/Windows/macOS (VMs)
vCPU 4
RAM 8GiB
Disk 100GB

Resource Packs Included in Membership

Eclipse Foundation Member Organizations have access to Resource Packs based on their membership level.

Membership Level and Resource Packs
Not a member Solution Enterprise Strategic
[$0, $15k) [$15k, $20k] [$50k, $100k] [$25k, $50k) [$50k, $100k) [$100k, $500k]
Resource packs 1 1 2 4 3 5 10
Dedicated Agents N/A N/A N/A 0 0 0 2
Important.png
Windows and macOS agents
Note that Windows and macOS agents (shared and headless - not suitable to run UI tests) can be added to your Jenkins instance. In this case, each shared headless agent count for 1 resource packs. These agents have common dependencies like JDK, Maven, etc installed. If your build requires specific/special dependencies you might need to setup your own external build agent instead.


Assigning Resource Packs to a Project

Resource Packs are assigned by Member organizations of the Eclipse Foundation to Eclipse Projects they sponsor. Packs are assigned as a whole to a single project (i.e., can’t split Packs across multiple projects). A member can assign several packs to a single project.

Note.png
Important
When asking for packs for your project, please ensure that project leads and your organization representatives are copied to the bugzilla ticket. We require approval from project leads but assume immediate approval from organization representatives. We strongly advise you seek authorization internally to your organization before opening such a request though. Should conflictual requests arise, the organization representatives will be asked to actively arbitrate.


To assign a pack to a project, please file a Bug.

By default, resource packs are assigned build agents. In some cases, it may be required to scale up the Jenkins master. In such case, we can allocate resource packs to the master instance.

We maintain a public list of sponsored projects. We also maintain a list of the Member Organizations benefits. On this last spreadsheet, you can also check how many Resource Packs each organization have left for project sponsoring.

CI Environment (Third-party)

If you host your Eclipse project git repository at Github, we also support some third-party services:

  • TravisCI. TravisCI is free to use for open source projects, but you should be aware that it limits to 5 the number of executors per organizations. It means that only 5 builds can run concurrently in any organization. If your repositories are part of the Eclipse organization, you may face long delays between the trigger and the actually start of the job. If it's your case, and would like to stick to TravisCI for building, we suggest you ask to move to a dedicated organization.
  • CircleCI. https://circleci.com/
  • Azure Pipelines. https://azure.microsoft.com/en-us/services/devops/pipelines/

If an environment is not listed in the unsupported list, you can ask whether it can be supported by opening a bug.

Important.png
Third party build services (e.g., Travis) limitations
The Eclipse Foundation's IT Team is responsible for the credentials (OSSRH, SSH and GPG keys...) it creates for projects. Keeping track of where credentials are disseminated is crucial for security reasons. As long as we have no automation in place to set/update/remove credentials on 3rd party build services, we don't put any credential on such services. For instance, it means that you can't use them to publish to Maven Central, or to upload build results to download.eclipse.org.


GitHub and Bot Accounts

The Eclipse Foundation can create a bot user that can be used from your Jenkins (JIPP) instance to build branches and pull requests, push, tag, and comment on the repo.

We don't share the credentials of those bots for external usage (e.g., to use it with a private, behind firewall Jenkins instance, or with third-party services). Those bots have the same set of permissions as a committer and it would be against the Eclipse Development Process to give those permissions to anybody. Likewise, we don't grant any elevated permissions on repositories or organizations to any other bot account.

Webhooks

However, we can setup webhooks for you. We create webhooks that can only receive the following events:

  • Branch or tag creation (Branch or tag created)
  • Branch or tag deletion (Branch or tag deleted)
  • Pull requests (Pull request opened, closed, reopened, edited, assigned, unassigned, review requested, review request removed, labeled, unlabeled, synchronized, ready for review, locked, or unlocked)
  • Pushes (Git push to a repository)

See the GitHub developer documentation for more information.

Personal Access Token

We can also create personal access token for the bot account. We create tokens with only the following scopes:

  • repo:status
  • repo_deployment
  • write:discussion

See the GitHub developer documentation about scopes and OAuth for more information.

Getting admin permissions on project's GitHub repositories

Project leads may require admin level access to their project's GitHub repositories. If that is the case, please open a new bug with your request. To be eligible to such permission level, your project must be in the mature phase and the request shall be highly motivated, so please tell us why you need that access (this information will be handy when sorting out the more general solution).

Docker Hub

The Eclipse Foundation owns the eclipse organization and a couple of other project specific organizations at https://hub.docker.com. You can ask to get a repository being created on one of these organizations. We will set permissions so that committers have write access to this repo (you will need to share your Docker Hub ID with us).

You can also ask us to create a project specific organization. The organization name needs to follow the pattern eclipse<projectname>.

Note that we don't grant admin permissions on any Eclipse Foundation owned organization. We recognize that this means that you will have to go through us for all new repo creation, but we can't grant organizations-wide admin permission assigned to committers for security reasons.

Nexus/Maven repository

See Services/Nexus.

Signing tool

Eclipse Platform / plugins specific tooling

CBI license bundle

We offer a P2 repository containing the `org.eclipse.license` bundle which is located at:

   http://download.eclipse.org/cbi/updates/license/

This URL is a composite P2 repo containing the license bundle.

If you use Tycho you can add the p2 repo to the <repositories> section of your pom.xml file. Something similar to this:

    <repository>
      <id>license-feature</id>
      <url>http://download.eclipse.org/cbi/updates/license/</url>
      <layout>p2</layout>
    </repository>

In any particular feature which you need the license you can use the usual feature.xml section:

<?xml version="1.0" encoding="UTF-8"?>
<feature
      id="org.eclipse.help"
      label="%featureName"
      version="2.0.0.qualifier"
      provider-name="%providerName"
      plugin="org.eclipse.help.base"
      license-feature="org.eclipse.license"
      license-feature-version="1.0.0.qualifier"/> 
....

p2 repo checks

A set of "tests" which create reports or can be ran as unit tests that check to correctness of p2 repositories. That is partially just "correctness" in general (such as, that jars are signed, etc.) but more so that repositories conform to the requirements of the Eclipse Simultaneous release (such as, that jars have correct "Provider names", licenses, etc.). For more information, see See CBI/p2repoAnalyzers/Repo Reports.

p2 repo aggregator

A tool to combine several p2 repositories. Among other things, it makes sure they all have consistent constraints (that is, can be "installed together") unlike a raw p2 mirror task. For more information see CBI/aggregator.

Resources

Back to the top