Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "CBI/Distribution"

< CBI
(Areas of concern)
 
(5 intermediate revisions by one other user not shown)
Line 3: Line 3:
 
==Background==
 
==Background==
  
Eclipse projects consume 3 types of software during software builds:
+
Eclipse projects consume a few main types of software during software builds:
 
# Approved unmodified third party software
 
# Approved unmodified third party software
 
# Approved modified third party software - with contentious bits removed to make it acceptable to Eclipse's IP policy
 
# Approved modified third party software - with contentious bits removed to make it acceptable to Eclipse's IP policy
# Eclipse software from other projects (or itself in the case of large projects with multiple parts)
+
# Eclipse software distributed (and consumed) as p2 repositories
 +
# Eclipse software distributed (and consumed) as maven artefacts
 +
# Other formats that will arrive in the future
  
NOTE: this page is not discussing run time install and dependency handling. AR: Sober second thought... not sure the two sides can be decoupled. For instance, anything that is consumed for build, will need to be bundled for runtime.
+
The [http://wiki.eclipse.org/LTS LTS] and [http://wiki.eclipse.org/Polarsys Polarsys] industry working groups provide a catalyst for new requirements.
 +
 
 +
NOTE: this page was originally not discussing run time install and dependency handling. In reality many of the same challenges need to be solved.
  
 
==Overview==
 
==Overview==
Line 43: Line 47:
  
 
==Areas of concern==
 
==Areas of concern==
 
===Scaling to large quantities of software===
 
 
Orbit provides a centralized clearing house for IP policy approved 3rd party dependencies. Orbit provides this software as a zip file containing bundles for all software. This would be better if it were componentized so that just what is needed can be consumed.
 
  
 
===Eclipse technology redistribution===
 
===Eclipse technology redistribution===
Line 64: Line 64:
 
# Generic re-usable components from the Community forge
 
# Generic re-usable components from the Community forge
  
This concern has a multiplier effect since there are multiple forges, potentially large numbers of releases supported, and potentially a number of companies involved with each release.
+
This concern has a multiplier effect since there are multiple forges, a much large number of releases active, and a number of people & companies involved with each release.
  
 
==Making a point==
 
==Making a point==
Line 72: Line 72:
 
# Orbit's distribution model doesn't scale as much as we need nor offer enough flexibility
 
# Orbit's distribution model doesn't scale as much as we need nor offer enough flexibility
 
# Orbit only covers a small portion of software consumption - third party. Thus even with Orbit, another mechanism is always needed
 
# Orbit only covers a small portion of software consumption - third party. Thus even with Orbit, another mechanism is always needed
# Orbit is unique to a subset of projects in the Eclipse ecosystem
+
# A subset of projects in the Eclipse ecosystem use Orbit
 
# Project, 3rd party (modified & unmodified) software needs to be in bundle format in any case to satisfy runtime demand.
 
# Project, 3rd party (modified & unmodified) software needs to be in bundle format in any case to satisfy runtime demand.
  
Line 87: Line 87:
 
# Select a candidate, capture tasks and create a project plan for deployment
 
# Select a candidate, capture tasks and create a project plan for deployment
 
# Implement it as part of the work for LTS, Polarsys, but with direct benefits back to today's Community forge.
 
# Implement it as part of the work for LTS, Polarsys, but with direct benefits back to today's Community forge.
 +
 +
 +
[[Category:CBI| ]]

Latest revision as of 09:15, 10 May 2017

This page tracks information related to how code, configuration, build artifacts, bundles, packages, and more are distributed at Eclipse.

Background

Eclipse projects consume a few main types of software during software builds:

  1. Approved unmodified third party software
  2. Approved modified third party software - with contentious bits removed to make it acceptable to Eclipse's IP policy
  3. Eclipse software distributed (and consumed) as p2 repositories
  4. Eclipse software distributed (and consumed) as maven artefacts
  5. Other formats that will arrive in the future

The LTS and Polarsys industry working groups provide a catalyst for new requirements.

NOTE: this page was originally not discussing run time install and dependency handling. In reality many of the same challenges need to be solved.

Overview

The following is an image depicting repositories of software at Eclipse, processes that consume and produce output, and the flows between processes.

Distribution today.png

Guide

Build process

1 - Hudson executes the build (or local developer initiates manually)

2 - Code and config is cloned from git

3 - Libraries are downloaded from either a) Orbit as a zip file b) maven.eclipse.org as maven repositories c) downloads.eclipse.org as p2 repositories

4 - The build executes, outputs p2 repositories

5 - Packaging takes place, outputs packages

Notes

Apologies for not representing all build technologies on this diagram. We recognize there are others. This was to serve as a useful framework for discussion rather than a complete inventory.

CQ - 3rd party libraries are reviewed and approved/rejected. Approved libraries are stored in Orbit

ECO - Eclipse technology is consumed by the ecosystem at large

Red arrows denote a path that would be useful, but may not exist today

Dashed arrows are strictly a manual process

Areas of concern

Eclipse technology redistribution

What about Eclipse technology? At the moment, projects consuming technology from other Eclipse projects do so in a variety of adhoc ways. People picking up Eclipse technology are left to discover the chain of transitive dependencies - both Eclipse based and third party. This is a brittle process and assumes a lot in terms of developers doing the right thing, even when it isn't always clear what the right thing is.

This is an issue for Eclipse projects hosted in the Foundation forge, and also for technology outside using Eclipse components.

Move to other forges

The LTS and Polarsys programs must ensure that all dependencies are available from an Eclipse Foundation controlled source decades into the future. This includes all 3 types of software as listed above. As well, the LTS forge is for members only. Binaries for software involved with LTS must only be available to members both for consumption in a build and runtime.

Thus, for a given developer from a specific company working on LTS software, their preference will be this order:

  1. Developer specific software
  2. Company specific software
  3. Generic re-usable components from the LTS forge
  4. Generic re-usable components from the Community forge

This concern has a multiplier effect since there are multiple forges, a much large number of releases active, and a number of people & companies involved with each release.

Making a point

Summary

  1. Orbit's distribution model doesn't scale as much as we need nor offer enough flexibility
  2. Orbit only covers a small portion of software consumption - third party. Thus even with Orbit, another mechanism is always needed
  3. A subset of projects in the Eclipse ecosystem use Orbit
  4. Project, 3rd party (modified & unmodified) software needs to be in bundle format in any case to satisfy runtime demand.

Conclusion

  1. We need a repository of some sort to unify software distribution for builds at Eclipse
  2. The same repository distributing between Eclipse projects should be proxied for LTS, Polarys, and other IWGs to satisfy their needs
  3. The same repository should also be used to feed Eclipse technology to anyone, via. the Internet
  4. The amount of effort to deploy this repository is likely similar to the effort to replicate the infrastructure above to LTS, Polarsys, etc. We'd be better off investing effort that simplifies our build environment and may even provide more capabilities.

Recommendation

  1. Start capturing must/should/would be nice requirements for this repository
  2. Review potential candidates for reuse to satisfy these requirements
  3. Select a candidate, capture tasks and create a project plan for deployment
  4. Implement it as part of the work for LTS, Polarsys, but with direct benefits back to today's Community forge.

Back to the top