Difference between revisions of "CBI/Distribution"

Revision as of 10:30, 17 February 2012

This page tracks information related to how code, configuration, build artifacts, bundles, packages, and more are distributed at Eclipse.

Background

Eclipse projects consume 3 types of software during software builds:

1. Approved unmodified third party software
2. Approved modified third party software - with contentious bits removed to make it acceptable to Eclipse's IP policy
3. Eclipse software from other projects (or itself in the case of large projects with multiple parts)

NOTE: this page is not discussing run time install and dependency handling. AR: Sober second thought... not sure the two sides can be decoupled. For instance, anything that is consumed for build, will need to be bundled for runtime.

Overview

The following is an image depicting repositories of software at Eclipse, processes that consume and produce output, and the flows between processes.

Guide

Build process

1 - Hudson executes the build (or local developer initiates manually)

2 - Code and config is cloned from git

3 - Libraries are downloaded from either a) Orbit as a zip file b) maven.eclipse.org as maven repositories c) downloads.eclipse.org as p2 repositories

4 - The build executes, outputs p2 repositories

5 - Packaging takes place, outputs packages

Notes

Apologies for not representing all build technologies on this diagram. We recognize there are others. This was to serve as a useful framework for discussion rather than a complete inventory.

CQ - 3rd party libraries are reviewed and approved/rejected. Approved libraries are stored in Orbit

ECO - Eclipse technology is consumed by the ecosystem at large

Red arrows denote a path that would be useful, but may not exist today

Dashed arrows are strictly a manual process

Areas of concern

Scaling to large quantities of software

Orbit provides a centralized clearing house for IP policy approved 3rd party dependencies. Orbit provides this software as a zip file containing bundles for all software. This would be better if it were componentized so that just what is needed can be consumed.

Eclipse technology redistribution

What about Eclipse technology? At the moment, projects consuming technology from other Eclipse projects do so in a variety of adhoc ways. People picking up Eclipse technology are left to discover the chain of transitive dependencies - both Eclipse based and third party. This is a brittle process and assumes a lot in terms of developers doing the right thing, even when it isn't always clear what the right thing is.

This is an issue for Eclipse projects hosted in the Foundation forge, and also for technology outside using Eclipse components.

Move to other forges

The LTS and Polarsys programs must ensure that all dependencies are available from an Eclipse Foundation controlled source decades into the future. This includes all 3 types of software as listed above. As well, the LTS forge is for members only. Binaries for software involved with LTS must only be available to members both for consumption in a build and runtime.

Thus, for a given developer from a specific company working on LTS software, their preference will be this order:

1. Developer specific software
2. Company specific software
3. Generic re-usable components from the LTS forge
4. Generic re-usable components from the Community forge

This concern has a multiplier effect since there are multiple forges, potentially large numbers of releases supported, and potentially a number of companies involved with each release.

Making a point

Summary

1. Orbit's distribution model doesn't scale as much as we need nor offer enough flexibility
2. Orbit only covers a small portion of software consumption - third party. Thus even with Orbit, another mechanism is always needed
3. Orbit is unique to a subset of projects in the Eclipse ecosystem
4. Project, 3rd party (modified & unmodified) software needs to be in bundle format in any case to satisfy runtime demand.

Conclusion

1. We need a repository of some sort to unify software distribution for builds at Eclipse
2. The same repository distributing between Eclipse projects should be proxied for LTS, Polarys, and other IWGs to satisfy their needs
3. The same repository should also be used to feed Eclipse technology to anyone, via. the Internet
4. The amount of effort to deploy this repository is likely similar to the effort to replicate the infrastructure above to LTS, Polarsys, etc. We'd be better off investing effort that simplifies our build environment and may even provide more capabilities.

Recommendation

1. Start capturing must/should/would be nice requirements for this repository
2. Review potential candidates for reuse to satisfy these requirements
3. Select a candidate, capture tasks and create a project plan for deployment
4. Implement it as part of the work for LTS, Polarsys, but with direct benefits back to today's Community forge.