Kepler Working Proposal
- Click here to go to the main Kepler Project wiki page.
- This page is an adaptation of the original Kepler Project proposal.
Community-oriented software development is widely seen as one of the most efficient, effective ways to promote both quality and reuse in the development process. Today's open source software communities provide a means to logically break down and organize otherwise monolithic collections of libraries into functionally-focused modules that are self-describing, discrete, and reusable. This community-oriented development model enables a collaborative approach to providing the common subsystems that many applications need. Harnessing the efforts of a diverse development community allows a shared project to accumulate skill, experience, and testing from all of the applications using it. By building an ecosystem of such software communities, each focused on delivering a single, well-articulated feature set, developers are free to seek the best implementation possible for each subsystem, then reuse it. Focusing each project on a single, well-defined problem promotes clearer architecture through a narrowed project scope. Such projects also tend to render a more reliable solution, since they draw on a larger set of use cases to produce a more diverse set of project tests. Additionally, as bugs surface and are fixed within these projects, the improvements benefit the entire community. Finally, the shared infrastructure code provided by these projects leads to greater consistency across grouped or related applications , and makes it easier for developers to integrate these projects with one another. Organizations can reap great rewards in terms of improved application quality and enhanced developer effectiveness by fostering rich, community-oriented development environments.
Unfortunately, today's development processes often impose a high price in the form of complex infrastructure and provisioning requirements, making the efficiency benefits of a community-oriented development model nearly impossible to realize. Infrastructure-related complexity is manifested in the many, fragmentary systems that currently inhibit information flow throughout the development life cycle. Further complicating matters is the considerable overhead imposed on engineering through the the micro-management of tasks and information specified by most traditional development methodologies. Every software project has the same essential requirements, such as supporting its users (architects, developers, project managers, etc.), tracking software issues and their respective resolution activities, managing requirements, and generating system designs. Whenever a new project is formed, the complexity or inefficiency involved in the existing development processes is propagated, further increasing the volume of noise that pollutes the engineering environment, inhibiting effective collaboration. In the end, the information overload suffered by community participants makes the cost of implementing a highly modular software ecosystem prohibitive. With today's project-infrastructure requirements, the ideal concept of community-oriented software development simply doesn't scale.
The purpose of Kepler is to address the complexities involved with provisioning, managing, and using a shared infrastructure in order to support a community-oriented development model. Our strategy will consist of the following:
- Derive an abstract set of tasks required to support software communities, based on tools commonly used today.
- Develop a framework to orchestrate the execution of related tasks and task groups, with the goal of enabling discovery, access, and control of any third-party, community-infrastructure system from within Kepler.
- Provide access to each set of framework features through a rich, Eclipse-based environment, thereby enabling users to participate in community activities without leaving their development environment.
Kepler will provide extensible frameworks and exemplary tools for participation in and management of a large, diverse software community environment. In pursuit of this mission, Kepler will be guided by the following principles:
We intend to create project-metadata management frameworks that provide equal access to all tools. Our goal is to integrate and simplify the development process in existing environments, and the best way to achieve this is to design a framework which allows existing products to integrate easily.
Open, Standards-based Orientation
This project will strive to use open standards for all information storage and communications APIs, in order to facilitate tighter integration with third-party tools. Where existing standards exist, we will attempt to use them. Where they do not exist, or are deemed unworkable, data formats used will be published and will reflect industry best practices for such designs.
Eclipse Community Involvement
We believe the success of Kepler is contingent upon successful integration with the rest of the Eclipse community. In order to achieve the level of usability required to serve the user community properly, it is critical to engage the broader Eclipse development community and reuse whatever components, technologies, and techniques that make sense.
User Community Involvement
The purpose of Kepler is to make community-oriented software development simpler. Moreover, we believe it is essential that Kepler integrate well with existing Community Infrastructure Tools tool-sets. In order to support this, it is critical to maintain tight bonds with our user community, to be characterized by constant feedback and discussion between project developers and project users.
Perfect integration of third-party tools is of very limited value if the integrated environment is itself unusable. Therefore, Kepler will maintain a strong focus on the experience of the target audience when designing both user-interface components and the integration framework itself.
In the tradition of the Eclipse community, key discussions and documentation will be easily accessible to the foundation membership at large.
Due to the many potential aspects of community-oriented development which this project may improve, we believe that it is critical to provide concrete, usable features as they become available, without waiting for a complete feature set. In this way, incremental roll-out of successively larger feature sets will give the developer and user communities an earlier opportunity to shape Kepler, and make it the best it can be. MUST, SHOULD, MAY Specification Semantics
In order to support the most advanced features of the most advanced tools applicable to each framework, each feature will strive to mark each requirement as one of MUST, SHOULD, or MAY to indicate the features that are required for minimal participation via the framework, and which additional features are supported by the framework for tools that can provide them. These feature annotations are provided in an attempt to avoid the least-common denominator effect, where a framework is limited by the feature-set of the most rudimentary tool to be integrated.
Kepler intends to start with small, high-impact steps to make concrete progress toward its goals as soon as possible. As new opportunities to support software communities are identified, the implementation of the overall Kepler project will gradually expand within each of its main feature categories to provide a richer toolset to its users. However, regardless of the scope of implementation, the project will remain true to one goal: minimizing the complexity involved with managing and participating in an ecosystem of distinct, but interdependent, software communities. The following are an initial list of feature categories to address some areas of community-oriented development in which we believe Kepler can make significant early progress:
Community Project Model
In order to provide frameworks for working with community-oriented project infrastructure, and in order unite the frameworks of Kepler behind the common goal of lowering the barriers to a community-oriented development model, it is essential that Kepler frameworks operate on a common model representing community-oriented projects. To this end, Kepler will introduce a community-project model, to define and maintain the specification for projects which are candidates for community-infrastructure support. While this model may implement a concrete API reading and writing to some "native" format, it will concern itself primarily with providing a means of building a project model in memory, and adapting other systems to provide project information that can be consumed by other participants in the Kepler framework.
The community project model will implement the following requirements:
- MUST define appropriate extension points to enable third-party tool configuration to be added to the model.
- MUST provide mechanisms for extensibility, versioning, and backward compatibility in the project model.
- MUST provide dependency information
- SHOULD support dependency searching
- MAY support optional (user-configured) substitution of workspace projects for stated binary dependencies
- MUST provide project-identification information for integration of the project into the larger software ecosystem
- MUST track the origins of project-model metadata
- SHOULD provide mechanisms for modifying project-model metadata files
- MAY provide a mechanism for writing changed metadata back to the appropriate file using existing metadata-model APIs
- MAY provide a compound navigator or editor (embedding or linking to existing metadata editors) which allows users single-point access to all project-model data elements
- SHOULD provide integration metadata for build servers, where applicable
- SHOULD include any applicable notification configurations, to enable build servers to provide feedback to the community
- SHOULD provide all applicable provisioning information for a project, including:
- mailing list addresses (subscribe, unsubscribe, archive; one set per mailing list)
- SCM location (developer access, anonymous access, and browsing locations, where applicable)
- issue tracker information (system identifier and location)
- web-site locations
- forum location
- committer information (including authorization or role where applicable)
Sources of inspiration for the community project model will include Apache Maven's POM (http://maven.apache.org/ref/current/maven-model/maven.html), Buckminster's Component Specification model (CSPEC), DOAP (http://usefulinc.com/doap) and Eclipse's own plugin.xml, feature.xml, and .project/.classpath/etc. project binding files, among others.
- SHOULD support configuration of dependency-search sources (globally, or per-project)
- (see Community Project Model section above for more on dependencies)
- MUST support tool-specific preferences, if they are provided by the third-party integration plugin
- (tool configuration refers to things settings required to run the build tool, such as ANT_HOME)
- SHOULD manage tool configuration per-project, or globally (per-workspace?)
One key feature commonly found in successful community-oriented software initiatives is the ability to build the project from source code with a minimum of required specialized knowledge or configuration effort. Aside from supporting users who choose to build from source, this feature is critical for all developers in the community, to ensure that when new code is committed, it does not introduce major problems. The tighter the integration between the project's build system and the developer's environment, the more likely developers are to run project builds, and the less likely that an integration build will fail as a result of code changes. To this end, one of the first features of Kepler will be concerned with integrating build tools into the development environment.
The following are some initial build-management feature requirements for the Kepler framework:
- MUST integrate with project-builder API
- MUST notify of build events:
- SHOULD support filtering
- MUST integrate with Problems and Errors views
- MUST provide bidirectional synchronization of project changes to and from the build metadata
- MUST detect supported build metadata files, and add appropriate tool-configuration sections to the project model
- SHOULD manage project code and non-code source directories
- MAY manage token-filtering configuration for resource files in non-code source directories
- (substitution of "live" values for @TOKEN@ in resource files)
- MAY manage token-filtering configuration for resource files in non-code source directories
- SHOULD provide support for debugging within a build
- (specifically, debugging unit tests, etc.)
- MAY handle accessory builds : that is, builds consisting of tasks different from the main build process
- MAY support tool-specific catalogs for selection of accessory (non-default) build steps to augment the main build
- MAY support tool-specific interactive update for build-step catalogs
- MAY support for build modifier activation: these alter the build metadata or environment under certain circumstances
- (often, these are used to modify the build process, dependencies, etc. in response to a detected platform, or similar)
- SHOULD support editing of build metadata using graphical tools
- MUST support for user interactivity in the build process: support builds that prompt for input
Build Server Management
Another common feature of successful community-oriented software projects is the implementation of continuous integration (CI) and other server-side build infrastructure. Performing ongoing, consistent build-and-test cycles for a project - rather than just building the software when it's time to release a new version - reduces the risks associated with a project release, and helps to ensure that the community always has up-to-date information about the health and progress of the project. Additionally, since CI builds happen frequently, developers get much quicker feedback when something breaks. With the timely feedback provided by a CI environment, developers can resolve problems as they surface in the build, while they're still relatively easy to diagnose and fix. Finally, server-side build infrastructure enables several advanced development practices, such as proactive detection of errors due to dependency upgrades and binary promotion of project build results. Unfortunately, installation and maintenance of a server-side build environment can be a fairly complicated proposition. To address this complexity, another early feature set of Kepler will be concerned with provisioning, maintenance, and usage of server-side build environments.
The following are some initial build-server-management feature requirements for the Kepler framework:
- SHOULD configure an existing build server instance or farm
- MAY support maintenance tasks such as shutdown, restart, and status reporting for existing build server instances or farm
- MUST support add, remove, or configure project builds from one or more build servers
- MUST support directed project builds within the build environment (builds triggered manually by the developer)
- MUST notify of build events
- SHOULD support event filtering
- SHOULD extract or generate build reports from the build environment
- MAY support remote debugging of directed build builds
- MAY support on-demand building of user-submitted project bundles, separate from SCM
- MAY support building a project as a precondition of SCM commits.
- MAY support building projects against latest snapshots of all dependencies
- MAY support binary promotion
- MAY support full binary release process
Project Provisioning and Maintenance
The ability to provision and manage infrastructure-related configuration for multiple projects in a consistent manner is one of the most intensive overhead activities for a community-oriented software ecosystem. When the ecosystem allows the promotion of community members from user to developer, such as in the open-source realm, maintaining these community-support systems becomes all the more difficult. Therefore, to lessen the burden of adopting a community-oriented development model on large scales, Kepler will introduce several project provisioning features.
The following are some initial project-provisioning feature requirements for the Kepler framework:
- MUST support existing community-infrastructure tools
- MUST provision new services for existing projects as needed
Initializes the following, using existing infrastructure (shared by multiple projects):
- MUST: continuous integration (single-instance, clustered, and farmed scenarios)
- MUST: mailing lists and associated rules
- MUST: forum
- MUST: issue tracking
- SHOULD: wiki
- MUST: version control (CVS, Subversion, etc.)
- SHOULD: static web sites
- MUST: build scripts
- MUST: Eclipse IDE configuration for the project
- MUST support addition and removal of developers (committers)
- SHOULD support modification of developer authorization/roles
NOTE: Our initial goal for this feature of Kepler will be to support the community infrastructure of the Eclipse Foundation itself. In order to achieve this, Kepler development team will need to work closely with the Eclipse IT team. Since the IT team is responsible for maintaining and provisioning infrastructure for the Eclipse Foundation communities, they represent a rich source of knowledge about the workflows and current tools used to manage these systems. Additionally, any implementation of the project provisioning feature which aims to serve the Eclipse Foundation will have to have access and buy-in from the IT team.
One of the great costs associated with subdividing a monolithic code base into modules is the complexity involved with participating in the resulting menagerie of projects. Instead of interacting with a single system for each life-cycle task, developers must manage information in dozens. For example, where a monolithic project might house all of its open issues in a single, application-wide tracking interface, issues logged against an ecosystem of community-oriented projects are housed in multiple separate issue-tracking repositories; one per project. Needless to say, modularization under these circumstances severely complicates the development and project management tasks of any application based on such an ecosystem. Similarly, community participation may take place largely on project-specific mailing lists, making it more difficult to maintain a single conversation thread that addresses multiple projects. Even the process of searching for documentation can become quite complex, as each project may host separate forums, FAQs, and project-documentation web sites.
In order to overcome the scalability problem and realize the benefits of a community-oriented development process, Kepler will reduce the burden incurred through a multi-project infrastructure by introducing a multi-community visibility feature. This project will deliver a framework and exemplary tooling that provide integrated, read-write views of information flow among and through multiple software communities.
The following are some initial multi-community feature requirements for the Kepler framework:
- MUST provide issue-tracker reporting (including open issues, all issues assigned to a particular developer, etc.)
- MUST aggregate and organize mailing list traffic (including easy cross-posting)
- MUST support forum searches (including aggregation of results)
- SHOULD support web-site documentation browsing (and search?)
- SHOULD support aggregation of structured documents, like FAQs
- MUST support aggregation of CI results
- MUST support aggregated management of and reporting on project binaries
This project proposes to work closely with individuals and organizations, both inside and outside the Eclipse Foundation, in order to arrive at an architecture that is relevant to as wide a community as is practically possible. To begin, the Buckminster project at Eclipse has expressed interest in exploring opportunities to integrate with and help shape the features detailed above. Proposed Project Lead and Initial Committers
- Carlos Sanchez, DevZuz - Project Lead
- Philip Dodds, DevZuz
- Brett Porter, DevZuz
- Mike Lim, Exist
- Glen Gonzales, Exist
- Maria Odea Ching, Exist
- Jonas Lim, Exist
- Cata Pineda, Exist
- Erle Mantos, Exist
- Genesis Deiparine, Exist
- Thomas Hallgren, Cloudsmith
- Henrik Lindberg, Cloudsmith
The following organizations and individuals have expressed interest in this project:
- Europa Build Workshop Members ( http://wiki.eclipse.org/index.php/Europa_Build_Workshop_Report )
- Apache Maven ( http://maven.apache.org )
- Eclipse Buckminster ( http://www.eclipse.org/buckminster )
- DevZuz ( http://www.devzuz.com )
- Intervoice ( http://www.intervoice.com )
- Exist ( http://www.exist.com )
- Simone Bordet simone _dot_ bordet *at* gmail _dot_ com