Difference between revisions of "Development Resources/HOWTO/Release Reviews"
|Line 4:||Line 4:|
Revision as of 11:35, 18 October 2013
The purposes of a Release Review are: to summarize the accomplishments of the release, to verify that the IP Policy has been followed and all approvals have been received, to highlight any remaining quality and/or architectural issues, and to verify that the project is continuing to operate according to the Principles and Purposes of Eclipse.
- 1 Overview
- 2 Checklist
- 3 What are the Requirements?
- 4 What Does a Release Review Cover?
- 5 Preparing the Documentation
- 6 The Review Process
- 7 After a Successful Review
Please see the release cycle, which includes a nice (simple) overview of the [[Development Resources/HOWTO/Release Cycle#Release_Review|release review process].
Helpful Documentation from the Eclipse Development Process
Reviews are Not Required for Service Releases
Note that a release review is not required for a service--or "bug fix only"--release. If your release contains no new features, APIs, or significant changes in functionality, you may release it without engaging in a community review. Service releases are indicated by an increment in the third segment of the version number (e.g. 2.3.2).
Note also that projects are not required to submit IP logs for review by the IP Team for service releases. Projects are required to ensure that they have the appropriate coverage for all intellectual property (e.g. CQs and contribution records) at all times.
Before you can consider a Release Review, all of the relevant CQs must be approved by the Eclipse Legal team. We cannot schedule a Review before the Legal team has completed their work. If you are waiting for CQs, please review where your CQs are, and when they are scheduled to be reviewed, in the IP team work queue.
After you have verified that all relevant CQs are approved, the following items must be completed at least one week before the scheduled review date:
- PMC Approval (Can occur in parallel with, prior to, and after, the IP clearance);
- IP Log Approval This is essential, and no release review can proceed without it!; and
- Review document.
IP Log Approval
Because IP Log approval is essential and can take varying amounts of time (depending on the work queue of the IP Team), projects must have an approved IP Log before their review date is confirmed. After you have submitted the IP Log to Eclipse Legal for approval, the review is added to the schedule with the notation Waiting for IP Log. After EMO is notified that the IP Log is approved, that notation is replaced with the date of the review.
Please use the automated IP Log Tool to update and and submit your IP Log. You can use the IP Log Generator for this (the Submitting the IP Log to Eclipse Legal section explains how to submit the IP Log). The URL for using the tool for your project uses your project ID, in this format: http://www.eclipse.org/projects/ip_log.php?projectid=project ID.
Before submitting your IP Log, we recommend that you review the contents of your project's download directory to ensure that you have the necessary CQs in place. The Project Downloads Review tool can help you in with this review.
While you are waiting for IP Log approval, we suggest that you obtain PMC approval for the review and begin work on your documentation.
Please forward an email showing that you have the approval of your PMC for the review and corresponding review documentation. The easiest way to do this is to request approval on your PMC mailing list (which requires subscription) and forward the response to EMO.
Note that you do not have to wait for the IP Log to be approved by the IP Team. You can seek PMC review/approval of your review documentation while the IP Log separate from the IP log approval process.
Your project plan must be current and available for public consumption.
The content requirements for review documentation are detailed below. The preferred means of providing review documentation is to enter it directly into the record associated with your release in the project metadata.
In the past, a presentation file (PPT or ODP) rendered in PDF from was the preferred format for review documentation (we used to host actual review calls for which the presentation format was appropriate). Presentation files are still acceptable (and are quite common). Other open formats, including HTML, a wiki page, or a document file rendered in PDF format, are also acceptable.
Please submit your document via email to the EMO. Include the review document as either a link or an attachment.
Many people underestimate the time and effort needed to create the document, so be sure to allow enough time for this task. The "official" due date for the document is one week before the scheduled start of the review period. However, the document need to be reviewed by the EMO before posting. If you wait until the due date to submit the first draft and the EMO requests changes, then you'll probably miss your deadline and your review will be postponed.
This looks scarier than it is. Be brave.
- Project website up-to-date:
- Project incubation status (if applicable) correctly noted
- Doing the right sorts of things to facilitate community development
- Project Metadata is up-to-date
- Release metadata is correct and up-to-date
- Includes well-written project description (see The Whats and Whys to creating project descriptions).
- Project plan is up-to-date and in the standard format
- Code and builds are in order:
- All project code available in eclipse.org VCS
- At least one milestone build available
- Builds hosted on download.eclipse.org
- archived builds hosted on archive.eclipse.org
- Naming Conventions followed:
- Bundle and package names e.g.
- Bundle and package names e.g.
- Version Numbering rules followed
- Bundle manifests contains a "Bundle-Vendor" entry set to "Eclipse <project name>"
- Every bundles contains an about.html file as required by A Guide to the Legal Documentation for Eclipse-Based Content.
- Features names and descriptions are captured, are spelled correctly, use proper grammar, and contain content that is actually useful for the intended audience
- (If in incubation phase) Incubation branding on distributed artefacts
- Well-defined retention policy for distributed artefacts (e.g. p2 repositories)
- Review security-related bugs that may affect your project and prepare mitigation strategy
- Every download includes the necessary legal documentation (including a copy of the project licenses) as required by A Guide to the Legal Documentation for Eclipse-Based Content.
- Intellectual Property (IP) Logs in order
- Approved Contribution Questionnaire (CQ) for project's initial contribution;
- Approved CQs for all third-party libraries used directly by project code (regardless of whether or not the library is directly distributed by the project)
- Use the Project Downloads Review tool to confirm that the necessary CQs are in place;
- Distribution channels cleaned up (e.g. project downloads)
- Have older releases been moved to the archive server?
- Have older nightly and integration builds been removed?
- IP Logs submitted for review by the Eclipse IP Team at least one week in advance of the review period
- PMC approval obtained (generally obtained through project mailing list)
- Review Documentation (described below in greater detail)
- PMC-approved review Documentation submitted to EMO in advance of the start of the review period (one week preferred)
If you have any questions, please ask EMO.
Note that the lead time required to process IP Logs and review documentation may change based on current load. We typically ask, for example, that IP Logs be submitted a full month in advance for projects participating in the annual simultaneous release.
What are the Requirements?
The Release Review is conducted before each major release to verify that the key goals of the release have been accomplished and to verify that all intellectual property rights issues have been resolved. For Eclipse projects with typical six week milestones, a plausible time for a release review is one to two weeks into the final six week period, i.e., four or five weeks before the release is scheduled to go live.
What Does a Release Review Cover?
The Release Review is the final review of the project, process, release and most importantly, the IP issues, against the characteristics that help ensure Eclipse Quality.
A Release Review is a fairly comprehensive process. We anticipate that gathering the material for the review and preparing the documentation is a non-trivial effort, but we believe that the introspection offered by this exercise is useful for the project and that the results are very useful for the community.
The items to be covered in the Review include the following. If the document is a slide deck (there's a template available), it will probably average one or two slides on each of these issues. A text document (provided in an open format such as HTML) should have one or two paragraphs, or a handful of bullets on each issue. Note the use of the word "summarize" throughout this agenda; judgment must be used to determine how much to include in each of these bullet items.
Summarize the major features of this release as well as any other features that have generated significant discussion amongst the community during the development cycle. Compare the features against the Roadmap to understand the project's conformance or divergence.
Reason: The community will use this release and the ecosystem will build products on top of this release, and both need to know what features were included or excluded.
References to existing New & Noteworthy documentation is a useful addition to this summary.
Summarize the state of the non-code aspects of the release including: user documentation, localization/externalization, examples, tutorials, articles, and so on. Have the existing artifacts been updated? Are there new artifacts? Have the obsolete ones been retired or at least marked as pertaining only to older material?
Reason: The non-code aspects are essential for the wide-spread adoption of the release (Reference Eclipse Ecosystem).
Certify that the APIs in this release are Eclipse Quality. The project lead will personally certify that the requirements for quality have been met and/or discuss any deficiences.
Reason: Eclipse members build commercial tools on top of the extensible frameworks and thus the quality of the APIs isextremely important.
Summarize the architectural quality of the release. Discuss the intrinsic nature of being extensible embodied by this project. Discuss issues such as unresolved overlap with other projects, unpaid "merge debt" from incorporating various components, and so on.
Reason: Eclipse members build commercial tools on top of the extensible frameworks and thus the quality of the architecture is important.
Prior to your release, check Bugzilla for the list of resolved vulnerabilities and currently committer-private vulnerabilities (there may be overlap between the two) to determine if any affect your project's code. Also check Bugzilla for bugs that may not have been marked for security (e.g. perform a simple search for words like "vulnerability" and "security" in the bug summary.
In this section of the document, describe the work that you've done to remediate any issues that may arise as a result of these vulnerabilities.
Summarize the usability of the tools. Usability in this sense is about using the tools to solve development problems, not the more academic sense of UI evaluation/testing.
Reason: Without usable tools, the project will not attract the user community necessary to enable the ecosystem (Reference Three Communities).
Summarize the features (APIs and any significant user features) from previous releases that are being end-of-life'd in this release. End of life includes both deprecation and actual removal.
Reason: The community builds products that rely on features and so they need to know when these features are changing (Reference Three Communities).
Summarize the bugzilla situation. How many bug records (defects and enhancements) have been opened/closed/deferred/new, etc? How many P1, P2, ..., bug records are outstanding?
Reason: Summaries of the bugzilla records offer a glimpse into the project productivity. They also offer an estimate of the outstanding risk. And the summary is used to alert the community to known issues (Reference ).
Summarize the standards compliance of this release. If the features are based on defined, pending, or ad-hoc standards, what is the state of those standards and what is the state of the support for those standards in this release.
Reason: Eclipse is about building frameworks and tools based on standards, so we need to make sure that we are conforming to the appropriate standards (Reference ).
Summarize the user interface usability and the conformance to the User_Interface_Guidelines. Include section 508 compliance, language pack conformance (does the code support multiple languages), etc. Explain any deviations from the user interface guidelines and standards.
Reason: The user community is larger than just mouse-wielding, English-speaking, computer jockeys. We need to support that larger community (Reference User_Interface_Guidelines).
Discuss the initial schedule and any changes to the schedule over the course of the release, i.e., what the project team achieved. Discuss whether milestones were met or slipped.
Reason: The community relies on consistent schedules from Eclipse so that projects and products can plan for the correct dependencies.
Summarize the project's development of its three communities. Consider the interactions on bugzilla, the mailing lists, the newsgroups, public conference calls, blogs, public-relations activities, code camps, conference tutorials, coordinating with other Eclipse projects and other open source projects (Apache, ObjectWeb, etc), ...
Reason: It is important for Eclipse projects to build a community around the project, not just deliver code for a project. This review item is about the success of building a community (Reference Three Communities).
As per the Eclipse IP Policy, these steps must be done:
- The project leadership verifies that:
- ... that the about files and use licenses are in place as per the Guidelines to Legal Documentation
- ... all contributions (code, documentation, images, etc) has been committed by individuals who are either Members of the Foundation, or have signed the appropriate Committer Agreement. In either case, these are individuals who have signed, and are abiding by, the Eclipse IP Policy.
- ... that all significant contributions have been reviewed by the Foundation's legal staff. Include references to the IPZilla numbers of all clearances.
- ... that all non-Committer code contributions, including third-party libraries, have been documented in the release and reviewed by the Foundation's legal staff. Include references to the IPZilla numbers of all clearances.
- ... that all Contribution Questionnaires have been completed
- ... the "copyright" field of each feature is set to the copyright owner (the Eclipse Foundation is rarely the copyright owner).
- ... that any third-party logos or trademarks included in the distribution (icons, help file logos, etc) have been licensed under the EPL.
- ... that any fonts or similar third-party images included in the distribution (e.g. in PDF or EPS files) have been licensed under the EPL.
- The PMC provides a Project Log that enumerates:
- every piece of third party software including information on the license
- every major contribution
- the name of every contributor including non-committer contributions via bug fixes with bug #'s
- the About files which contain any non-standard terms (e.g., a reference to a license other than the EPL, etc)
- The EMO will validate for (a) and (b) that Contribution Questionnaires have been properly submitted and EMO approvals have been completed.
- A frozen copy of the reviewed-and-approved-by-Eclipse-legal Project Log is part of the Release Review documentation. It can be included in the documentation or as a separate document. (Reference )
The provider name for contributed features/bundles should always start with "Eclipse ..." followed by project name. It is up to each PMC to decide how best to describe their project in a meaningful, balanced fashion (either as one top level one or as multiple pieces). Avoid special symbols (e.g. ':' or "-") unless they are part of the project name. Avoid acronyms (unless in common use in software industry, such as XML, or PHP). Capitalization should follow "headline-sytyle capitalization": The only words that are not capitalized are articles (except as the first word), coordinating conjunctions (for example, and, but, and or), prepositions (except as the first or last word), and the to in an infinitive.) For discussion, see Bug 252813.
IP Issues Speak-Up-Now
The EMO explicitly asks during the Release Review if any Member would like to assert that this release infringes their IP rights. If so, the EMO and the project will follow the Eclipse IP Policy in discussions with that Member.
Reason: One of the important benefits that the Eclipse Foundation provides for its members is the consistent application of the Eclipse IP Policy which helps ensure (but does not guarantee) that the framework and tools are useable in commercial products.
Per Board resolution of December 11, 2008, to pass a Continuation Review, Graduation Review or Release Review a project must have a current project plan, in the format specified by the EMO, available to the community.
If there is a Project Plan (full or even a draft) for the next release, the final issue to cover in the Release Review is the unveiling of the new plan.
Preparing the Documentation
The documentation must cover the issues from (2) above.
The Review Process
Once the review content has been created, the operational side of a Release Review is described here: Development_Resources/HOWTO/Review_Process
After a Successful Review
After the successful advisory vote and the approval of the Release Review by the EMO:
- the EMO will send an email to the project development list affirming the positive result
- then, when the final release files are ready, the project may post them to the download server and let the powerful replication system distribute the bits throughout the known universe
Other possible results of the Release Review include:
- Postpone - issues have arisen that must be solved before the release proceeds
- Pass w/ Notes - the release has some issues that need to be addressed, but they can be addressed with release notes and/or documentation
- Pass w/ Issues - the release has some issues, but they can be addressed in a future release as agreed to in the review
- Fail - there are significant problems with the release and/or the project.
This page is moderated by the EMO