Difference between revisions of "SimRel/Simultaneous Release Requirements"
|Line 259:||Line 259:|
==== Document APIs provided and publish API policy ====
==== Document APIs provided and publish API policy ====
APIs should be provided, so others can properly extend functionality, and presumably most of that detail is defined in JavaDoc. But, there should be a high level statement of how consuming projects can determine what is API; how 'APIs' are distinguished from non-API and 'provisional' API, if any. It is recommended that non-API be marked with x-internal in the
APIs should be provided, so others can properly extend functionality, and presumably most of that detail is defined in JavaDoc. But, there should be a high level statement of how consuming projects can determine what is API; how 'APIs' are distinguished from non-API and 'provisional' API, if any. It is recommended that non-API be marked with x-internal in the manifest. Also, API policy should include what the commitment is to API, how long maintained after deprecated, etc. As one example, see [[WTP API Policy|WTP API Policy]].
=== Retention Policy ===
=== Retention Policy ===
Revision as of 07:57, 17 October 2012
The Eclipse Simultaneous Release Requirements
Updated October 3, 2011
Authored and maintained by the Eclipse Planning Council
Contact: David Williams
This document defines the rules and criteria for participating in the yearly Simultaneous Release. There are more criteria than when releasing at other times. There are more requirements partially because there are more projects releasing at once, so the workload needs to streamlined and made uniform. But also, the extra criteria are included by mutual agreement between projects (via their representatives to Planning Council) so that as a whole, the release will be of better quality, maintainability, and improved consumability.
The spirit of this document is not be so much as a "contract" of what has to be done to release, but instead a statement of what minimally is necessary to make the Yearly Release good, if not great! While each Project does their individual things to make the Release great, this document describes how we, as a group, do that by our voluntary agreement to participate and provide these minimum requirements. We are always open to better documentation and more meaningful agreements, so please feel to make suggestions on how to make our yearly release better from year to year (preferably through your Planning Council representative). Changes may be made to this document throughout the development cycle for clarity or to improve reference links, but nothing new will be added after M4 (that is, things that would affect workload).
To allow for some flexibility for special cases, exceptions to these requirements are allowed, but to provide balance and foster good communication, any exceptions to the items or deadlines must follow the Planning Council Exception Process.
The requirements are divided into three categories:
- The minimum requirements to be released as part of the "yearly release" ... that is, to able to "claim" you were part of the yearly release. These are the normal release requirements, but done earlier than usual.
- The "must do" requirements to be part of the common software repository and, consequently, the minimum requirements to be part of an EPP package.
- Requirements to improve adoption and demonstrate good Eclipse Citizenship, following "the Eclipse Way". These are requirements you do not have to do, exactly, but they are recommended, encouraged, and the thing that you do have to do is to document if and how you do them.
Changes for Juno Release
There were a number of wording and reorganization changes, but the substantial changes were:
- removed 2 items
- added 3 items to request information in standard plans (two universal, one specific to Juno)
- added 1 item to request projects make it easier to get source used in a release
Normal release requirements ... but needed earlier than usual
The requirements and conditions stated in this section are the basic minimum required for a project to claim they are part of the yearly Simultaneous Release.
State intent early (M4)
To join a Simultaneous Release, Projects must have stated their intent to do so by M4, at the latest. The "statement of intent" is done formally by marking the "Simultaneous Release Flag" in the project's Portal meta-data. For a normal release, a month's notice is typical, but we ask for more advanced planning for the yearly release.
Formal (standard format) plans, early (M4)
All projects must have their project plan in the Eclipse Foundation standard XML Format (a normal Eclipse requirement). Committing to be in the Simultaneous Release means you commit to having these plans available early: by M4 at the latest. Naturally, plans will change as development continues, and we encourage teams to update them periodically, such as every milestone, to reflect reality and progress, but an initial version is required by at least M4 and the final version, due by the release in June, should be a clear statement of what was planned, what was achieved, and what was deffered. The plans are collected in January or February (approximately M5) and become part of the "road map" document that goes before the Board of Directors for review and approval. Every plan, for any release, should have some specific items covered, such as Target Environments and Compatibility with Previous Releases but we give some specific guidance here since these are so important to adoption. In addition, we do ask for one extra "theme" item, that is technically required only for the Simultaneous Release. What you plan, is up to each project, we just want to be sure its clear for adopters and downstream projects.
Exactly what platforms and runtimes a project supports is up to them and their community, but it is required all projects document what platforms they support, especially if they have native (non-Java) code and especially if it is different than the set of platforms supported by the Eclipse Platform itself.
It is a similar case for specifying what minimum (or, maximum) Java level to use (such as Java 5, Java 6, Java 7). We do not mandate any particular level, but the question often gets asked, so we simply clarify here that projects should well document what version of Java they generally write to and test with. (Note: individual bundles have their own specific BREE's, but that is a very specific "micro" level specification, and does not reflect the overall level of Java targeted, or tested.)
It would be anticipated that this year, for Juno, many Eclipse users would still be using Java 6, perhaps a few on Java 5, but projects should test extensively on Java 7 since its use will likely grow and become common place by the time Juno ships, especially over Juno's life time. In other words, in general, it is recommended to write your code to the lowest level possible, but test on the highest level possible.
Of course, the "target environments" you focus on, in your plans, depends on your project. For example, some runtime projects might focus on what servers they work with and test, some projects might focus on what browsers they target, test, and support, or similar.
To satisfy this requirement, please document your supported platforms in your standard project plan by having a section labeled, exactly, as <target_environments>. See template in standard plan reference and for an example, other than the Platform's, see the subject section in Web Tools plan.
Be sure to raise any "platform level" questions or issues to the cross-project list, especially if a project is "moving up" and might affect projects that depend on it.
Compatibility with Previous Releases
It should be part of every project's plan to have a section detailing compatibility with previous releases. This should not only include commitments to API and binary compatibility, but ideally would also include plans for source compatibility, workspace compatibility, and project "coexistence" compatibility. See the template in standard plan reference and for examples, see the plans for the Eclipse Platform and the Web Tools Platform project.
Note: this primarily applies to projects that have "released before", but even those releasing for the first time should be aware of this requirement as there could be things done "now" for future compatibility and migration ... these need to be "thought through" even for your first release, to make sure your second release maintains appropriate compatibility.
The intent, here, is for projects to document their intent and whether or not bugs about compatibility would be considered valid. For example, a project might document "we support only a one-time migration of projects, and if it is a shared project, all developers must move to new version at the same time". While that level of compatibility is less than ideal, at least then adopters know what to expect and can make their plans based on that information.
Support for Eclipse 3.8 workbench
For Juno, each project plan needs to have a section (or theme) in their standard format plan, on how they intend to support Eclipse 3.8. We will have 4.2 as primary platform (hence the one in common repo, and the one used for EPP Packages) but we ask participating projects to have a plan theme item titled, exactly, "Support for Eclipse 3.8 workbench". We ask for that exact wording, to enable being able to automatically produce a "summary" of this auxiliary support. To be more exact, there should be an item in your "<p:themes_and_priorities>" section, hence you would have an element there named "<p:theme name="Support for Eclipse 3.8 workbench">".
Possible topics and descriptions to be covered in your plan description should include things like how tested, if bugs against 3.8 are considered valid, etc. Some statements in your plan might be similar to:
- We do not support for 3.8 based apps.
- We will accept bugs against 3.8 based apps, but do not test or compile against it.
- We will compile against 3.8 (or, run API checks to make sure compatible) and somewhat test 3.8, though 4.2 is primary.
- We will support 3.8 as well as 4.2, but the exact functionality may differ.
- We will support 3.8 and 4.2 equally, and all the functionality will be the same.
- Clarification on 02/02/2012: Projects may state their plan is to "support only 3.8", but then be subject to other constraints of Juno plans and requirements, such as not being in the common repo.
For one example of a plan, see the WTP statement on support for 3.8 workbench.
IP Documentation and Logs (RC1)
Projects must have their IP logs approved (a normal Eclipse requirement) but follow the earlier deadlines set by EMO and IP staff. The IP log deadline is typically mid-week RC1.
Projects are encouraged to created drafts of the Projects IP Logs even earlier. The development process requires the IP Log to always be accurate, but experience shows there often are not, until examined for a release, and there are typically many issues to resolve, or fixed, before the final submission to the IP staff. For example, sometimes a CQ might have the wrong flag, and cause it to not show up in the Auto IP Log, or perhaps a common, already approved 3rd party jar was was used, but no CQ for that specific project entered. The purpose of having these early drafts is so that projects get familiar with what is required, and do not allow work to build up at the end, since that could cause a "bottleneck" at a critical time and jeopardize having the ability to resolve all issues in time to be released. Also, some adopters will want to look at early drafts to see what 3rd party requirements are associated with the code they are planning to adopt.
A good guideline is to have a draft of your IP Log by M5, and plan for it to be complete for M7.
Note that being in the Simultaneous Release will give your IP CQ requests some higher priority in getting evaluated, in order to make the date. The higher priority treatment is only for the 5 months or so before the release (after the deadline for CQs, typically M5). The reason being, of course, is that the rest of the year the IP staff must also get work done for maintenance releases and projects not on the release train. During that part of the year (roughly July to February every year) all CQs are prioritized in a uniform way. But, there is a deadline for CQs for the yearly release, usually in February, end of M5, to have all required CQs submitted, on file (but not necessarily approved by then).
Release Review and compliance to requirements documentation (RC3)
The release review documentation must be complete by the date specified by the EMO, which is earlier than it would be for other releases. (Typically mid-week during RC3.)
[removed 12/05/2011. It never accomplished the desired "streamlining", and just confused people]
A Project's PMC must approve the projects request for review (a normal Eclipse requirement). It is recommended, to help organize and streamline the yearly Simultaneous release, that a PMC provide their approval in writing, in the form of a short summary of their projects that are requesting review and summary of the PMC's discussion or method of approving them. (This is meant to be very brief, such as 1/2 page). The short summary can be documented in a mailing list, PMC Meeting notes, or even a wiki document.
[added 12/05/2011] Normally, in release review documentation, Project Leads must document and state that the project's release complies with all the normal release requirements, such as IP requirements. In addition, for a Simultaneous Release, Project Leads must document their verification that the project complies with all extra requirements of this Simultaneous Release document, as they apply to their project, and document any exceptions, there in the release review documentation. This is intended to be a few short sentences or paragraphs ... not a detailed checklist. [This is the replacement or "low cost alternative" to tracking on Foundation Portal, which was deemed unmaintainable. Eventually some of the URLs etc. requested in the old tracking system will be in the new "release portal" but (likely) not in time for Juno (see bug 363524.]
Extra requirements, to be in common repository
The requirements in this section must be met for a project to be on the common, central repository (e.g. /releases/juno) for end users to discover easily and therefore (per EPP Policy) are the minimum requirements to be included in EPP Packages. The criteria in this section are designed to make sure projects work relatively well, and work well together. This is especially required for adopters who may be using these projects in complicated, interwoven ways so each piece of the puzzle must fit together well and be dependable and be maintainable, as well as being on time and IP clean.
Integrate Early and Often
First-time participants are expected to be in an aggregation build by M4, at the latest. Then, once in, always in. This firstly and mostly means by agreeing to be in the yearly release, in June, you will also participate in the two planned Service Releases. But, even more than that, it is assumed that once you are in one Simultaneous Release, you will continue to be, so the following year, it is assumed you will be in M1 ... that is, you should not wait until M4 every year, even though that is the deadline for first-timers. Put another way, being part of the Simultaneous Release is not a "one time" activity, covering only the release part of the development cycle. Instead it is a commitment to stay "simultaneous" on an on-going basis. Once in, if a project decides to not be part of future simultaneous releases, they need to communicate that widely, and as early as possible, since could affect adopters or downstream projects.
At least one person from each project in a Simultaneous Release must subscribe to cross-project mailing list, since that is the primary communication channel for issues related to the Simultaneous Release. Also, at least one person from each project must subscribe to cross-project bugzilla inbox (add email@example.com to the "Add users to my watch list" box at the bottom of your Bugzilla email preferences page), as that is the primary bugzilla components for bugs that are truly cross-project, or bugs which are not known to be in one particular component.
Your representative to the Planning Council, either from PMC or Strategic Member, must attend Planning Council meetings and represent you there. Presumably, of course, after meeting or communicating with you and the other projects they represent, so they can fairly bring forward concerns and vote on issue that affect all projects, if required. Put another way, by committing to be in the Simultaneous Release, you agree to abide by all the Planning Council decisions and rules, so be sure your representative understands your project and your situation.
Build-team members (or their designated alternates) from each project may be asked to provide direct communication channels: at least email (if not phone, IM, IRC) and at least one build team member must be "on call" during the integration periods.
Required Bundle forms and formats
Version Numbering (tested)
Projects must use 4-part version numbers following the common semantics described in the Eclipse version numbering document.
OSGi bundle format
All plug-ins (bundles) must use the true bundle form. That is, provide a manifest.mf file, and not rely on the plugin.xml file being 'translated' into a manifest.mf file at initial startup. With that, empty plugin.xml files in the presence of a manifest.mf file should not be included in a bundle. (For some old history, see bug 130598.)
Execution Environment (tested)
All plug-ins (that contain Java code) must correctly specify their Bundle Required Execution Environment (BREE).
Projects must use signed plugins using the Eclipse certificate.
Projects must use jarred plug-ins (with unpack=false) unless there are technical reasons not to do so and require the directory form. [TODO: are nested jars still an issue for PDE? Something to avoid? Perhaps fixed?] Also, nested jars should be avoided if possible since it creates problems for projects that has dependencies to such plug-ins. The OSGi runtime is fine with it but the PDE environment is not able to handle classpaths that contain nested jars. Exceptions to these principles should be documented by the project, so we can learn the reasons and extent of unjarred bundles.
License text consistency (tested)
Use standard forms of license documents so it is displayed in the most usable, and concise way during install and update. It is a normal requirement to use a standard Eclipse Foundation "about" template, but where those templates are edited by each project, care must be taken to be sure they are edited in similar ways. That is, substantial differences are fine, if required, but we need to avoid minor differences based on case, dates, and formatting. Note that the Eclipse Foundation's license or user agreement files may change from year to year (such as, see bug 316152 but ideally in Indigo and future releases, it will be easier to point to a "symbolic" representation of the license, so it will be accurate with less manual updates from each project (see bug 306818).
Any third-party plug-ins that are common between projects must be consumed via Orbit. The Simultaneous Release must not have duplicate third-party libraries (note that this only applies to versions of the libraries; thus if project A requires foo.jar 1.6 and project B uses foo.jar 1.7, that's normally ok, different service versions a little less ok, such as 1.7.1 vs 1.7.2 (those should be explained, if required), and a qualifier-only difference is definitely not ok).
Note: the "partially tested", for this case, means there is a report of "Non Unique Versions used in repository" which can catch issues of not using common bundles. See current report for an example.
Provide optimized p2 repository (partially tested)
Projects must provide their own project p2 repository for their own project and updates. Projects must optimize their p2 repositories to reduce bandwidth utilization and provide a better install and update experience for users.
In addition, they must provide their artifacts and metadata in a specified format and method to allow at least parts of their repository to be aggregated and mirrored to a common repository. The current process may be modified throughout the year, if improvements can be made.
Clarification on 03/31/2010: Note that a project's repositories must contain original (conditioned) jars, and pack.gz files (where original jar means the jar produced by the build, but which has been conditioned for pack200). This is mentioned since in some scenarios, only the pack.gz files needs to be left there ... but, that practice is controversial for now, so we ask for both.
Clarification on 11/08/2010: feature "includes" must be strict, that is "include" an exact version of that other feature. This is required so installs and builds can be repeatable independent of the exact day of the install or the exact repos enabled. This is the way things are, and have been for years, and this statement is just making it explicit. While there may, in the future, be new mechanisms that allow some "line up collection" to be specified, it will be something new, not the feature "includes" element.
Clarification on 02/01/2012: the repositories produced and contributed must use p2 publishers that produce greedy='false', by default, in the content metadata. See bug 247099 and the p2 Publisher wiki for some history and details on this issue of greedy vs. non-greedy requirements.
Clarification on 02/25/2012: everyone's p2 repositories should make use the of p2.mirrorsURL property. For "how to" information, see p2.mirrorsURL wiki. Note: this is not really a "Simultaneous Release Requirement" but is required of any p2 repository on Eclipse Foundation infrastructure, and is just documented here to help spread the word and educate new comers.
Clarification on 03/01/2012: Similar to p2.mirrorsURL attribute, a well behaved, well optimized p2 repository should contain a p2.index file. This is especially important for "composite repos" and prevents unnecessary "round trips" to server looking for files. See bug 347448 for history and for how-to instructions, see the p2 wiki. Again, this is not so much a "Simultaneous Release Requirement" but is recommended of any p2 repository on Eclipse Foundation infrastructure, and is just documented here to help spread the word and educate new comers.
Each major project (as determined by participating PMCs) must have an 'About' dialog icon with hover text that displays the provider name. Every plug-in and feature must specify a descriptive provider-name (for features), or Bundle-Vendor header (for plug-ins), as determined by the project's PMC (e.g. "Eclipse Modeling Project" rather than "Eclipse.org"). Also, Projects must contribute to the welcome page when appropriate.
Do No Harm
Projects must work together in any combination of any install. Put another way, this means that users can install any subset of the projects participating in Simultaneous Release, and each of the installed projects will work as well as if it had been loaded independently. If such a problem is identified, the affected projects must track down and fix the problem.
Support Primary Eclipse Platform to be in common repo (and EPP Packages).
For Juno, Eclipse 4.2 will be the primary platform. Thus, projects in the common repo must support that version of Eclipse and prereqs based on that version. By EPP Policy, only features in common repo can be used in EPP packages. This is called out as a separate required item, since, in theory, there could be a "giant repo" where some Juno feature or bundle could pull in a 3.8 (or even an Indigo) version of Eclipse or some prereq ... so, this "requirement" is just to say that scenario is not supported. We assume only 4.2 as the base for all Simultaneous Release projects and all projects (in common repo) agree to work towards that goal (such as, test, fix bugs, etc. against 4.2).
Required for good adoption
The items in this category are, in a sense, optional. That is, what, exactly, is done by a project is up to that project, but it is required for projects to document what they do. These are often "best practices" that many projects have found essential at driving good adoption, plus the items sometimes speak to the quality of the project, as an Eclipse "good citizen", as opposed to their code quality or architecture. But, their importance is not be as universally relevant to all projects. Hence, it is required that each project document what they do for these items, but exactly what they do is up to the best judgment of the project and their community.
The Project should actively engage their community to get feedback on milestone builds, and document how they do that. One way to do this is to have a New & Noteworthy for each milestone. New and Noteworthy documents should be something readable and usable not just a static list of all the bugs. Corollary: individual new & noteworthy should be linked in to a collective New & Noteworthy.
Project should have measurable performance criteria that are regularly tested against. Projects should devote at least one milestone to performance and scalability improvements. This is important to the Simultaneous Release since a project or plugin by itself might seem to perform ok, taken together as part of hundreds of bundles, might result in unusable performance.
Each project should provide basic capability/activity definitions to allow for their UI contributions to be hidden. These can be provided in a separate plugins and feature to facilitate inclusion and reuse by consumers, or ... as most projects do ... simply document some examples, so adopters can create their own, or reuse via copy/paste. Ideally, projects should also provide triggers to facilitate progressive discovery of functionality (but, not many do, other than the Platform). As with other "good citizen" items, projects are free to document "we don't do this" ... but, then at least it is known and better communicated.
[removed 11/23/11] Good idea, but there is nothing in "release train" that depends on this (only one EPP package uses capabilities and does not depend on this requirement), and adopters that use it likely already understand how to do it, so removed requirement to simplify list. And, to emphasize, that's not to say the concepts behind products using capabilities are not important ... they are ... just nothing as a release train we can do to help much.
Projects must have a mature, stable build process: documented, scripted, repeatable, and executable by others on their own system. Ideally there would be just a few commands to run from a Linux shell to accomplish the build, but there might need to be a page of "set up" instructions of what's required (e.g. if CVS or GIT needs to be installed, certain version of Java, etc.) Of course, there may be some parts of it that do not work completely (such as, being signed by the Eclipse code certificate would not work when ran by others, on non-Eclipse.build systems) but the build should not fail if it does not sign, and otherwise should produce essentially the same functional output as the "officially signed build". Note, this build should include both producing the code and producing the runnable unit tests.
Projects should have unit tests with enough documented instructions that others (adopters or extenders) can run in their own environments and configurations, independent of doing a build. Ideally this could be done with just a few commands, but as with builds, might require a description of machine set-up.For one, example, see WTP Stand alone unit tests.
Ramp Down Planned and Defined
Projects must have a written ramp down policy by M6, at the latest, and provide link. The plan should describe when the project plans to be feature complete, have API frozen, and similar. See Platform 3.5 Endgame plan as a guideline. See also Indigo Final Daze.)
Projects should design and test for accessibility compliance, following established guidelines and Eclipse fundamental techniques to achieve accessibility. Projects must document their accessibility work and compliance. Ideally this would be by using a publicly available checklists, such as
See the WTP Accessibility Page for how one project documented their checklists.
Note, projects can document their work or compliance as a negative, such as "We do not do any accessibility work or testing and do not know the degree of our compliance. While we will consider accessibility bugs as valid, it is unlikely we would give them a high priority". But its important to document if that is the case, so adopters know.
If possible, and appropriate, accessibility testing tools can be leveraged such as NVDA.
The main accessibility article at Eclipse Corner is also very helpful. (thanks goes to Todd Creasey).
Use only APIs
Projects should leverage only published APIs of dependencies. All deviations must be documented in bugzillas. These bugzillas may be of the type that a dependent project should provide a required API, or of the type that a consuming project must move to some API that already exists. Note that technically there is no obligation for consumed projects to provide API that is requested ... that depends on many things ... but the main goal of requiring these bugzilla entries is to provide some documentation and measure of the amount of risk associated with non-API use.
Document APIs provided and publish API policy
APIs should be provided, so others can properly extend functionality, and presumably most of that detail is defined in JavaDoc. But, there should be a high level statement of how consuming projects can determine what is API; how 'APIs' are distinguished from non-API and 'provisional' API, if any. It is recommended that non-API be marked with x-internal in the bundle's manifest. Also, API policy should include what the commitment is to API, how long maintained after deprecated, etc. As one example, see WTP API Policy.
Projects should provide some summary metrics, such as number of bundles, number of committers, lines of code, number of bugs opened and fixed. This is so some statements can be made and tracked year-to-year about the size of the simultaneous release.
[removed 11/23/2011] Again, a good idea, but nothing in "release train" depends on this, nor do adopters need anything other than what is provided by Eclipse Foundation itself, via "dash" and project pages. Some projects submit their repository to Ohloh (or similar) and let it produce project stats.
Make it easy to get released source from repository
Projects should make it easy for potential contributers (and adopters) to get the source used in the release, from the repository (that is, source that exactly matches what was used in a build or release). This might be done by some form of documentation, or tag, or by producing a "team project set" during your build and making that available from your downloads page. Another good way to accomplish this, while maybe not suitable for all projects, is to make your bundle's source repository "self documenting" by using the Eclipse-SourceReference directive in your manifest.mf file (its very easy to have this added if your build makes use of PDE build or its properties). This requirement serves two purposes. First, we want to be sure adopters can get your source used in a release, in case they need to create some fix, even at some distant point in the future. But, also, this serves the community purpose of making it easier for contributors to provide patches for bugs.
Excel in National Language support
Use the best NL Java Libraries (ICU4J)
Projects should use ICU4J, where they have "user strings", in order to excel in NL support. (The latest ICU4J bundles will be in Orbit).
Use the most efficient message bundles
Projects must use Eclipse message bundles unless there are technical reasons not to, since (with or without translations) these are known to be more memory efficient that some other forms of handling UI-releated strings. This is important to Eclipse and adoption is some adopters end up using thousands of bunldes ... so, that's a lot of string.
All strings must be externalized, and projects must participate in Babel, meaning it is registered and available for string translation, etc. Projects must freeze the UI sufficiently early to allow the Babel project time to translate strings so there can be simultaneous release of translated versions. This typically means the UI should be frozen by M6 (a "freeze" means all major changes and additions are done by M6, and changes after that are done in a controlled, well documented, well reviewed fashion, so Babel translators can more easily keep up with late changes). The project should use the Babel Pseudo Translation Test to verify their translatability. See bug 217339.
Enable development for all languages
Projects should design and test for enabling development for all languages including bidi, unicode characters, etc. This is different than "translating" the User Interface software and would apply even to "runtime only" components. For one example, while using an English version of Eclipse Web Tools Platform, someone should be able to create a Chinese language web application and have it (and debug logs) displayed correctly.
Planning Council Exception Process
Exceptions for any rule or schedule can be made if there are good enough reasons to. This same exception process will be followed for things like "requests to respin" a build after a deadline. The process to get any exception must be open and well documented and follow these steps:
First, the Project's PMC must approve the request for an exception and it is the PMC (not the Project) that makes the request to the Planning Council. The Planning Council member that represents the PMC would bring the issue forward to the Planning Council.
Second, the exception requires at least 3 positive votes from active Planning Council members and no negative votes. When time is a factor (e.g. requests for rebuilds) the deadline for voicing a negative vote is basically by the time 3 votes are documented. But when time is not a factor, such as when requesting an exception to one of the criteria, then a period of one week will pass before being final, to allow times for concerns or negative votes to be voiced even after 3 positive votes. If there are not enough positive votes within one week, then the request for exception will be considered 'failed'. Note that "3" was chosen under the assumption that the Planning Council member representing the PMC would vote for it (since that PMC must approve it initially) so that means 2 others must also vote for it, for 3 total.
Depending on the timing, the issue and votes will be documented in either the Planning Council Meeting minutes, or on the Planning Council mailing list. If possible, some automation may be added to the release reporting tool to aid this documentation.
Testing of Common Repository
Some of the items marked "(tested)" means that there is some degree of automatic checking and reporting done on the common repository. The reports, for example, see the Juno Reports do not test "project by project" but as a whole, give some idea of where there are violations or problems. Ideally, the "common repo" tests should be simply a final sanity check, and projects can include their own automatic testing during their own build processes. See reporting FAQs for more information. Projects themselves (and/or adopters) are responsible for checking these reports ... that is, the Planning Council does not "police" compliance, the projects and PMCs do.