Difference between revisions of "SimRel/Simultaneous Release FAQ"
(→Mark "offset" category in portal tracking data)
|Line 268:||Line 268:|
[[Category:Juno| ]] [[Category:Indigo| ]] [[Category:Coordinated]]
[[Category:Juno| ]] [[Category:Indigo| ]] [[Category:Coordinated]]
Revision as of 12:21, 1 July 2012
- 1 Introduction
- 2 How do we join the simultaneous release train
- 3 What are the most important URLs
- 4 Where can the common repository be tested, before it is rolled out for a milestone or release?
- 5 What is the staging repository?
- 5.1 What is staging, for the current, most forward looking, yet-to-be-released stream?
- 5.2 What is staging, for the maintenance stream?
- 5.3 What's the best way to test with the staging repository?
- 6 Once I update my .b3aggrcon file, how can I start a build?
- 7 A build failed message says it can not find xyz.feature.group, but I have nothing with "feature.group" in the name?
- 8 A build failed message says it can not find version 1.2.3.v9 but I can see 1.2.3.v9 on the file system?
- 9 How is a final build made "invisible" until release?
- 10 I see that xyz is being pulled into the repository, how can I see who is pulling it in?
- 11 What if we need a rebuild, past our deadline?
- 12 Policy FAQs
This page is to document answers to frequently asked questions about the yearly Simultaneous Release process or build.
How do we join the simultaneous release train
Pretty much any project that wants to join the yearly release train can and are encouraged to. Of course, it does sometimes take more work ... it is not just a matter of timing. But, often makes things much easier for your adopters and users. You should discuss with your mentors and PMC if you have any doubts or to make sure they agree its appropriate for your project.
Be aware, too, that there are conceivably "limits to growth", but hopefully we will not encounter such limits that can not be solved by improved resources and processes. A rough document of [SimRel/Priorities priorities for the yearly simultaneous release] has been drafted by the Planning Council, to help communicate and be open and transparent about some of the potential future limits.
Mark simultaneousrelease flag in portal =
You need to mark the simultaneousrelease flag in the Foundation's Portal metadata for your project ... by M4, at the latest.
Mark "offset" category in portal tracking data
After you mark that simultaneousrelease flag, the project lead should see some extra "tracking" metadata at the Foundation Portal.
Click "track" next to your project. As of Juno, we've deprecated most of the information in the Simultaneous Release Tracker. The one bit of information that you really need to provide is the offset. You can safely ignore everything else (the EMO is working on a replacement implementation that is expected to be available in the Kepler timeframe).
The information that you provide here is displayed on the Releases page and the Release Descriptions page. Note that the descriptions are mined from the project description. The Project Metadata page provides help on specifying and updating your project's description (note that the EMO is also working on making this a lot easier).
The Releases page provides a listing of projects participating in a particular release. You can navigate to previous releases directly on the page (it shows the current simultaneous release by default).
Note that the project release number is displayed along with the offset. This page picks up the project releases provided in the project metadata that occur on the date of the simultaneous release. Note that the name provided for the release is normalized to a three part version number, release names that do not contain numbers, or are labelled as a milestone (e.g. "1.0.0M2") are ignored.
How to decide if offset is +N category
It partially depends on who you depend on. The Platform is +0, so nearly everyone else is +1, +2, or +3. For example, if you highly depended on "webtools" (which is +2) then you'd have to be a +3.
It also depends on "who depends on you". If no one else, in release train, depends on building with you as a pre-requisite, then +3 would be a good choice.
Of course, the lines are not always clear and pure ... such as part of emf is required at +0 but part of it is later at +1. In such cases, projects usually mark themselves as +1 and "work out the details" with those that need things at +0. Another example, part of a project such as WTP might depend on part of DTP and DTP might depend on part of WTP, but both need to be +2 to "fit in". In cases such, the projects might have to work out plans or agreements about having the sensitive, overlapping parts to be stable early (in both API and version numbers) so neither has to build against the literal final delivery of the other.
Keep in mind, the +N category designation just signifies the last possible time to drop (at least, without notifying others on cross project list). You are welcome and encouraged to have a "warm up drop" a bit earlier, with your "near final" bits, just to see if anything breaks or effects others.
That's too easy, how do we really join the simultaneous release train
Setting the simultaneousrelease flag and offset category is the first step, the formal step in declaring your intent. Of course, your PMC should be well aware of your plans, as always. Some people like to post a note to cross-project list, which is not necessary, but does not hurt anything either, if you just want to keep everyone informed. Then, of course, the real work begins.
There is a whole list of things you have to do, some required, some recommended, and, you need to get yourself into the common repository aggregation build, by M4, at the latest for new projects (existing projects continuing from previous years should already be in, as part of the intent it to be "simultaneous" every milestone, not just at the end. You will want to read through these documents, and others at the main wiki category (e.g. for Juno) as there is a lot of work involved ... more than just "releasing at the same time of year". And, if it is after M4, there is an exception processes, but you should talk to your PMC and Planning Council representative to see if you have a reasonable case for your Planning Council representative to take to the whole council.
What are the most important URLs
Where can the common repository be tested, before it is rolled out for a milestone or release?
The best place is the "staging area", by adding this URL to your "available software sites" list.
Note: to get the best test, disable all other repositories in your list, or you might end up pulling something from some other repository, not staging. Be aware that moving a specific build to "staging" may happen only every few days (if you need something promoted more urgently, just ask on cross-project list).
What is the staging repository?
Conceptually, it is simply a place to hold a repository temporarily ... until the repository is promoted to a released location.
What is staging, for the current, most forward looking, yet-to-be-released stream?
Not every build-repository makes it to this staging repository, but they do fairly frequently. Before the actual yearly release, a common milestone aggregation build is moved from 'staging' to the 'releases' area. For example, once, for each final milestone, staging is moved to
What is staging, for the maintenance stream?
Not every build-repository makes it to this staging repository, but they do fairly frequently. There are no milestones for maintenance streams, so this is as far as a maintenance repository gets, until it is time for SR1 or SR2 to be declared at which time 'maintenance' is promoted to be part of the composite for the release, such as
What's the best way to test with the staging repository?
There are a couple of good tests to do, before a staging repository is released:
Test staging all by itself
Just add the staging repository to the available software sites, using preferences, and ... very important ... make sure all other sites are disabled. This then gives you the best view that everything in that one repository is correct, and all dependencies can indeed be found, and things show up in categories as expected.
After confirming the categories are as expected, usually the next test is just to install things, fresh, but often it is a good idea to test various update sceneries, to make sure things work as expected (for example, if you already have M6 installed, then once M7 is ready, test to be sure it updates to M7 as expected.
Test staging as a pseudo composite
After confirming staging repository is correct, its often useful to then also (re)enable the released repository. Then, on install dialog, you can select "all available sites" and this effectively simulates what the eventual, final composite repository will look like to end users. There are cases, especially during initial development, where invalid features will show up. For example, if a feature was removed, renamed, its version reduced, or its category changed from one milestone to the next, it might still show up in the composite, and that might interfere with correct installation (see bug 314165), if not merely be confusing to end users. If there is a serious problem due to the composite, please open a bug in cross-project component and we'll decide if the composite should be changed or reduced to allow for correct installation.
Test EPP Package updates
For a release, EPP packages are added to the common repository via composite, so its exact (final) location is transparent to end-users.
But before a release, similar to above, you can use a pseudo composite to test if an EPP Package updates as expected. You'd need to follow (or ask on) the epp-dev list to know details, but in addition to the above staging repo, you would add (and enable) EPP's staging repo, which would usually have a form similar to the following:
Why do we use composites anyway, if there are potential problems with them?
Two reasons. While the eventual, initial released repository, in June, will definitely consist of just one set of released, consistent features, later in September, and then February, we use composites to add maintenance, so testing composites early is a good idea to make sure there are no bugs in p2, etc., that need to be fixed.
Secondly, someone may be using a previous milestone as a runtime target, and once we remove it, it will invalidate that target, so we do not want to force everyone to "move up" all at the the same time. Some developers may need a few weeks to transition to the latest code. To save space, we do not try and save all previous milestones, though. Our goal is to maintain about 3 milestones in the composite. For example, if we had M4, M5, and M6 in the composite, once M7 came out, we'd only have M5, M6, and M7. But the number can not be guaranteed. If there are serious problem with providing the composite we will reduce it to just two, or even to just one, to make sure the repository is usable for testing and continued development.
Once I update my .b3aggrcon file, how can I start a build?
You don't need to. The build will start automatically, once you check in a .build file. The .build files are checked every 15 minutes to see if any have changed, and if so an aggregation build will start. It takes approximately 2 hours to run.
But what if I really want to kick off the build myself?
If you can't wait 15 minutes, you can start the build your self. Anyone that has authorization to check in a .build file, should have authority to manually start a build.
Plus, there are some cases where someone may need to kick off a build manually. For example, if a build fails due to network issues. Another common case is that a build may fail, even though the .build file is correct, the repository it points to might have had an error. Once the repository is corrected, there's no automatic mechanism to detect that change, so after the repository is corrected, a new build has to be manually started (that, or the .build file "touched" and then checked in again). To manually start a build, just click the "Schedule a Build" button at the build status control page.
You need to be sure to login (with your committer ID):
Then click the "Schedule a Build" buttton:
A build failed message says it can not find xyz.feature.group, but I have nothing with "feature.group" in the name?
The suffix ".feature.group" is added to feature names, to refer to the whole feature ... the feature files themselves, but also all its included bundles and features ... to help distinguish it from the literal feature files. So "xyz.feature.group" just means the "xyz feature", conceptually. See Eclipse Help for detailed information about metadata.
A build failed message says it can not find version 1.2.3.v9 but I can see 1.2.3.v9 on the file system?
The key file, the one that "drives" P2, is the content.jar/xml file. Be sure to check the version numbers there. If, inside it, the installable unit (often a feature) says version="1.2.3.v8", then P2 will look no further and conclude that the 'v9' feature it is looking for is not there. This is usually a sign your meta data needs to be re-generated to match the contents on the file system.
How is a final build made "invisible" until release?
Web download pages?
You can put the zips in their download directory, so those large items can replicate to mirrors, but don't use any HTML that would cause them to be displayed as links to an end user. True, if someone knew the exact URL they could still get it, but the idea is that the URL is not widely announced or visible, so even if a few download it, it is not hit by thousands of downloads. Then, on release day, you'd update the HTML pages to make the downloads visible to browsers and click-able by users.
Tip : you can hide some particular builds from download pages using the hidden.txt file in downloads directory ( see for example emft : http://dev.eclipse.org/viewcvs/viewvc.cgi/www/modeling/emft/downloads/hidden.txt?root=Eclipse_Website&view=log )
There is a few ways to accomplish this, depending on if you have composite or simple repositories, but they all involve promoting the "main" parts of the repository (the artifacts, usually "plugins" and "features" directory) to their final location so those meaty parts can be replicating to mirrors, but do not put the metadata (usually content.jar and artifacts.jar) to their final location until "release day". p2 doesn't "see" the artifacts, until it can read the metadata.
I see that xyz is being pulled into the repository, how can I see who is pulling it in?
There are times when someone may see an old, or inappropriate, bundle being "pulled in" to the aggregation build, and they want to find out how or why it is being included. While the full rules for how p2 decides something is required, or satisfies a requirement, is beyond the scope of this FAQ, there are two ways to get started narrowing it down. Just to have a concrete example, I will use the use-case provided by bug 357171 to show a specific case, where some old (unsigned) version of org.eclipse.emf.teneo.hibernate.libraries was being included.
The quick approximate check of where its coming from
Check the console log for a message about mirroring the problematic bundle, and then look back "up" in the log, to see what repository is being processed at that point.
For, our example, searching for 'org.eclipse.emf.teneo.hibernate.libraries' we find the following line:
[exec] - mirroring artifact osgi.bundle,org.eclipse.emf.teneo.hibernate.libraries,1.0.1.v200907090915
Then, looking back "up" in the log, we see the following repositories were being used to pull things from:
[exec] Mirroring meta-data from from file:///home/data/httpd/download.eclipse.org/modeling/emf/updates/interim [exec] - mirroring meta-data reference http://download.eclipse.org/modeling/mdt/updates/ [exec] - mirroring artifacts reference http://download.eclipse.org/modeling/mdt/updates/ [exec] - mirroring meta-data reference http://download.eclipse.org/modeling/emf/updates/ [exec] - mirroring artifacts reference http://download.eclipse.org/modeling/emf/updates/ [exec] - mirroring meta-data reference http://download.eclipse.org/modeling/emft/updates/ [exec] - mirroring artifacts reference http://download.eclipse.org/modeling/emft/updates/ [exec] Mirroring artifacts from from file:///home/data/httpd/download.eclipse.org/modeling/emf/updates/interim
In some cases, this might be sufficient to track down the problematic contributor.
The detailed, exact check on requirements
If the log doesn't help, the definitive source of "requirements" can be seen in the content.jar/xml. It can be tricky to "find" the content.jar/xml file, depending on composites repositories, if compressed or not, etc., but for our example, from a temporary maintenance repository, you can download it from
Once unzipped, you can search for your bundle, in our case. searching for 'org.eclipse.emf.teneo.hibernate.libraries'.
You'll certainly find the IU that defines the bundle
<unit id='org.eclipse.emf.teneo.hibernate.libraries' version='1.0.1.v200907090915' singleton='false'>
That IU, by itself, doesn't help narrow down who is requiring it. But, continue to search.
The easy case of "Require-Bundle"
If you find a match in a "requires" element, you can see what bundle "requires" your bundle. If found, those matches would start with "<required" element and include the name of the bundle. In these cases, there is some bundle that uses "Require-Bundle" and should be easy to resolve with who ever is doing that.
The harder case of "Import-Package"
If you find no direct "<required" match, then that means p2 is deciding your bundle "fits" the requirements due to the packages it provides (exports), and some other bundle (or bundles) "imports" those packages. If there are just a few, uniquely named packages provided by your mystery bundle, you can search the content.xml file for who "requires" those packages and figure out what to do from there.
The hardest case
But, our example does not have such a happy ending. The IU 'org.eclipse.emf.teneo.hibernate.libraries' provides 264 packages! Some of those are fairly distinctly named, but some are very common, and very commonly "imported" by other bundles. While I did not try, it would be hard to narrow down to just a few other bundles that might be requiring the many exported packages. I do not know the exact details, but p2 does have some heuristics where it will give extra weight to include a bundle that satisfies the most requirements -- so, any bundle that provides (exports) a huge number of common packages is in danger of getting included unintentionally by p2.
Back to the quick approximate findings
Another approach is to go back to those repositories implied in the "The quick approximate check" above.
Ideally, it would be possible to provide a more specific repository for the b3 aggregator so it only looks in more currently correct repositories. For example, (and, I emphasize 'example', since I am unsure of Modeling project's policies and procedures) a repository for "../emf/updates/indigo/" might be more appropriate for indigo contributions than a high level "../emf/updates/" repository. Or, as another example, perhaps "../emf/updates/interim" repository is actually incorrect for Indigo contributions. In that case, you could search the b3aggrecon files to see who is using or providing that repository and work out a better alternative.
A caution on how not to solve
For cases such as this, where some old, inappropriate bundle is being included, one solution, one might think, would be to surgically remove it from its original repository. This is almost never a good idea, since someone, somewhere, might be depending on it being there ... released repositories are forever, or, at least, are supposed to be ... so great care is needed in changing released repositories.
What if we need a rebuild, past our deadline?
Everyone knows plans are just plans, and occasionally exceptions are required. While we can't cover every exception (after all ... they are exceptions) there are the some general principles of deciding if an exception is required. The same ideas apply to milestones, release candidates, and service releases, but the impact levels are higher the further along a cycle is. If possible, naturally, it is best to handle problems within your own project, say, by providing a patch feature or update to your own project's software repository, but in some cases, that's not possible ... this FAQ entry is about those cases.
Questions to ask yourself (and answer, in your request for an exception or rebuild)
Not necessarily an exhaustive list ... but, some common questions:
- Is the bug something that effects IP or similar "legal requirements" from the Eclipse Foundation?
- Is the bug something that prevents other projects from working correctly?
- Is the bug something that causes install, or update, to fail, or otherwise leave an installation in a bad, unfixable state?
- Is the bug something that can not be solved by a patch feature, applied by users or adopters after the release?
- Is the bug a regression, from previous release?
- Do other projects have to recompile, once a bug is fixed? For example, are constants changed? APIs? version numbers?
- Do other projects need to retest, once a bug is fixed? That is, is it something that could potentially effect others ... such as a change in timing or synchronization of some "notification" to listeners? Or something clearly "internal" to your own code?
- Do you have the support, approval, or review of your Project Lead and PMC? (Or, otherwise follow what ever your project's rampdown process is?)
- Does it affect an EPP Package? Or just the common repository?
Past our +n day, but before window closes?
If we are still within the "drop window" for a deadline, but you are past your particular +n day, simply post a note to cross-project list, with any relevant questions and answers from above list, the bug number, and then just do it. (No need for further approval or coordination.)
Keep in mind that spinning new builds past your deadline can result in a lot of work for downstream projects and consumers as they make last minute adjustment to your change. Projects are generally willing to accommodate these changes as much as they can, but please keep this in mind and only do it when absolutely necessary.
Past the drop window?
In this case, it is completely past the drop window, after EPP packages have been built. In this case, you still need to post to cross-project list, with bug number, and relevant questions and answers from above list, but now explicit review/permission from Planning Council is also required. Please follow the Planning Council Exception Process, but in cases of "tight timing", the Planning Council has authorized the Planning Council Chair (currently David Williams) to make the initial decision and allow others to review later or in parallel.
Can a new project or feature join a Simultaneous Service Release (SR1 or SR2)?
Yes, but still subject to all the other rules of the Eclipse Development Process and the Simultaneous Release Requirements. For example, if a "Release Review" is required by the EDP, it is still required, before being in an SR. A release review is normally not required for a maintenance release, if the maintenance is "bug fixes" only, but if a new feature is added (including an increase in minor version number) then according to the Eclipse Development Process a release review is required. All that is independent of the "Simultaneous Service Release" and Planning Council ... just normal Eclipse business. Similarly, something "new" must still meet all the other Simultaneous Release requirements, such as signed jars, 4 part versioning, etc. In particular, the addition must "do no harm". In practice this means it is relatively easy to change "leaf components", but more care and effort is required to change a "low level" feature.
Can a Simultaneous Release project include bundles or features from a project not in the Simultaneous Release?
It can "include" another Eclipse Project that is not in the Simultaneous Release, if that other other Eclipse Project has been released before, and if that other Eclipse project (still) meets all the requirements of a Release (such as correct about.html files, etc.) and also the extra requirements of Simultaneous Release (such as signed jars, etc.,). This is analogous to the use of "third party" bundles from Orbit, where the original authors clearly do not "participate" in the release ... but the bundles have all been through the IP process, are well formed, etc. Of course, it it only natural and polite to let the other Eclipse Project know that you are planning on doing this, and give them the right to comment or object if they had some reason too (for example, they might say, "oh, we didn't know you were using us, ok, we'll join the Sim. Release too" or they might say, "Please don't, we plan to come out with a new, incompatible release in August and contribute that in SR1" ... just to give some hypothetical examples of the importance of communication.
But, if a project has not been released before, then another project can not "include" it in their own released code, even for a normal Eclipse Foundation Release, much less a Simultaneous Release. For the sake of history, I'll note this topic was discussed at length in bug 370974.