Equinox p2 Meetings

Week of 20071008 - Thanksgiving Day in Canada

Agenda

• M3 Status
• Recap on entry points - Susan
• Eliminating XStream - Dave
• Other topics

Attendees

• Darin Wright
• Curtis Windatt
• Dave Stevenson
• Jim Miles
• Susan McCourt
• Tim Webb
• Stefan Liebig
• Scott Lewis
• Ted Habeck

Week of 20071001

Agenda

• M3 Plan recap
• Updates from the Equinox Summit
• Entry points

Attendees

• Darin Wright
• Curtis Windatt
• Dave Stevenson
• Jeff McAffer
• Jim Miles
• John Arthorne
• Pascal Rapicault
• Simon Kaegi
• Susan McCourt
• Tim Webb
• Stefan Liebig
• Scott Lewis
• DJ Houghton

Minutes

M3 Planning

• M3 about defining a final shape for things
• quick recap of plan items see Equinox p2 Plan items for M3
• We want to be in the position to self-host between integration builds
• We're integrated with the main build process now. Pascal to tag p2 projects for integration builds.
• Jim and Dave ready to work on NativeTouchpoints
• Dave interested in looking at replacing XStream

Updates from the Equinox Summit

• Governor shapes context of an install - Tim to add some API info (Some security context?)
• Discussion around Transport support for cancel and resume -- being worked on
• Need discussion on mailing list for Touchpoint Actions, Phases, replacement for JavaScript
• Need to consider commands infrastructure. IUndoableOperations seem good fit for touchpoint actions.
• Update on work on ProcessingSteps -- good shape. Fully in this week.

Entry Points

Currently there are a number of concerns that entry points seek to address:

• Distinguish what the user selected to install, versus what got installed indirectly
• Helps in deciding what to update automatically
• Helpful when the user needs to be prompted to make decisions later on, it can be expressed in terms of the IUs that the user chose to install.
• Grouping of related IUs together ("C tools" to group together "CDT" and "Autoconf tools").
• Renaming of what was installed to something more familiar to user (nickname)

Problems:

• Obscure the names of the IUs that were installed (Feature name changes when it goes from "Available Features" to "Installed Features" tabs).
• There is a version numbering problem when IUs are grouped together

Should the user be prompted when installing a new feature requires updates to features they already have installed? Should they only be prompted if entry points were modified, but not if an upgrade is required to something that was only installed indirectly?

Don't want to force the user to answer difficult questions: "Installing X requires updating Y. Continue?" There are ways to avoid putting this in the user's face, for example have a "Preview" or "Details" button that allows the power user to see what's happening, without forcing all users to make a decision.

Direction for M3:

• Entry points are just a flag to distinguish what the user chose to install from things installed indirectly.
• The concept is purely internal, and no new concept/metaphor is needed in the UI
• Grouping is a separate concern that we can add independently from entry points if the need arises (not for M3).
• We need some more exploration to decide exactly how categories will work and their relationship to entry points

Metadata repo categories

• Producer/consumer problem with category definition
• How to merge categories when multiple repositories are being browsed

Week of 20070924

The weekly meeting was canceled due to the Equinox Summit 2007. See Equinox Summit 2007 Results for discussion minutes related to p2.

Week of 20070917

Agenda

• M2 -5, what are we missing, what are the actual delivery?

Attendees

• Andrew Overholt
• Dave Stevenson
• DJ Houghton
• Jeff McAffer
• Jim Miles
• John Arthorne
• Pascal Rapicault
• Simon Kaegi
• Susan McCourt
• Ted Habeck
• Tim Webb

Minutes

Repository API Update

Large batch of changes to repository APIs and implementation has been released. This includes new processing step support in the artifact repository. API between metadata and artifact repository has been harmonized. Repository URLs for local repositories now refer to the root directory, not the artifact.xml or content.xml file (those file names are an implementation detail). Jeff hopes to get pack200 support working for M2 as proof of concept for processing steps.

3.4 M2 status

• DJ is close to getting relative paths working, so we don't need to hard-code agent path, etc. Need to figure out the story for how touchpoints contribute actions. Would like to add more engine tests for M2.
• Andrew will have a patch ready soon for shared install support
• Simon has released initial cut of new Engine API. Working on transaction support - rollback/undo.
• Some minor bugs in UI to sort out, but overall in good shape.

What do we want to package for M2?

• We want to make available the prototype end user UI as well as the admin UI.
• Could just include the end-user UI in the admin UI for M2 - showcase example is to use the end user UI to update bundles in the admin UI RCP application.
• Deliver for M2: Generate metadata for Eclipse update site. Be able to install the SDK using the bundle content that is already in the Eclipse update site. Also generate metadata/artifacts for p2 bundles, and zip file for agent RCP application.

Week of 20070910

Agenda

• Artifact key format: should it be a hash, or a set of strings like it is today?
• Self hosting: who is running in self hosting mode, is it stable/working, when should one bother to run it (Susan)
• Reconciler for shared install: what object to use as the "shared" profile? (Tim)

Attendees

• Pascal Rapicault
• Simon Kaegi
• John Arthorne
• Dave Stevenson
• DJ Houghton
• Tim Webb
• Dennis Vaugn
• Ted Habeck
• Susan McCourt
• Andrew Overholt
• Jeff McAffer
• James Miles (Jim)

Minutes

Reconciler / Shared Install

• Main issue is reconciliation at startup
• Some discussion around treating Profile as an IU (or at least a mechanism to serialize a Profile as an IU)
• Shared portion is definitive.

Become

• Some discussion on difference between "become" and "flush and install of toplevel IU".
• "become" should fail if the IU is not a profile.

Self Hosting

• limited use (Jeff and Susan) but eventually there should be more use.
• When metada format changes a flush of existing metadata is required.
• Potential problems around restoring persistent info. DJ to look at adding persistentce to repos added to the managers.

Artifact Key

• Ultimately unresolved and still requires further discussion.
• Is the Key a true opaque URI? How can we do mappings from the Key to the actual resource in a repository.

Week of 20070903

Agenda

• Handling of fragments: there are fragments that are mandatory (e.g. SWT), fragments that are recommended (e.g. org.eclipse.core.filesystem ones) and fragments that are optional (e.g. the ant.optional.junit obtained from CDT). How can we distinguish between these fragments?
• Event Admin, should we use it or not (https://bugs.eclipse.org/bugs/show_bug.cgi?id=184021)?
• 3 weeks from M2, where are we?

Attendees

• Pascal Rapicault
• Simon Kaegi
• James Miles (Jim)
• Stefan Liebig
• John Arthorne
• Susan McCourt
• Dave Stevenson
• DJ Houghton
• Jeff McAffer
• Andrew Overholt

Minutes

Engine API

• Recap of Engine API discussion that was held after the call
• API changes in place by end of next week

EventAdmin discussion

• We would use it only as a backing implementation
• Continue using our own APIs Event Types e.g. we would not use the EventAdmin API directly
• Events destined for a particular topic should be of one type so that we avoid instanceof checks

Fragments optional vs. requirement

• Cardinality problem e.g. SWT requires at least one (or perhaps exactly one) fragment to provide native integration
• Another examples are the NL case, filesystem
• Is there a generic mechanism to describe these?
• Filters might be a way to support this. The "host" would have to specify the "must" requirement.
• Decision types: "must", "should", "may", "not?" -- no nots seemed to be consensus
• There is nothing specific to fragments with this problem. e.g. it applies to the establishing of "any" relationship between IUs

Week of 20070827

Agenda

• Entry points and profiles, those higher-order level IUs we need to be able to manipulate
• Engine what is the API? How do we modify Profiles?
• Transactions, how pervasive are they, how are they performed? What do we need?

Attendees

• Pascal Rapicault
• Jeff McAffer
• Simon Kaegi
• James Miles (Jim)
• Andrew Overholt
• Timothy Webb (Tim)
• Stefan Liebig

Minutes

Engine API and Transactionality

• We discussed the change in model for how changes are applied to a Profile. Instead of one operation and many operands we now use many self contained operations that apply to a single profile.
• Early discussion of how we can create a Session to encapsulate a transaction.
• Phases and operations and how they relate were talked about. One thing we have to be careful about is mixing up the use of the Engine API with the Director API.
• Various ideas around using operation where we run a partial set of phases.
• Pascal, Jeff, Jim, and Simon to discuss further and update this page later.
• (Update) Key findings:
  • Install, Uninstall, Update operations for now.
  • Fixed set of phases
  • Phases are executed _breadth_ wise
  • Session is created from Engine and is used to create a transactional scope.

Entry Points

• Entry Points are an encapsulation of the IUs installed at the director level
• There are tough problems around handling of entry points are IUs that group others. Shallow vs. deep search for updates.
• Discussion around if Profiles can be treated as IUs. Key use cases include the desire for one developer to use the same profile as another.

Naming

• Candidates: QM, Hive, Provo, p2, Phoenix
• p2 won. Renaming can be done post M2 -- the day after preferably.

Week of 20070820

Agenda

• We need a lot of events, some are core to the functioning of the system, others are to help drive the UI. Currently we have our own event bus. Should we migrate to the osgi event service?
• Related to events, we need some event generation guidelines. See the concerns raised in bug 199806.
• Naming. We have had some discussion around the naming of the work we are doing. Currently it is "The Equinox Provisioning support" or some such. This is very cumbersome for discussions, presentations, documents, ... The package name we've been using is o.e.equinox.prov.*. Before we get too too far we need a real name that we can live with through graduation. By the time of the meeting hopefully we will have had some discussion on the mailing list and be able to conclude with a decision.
• Carried over from last week's agenda: shared install

Attendees

• John Arthorne
• DJ Houghton
• Simon Kaegi
• Susan McCourt
• James Miles (Jim)
• Andrew Overholt
• Pascal Rapicault
• Thomas Watson
• Timothy Webb (Tim)

Meetings

Naming

The naming discussion was deferred because Jeff McAffer, who proposed the item, was not present

Event handling

• The OSGi event service is not currently included in the SDK, so we would need to add it. Adding a listener requires registering a service.
• Could add our own event API and use Equinox service as an implementation detail
• Want to be able to batch fine-grained events
• Concurrency issues, syncExec into UI thread, locking, etc
• John to take a look at OSGi event API

Shared Install

• Lots of different kinds of shared install setups. Spectrum from complete locked down where each user must have exactly the same thing, to a more permissive approach where there is a "profile" of shared stuff, and then each user can add custom things on top
• Terminology point: Maya calls the shared stuff the "system profile"
• Need to figure out semantics of relationship between profiles - profile hierarchy, inheritance, etc
• Reconciling changes to shared profile on startup
• Tom raised issue that cascading Equinox profiles is problematic. Profile inheritance doesn't necessarily imply multiple Equinox profiles though
• Various parties want to plug in and take over the process of reconciling the profile on startup (Maya, Lotus, etc)
• In some shared installs, want to unequivocally upgrade the shared components and throw away the old version. Users are no longer allowed to run on old versions. In other installs, policy may be more lenient and allow the user some control over if/when they upgrade. We should support both.
• Discussion around profiles - whether Equinox profiles and Maya profiles are the same. Can a profile be represented as an installable unit?

Week of 20070813

Agenda

• Separation of concerns around the artifact repository (https://bugs.eclipse.org/bugs/show_bug.cgi?id=197644).
• Shared install:
  • How much of the things noted in the discussion (http://dev.eclipse.org/mhonarc/lists/equinox-dev/msg02692.html) are provisioning related vs runtime related (location of preferences, log files, cache, etc.)
• Controlling the Engine:
  • The engine currently takes an operation to install, or uninstall IUs. Can this separation of installation / uninstallation / update / etc. be kept? How do we reconcile?

Attendees

• DJ Houghton
• Jeff McAffer
• Simon Kaegi
• Pascal Rapicault
• John Arthorne
• Susan McCourt
• Dave Stevenson
• James Miles
• Patrick Dempsey
• Stefan Liebig
• Ted Habeck
• Joe Rusnak

Minutes

We initially attempted an Asterisk conference call, but the line quality was not acceptable. We moved all participants to a private number until we find an acceptable way to do a public call.

Artifact repository

We need to be able to perform post-processing steps after download, such as unpack-200, re-assembling compressed deltas. Should the repository be aware of these post-processing steps, or should it be done independently? When mirroring between repositories, some post-processing steps are not necessary (for example unpacking zips/jars

In our discussion, we distinguish between original bytes (as they are added to the repository from build system), and stored bytes (the actual bytes stored on disk in the repository). When mirroring between repositories it is more efficient to directly mirror the stored bytes, rather than the original bytes.

We should have different artifact descriptors for each available representation of an artifact (jarred version, delta compressed version, etc). Leave it up to download manager or some other entity on the client to decide which artifact descriptor it wants to load for its current circumstances. The artifact key uniquely identifies the original bytes, whereas the artifact descriptor identifies various flavors of stored bytes.

Two main methods for artifact repository API: getArtifactBytes(IArtifactKey), returning the "original bytes", and getArtfactDescriptors(IArtifactKey), which returns all the available descriptors corresponding to the given key.

Tangible next steps are to implement post-processing steps for pack200 and signature verification.

Shared Install

Discussion skipped because Andrew Overholt is leading this work and wasn't present

Engine discussion

Currently the engine API only allows a single operation to be performed at once. We may want to be able to pass a batch of operations to the engine and have them be performed together (atomically).

Currently the interesting units are ProvisioningOperand and Phase. An EngineOperation just provides an ordered list of phases. The operands actually provide the interesting behavior (install, uninstall, update).

Could we specify the engine API instead in terms of states, and an operation is just a request to transition from one state to another. The precise phases required for that state transition are implementation details.

On a multi-user install, some phases are no-ops because they have already been performed before by another user.