View source for Equinox/p2/Plan/3rd party installers on Linux

= P2 with 3rd party installers =

I'd like to make this OS independent, but some issues are very distro-specific, I'll mark them accordingly.

== Linux background ==

Linuxes have a policy of one library per system, which means that each jar is physically installed only once by a package that contains it, and all other packages using it have to 'Require' it and then properly symlink.
Eclipse is no different. In Fedora, RHEL and Mageia it comes as a set of org.eclipse bundles and a set of symlinks to other dependencies.
Example of symlinking:

sat4j is shipped as a separate package (sat4j), with files:
/usr/share/java/org.sat4j.core.jar
/usr/share/java/org.sat4j.pb.jar

Now after Eclipse is build, jars are symlinked:
org.sat4j.core_2.3.5.v20130405.jar -> /usr/share/java/org.sat4j.core.jar
org.sat4j.pb_2.3.5.v20130405.jar -> /usr/share/java/org.sat4j.pb.jar

Therefore if there is a security problem with sat4j it is enough to update one library (in most cases).

Yes, Eclipse is split amongst a number of packages which can be independently updated.


= Current problems =
# Installing Eclipse by P2 director makes the installation unverifiable and opens a security hole (some files/directories can't be checked with crc), so does running Eclipse as root (Bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=379102)
# [Linux specific] Security update of a symlinked bundle (each jar is located in the linux only once) breaks the installation. This is something I have no idea how to fix.
# Diagnosing dropins is a nightmare (dropins are used by RPM to deploy components), user is not notified if something is not installed.

= Use cases =

== Eclipse Base installation ==
Usually only platform is installed into a read-only area of the system.

== Additional features management ==
Additional features are separate projects and separate packages. After deploying them to certain folders, they should start working after next Eclipse restart. When those folders are deleted, features should disappear from Eclipse installation. No additional files should be modified (like history, timestamps, etc). This is because an admin should be able to verify that his system is not compromised by doing checksums.

== Eclipse updates [I] ==
From time to time, Eclipse or eclipse packages are updated, and the new installations overwrite the old ones. After being started, Eclipse should discover changes and update new things accordingly without bugging the user too much.

== Eclipse updates [II] ==
Eclipse should not allow for mixing RPM installed features and traditional P2 updates from update sites. What is managed by external installer, should stay such, and P2 should not try to install/update/uninstall those things. This affects especially Eclipse updates, Migration Wizard and remediation dialog.

== Eclipse updates [III] ==
If a user chooses to install things from the Eclipse Foundation, and they are not installed using the external installer, he should be allowed to do so. If he later installs the same package (even older one) using the installer, the installer version should take precedence over the P2 installed one.

== [LINUX] Security updates  ==
From time to time, CVE are discovered and certain dependencies are rebuild (tomcat, jetty, axis for mylyn etc). This causes a situation where a symlink target changes (usually MANIFEST.MF gets new version). Eclipse should discover that and reinstall as many things as possible.


= Implications =
== External dependencies ==
* feature.xml may contain version ranges rather than specific version when it comes to the external dependencies.
* feature patches may be generated on the fly.

= Solution description I =
# A droplet should be a repository in a runnable form, because 
## generating repositories on the fly is very expensive. See the existing dropins mechanism, featuring 30sec delay in some cases.
## generating repositories is also expensive in terms of the development code - there is no code we could reuse here, except dropins, which seems to generate a repo per plugin.
# After a repository is read, it will be queried for all installable units. Fake installable unit will be created to wrap all installable units from the repo. All installable units will get a requirement to the fake unit to achieve the all-or-nothing scenario.
# Security updates scenarios are postponed now.

= P2 requirements processing =
Dropins install everything as optional, which rules out the problem with duplicated bundles. This will not work for droplets, because removing one bundle would cause a removal of the entire droplets. So it is necessary to group artifacts that appear more than once in an ORed "tier", and create a dependency from the Droplet IU to the tier instead of to the particular bundle.

Considering that we have available following repos (with installable units):
* Profile  (a,b,c,d)
* Droplet1 (d,e,f)
* Droplet2 (d,e,g)
* Droplet3 (h,i)
we should get following installation requirements:
* common-d (Droplet1:d OR Droplet2:d OR Profile:d) is installed
* common-e (Droplet1:e OR Droplet2:e OR Profile:e) is installed
* Droplet1 (common-d AND common-e AND Droplet1:f) is installed
* Droplet2 (common-d AND common-e AND Droplet2:g) is installed
* Droplet3 (Droplet3:h AND Droplet3:i) is installed

= Solution description II =
P2 will never be fully interoperable with an external installer - unless the latter fully mimics P2 behaviour, which is probably not desired (if you have a product with well-defined components, you don't want to have the installation changed in an unexpected way, as part of your components may either stop working or be not installable anymore).
In Kepler, P2 got support for shared installation, and the Equinox team did not negated the need to restrict Eclipse from running in a root mode. In that light, it is reasonable to assume that only the external installer could deploy master Eclipse installation, and the user would be allowed to run it only in a shared mode. The user wouldn't be able to confuse the external installer, and user configuration would be dropped after each master installation change.
Yet another problem is how to get components deployed - I believe it could be done at the build time - two places are really important:
* bundles.info - can't be one file in a master installation anymore. It is a read-only item, and there could be one bundles.info per component. Those bundles.info files would be merged at startup, causing *very* little performance overhead (reading a small number of small text files), and give huge performance win against dropins.
* profiles - this needs to be split, too, in a similar way as bundles.info.
The protection of master installation removes a very problematic scenario - what would happen if P2 tries to write to multiple *.info or profiles. This will not happen.
== Solving security updates (Linux specific). ==
bundles.info could support 'volatile' attribute (for symlinks), and delegate handling of it to a linux specific fragment that would:
* try to load the bundle based on regexp (f.e. org.eclipse.emf.core_*.jar) - and report an error if the bundle can't be found.
* verify that the version of a file matches one stored in the bundles.info - and report an error if it doesn't, but still load it.
We would get at least diagnosed and working IDE. although with broken P2, at least until proper update of the IDE arrives. If it will be possible, this fragment could generate feature patches on the fly - but I'm not really sure if I want to implement that - I'd prefer to make it possible for users to work, but ask them to report errors.
== Tooling ==
Tooling will be required to cut full installation into components (tycho plugin, command line or maybe UI).