Equinox/p2/Proposals/Query Management and Optimization

While p2 was designed to support a number of different repositories, there are a few areas of p2 that could be enhanced to improve the performance of database backed repositories. This (working draft) proposal will help identify these areas and present concrete suggestions on how to address these issues.

There are currently six (6) areas that we have identified that can be improved:

1. Define standard queries as "non-opaque" meaning repository implementors could supply custom implementations using SQL or some other query language
2. Remove the complexities surrounding collectors. Many collectors act as queries, hampering the repositories ability to create custom implementations
3. Make query results queryable. In many instances, it would be convenient to re-query a set of query results. This may have performance implications with respect to the non-opaque queries.
4. Add "lazy" or "asynchronous" query results
5. Query result life cycle. Callers of a query will need to specify when / how the results are available, and what happens if the query gets restarted, or enhanced
6. Provide query result Status (likely MultiStatus)

Non Opaque Queries

To help repository implementors craft custom queries to represent many of the "standard" p2 queries, a set of non-opaque (i.e. transparent) queries should be provided. These queries should have well documented semantics and allow query implementors to access the properties of the query without explicitly depending on the Query itself.

To support alternative implementations of Queries, we propose adding the following two methods to Query.java:

/**
 * Indicates whether or not this Query is Transparent.  The properties of Transparent queries can be
 * accessed via the <link>getProperty</link> method.  Each transparent query should provide a 
 * get<PropertyName>() method for each parameter. In addition to this, each property should be describe as a 
 * constant at the top of the class.  For example, the CapabilityQuery should define:
 * 
 * public static final String REQUIRED_CAPABILITIES = "RequiredCapabilities";
 *
 * public IRequiredCapability[] getRequiredCapabilities();
 *
 * This query should also override isTransparent to return "true".
 *
 */
public boolean isTransparent {
    return false;
}
 
/**
 * Returns a particular property of a given Query.  
 */
public Object getProperty(String property) {
  // Reflectively look up the property and call the getter to get it
  // If it fails, return null;
}

Implementors of IQueryable can use these methods to construct alternative mechanisms of querying their data. For example, a DB backed IQueryable may use these methods to construct an SQL statement.

class MyRepository implements IQueryable {
  public Collector query(Query query, Collector collector, IProgressMonitor monitor) {
    if (query.isTransparent() && query.getClass().getName().equals("CapabilityQuery") {
        Object o = query.getProperty("RequiredCapabilities");
        if ( o != null ) {
           IRequiredCapability[] capabilities = (IRequiredCapability[]);
           SQLStatement statement = constructCapabilityQuery(capabilities);
           executeSQL(statement);
           return results;
        }
    }
    return query.perform(getIterator, collector);
  }
}

Simplify the Collectors

Currently, many collectors have been implemented as "Queries". That is, the collectors have the logic of whether or not something should be included in the result set. This has a number of drawbacks, including:

1. Query implementors cannot account for this when creating alternative query implementations.
2. This adds complexity when trying to design asynchronous query results
3. Limits reuse as some queries use the Query class isMatch method while other use the collector accept method.

To address this issue, we propose re-writing complex collectors (those that override the accept method), in terms of a Query. In particular, this affects:

• LatestIUVersionCollector
• AvailableIUCollector
• LatestIUVersionElementCollector
• CategoryElementCollector
• InstalledIUCollector
• ProductQuery.Collector
• IUPropertyUtils .localeFragmentCollector
• IUPropertyUtils.hostLocalizationCollector

There are also a few other queries that provide a custom mechanism for storing the results (i.e. group IUs into categories, etc...). These should also be reviewed.

In order to support the migration of Collectors to Queries, Collectors themselves will likely need to be Queryable.