Difference between revisions of "Equinox/p2/Proposals/Query Management and Optimization"

Revision as of 15:20, 5 January 2009

While p2 was designed to support a number of different repositories, there are a few areas of p2 that could be enhanced to improve the performance of database backed repositories. This (working draft) proposal will help identify these areas and present concrete suggestions on how to address these issues.

There are currently four (4) areas that we have identified that can be improved:

1. Define standard queries as "non-opaque" meaning repository implementors could supply custom implementations using SQL or some other query language
2. Remove the complexities surrounding collectors. Many collectors act as queries, hampering the repositories ability to create custom implementations
3. Make query results queryable. In many instances, it would be convenient to re-query a set of query results. This may have performance implications with respect to the non-opaque queries.
4. Design QueryResults (Collectors) as a Future, that is, something that can continually gather results and the caller can check available results, listen for results and get the results. This also includes QueryStatus,

Non Opaque Queries

To help repository implementors craft custom queries to represent many of the "standard" p2 queries, a set of non-opaque (i.e. transparent) queries should be provided. These queries should have well documented semantics and allow query implementors to access the properties of the query without explicitly depending on the Query itself.

To support alternative implementations of Queries, we propose adding the following two methods to Query.java:

/**
 * Indicates whether or not this Query is Transparent.  The properties of Transparent queries can be
 * accessed via the <link>getProperty</link> method.  Each transparent query should provide a 
 * get<PropertyName>() method for each parameter. In addition to this, each property should be describe as a 
 * constant at the top of the class.  For example, the CapabilityQuery should define:
 * 
 * public static final String REQUIRED_CAPABILITIES = "RequiredCapabilities";
 *
 * public IRequiredCapability[] getRequiredCapabilities();
 *
 * This query should also override isTransparent to return "true".
 *
 */
public boolean isTransparent {
    return false;
}
 
/**
 * Returns a particular property of a given Query.  
 */
public Object getProperty(String property) {
  // Reflectively look up the property and call the getter to get it
  // If it fails, return null;
}

Implementors of IQueryable can use these methods to construct alternative mechanisms of querying their data. For example, a DB backed IQueryable may use these methods to construct an SQL statement.

class MyRepository implements IQueryable {
  public Collector query(Query query, Collector collector, IProgressMonitor monitor) {
    if (query.isTransparent() && query.getClass().getName().equals("CapabilityQuery") {
        Object o = query.getProperty("RequiredCapabilities");
        if ( o != null ) {
           IRequiredCapability[] capabilities = (IRequiredCapability[]);
           SQLStatement statement = constructCapabilityQuery(capabilities);
           executeSQL(statement);
           return results;
        }
    }
    return query.perform(getIterator, collector);
  }
}

Simplify the Collectors

Currently, many collectors have been implemented as "Queries". That is, the collectors have the logic of whether or not something should be included in the result set. This implementation allows collectors to essentially be used as "composite queries". This has a number of drawbacks, including:

1. Query implementors cannot account for this when creating alternative query implementations.
2. This adds complexity when trying to design asynchronous query results
3. Limits reuse as some queries use the Query class isMatch method while other use the collector accept method.

To address this issue, we propose re-writing complex collectors (those that override the accept method), in terms of a Query. In particular, this affects:

• LatestIUVersionCollector
• AvailableIUCollector
• LatestIUVersionElementCollector
• CategoryElementCollector
• InstalledIUCollector
• ProductQuery.Collector
• IUPropertyUtils .localeFragmentCollector
• IUPropertyUtils.hostLocalizationCollector

There are also a few other queries that provide a custom mechanism for storing the results (i.e. group IUs into categories, etc...). These should also be reviewed.

In order to support the migration of Collectors to Queries, Collectors themselves will likely need to be Queryable.

Make Query Results (Collectors) Queryable

To help simplify the collectors while still allowing them to be used as "composite queries", we propose making Query Results, (i.e. collectors) Queryable. Of course, this still limits alternative query implementations (transparent queries), and composing queries in this manner will not be enouraged. Over time, we will investigate the possibility of constructing proper "CompoundQueries".

Design Query Results (Collectors) as a Future