Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Equinox/p2/Query Language for p2"

< Equinox‎ | p2
(An alternative way of doing things)
(An alternative way of doing things)
Line 101: Line 101:
 
Passing instances of the IQuery to an IQueriable is problematic since it is very difficult for the implementer to figure out what the IQuery does. When the implementation is based on a database or a remote server, the query needs to be converted into some kind of generic expression. Some special code would need to be written that would make a best effort. Perhaps it could recognize some queries and then process others locally. Often with very varying performance. Executing a query locally means passing all data over the net or extracting all tuples from a database. For huge repositories, that will be a very resource intensive operation.
 
Passing instances of the IQuery to an IQueriable is problematic since it is very difficult for the implementer to figure out what the IQuery does. When the implementation is based on a database or a remote server, the query needs to be converted into some kind of generic expression. Some special code would need to be written that would make a best effort. Perhaps it could recognize some queries and then process others locally. Often with very varying performance. Executing a query locally means passing all data over the net or extracting all tuples from a database. For huge repositories, that will be a very resource intensive operation.
  
The IQuery does have one big advantage though, it can contain java code that does virtually anything to filter the result or even perform actions based on the result. Needless to say, such filtering, or such actions, cannot be performed by a database or a remote server. So clearly, the concept currently encapsulated by the IQuery/IQueryable is needed.
+
The IQuery does have one big advantage though. It can contain java code that does virtually anything to filter the result or even perform actions based on the result. Needless to say, such filtering, or such actions, cannot be performed by a database or a remote server. So clearly, the concept currently encapsulated by the IQuery/IQueryable is needed.
  
 
Here is a proposal on an alternative solution that would get the best of both worlds. In essence, each query would start with the evaluation of a query expression. Then, if needed, the query result is further processed by a construct very similar to the current IQuery/IQueryable.
 
Here is a proposal on an alternative solution that would get the best of both worlds. In essence, each query would start with the evaluation of a query expression. Then, if needed, the query result is further processed by a construct very similar to the current IQuery/IQueryable.

Revision as of 17:25, 16 November 2009

Background

p2 has a query mechanism today that makes it very hard to create an implementation that is based on a database. It is also very hard to create an efficient client/server solution. The reason for this is that most of the queries are written as java code in a callback and there is often no good way to extract the semantics of the query.

We discussed this at length on the p2 meeting on November 9. This resulted in the bugzilla Create a QueryLanguage for p2. I've spend some time on this now and today I have a fully functional IQuery implementation.

Considerations

When designing the language I tried to consider two things. Who is the primary user and which are the most common queries?

Java style operators

The primary user is typically well aqquinted with Java so I opted for using java operators such as '&&' and '||' rather then SQL'ish keywords like OR and AND. I also use '.' for member access and '[' ']' to get indexed access. The latter can also be used to get keyed access.

Matches operator

A large amount of queries involve versions, version ranges, and capability matching. So managing that is important. Another thing that is important is to be able to support filtering based on ID's. All of this is now also built in to the language through the matches operator '~='. It can be though of as the 'satisfies' method on an installable unit or provided capability, as 'is included in' when used with a version and version range, as a plain matches when used with a string and a pattern, or as 'instanceof' when comparing with a class.

The following things can be matched using the operator.

RHS LHS Implemented as
IProvidedCapability IRequiredCapability lhs.satisfies(rhs)
IInstallableUnit IRequiredCapability lhs.satisfies(rhs)
Version VersionRange rhs.isIncluded(lhs)
IInstallableUnit Filter rhs.matches(lhs.properties)
Map Filter rhs.match(lhs)
String Pattern rhs.matcher(lhs).matches()
<any> Class rhs.isInstance(lhs)
Class Class rhs.isAssignableFrom(lhs)

Latest

Some queries must make sure that the result only contains the latest version of each found IU. I added a special keyword 'latest' to make that possible.

Combined queries

It must also be possible to combine queries (hence && and ||).

Query parameters

The query must be parameterised so that expression parsing can be done once and then executed multiple times.

Java API

The expression tree created by the parser must be well documented and easy to use so that queries can be created programmatically.

Performance

Both the parser and the evaluator must be very fast. Performance must be comparable with the current solution.

Examples

Think of the query string as something that would be entered after the WHERE keyword in an SQL query, i.e.

SELECT self FROM repository WHERE ...

The query language will only cover the '...' part since the rest is implicit.

Query for all IU's that has an id:

id = :1

Query for the latest IU of some specific id:

latest id = :1

Java code to query for the latest IU that matches a specific version range (note the matches operator):

IQuery query = new ExpressionQuery("latest id = :1 && version ~= :2", id, range);

Java code to query for an IU that has a specific property set:

IQuery query = new ExpressionQuery("properties[:1] = :2", key, value);

The same query, but this time for multiple possible values:

IQuery query = new ExpressionQuery("properties[:1] = ANY :2", key, new Object[] { v1, v2, v3 });

Query for all categories found in the repository:

IQuery query = new ExpressionQuery("properties[:1] = 'true'", IInstallableUnit.PROP_TYPE_CATEGORY);

Query for all IU's that fulfil at least one of the requirements from another IU.

IQuery query = new ExpressionQuery("ANY providedCapabilities ~= ANY :1", iu.getRequiredCapabilities());
OR
IQuery query = new ExpressionQuery("self ~= ANY :1", iu.getRequiredCapabilities());

Query for the latest version of all patches:

IQuery query = new ExpressionQuery("latest self ~= :1", IInstallableUnitPatch.class);

Query for all IU's affected by a patch:

IQuery query = new ExpressionQUery("self ~= ANY ALL :1", patch.getApplicabilityScope());

I think it would be valuable if users of p2 that have the need for specific queries could list them here. It is not unlikely that some queries might motivate some extension to the current syntax or to how the evaluator works.

An alternative way of doing things

Passing instances of the IQuery to an IQueriable is problematic since it is very difficult for the implementer to figure out what the IQuery does. When the implementation is based on a database or a remote server, the query needs to be converted into some kind of generic expression. Some special code would need to be written that would make a best effort. Perhaps it could recognize some queries and then process others locally. Often with very varying performance. Executing a query locally means passing all data over the net or extracting all tuples from a database. For huge repositories, that will be a very resource intensive operation.

The IQuery does have one big advantage though. It can contain java code that does virtually anything to filter the result or even perform actions based on the result. Needless to say, such filtering, or such actions, cannot be performed by a database or a remote server. So clearly, the concept currently encapsulated by the IQuery/IQueryable is needed.

Here is a proposal on an alternative solution that would get the best of both worlds. In essence, each query would start with the evaluation of a query expression. Then, if needed, the query result is further processed by a construct very similar to the current IQuery/IQueryable.

Simplified, the solution would be based on the following two interfaces:

public interface IQueryable {
   Iterator query(String expression, Object[] args, IProgressMonitor monitor);
   Iterator query(Expression expression, Object[] args, IProgressMonitor monitor);
}

public interface ICollector {
   Collector collect(Iterator iterator, Collector collector);
}

The ICollector is basically the same thing as the current IQuery. The key thing here is the separation of concern. An IQueriable can return a result based on a predicate. The ICollector does something with that result.

Today, you would write something like:

MyQuery query = new MyQuery(collectorArgs, queryArgs);
Collector coll = repo.query(query, new Collector(), monitor);

If the query does things that must be done locally and is not covered by the query expression language, this would now be replaced by:

MyCollector collector = new MyCollector(collectorArgs);
Collector coll = collector.collect(repo.query("some expression", queryArgs, monitor), new Collector());

The BNF

p2query : 'latest'? orExpression ;

orExpression : andExpression ( '||' andExpression )* ;

andExpression : notExpression ( '&&' notExpression )* ;

notExpression
   : '!' notExpression
   | binaryExpression
   ;

op : '=' | '!=' | '>' | '>=' | '<' | '<=' | '~=' ;

binaryExpression : sideExpression op sideExpression ;

sideExpression : ('any' | 'all')* memberExpression ;

memberExpression : unaryExpression ( ( '.' ID ) | ( '[' memberExpression ']' ) )* ;

unaryExpression
   : '(' orExpression ')'
   | '[' memberExpression ( ',' memberExpression )* ']' // #array construct
   | '/' regexpPattern '/'
   | STRING
   | INT
   | parameter
   | 'self'
   | 'null'
   | 'true'
   | 'false'
   | ID // # implies self.identifier
   ;

   parameter
   : ':' INT
   ;

The expression tree model

P2ql.png

Back to the top