Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "VIATRA/Query/UserDocumentation/API/Advanced"

(Using Filtered Input Models During Pattern Matching)
(44 intermediate revisions by 5 users not shown)
Line 1: Line 1:
= Advanced EMF-IncQuery API Features =
+
= Overview  =
  
== Lifecycle management ==
+
This page overviews advanced use-cases for EMF-IncQuery. The topics cover
  
=== Unmanaged engines ===
+
*the '''Generic API''', to be used when the features supported by the generated API do not suffice (e.g. when working with dynamically defined patterns, or patterns whose handles are not known at compile time)
 +
*'''Advanced change processing APIs''', and '''advanced lifecycle management techniques''' to be used for performance-critical and/or resource-constrained applications
 +
*'''IncQuery Base''', the low-level query and indexer layer underneath EMF-IncQuery pattern matchers
 +
*'''logging in EMF-IncQuery''', which is provided by hierarchic Log4J loggers.
  
== Generic API  ==
+
== Running example  ==
 +
 
 +
All the code examples and explanations will be given in the context of the [[EMFIncQuery/UserDocumentation/HeadlessExecution|Headless]] example. The up-to-date sample source code to this page is found in Git here: http://git.eclipse.org/c/incquery/org.eclipse.incquery.examples.git/tree/headless Most notably,
 +
 
 +
*the patterns are found in [http://git.eclipse.org/c/incquery/org.eclipse.incquery.examples.git/tree/headless/headlessQueries.incquery/src/headless/headlessQueries.eiq headlessQueries.eiq]
 +
*and the API usage samples are found in [http://git.eclipse.org/c/incquery/org.eclipse.incquery.examples.git/tree/headless/org.eclipse.incquery.application/src/org/eclipse/incquery/application/common/IncQueryHeadless.java IncQueryHeadless.java] and [http://git.eclipse.org/c/incquery/org.eclipse.incquery.examples.git/tree/headless/org.eclipse.incquery.application/src/org/eclipse/incquery/application/common/IncQueryHeadlessAdvanced.java IncQueryHeadlessAdvanced.java]
 +
 
 +
== Javadoc ==
 +
 
 +
The most up-to-date Javadocs for the EMF-IncQuery API can be found online at http://eclipse.org/incquery/javadoc/
 +
 
 +
= The IncQuery Generic API  =
  
 
The "generic" API differs from the generated one in two key aspects:  
 
The "generic" API differs from the generated one in two key aspects:  
Line 12: Line 26:
 
*the generic API is not "type safe" in the sense that the Java types of your pattern variables is not known and needs to be handled dynamically (e.g. by instanceof - typecase combos).
 
*the generic API is not "type safe" in the sense that the Java types of your pattern variables is not known and needs to be handled dynamically (e.g. by instanceof - typecase combos).
  
=== Match base interface ===
+
== Initializing matchers and accessing results ==
 
+
In EMF-IncQuery 0.8 the generic API has changed significantly. Most importantly, it has been moved into the patternlanguage.emf plug-in, and there is no UI dependency required anymore. For more details, see [[EMFIncQuery/Releases/MigrateTo0.8|the migration guide]]; for the old version of this content see [[EMFIncQuery/UserDocumentation/API/Advanced/Archive]]
 +
=== Sample code ===
 +
Using the Generic API:
 
<source lang="java">
 
<source lang="java">
/**
+
public String executeDemo_GenericAPI_LoadFromEIQ(String modelPath, String patternFQN) {
* Generic interface for a single match of a pattern. Each instance is a (partial) substitution of pattern parameters,
+
  final StringBuilder results = new StringBuilder();
* essentially a parameter to value mapping.
+
  Resource resource = loadModel(modelPath);
*
+
  if (resource != null) {
* Can also represent a partial match; unsubstituted parameters are assigned to null. Pattern matchers must never return
+
     try {
* a partial match, but they accept partial matches as method parameters.
+
      // get all matches of the pattern
*
+
      // create an *unmanaged* engine to ensure that noone else is going
*/
+
      // to use our engine
public interface IPatternMatch extends Cloneable /* , Map<String, Object> */{
+
      AdvancedIncQueryEngine engine = AdvancedIncQueryEngine.createUnmanagedEngine(resource);
    /** @return the pattern for which this is a match. */
+
      // instantiate a pattern matcher through the registry, by only knowing its FQN
    public Pattern pattern();
+
      // assuming that there is a pattern definition registered matching 'patternFQN'
    /** Identifies the name of the pattern for which this is a match. */
+
    public String patternName();
+
    /** Returns the list of symbolic parameter names. */
+
    public String[] parameterNames();
+
     /** Returns the value of the parameter with the given name, or null if name is invalid. */
+
    public Object get(String parameterName);
+
    /** Returns the value of the parameter at the given position, or null if position is invalid. */
+
    public Object get(int position);
+
    /** Sets the parameter with the given name to the given value.    */
+
    public boolean set(String parameterName, Object newValue);
+
    /** Sets the parameter at the given position to the given value.     */
+
    public boolean set(int position, Object newValue);
+
    /** Returns whether the match object can be further modified after its creation. Setters work only if the match is mutable.      */
+
    public boolean isMutable();
+
    /** Converts the match to an array representation, with each pattern parameter at their respective position.      */
+
    public Object[] toArray();
+
    /** Prints the list of parameter-value pairs. */
+
    public String prettyPrint();
+
}
+
</source>
+
  
<br>
+
      Pattern p = null;
 +
 +
      // Initializing Xtext-based resource parser
 +
      // Do not use if EMF-IncQuery tooling is loaded!
 +
      new EMFPatternLanguageStandaloneSetup().createInjectorAndDoEMFRegistration();
  
=== Matcher base interface ===
+
      //Loading pattern resource from file
 +
      ResourceSet resourceSet = new ResourceSetImpl();
 +
      URI fileURI = URI.createPlatformPluginURI("headlessQueries.incquery/src/headless/headlessQueries.eiq", false);
 +
      Resource patternResource = resourceSet.getResource(fileURI, true);
 +
   
 +
      // navigate to the pattern definition that we want
 +
      if (patternResource != null) {
 +
        if (patternResource.getErrors().size() == 0 && patternResource.getContents().size() >= 1) {
 +
          EObject topElement = patternResource.getContents().get(0);
 +
          if (topElement instanceof PatternModel) {
 +
            for (Pattern _p : ((PatternModel) topElement).getPatterns()) {
 +
              if (patternFQN.equals(CorePatternLanguageHelper.getFullyQualifiedName(_p))) {
 +
                p = _p; break;
 +
              }
 +
            }
 +
          }
 +
        }
 +
      }
 +
      if (p == null) {
 +
        throw new RuntimeException(String.format("Pattern %s not found", patternFQN));
 +
      }
 +
      // A specification builder is used to translate patterns to query specifications
 +
      SpecificationBuilder builder = new SpecificationBuilder();
 +
   
 +
      // attempt to retrieve a registered query specification    
 +
      IncQueryMatcher<? extends IPatternMatch> matcher = engine.getMatcher(builder.getOrCreateSpecification(p));
 +
 +
      if (matcher!=null) {
 +
        Collection<? extends IPatternMatch> matches = matcher.getAllMatches();
 +
        prettyPrintMatches(results, matches);
 +
      }
 +
 +
      // wipe the engine
 +
      engine.wipe();
 +
      // after a wipe, new patterns can be rebuilt with much less overhead than
 +
      // complete traversal (as the base indexes will be kept)
  
<source lang="java">
+
      // completely dispose of the engine once's it is not needed
/**
+
      engine.dispose();
* Interface for an EMF-IncQuery matcher associated with a graph pattern.
+
      resource.unload();
*
+
    } catch (IncQueryException e) {
* @param <Match>
+
      e.printStackTrace();
*            the IPatternMatch type representing a single match of this pattern.
+
      results.append(e.getMessage());
* @author Bergmann Gábor
+
    }
*/
+
  } else {
public interface IncQueryMatcher<Match extends IPatternMatch> {
+
     results.append("Resource not found");
     // REFLECTION
+
  }
    /** The pattern that will be matched. */
+
  return results.toString();
    public abstract Pattern getPattern();
+
}
 +
</source>
  
    /** Fully qualified name of the pattern. */
 
    public abstract String getPatternName();
 
  
    /** Returns the index of the symbolic parameter with the given name. */
 
    public abstract Integer getPositionOfParameter(String parameterName);
 
  
    /** Returns the array of symbolic parameter names. */
+
=== API interfaces  ===
    public abstract String[] getParameterNames();
+
  
    // ALL MATCHES
+
*IPatternMatch [http://eclipse.org/incquery/javadoc/milestones/m2/org/eclipse/incquery/runtime/api/IPatternMatch.html Javadoc]
    /**
+
**reflection: pattern(), patternName()
    * Returns the set of all pattern matches.
+
**getters and setters
    *  
+
**utility functions (toArray, prettyPrint)
    * @return matches represented as a Match object.
+
*IncQueryMatcher [http://eclipse.org/incquery/javadoc/milestones/m2/org/eclipse/incquery/runtime/api/IncQueryMatcher.html Javadoc]
    */
+
**reflection
    public abstract Collection<Match> getAllMatches();
+
**get all matches  
 +
**get single/arbitrary match
 +
**check for a match
 +
**number of matches
 +
**process matches
 +
**access change processing features
 +
**create a new Match for input binding
 +
**access projected value sets
  
    /**
+
== The Pattern Registry and Matcher Factories  ==
    * Returns the set of all matches of the pattern that conform to the given fixed values of some parameters.
+
    *
+
    * @param partialMatch
+
    *            a partial match of the pattern where each non-null field binds the corresponding pattern parameter to
+
    *            a fixed value.
+
    * @return matches represented as a Match object.
+
    */
+
    public abstract Collection<Match> getAllMatches(Match partialMatch);
+
  
    // variant(s) with input binding as pattern-specific parameters: not declared in interface
+
TODO The IMatcherFactory interface [http://eclipse.org/incquery/javadoc/milestones/m2/org/eclipse/incquery/runtime/api/IMatcherFactory.html Javadoc]
  
    // SINGLE MATCH
+
*reflection
    /**
+
*initialize the matcher
    * Returns an arbitrarily chosen pattern match. Neither determinism nor randomness of selection is guaranteed.
+
    *
+
    * @return a match represented as a Match object, or null if no match is found.
+
    */
+
    public abstract Match getOneArbitraryMatch();
+
  
    /**
+
<br>
    * Returns an arbitrarily chosen match of the pattern that conforms to the given fixed values of some parameters.
+
    * Neither determinism nor randomness of selection is guaranteed.
+
    *
+
    * @param partialMatch
+
    *            a partial match of the pattern where each non-null field binds the corresponding pattern parameter to
+
    *            a fixed value.
+
    * @return a match represented as a Match object, or null if no match is found.
+
    */
+
    public abstract Match getOneArbitraryMatch(Match partialMatch);
+
  
    // variant(s) with input binding as pattern-specific parameters: not declared in interface
+
= Advanced query result set change processing  =
  
    // MATCH CHECKING
+
== Match update callbacks  ==
    /**
+
    * Indicates whether the given combination of specified pattern parameters constitute a valid pattern match, under
+
    * any possible substitution of the unspecified parameters (if any).
+
    *
+
    * @param partialMatch
+
    *            a (partial) match of the pattern where each non-null field binds the corresponding pattern parameter
+
    *            to a fixed value.
+
    * @return true if the input is a valid (partial) match of the pattern.
+
    */
+
    public abstract boolean hasMatch(Match partialMatch);
+
  
    // variant(s) with input binding as pattern-specific parameters: not declared in interface
+
<source lang="java">
 +
private void changeProcessing_lowlevel(final StringBuilder results, IncQueryMatcher<? extends IPatternMatch> matcher) {
 +
// (+) these update callbacks are called whenever there is an actual change in the
 +
// result set of the pattern you are interested in. Hence, they are called fewer times
 +
// than the "afterUpdates" option, giving better performance.
 +
// (-)  the downside is that the callbacks are *not* guaranteed to be called in a consistent
 +
// state (i.e. when the update propagation is settled), hence
 +
//  * you must not invoke pattern matching and model manipulation _inside_ the callback method
 +
//  * the callbacks might encounter "hazards", i.e. when an appearance is followed immediately by a disappearance.
 +
engine.addMatchUpdateListener(matcher, new IMatchUpdateListener<IPatternMatch>() {
 +
  @Override
 +
  public void notifyDisappearance(IPatternMatch match) {
 +
  // left empty
 +
  }
 +
  @Override
 +
  public void notifyAppearance(IPatternMatch match) {
 +
  results.append("\tNew match found by changeset low level callback: " + match.prettyPrint()+"\n");
 +
  }
 +
}, false);
 +
}
 +
</source><br>
  
    // NUMBER OF MATCHES
+
== Using the EVM  ==
    /**
+
    * Returns the number of all pattern matches.
+
    *
+
    * @return the number of pattern matches found.
+
    */
+
    public abstract int countMatches();
+
  
    /**
+
The [[EMFIncQuery/DeveloperDocumentation/EventDrivenVM|Event-driven VM]] can also be used for this purpose.
    * Returns the number of all matches of the pattern that conform to the given fixed values of some parameters.
+
See the section called "Efficiently reacting to pattern match set changes".
    *
+
    * @param partialMatch
+
    *            a partial match of the pattern where each non-null field binds the corresponding pattern parameter to
+
    *            a fixed value.
+
    * @return the number of pattern matches found.
+
    */
+
    public abstract int countMatches(Match partialMatch);
+
  
    // variant(s) with input binding as pattern-specific parameters: not declared in interface
+
= Advanced IncQuery Lifecycle management  =
  
    // FOR EACH MATCH
+
== Managed vs. unmanaged IncQueryEngines  ==
    /**
+
    * Executes the given processor on each match of the pattern.
+
    *
+
    * @param action
+
    *            the action that will process each pattern match.
+
    */
+
    public abstract void forEachMatch(IMatchProcessor<? super Match> processor);
+
  
    /**
+
== Disposing  ==
    * Executes the given processor on each match of the pattern that conforms to the given fixed values of some
+
    * parameters.
+
    *
+
    * @param parameters
+
    *            array where each non-null element binds the corresponding pattern parameter to a fixed value.
+
    * @param processor
+
    *            the action that will process each pattern match.
+
    */
+
    public abstract void forEachMatch(Match partialMatch, IMatchProcessor<? super Match> processor);
+
  
    // variant(s) with input binding as pattern-specific parameters: not declared in interface
+
If you want to remove the matchers from the engine you can call the wipe() method on it. It discards any pattern matcher caches and forgets the known patterns. The base index built directly on the underlying EMF model, however, is kept in memory to allow reuse when new pattern matchers are built. If you don’t want to use it anymore call the dispose() instead, to completely disconnect and dismantle the engine.
  
    // FOR ONE ARBITRARY MATCH
+
*wipe
    /**
+
*dispose
    * Executes the given processor on an arbitrarily chosen match of the pattern. Neither determinism nor randomness of
+
    * selection is guaranteed.
+
    *
+
    * @param processor
+
    *            the action that will process the selected match.
+
    * @return true if the pattern has at least one match, false if the processor was not invoked
+
    */
+
    public abstract boolean forOneArbitraryMatch(IMatchProcessor<? super Match> processor);
+
  
    /**
+
= IncQuery Base  =
    * Executes the given processor on an arbitrarily chosen match of the pattern that conforms to the given fixed
+
    * values of some parameters. Neither determinism nor randomness of selection is guaranteed.
+
    *
+
    * @param parameters
+
    *            array where each non-null element binds the corresponding pattern parameter to a fixed value.
+
    * @param processor
+
    *            the action that will process the selected match.
+
    * @return true if the pattern has at least one match with the given parameter values, false if the processor was
+
    *        not invoked
+
    */
+
    public abstract boolean forOneArbitraryMatch(Match partialMatch, IMatchProcessor<? super Match> processor);
+
  
    // variant(s) with input binding as pattern-specific parameters: not declared in interface
+
EMF-IncQuery provides a light-weight indexer library called Base that aims to provide several useful (some would even argue critical) features for querying EMF models:  
  
    // CHANGE MONITORING
+
*inverse navigation along EReferences
    // attach delta monitor for high-level change detection
+
*finding and incrementally tracking all model elements by attribute value/type (i.e. inverse navigation along EAttributes)
    /**
+
*incrementally computing transitive reachability along given reference types (i.e. transitive closure of an EMF model)  
    * Registers low-level callbacks for match appearance and disappearance on this pattern matcher.
+
*getting and tracking all the (direct) instances of a given EClass
    *
+
    * <p>
+
    * This is a low-level callback that is invoked when the pattern matcher is not necessarily in a consistent state
+
    * yet. Importantly, no model modification permitted during the callback. Most users should use the agenda and trigger engine instead. TODO reference
+
    *  
+
    * <p>
+
    * Performance note: expected to be much more efficient than polling at {@link #addCallbackAfterUpdates(Runnable)},
+
    * but prone to "signal hazards", e.g. spurious match appearances that will disappear immediately afterwards.
+
    *
+
    * <p>
+
    * The callback can be unregistered via {@link #removeCallbackOnMatchUpdate(IMatchUpdateListener)}.
+
    *  
+
    * @param fireNow
+
    *            if true, appearCallback will be immediately invoked on all current matches as a one-time effect. See
+
    *            also {@link IncQueryMatcher#forEachMatch(IMatchProcessor)}.
+
    * @param listener
+
    *            the listener that will be notified of each new match that appears or disappears, starting from now.
+
    */
+
    public abstract void addCallbackOnMatchUpdate(IMatchUpdateListener<Match> listener, boolean fireNow);
+
  
    /**
+
The point of IncQuery Base is to provide all of these in an incremental way, which means that once the query evaluator is attached to an EMF model, as long as it stays attached, the query results can be retrieved instantly (as the query result cache is automatically updated). IncQuery Base is a lightweight, small Java library that can be integrated easily to any EMF-based tool as it can be used in a stand-alone way, without the rest of EMF-IncQuery.  
    * Unregisters a callback registered by {@link #addCallbackOnMatchUpdate(IMatchUpdateListener, boolean)}.
+
    *
+
    * @param listener
+
    *            the listener that will no longer be notified.
+
    */
+
    public abstract void removeCallbackOnMatchUpdate(IMatchUpdateListener<Match> listener);
+
  
    /**
+
We are aware that some of the functionality can be found in some Ecore utility classes (for example ECrossReferenceAdapter). These standard implementations are non-incremental, and are thus do not scale well in scenarios where high query evaluation performance is necessary (such as e.g. on-the-fly well-formedness validation or live view maintenance). IncQuery Base has an additional important feature that is not present elsewhere: it contains very efficient implementations of transitive closure that can be used e.g. to maintain reachability regions incrementally, in very large EMF instance models.  
    * Registers a new delta monitor on this pattern matcher. The DeltaMonitor can be used to track changes (delta) in
+
    * the set of pattern matches from now on. It can also be reset to track changes from a later point in time, and
+
    * changes can even be acknowledged on an individual basis. See {@link DeltaMonitor} for details.
+
    *
+
    * @param fillAtStart
+
    *            if true, all current matches are reported as new match events; if false, the delta monitor starts
+
    *            empty.
+
    * @return the delta monitor.
+
    */
+
    public abstract DeltaMonitor<Match> newDeltaMonitor(boolean fillAtStart);
+
  
    /**
+
The detailed documentation is covered in [[EMFIncQuery/UserDocumentation/API/BaseIndexer]]
    * Registers a new filtered delta monitor on this pattern matcher. The DeltaMonitor can be used to track changes
+
    * (delta) in the set of filtered pattern matches from now on, considering those matches only that conform to the
+
    * given fixed values of some parameters. It can also be reset to track changes from a later point in time, and
+
    * changes can even be acknowledged on an individual basis. See {@link DeltaMonitor} for details.
+
    *
+
    * @param fillAtStart
+
    *            if true, all current matches are reported as new match events; if false, the delta monitor starts
+
    *            empty.
+
    * @param partialMatch
+
    *            a partial match of the pattern where each non-null field binds the corresponding pattern parameter to
+
    *            a fixed value.
+
    * @return the delta monitor.
+
    */
+
    public abstract DeltaMonitor<Match> newFilteredDeltaMonitor(boolean fillAtStart, Match partialMatch);
+
  
    /**
+
== Extracting reachability paths from transitive closure  ==
    * Registers a callback that will be run each time EMF-IncQuery match sets are refreshed after a model update.
+
    * Typically useful to check delta monitors. When the callback is issued, the pattern match sets are guaranteed to
+
    * reflect the post-state after the update.
+
    * <p>
+
    * Callbacks are issued after each elementary change (i.e. possibly at incomplete transient states). This can have a
+
    * negative effect on performance, therefore clients are advised to use it as a last resort only. Consider
+
    * coarser-grained timing (e.g EMF Transaction pre/post-commit) instead, whenever available.
+
    *
+
    * @param callback
+
    *            a Runnable to execute after each update.
+
    * @return false if the callback was already registered.
+
    */
+
    public boolean addCallbackAfterUpdates(Runnable callback);
+
  
    /**
+
TODO
    * Removes a previously registered callback. See addCallbackAfterUpdates().
+
    *
+
    * @param callback
+
    *            the callback to remove.
+
    * @return false if the callback was not registered.
+
    */
+
    public boolean removeCallbackAfterUpdates(Runnable callback);
+
  
    /**
+
= Logging in EMF-IncQuery =
    * Registers a callback that will be run each time the EMF-IncQuery engine is wiped or disposed. Typically useful if
+
    * delta monitors are used, especially of the {@link IncQueryEngine} is managed.
+
    *
+
    * <p>
+
    * When the callback is issued, the wipe has already occurred and pattern matchers will continue to return stale
+
    * results.
+
    *
+
    * @param callback
+
    *            a Runnable to execute after each wipe.
+
    * @return false if the callback was already registered.
+
    */
+
    public boolean addCallbackAfterWipes(Runnable callback);
+
  
    /**
+
EMF-IncQuery logs error messages and some trace information using log4j. If you need to debug your application and would like to see these messages, you can set the log level in different hierarchy levels.
    * Removes a previously registered callback. See {@link #addCallbackAfterWipes()}.
+
Since we use standard log4j, you can configure logging both with configuration files or through API calls.
    *
+
    * @param callback
+
    *            the callback to remove.
+
    * @return false if the callback was not registered.
+
    */
+
    public boolean removeCallbackAfterWipes(Runnable callback);
+
  
    /**
+
* All loggers are children of a top-level default logger, that can be accessed from IncQueryLoggingUtil.getDefaultLogger(), just call setLevel(Level.DEBUG) on the returned logger to see all messages (of course you can use other levels as well).
    * Returns an empty, mutable Match for the matcher.
+
* Each engine has it's own logger that is shared with the Base Index and the matchers as well. If you want to see all messages related to all engines, call IncQueryLoggingUtil.getLogger(IncQueryEngine.class) and set the level.
    * Fields of the mutable match can be filled to create a partial match, usable as matcher input.  
+
* Some other classes also use their own loggers and the same approach is used, they get the loggers based on their class, so retrieving that logger and setting the level will work as well.
    * This can be used to call the matcher with a partial match
+
    * even if the specific class of the matcher or the match is unknown.
+
    *
+
    * @return the empty match
+
    */
+
    public abstract Match newEmptyMatch();
+
  
    /**
+
== Configuration problems ==
    * Returns a new (partial) Match object for the matcher.
+
    * This can be used e.g. to call the matcher with a partial
+
    * match.
+
    *
+
    * <p>The returned match will be immutable. Use {@link #newEmptyMatch()} to obtain a mutable match object.
+
    *
+
    * @param parameters
+
    *            the fixed value of pattern parameters, or null if not bound.
+
    * @return the (partial) match object.
+
    */
+
    public abstract Match newMatch(Object... parameters);
+
  
    /**
+
log4j uses a properties file as a configuration for its root logger. However, since this configuration is usually supplied by developers of applications, we do not package it in EMF-IncQuery.
    * Retrieve the set of values that occur in matches for the given parameterName.
+
This means you may encounter the following on your console if no configuration was supplied:
    *
+
    * @param parameterName
+
    *            name of the parameter for which values are returned
+
    * @return the Set of all values for the given parameter, null if the parameter with the given name does not exists,
+
    *        empty set if there are no matches
+
    */
+
    public abstract Set<Object> getAllValues(final String parameterName);
+
  
    /**
+
log4j:WARN No appenders could be found for logger (org.eclipse.incquery.runtime.util.IncQueryLoggingUtil).
    * Retrieve the set of values that occur in matches for the given parameterName, that conforms to the given fixed
+
log4j:WARN Please initialize the log4j system properly.
    * values of some parameters.
+
    *
+
    * @param parameterName
+
    *            name of the parameter for which values are returned
+
    * @param partialMatch
+
    *            a partial match of the pattern where each non-null field binds the corresponding pattern parameter to
+
    *            a fixed value.
+
    * @return the Set of all values for the given parameter, null if the parameter with the given name does not exists
+
    *        or if the parameter with the given name is set in partialMatch, empty set if there are no matches
+
    */
+
    public abstract Set<Object> getAllValues(final String parameterName, Match partialMatch);
+
  
    /**
+
There are several cases where this can occur:
    * Returns the engine that the matcher uses.
+
* '''You have Xtext SDK installed''', which has a plugin fragment called org.eclipse.xtext.logging that supplies a log4j configuration. Make sure that the fragment is selected in your Runtime Configuration.
    *  
+
* '''You are using the tooling of EMF-IncQuery without the Xtext SDK''', you will see the above warning, but since the patternlanguage.emf plugins also inject appenders to the loggers of EMF-IncQuery, log messages will be correctly displayed.
    * @return the engine
+
* '''You are using only the runtime part of EMF-IncQuery''' that has no Xtext dependency. You have to provide your own properties file (standalone execution) or fragment (OSGi execution), see http://www.eclipsezone.com/eclipse/forums/t99588.html
    */
+
* Alternatively, if you just want to make sure that log messages appear in the console no matter what other configuration happens, you can call <code>IncQueryLoggingUtil.setupConsoleAppenderForDefaultLogger()</code> which will do exactly what its name says. Since appenders and log levels are separate, you will still have to set the log level on the loggers you want to see messages from.
    public abstract IncQueryEngine getEngine();
+
* If you wish to completely turn the logger of, call <code>IncQueryLoggingUtil.getDefaultLogger().setLevel(Level.OFF);</code>.
}
+
</source>  
+
  
== The Pattern Registry and Matcher Factories ==
+
= Using Filtered Input Models During Pattern Matching =
  
=== Matcher Factory base interface  ===
+
In several cases it is beneficial to not include all Resources from a ResourceSet during pattern matching, but consider more than one. Such cases might include Xtext/Xbase languages or JaMoPP[http://www.jamopp.org/index.php/JaMoPP]-based instances that include resources representing the classes of the Java library.
  
<source lang="java">
+
EMF-IncQuery includes some options to filter out some subtrees from being indexed both by the Base Indexer and the Rete networks, by providing a filter implementation to the IncQuery Engine. These options include the IBaseIndexResourceFilter and IBaseIndexObjectFilter instances that can be used to filter out entire resources or containment subtrees, respectively.
/**
+
* Interface for an IncQuery matcher factory. Each factory is associated with a pattern. Methods instantiate a matcher
+
* of the pattern with various parameters.
+
*
+
* @author Bergmann Gábor
+
*
+
*/
+
public interface IMatcherFactory<Matcher extends IncQueryMatcher<? extends IPatternMatch>> {
+
  
    /**
+
Sample usage (by filtering out Java classes referred by JaMoPP):
    * @throws IncQueryException
+
    *            if there was an error loading the pattern definition
+
    * @returns the pattern for which matchers can be instantiated.
+
    */
+
    public Pattern getPattern();
+
  
    /**
+
<source lang="java">
    * Identifies the pattern for which matchers can be instantiated.
+
ResourceSet resourceSet = ...; //Use a Resource Set as the root of the engine
    */
+
BaseIndexOptions options = new BaseIndexOptions().withResourceFilterConfiguration(new IBaseIndexResourceFilter() {
    public String getPatternFullyQualifiedName();
+
  
    /**
+
  @Override
    * Initializes the pattern matcher over a given EMF model root (recommended: Resource or ResourceSet). If a pattern
+
  public boolean isResourceFiltered(Resource resource) {
    * matcher is already constructed with the same root, only a lightweight reference is created.
+
    // PathMap URI scheme is used to refer to JDK classes
    *
+
    return "pathmap".equals(resource.getURI().scheme());
    * <p>
+
  }
    * The scope of pattern matching will be the given EMF model root and below (see FAQ for more precise definition).
+
});
    * <p>
+
//Initializing scope with custom options
    * The match set will be incrementally refreshed upon updates from this scope.
+
EMFScope scope = new EMFScope(resourceSet, options);
    *
+
IncQueryEngine engine = IncQueryEngine.on(scope);
    * <p>
+
    * The matcher will be created within the managed {@link IncQueryEngine} belonging to the EMF model root, so
+
    * multiple matchers will reuse the same engine and benefit from increased performance and reduced memory footprint.
+
    *
+
    * @param emfRoot
+
    *            the root of the EMF tree where the pattern matcher will operate. Recommended: Resource or ResourceSet.
+
    * @throws IncQueryException
+
    *            if an error occurs during pattern matcher creation
+
    */
+
    public Matcher getMatcher(Notifier emfRoot) throws IncQueryException;
+
 
+
    /**
+
    * Initializes the pattern matcher within an existing {@link IncQueryEngine}. If the pattern matcher is already
+
    * constructed in the engine, only a lightweight reference is created.
+
    * <p>
+
    * The match set will be incrementally refreshed upon updates.
+
    *
+
    * @param engine
+
    *            the existing EMF-IncQuery engine in which this matcher will be created.
+
    * @throws IncQueryException
+
    *            if an error occurs during pattern matcher creation
+
    */
+
    public Matcher getMatcher(IncQueryEngine engine) throws IncQueryException;
+
}
+
 
</source>
 
</source>
  
=== Loading an instance model ===
+
'''Important!''' there are some issues to be considered while using this API:
With the Generic API:
+
* If a Resource or containment subtree is filtered out, it is filtered out entirely. It is not possible to re-add some lower-level contents.
 +
* There is a known issue with result matches that are only partially in the filtered area. Be careful with queried edges that are connecting filtered and non-filtered elements. See {{bug|398907}} for details.
  
 +
= Using alternative search algorithms =
 +
 +
Since version 0.9, there is a possibility to refer to alternative search engines in addition to Rete-based incremental engines; version 1.0 includes a local search based search algorithm usable with the EMF-IncQuery matcher API.
 +
 +
The most important steps to perform:
 +
* Add a dependency to the optional plug-in ''org.eclipse.incquery.runtime.localsearch''
 +
* Execute the following call '''before''' initializing the EMF-IncQuery engine (it is safe to execute it multiple times; but will not work after initializing the engine):
 
<source lang="java">
 
<source lang="java">
public String executeGeneric(String modelPath, String patternFQN) {
+
QueryBackendRegistry.getInstance().
StringBuilder results = new StringBuilder();
+
    registerQueryBackendFactory(LocalSearchBackend.class, new LocalSearchBackendFactory());</source>
Resource resource = loadModel(modelPath);
+
* Explicitly ask for a local search based matcher when initializing the matcher instance:
if (resource != null) {
+
<source lang="java">
  try {
+
IQuerySpecification<?> specification = ...;
  // get all matches of the pattern
+
QueryEvaluationHint hint = new QueryEvaluationHint(LocalSearchBackend.class, new HashMap<String, Object>());
  IncQueryMatcher matcher = MatcherFactoryRegistry.getMatcherFactory(patternFQN).getMatcher(resource);
+
AdvancedIncQueryEngine.from(engine).getMatcher(specification, hint);
  Collection<IPatternMatch> matches = matcher.getAllMatches();
+
  prettyPrintMatches(results, matches);
+
  } catch (IncQueryException e) {
+
  e.printStackTrace();
+
  results.append(e.getMessage());
+
  }
+
} else {
+
  results.append("Resource not found");
+
}
+
return results.toString();
+
}
+
 
</source>
 
</source>
 +
* After initialization, the existing pattern matcher API constructs can be used over the local search engine.
 +
 +
== Known limitations ==
 +
* A local search matcher cannot provide change notifications on pattern matches. If asked, an UnsupportedOperationException is thrown.
 +
* For now, it is not possible to combine different pattern matching algorithms. Either the entire search must use Rete or Local search based algorithms.
 +
* Currently, local search always creates new plans from scratch, saving and reusing plans is not implemented yet.

Revision as of 17:37, 24 July 2015

Overview

This page overviews advanced use-cases for EMF-IncQuery. The topics cover

  • the Generic API, to be used when the features supported by the generated API do not suffice (e.g. when working with dynamically defined patterns, or patterns whose handles are not known at compile time)
  • Advanced change processing APIs, and advanced lifecycle management techniques to be used for performance-critical and/or resource-constrained applications
  • IncQuery Base, the low-level query and indexer layer underneath EMF-IncQuery pattern matchers
  • logging in EMF-IncQuery, which is provided by hierarchic Log4J loggers.

Running example

All the code examples and explanations will be given in the context of the Headless example. The up-to-date sample source code to this page is found in Git here: http://git.eclipse.org/c/incquery/org.eclipse.incquery.examples.git/tree/headless Most notably,

Javadoc

The most up-to-date Javadocs for the EMF-IncQuery API can be found online at http://eclipse.org/incquery/javadoc/

The IncQuery Generic API

The "generic" API differs from the generated one in two key aspects:

  • it can be used to apply queries and use other IncQuery features without generating code and loading the resulting bundles into the running configuration. In other words, you just need to supply the EMF-based in-memory representation (an instance of the Pattern class)
  • the generic API is not "type safe" in the sense that the Java types of your pattern variables is not known and needs to be handled dynamically (e.g. by instanceof - typecase combos).

Initializing matchers and accessing results

In EMF-IncQuery 0.8 the generic API has changed significantly. Most importantly, it has been moved into the patternlanguage.emf plug-in, and there is no UI dependency required anymore. For more details, see the migration guide; for the old version of this content see EMFIncQuery/UserDocumentation/API/Advanced/Archive

Sample code

Using the Generic API:

public String executeDemo_GenericAPI_LoadFromEIQ(String modelPath, String patternFQN) {
  final StringBuilder results = new StringBuilder();
  Resource resource = loadModel(modelPath);
  if (resource != null) {
    try {
      // get all matches of the pattern
      // create an *unmanaged* engine to ensure that noone else is going
      // to use our engine
      AdvancedIncQueryEngine engine = AdvancedIncQueryEngine.createUnmanagedEngine(resource);
      // instantiate a pattern matcher through the registry, by only knowing its FQN
      // assuming that there is a pattern definition registered matching 'patternFQN'
 
      Pattern p = null;
 
      // Initializing Xtext-based resource parser
      // Do not use if EMF-IncQuery tooling is loaded!
      new EMFPatternLanguageStandaloneSetup().createInjectorAndDoEMFRegistration();
 
      //Loading pattern resource from file
      ResourceSet resourceSet = new ResourceSetImpl();
      URI fileURI = URI.createPlatformPluginURI("headlessQueries.incquery/src/headless/headlessQueries.eiq", false);
      Resource patternResource = resourceSet.getResource(fileURI, true);
 
      // navigate to the pattern definition that we want
      if (patternResource != null) {
        if (patternResource.getErrors().size() == 0 && patternResource.getContents().size() >= 1) {
          EObject topElement = patternResource.getContents().get(0);
          if (topElement instanceof PatternModel) {
            for (Pattern _p  : ((PatternModel) topElement).getPatterns()) {
              if (patternFQN.equals(CorePatternLanguageHelper.getFullyQualifiedName(_p))) {
                p = _p; break;
              }
            }
          }
        }
      }
      if (p == null) {
        throw new RuntimeException(String.format("Pattern %s not found", patternFQN));
      }
      // A specification builder is used to translate patterns to query specifications
      SpecificationBuilder builder = new SpecificationBuilder();
 
      // attempt to retrieve a registered query specification		    
      IncQueryMatcher<? extends IPatternMatch> matcher = engine.getMatcher(builder.getOrCreateSpecification(p));
 
      if (matcher!=null) {
        Collection<? extends IPatternMatch> matches = matcher.getAllMatches();
        prettyPrintMatches(results, matches);
      }
 
      // wipe the engine
      engine.wipe();
      // after a wipe, new patterns can be rebuilt with much less overhead than 
      // complete traversal (as the base indexes will be kept)
 
      // completely dispose of the engine once's it is not needed
      engine.dispose();
      resource.unload();
    } catch (IncQueryException e) {
      e.printStackTrace();
      results.append(e.getMessage());
    }
  } else {
    results.append("Resource not found");
  }
  return results.toString();
}


API interfaces

  • IPatternMatch Javadoc
    • reflection: pattern(), patternName()
    • getters and setters
    • utility functions (toArray, prettyPrint)
  • IncQueryMatcher Javadoc
    • reflection
    • get all matches
    • get single/arbitrary match
    • check for a match
    • number of matches
    • process matches
    • access change processing features
    • create a new Match for input binding
    • access projected value sets

The Pattern Registry and Matcher Factories

TODO The IMatcherFactory interface Javadoc

  • reflection
  • initialize the matcher


Advanced query result set change processing

Match update callbacks

private void changeProcessing_lowlevel(final StringBuilder results, IncQueryMatcher<? extends IPatternMatch> matcher) {
 // (+) these update callbacks are called whenever there is an actual change in the
 // result set of the pattern you are interested in. Hence, they are called fewer times
 // than the "afterUpdates" option, giving better performance.
 // (-)  the downside is that the callbacks are *not* guaranteed to be called in a consistent
 // state (i.e. when the update propagation is settled), hence
 //  * you must not invoke pattern matching and model manipulation _inside_ the callback method
 //  * the callbacks might encounter "hazards", i.e. when an appearance is followed immediately by a disappearance.
 engine.addMatchUpdateListener(matcher, new IMatchUpdateListener<IPatternMatch>() {
  @Override
  public void notifyDisappearance(IPatternMatch match) {
  // left empty
  }
  @Override
  public void notifyAppearance(IPatternMatch match) {
   results.append("\tNew match found by changeset low level callback: " + match.prettyPrint()+"\n");
  }
 }, false);
}

Using the EVM

The Event-driven VM can also be used for this purpose. See the section called "Efficiently reacting to pattern match set changes".

Advanced IncQuery Lifecycle management

Managed vs. unmanaged IncQueryEngines

Disposing

If you want to remove the matchers from the engine you can call the wipe() method on it. It discards any pattern matcher caches and forgets the known patterns. The base index built directly on the underlying EMF model, however, is kept in memory to allow reuse when new pattern matchers are built. If you don’t want to use it anymore call the dispose() instead, to completely disconnect and dismantle the engine.

  • wipe
  • dispose

IncQuery Base

EMF-IncQuery provides a light-weight indexer library called Base that aims to provide several useful (some would even argue critical) features for querying EMF models:

  • inverse navigation along EReferences
  • finding and incrementally tracking all model elements by attribute value/type (i.e. inverse navigation along EAttributes)
  • incrementally computing transitive reachability along given reference types (i.e. transitive closure of an EMF model)
  • getting and tracking all the (direct) instances of a given EClass

The point of IncQuery Base is to provide all of these in an incremental way, which means that once the query evaluator is attached to an EMF model, as long as it stays attached, the query results can be retrieved instantly (as the query result cache is automatically updated). IncQuery Base is a lightweight, small Java library that can be integrated easily to any EMF-based tool as it can be used in a stand-alone way, without the rest of EMF-IncQuery.

We are aware that some of the functionality can be found in some Ecore utility classes (for example ECrossReferenceAdapter). These standard implementations are non-incremental, and are thus do not scale well in scenarios where high query evaluation performance is necessary (such as e.g. on-the-fly well-formedness validation or live view maintenance). IncQuery Base has an additional important feature that is not present elsewhere: it contains very efficient implementations of transitive closure that can be used e.g. to maintain reachability regions incrementally, in very large EMF instance models.

The detailed documentation is covered in EMFIncQuery/UserDocumentation/API/BaseIndexer

Extracting reachability paths from transitive closure

TODO

Logging in EMF-IncQuery

EMF-IncQuery logs error messages and some trace information using log4j. If you need to debug your application and would like to see these messages, you can set the log level in different hierarchy levels. Since we use standard log4j, you can configure logging both with configuration files or through API calls.

  • All loggers are children of a top-level default logger, that can be accessed from IncQueryLoggingUtil.getDefaultLogger(), just call setLevel(Level.DEBUG) on the returned logger to see all messages (of course you can use other levels as well).
  • Each engine has it's own logger that is shared with the Base Index and the matchers as well. If you want to see all messages related to all engines, call IncQueryLoggingUtil.getLogger(IncQueryEngine.class) and set the level.
  • Some other classes also use their own loggers and the same approach is used, they get the loggers based on their class, so retrieving that logger and setting the level will work as well.

Configuration problems

log4j uses a properties file as a configuration for its root logger. However, since this configuration is usually supplied by developers of applications, we do not package it in EMF-IncQuery. This means you may encounter the following on your console if no configuration was supplied:

log4j:WARN No appenders could be found for logger (org.eclipse.incquery.runtime.util.IncQueryLoggingUtil).
log4j:WARN Please initialize the log4j system properly.

There are several cases where this can occur:

  • You have Xtext SDK installed, which has a plugin fragment called org.eclipse.xtext.logging that supplies a log4j configuration. Make sure that the fragment is selected in your Runtime Configuration.
  • You are using the tooling of EMF-IncQuery without the Xtext SDK, you will see the above warning, but since the patternlanguage.emf plugins also inject appenders to the loggers of EMF-IncQuery, log messages will be correctly displayed.
  • You are using only the runtime part of EMF-IncQuery that has no Xtext dependency. You have to provide your own properties file (standalone execution) or fragment (OSGi execution), see http://www.eclipsezone.com/eclipse/forums/t99588.html
  • Alternatively, if you just want to make sure that log messages appear in the console no matter what other configuration happens, you can call IncQueryLoggingUtil.setupConsoleAppenderForDefaultLogger() which will do exactly what its name says. Since appenders and log levels are separate, you will still have to set the log level on the loggers you want to see messages from.
  • If you wish to completely turn the logger of, call IncQueryLoggingUtil.getDefaultLogger().setLevel(Level.OFF);.

Using Filtered Input Models During Pattern Matching

In several cases it is beneficial to not include all Resources from a ResourceSet during pattern matching, but consider more than one. Such cases might include Xtext/Xbase languages or JaMoPP[1]-based instances that include resources representing the classes of the Java library.

EMF-IncQuery includes some options to filter out some subtrees from being indexed both by the Base Indexer and the Rete networks, by providing a filter implementation to the IncQuery Engine. These options include the IBaseIndexResourceFilter and IBaseIndexObjectFilter instances that can be used to filter out entire resources or containment subtrees, respectively.

Sample usage (by filtering out Java classes referred by JaMoPP):

ResourceSet resourceSet = ...; //Use a Resource Set as the root of the engine 
BaseIndexOptions options = new BaseIndexOptions().withResourceFilterConfiguration(new IBaseIndexResourceFilter() {
 
  @Override
  public boolean isResourceFiltered(Resource resource) {
    // PathMap URI scheme is used to refer to JDK classes
    return "pathmap".equals(resource.getURI().scheme());
  }
});
//Initializing scope with custom options
EMFScope scope = new EMFScope(resourceSet, options);
IncQueryEngine engine = IncQueryEngine.on(scope);

Important! there are some issues to be considered while using this API:

  • If a Resource or containment subtree is filtered out, it is filtered out entirely. It is not possible to re-add some lower-level contents.
  • There is a known issue with result matches that are only partially in the filtered area. Be careful with queried edges that are connecting filtered and non-filtered elements. See bug 398907 for details.

Using alternative search algorithms

Since version 0.9, there is a possibility to refer to alternative search engines in addition to Rete-based incremental engines; version 1.0 includes a local search based search algorithm usable with the EMF-IncQuery matcher API.

The most important steps to perform:

  • Add a dependency to the optional plug-in org.eclipse.incquery.runtime.localsearch
  • Execute the following call before initializing the EMF-IncQuery engine (it is safe to execute it multiple times; but will not work after initializing the engine):
QueryBackendRegistry.getInstance().
    registerQueryBackendFactory(LocalSearchBackend.class, new LocalSearchBackendFactory());
  • Explicitly ask for a local search based matcher when initializing the matcher instance:
IQuerySpecification<?> specification = ...;
QueryEvaluationHint hint = new QueryEvaluationHint(LocalSearchBackend.class, new HashMap<String, Object>());
AdvancedIncQueryEngine.from(engine).getMatcher(specification, hint);
  • After initialization, the existing pattern matcher API constructs can be used over the local search engine.

Known limitations

  • A local search matcher cannot provide change notifications on pattern matches. If asked, an UnsupportedOperationException is thrown.
  • For now, it is not possible to combine different pattern matching algorithms. Either the entire search must use Rete or Local search based algorithms.
  • Currently, local search always creates new plans from scratch, saving and reusing plans is not implemented yet.

Back to the top