VIATRA/Query/UserDocumentation/API/Advanced

Old information
This page is not updated anymore; for more up-to-date details look at the query api documentation at https://www.eclipse.org/viatra/documentation/query-api.html instead.

Overview

This page overviews advanced use-cases for VIATRA Query. The topics cover

• the Generic API, to be used when the features supported by the generated API do not suffice (e.g. when working with dynamically defined patterns, or patterns whose handles are not known at compile time)
• Advanced change processing APIs, and advanced lifecycle management techniques to be used for performance-critical and/or resource-constrained applications
• VIATRA Query Base, the low-level query and indexer layer underneath VIATRA Query pattern matchers
• logging in VIATRA Query, which is provided by hierarchic Log4J loggers
• Query specification registry, for accessing the central store of available query specifications.

The VIATRA Query Generic API

The "generic" API differs from the generated one in two key aspects:

• it can be used to apply queries and use other VIATRA Query features without generating code and loading the resulting bundles into the running configuration. In other words, you just need to supply the EMF-based in-memory representation (an instance of the Pattern class)
• the generic API is not "type safe" in the sense that the Java types of your pattern variables is not known and needs to be handled dynamically (e.g. by instanceof - typecase combos).

Initializing matchers and accessing results

Sample code

Using the Generic API:

public String executeDemo_GenericAPI_LoadFromEIQ(String modelPath, String patternFQN) {
  final StringBuilder results = new StringBuilder();
  Resource resource = loadModel(modelPath);
  if (resource != null) {
    try {
      // get all matches of the pattern
      // create an *unmanaged* engine to ensure that noone else is going
      // to use our engine
      AdvancedViatraQueryEngine engine = AdvancedViatraQueryEngine.createUnmanagedEngine(resource);
      // instantiate a pattern matcher through the registry, by only knowing its FQN
      // assuming that there is a pattern definition registered matching 'patternFQN'
 
      Pattern p = null;
 
      // Initializing Xtext-based resource parser
      // Do not use if VIATRA Query tooling is loaded!
      EMFPatternLanguageStandaloneSetup.createInjectorAndDoEMFRegistration();
 
      //Loading pattern resource from file
      ResourceSet resourceSet = new ResourceSetImpl();
      URI fileURI = URI.createPlatformPluginURI("headlessQueries.incquery/src/headless/headlessQueries.vql", false);
      Resource patternResource = resourceSet.getResource(fileURI, true);
 
      // navigate to the pattern definition that we want
      if (patternResource != null) {
        if (patternResource.getErrors().size() == 0 && patternResource.getContents().size() >= 1) {
          EObject topElement = patternResource.getContents().get(0);
          if (topElement instanceof PatternModel) {
            for (Pattern _p  : ((PatternModel) topElement).getPatterns()) {
              if (patternFQN.equals(CorePatternLanguageHelper.getFullyQualifiedName(_p))) {
                p = _p; break;
              }
            }
          }
        }
      }
      if (p == null) {
        throw new RuntimeException(String.format("Pattern %s not found", patternFQN));
      }
      // A specification builder is used to translate patterns to query specifications
      SpecificationBuilder builder = new SpecificationBuilder();
 
      // attempt to retrieve a registered query specification		    
      ViatraQueryMatcher<? extends IPatternMatch> matcher = engine.getMatcher(builder.getOrCreateSpecification(p));
 
      if (matcher!=null) {
        Collection<? extends IPatternMatch> matches = matcher.getAllMatches();
        prettyPrintMatches(results, matches);
      }
 
      // wipe the engine
      engine.wipe();
      // after a wipe, new patterns can be rebuilt with much less overhead than 
      // complete traversal (as the base indexes will be kept)
 
      // completely dispose of the engine once's it is not needed
      engine.dispose();
      resource.unload();
    } catch (ViatraQueryException e) {
      e.printStackTrace();
      results.append(e.getMessage());
    }
  } else {
    results.append("Resource not found");
  }
  return results.toString();
}

API interfaces

• IPatternMatch
  • reflection: pattern(), patternName()
  • getters and setters
  • utility functions (toArray, prettyPrint)
• ViatraQueryMatcher
  • reflection
  • get all matches
  • get single/arbitrary match
  • check for a match
  • number of matches
  • process matches
  • access change processing features
  • create a new Match for input binding
  • access projected value sets

Advanced query result set change processing

The Event-driven VM can be used for this purpose. See the section called "Efficiently reacting to pattern match set changes".

Advanced VIATRA Query Lifecycle management

Disposing

If you want to remove the matchers from the engine you can call the wipe() method on it. It discards any pattern matcher caches and forgets the known patterns. The base index built directly on the underlying EMF model, however, is kept in memory to allow reuse when new pattern matchers are built. If you don’t want to use it anymore call the dispose() instead, to completely disconnect and dismantle the engine.

VIATRA Query Base

VIATRA Query provides a light-weight indexer library called Base that aims to provide several useful (some would even argue critical) features for querying EMF models:

• inverse navigation along EReferences
• finding and incrementally tracking all model elements by attribute value/type (i.e. inverse navigation along EAttributes)
• incrementally computing transitive reachability along given reference types (i.e. transitive closure of an EMF model)
• getting and tracking all the (direct) instances of a given EClass

The point of VIATRA Query Base is to provide all of these in an incremental way, which means that once the query evaluator is attached to an EMF model, as long as it stays attached, the query results can be retrieved instantly (as the query result cache is automatically updated). VIATRA Query Base is a lightweight, small Java library that can be integrated easily to any EMF-based tool as it can be used in a stand-alone way, without the rest of VIATRA Query.

We are aware that some of the functionality can be found in some Ecore utility classes (for example ECrossReferenceAdapter). These standard implementations are non-incremental, and are thus do not scale well in scenarios where high query evaluation performance is necessary (such as e.g. on-the-fly well-formedness validation or live view maintenance). VIATRA Query Base has an additional important feature that is not present elsewhere: it contains very efficient implementations of transitive closure that can be used e.g. to maintain reachability regions incrementally, in very large EMF instance models.

The detailed documentation is covered in Base Indexer

Extracting reachability paths from transitive closure

Beyond the support for querying reachability information between nodes in the model, the TransitiveClosureHelper class also provides the functionality to retrieve paths between pairs of nodes. The getPathFinder method returns an IGraphPathFinder object, which exposes the following operations:

Deque<V> getPath(V sourceNode, V targetNode): Returns an arbitrary path from the source node to the target node (if such exists).
Iterable<Deque<V>> getShortestPaths(V sourceNode, V targetNode): Returns the collection of shortest paths from the source node to the target node (if such exists).
Iterable<Deque<V>> getAllPaths(V sourceNode, V targetNode): Returns the collection of paths from the source node to the target node (if such exists).
Iterable<Deque<V>> getAllPathsToTargets(V sourceNode, Set<V> targetNodes): Returns the collection of paths from the source node to any of the target nodes (if such exists).

Internally these operations use a depth-first-search traversal and rely on the information which is incrementally maintained by the transitive closure component.

Logging in VIATRA Query

VIATRA Query logs error messages and some trace information using log4j. If you need to debug your application and would like to see these messages, you can set the log level in different hierarchy levels. Since we use standard log4j, you can configure logging both with configuration files or through API calls.

• All loggers are children of a top-level default logger, that can be accessed from ViatraQueryLoggingUtil.getDefaultLogger(), just call setLevel(Level.DEBUG) on the returned logger to see all messages (of course you can use other levels as well).
• Each engine has it's own logger that is shared with the Base Index and the matchers as well. If you want to see all messages related to all engines, call ViatraQueryLoggingUtil.getLogger(ViatraQueryEngine.class) and set the level.
• Some other classes also use their own loggers and the same approach is used, they get the loggers based on their class, so retrieving that logger and setting the level will work as well.

Configuration problems

log4j uses a properties file as a configuration for its root logger. However, since this configuration is usually supplied by developers of applications, we do not package it in VIATRA Query. This means you may encounter the following on your console if no configuration was supplied:

log4j:WARN No appenders could be found for logger (org.eclipse.viatra.query.runtime.util.ViatraQueryLoggingUtil).
log4j:WARN Please initialize the log4j system properly.

There are several cases where this can occur:

• You have Xtext SDK installed, which has a plugin fragment called org.eclipse.xtext.logging that supplies a log4j configuration. Make sure that the fragment is selected in your Runtime Configuration.
• You are using the tooling of VIATRA Query without the Xtext SDK, you will see the above warning, but since the patternlanguage.emf plugins also inject appenders to the loggers of VIATRA Query, log messages will be correctly displayed.
• You are using only the runtime part of VIATRA Query that has no Xtext dependency. You have to provide your own properties file (standalone execution) or fragment (OSGi execution), see http://www.eclipsezone.com/eclipse/forums/t99588.html
• Alternatively, if you just want to make sure that log messages appear in the console no matter what other configuration happens, you can call ViatraQueryLoggingUtil.setupConsoleAppenderForDefaultLogger() which will do exactly what its name says. Since appenders and log levels are separate, you will still have to set the log level on the loggers you want to see messages from.
• If you wish to completely turn the logger of, call ViatraQueryLoggingUtil.getDefaultLogger().setLevel(Level.OFF);.

Query Scopes

VIATRA Query uses the concept of Scopes to define the entire model to search for results. For queries over EMF models, the EMFScope class defines such scopes. When initializing a ViatraQueryEngine, it is required to specify this scope by creating a new instance of EMFScope.

This instance might be created from one or more Notifier instances (ResourceSet: includes all model elements stored in the ResourceSet; Resource: includes all elements inside the corresponding Resource; EObject: includes all elements in the containment subtree of the object itself).

In most cases, it is recommended to include the entire ResourceSet as the query scope; however, if required, it is possible to

Using Filtered Input Models During Pattern Matching

In several cases it is beneficial to not include all Resources from a ResourceSet during pattern matching, but consider more than one. Such cases might include Xtext/Xbase languages or JaMoPP[1]-based instances that include resources representing the classes of the Java library.

In case of EMF models, the EMFScope instance may also set some base index options to filter out containment subtrees from being indexed both by the Base Indexer and the Rete networks, by providing a filter implementation to the VIATRA Query Engine. These options include the IBaseIndexResourceFilter and IBaseIndexObjectFilter instances that can be used to filter out entire resources or containment subtrees, respectively.

Sample usage (by filtering out Java classes referred by JaMoPP):

ResourceSet resourceSet = ...; //Use a Resource Set as the root of the engine 
BaseIndexOptions options = new BaseIndexOptions().withResourceFilterConfiguration(new IBaseIndexResourceFilter() {
 
  @Override
  public boolean isResourceFiltered(Resource resource) {
    // PathMap URI scheme is used to refer to JDK classes
    return "pathmap".equals(resource.getURI().scheme());
  }
});
//Initializing scope with custom options
EMFScope scope = new EMFScope(resourceSet, options);
ViatraQueryEngine engine = ViatraQueryEngine.on(scope);

Limitations

Important! there are some issues to be considered while using this API:

• If a Resource or containment subtree is filtered out, it is filtered out entirely. It is not possible to re-add some lower-level contents.
• In case of the query scope is set to a subset of the entire model (e.g only one EMF resource within the resource set), model elements within the scope of the engine may have references pointing to elements outside the scope; these are called dangling edges. Previous versions of VIATRA made the assumption that the model is self-contained and free of dangling edges; the behavior of the query engine was unspecified (potentially incorrect match sets) if the model did not have this property. In VIATRA 1.6, this behavior was cleaned up by adding a new indexer mode that drops this assumption, and (with a minor cost to performance) always checks both ends of all indexed edges to be in-scope. For backward compatibility, the old behavior is used by default, but you can manually change this using the corresponding base index option as below. For new code we suggest to use this option to drop the dangling-free assumption, as it provides more consistent and intuitive results in a lot of cases; in a future VIATRA release this will be the new default.

BaseIndexOptions options = new BaseIndexOptions().withDanglingFreeAssumption(false); 
ResourceSet rSet = new ResourceSetImpl();
EMFScope scope = new EMFScope(rSet, options);
ViatraQueryEngine engine = ViatraQueryEngine.on(scope);

Using alternative search algorithms

Since version 0.9, there is a possibility to refer to alternative search engines in addition to Rete-based incremental engines; version 1.0 includes a local search based search algorithm usable with the VIATRA Query matcher API. Local search documentation was moved into a separate page.

Providing Query Evaluation Hints

It is possible to pass extra information to the runtime of VIATRA Query using evaluation hints, such as information about the structure of the model or requirements for the evaluation. In version 1.4, the handling of such hints were greatly enhanced, allowing the following ways to pass hints:

1. The Query engine might be initialized with default hints using the static method AdvancedQueryEngine#createUnmanagedEngine(QueryScope, ViatraQueryEngineOptions). The hints provided inside the query engine options are the default hints used by all matchers, but can be overridden using the following options.
2. A pattern definition can be extended with hints, e.g. for backend selection in the pattern language. Such hints will be generated into the generated query specification code. TODO provide example
3. When accessing a new pattern matcher through the Query Engine, further override hints might be presented using AdvancedQueryEngine#getMatcher(IQuerySpecification, QueryEvaluationHint). Such hints override both the engine default and the pattern default hints.

In version 1.4 the hints are mostly used to fine tune the local search based pattern matcher, but their usage is gradually being extended. See classes ReteHintOptions and LocalSearchHints for hint options provided by the query backends.

Query specification registry

The query specification registry, available since VIATRA 1.3 is used to manage query specifications provided by multiple connectors which can dynamically add and remove specifications. Users can read the contents of the registry through views that are also dynamically updated when the registry is changed by the connectors.

Basic usage

The most common usage of the registry will be to get a registered query specification based on its fully qualified name. You can access the registry through a singleton instance:

IQuerySpecificationRegistry registry = org.eclipse.viatra.query.runtime.registry.QuerySpecificationRegistry.getInstance();
IQuerySpecification<?> specification = registry.getDefaultView().getEntry("my.registered.query.fqn").get();

The default view lets you access the contents of the registry, the entry returned is a provider for the query specification that returns it when requested through the get() method.

Advanced usage

Views

To get an always up to date view of the registry, you can either:

• request a default view that will contain on specification marked to be included in this view (e.g. queries registered through the queryspecification extension point)
• create a new view that may use either a filter or a factory for defining which specifications should be included in the view

IQuerySpecificationRegistry registry = QuerySpecificationRegistry.getInstance();
// access default view
IDefaultRegistryView defaultView = registry.getDefaultView();
 
// create new view
IRegistryView simpleView = registry.createView();
 
// create filtered view
IRegistryView filteredView = registry.createView(new IRegistryViewFilter() {
  @Override
  public boolean isEntryRelevant(IQuerySpecificationRegistryEntry entry) {
    // return true to include in view
  }
});
 
// create specific view instance
boolean allowDuplicateFQNs = false;
IRegistryView ownView = registry.createView(new IRegistryViewFactory() {
  return new AbstractRegistryView(registry, allowDuplicateFQNs) {
    @Override
    protected boolean isEntryRelevant(IQuerySpecificationRegistryEntry entry) {
      // return true to include in view
    }
  }
);

Once you have a view instance, you can access the contents of the registry by requesting the entries from the view or adding a listener that will be notified when the view changes.

Default views add a few additional utilities that are made possible by also restricting what is included in them. Default views will only contain entries that are marked explicitly to be included and will not allow different specifications with the same fully qualified name. In return, you can request a single entry by its FQN (since at most one can exist) and also request a query group that contains all entries.

Listening to view changes

The contents of the registry may change after a view is created. When you access the view to get its entries, it will always return the current state of the registry. If you want to get notified when the contents of your view change, you can add a listener to the view:

IQuerySpecificationRegistry registry = QuerySpecificationRegistry.getInstance();
IRegistryView myView = registry.createView();
IQuerySpecificationRegistryChangeListener listener = new IQuerySpecificationRegistryChangeListener() {
  @Override
  public void entryAdded(IQuerySpecificationRegistryEntry entry) {
    // process addition
  }
 
  @Override
  public void entryRemoved(IQuerySpecificationRegistryEntry entry) {
    // process removal
  }
});
myView.addViewListener(listener);
 
// when you don't need to get notifications any more
myView.removeViewListener(listener);

Important note: your code has to keep a reference to your view otherwise it will be garbage collected. The registry uses weak references to created views in order to free users from having to manually dispose views.

Adding specifications to the registry

The registry is supplied with specifications through sources. You can add your own source connector as a source and dynamically add and remove your own specifications.

IQuerySpecificationRegistry registry = QuerySpecificationRegistry.getInstance();
// initialize your connector
IRegistrySourceConnector connector;
 
// add connector
boolean sourceAdded = registry.addSource(connector);
 
// [...]
 
// remove your source when needed
boolean sourceRemoved = registry.removeSource(connector);

We already have some connector implementations for the most common use cases. For example, you can create a connector with a simple add and remove method for query specifications:

IRegistrySourceConnector connector = new SpecificationMapSourceConnector("my.source.identifier", false /* do not include these in default views */);
 
IQuerySpecification<?> specification = /* available from somewhere */
 
IQuerySpecificationProvider provider = new SingletonQuerySpecificationProvider(specification);
 
// add specification to source
connector.addQuerySpecificationProvider(provider);
 
// remove specification by FQN
connector.removeQuerySpecificationProvider(specification.getFullyQualifiedName());

Performance tuning with special modes of the engine

Query groups and coalescing model traversals

If you initialize a new query that requires the indexing of some EMF types for which the current engine instance has not yet built an index, then the base index of the VIATRA engine will traverse the entire scope to build the index. It can make a great difference if such expensive re-traversals are avoided, and the engine traverses the model only once to build indexes for all queries.

The easiest wax to do this would be to use IQueryGroup.prepare(engine) for a group of queries. Such a group is generated for every query file, and any other custom group can be manually assembled with GenericQueryGroup.

IQueryGroup queries = ...
ViatraQueryEngine engine = ...
queries.prepare(engine);

For advanced use cases, it is possible to directly control indexing traversals in an arbitrary code block, such that any index constructions are coalesced into a single traversal:

ViatraQueryEngine engine = ...
engine.getBaseIndex().coalesceTraversals(new Callable<Void>() {
    @Override
    public Void call() throws Exception {
        // obtain matchers, etc.
        return null;
    }
});

Delaying query result updates

As of version 1.6, the advanced query API now includes a feature that lets users temporarily "turn off" query result maintenance in the incremental query backend. During such a code block, only the base model indexer is updated, query results remain stale until the end of the block. The advantage is that it is possible to save significant execution time when changing the model in a way that partially undoes itself, e.g. a large part of the model is removed and then re-added.

AdvancedViatraQueryEngine engine = ...
engine.delayUpdatePropagation(new Callable<Void>() {
    @Override
    public Void call() throws Exception {
        // perform extensive changes in model that largely cancel each other out
        return null;
    }
});