View source for VIATRA/Addon/Query Based Features

{{caution|Old information|This page is not updated anymore; for more up-to-date details look at the language specification at https://www.eclipse.org/viatra/documentation/addons.html instead.}}
VIATRA supports the definition of efficient, incrementally maintained, well-behaving derived features in EMF by using advanced model queries and incremental evaluation for calculating the value of derived features and providing automated code generation for integrating into existing applications.

==Main scope of query-based features==

* Integrate model query results into EMF applications as structural features
* Replace low performance derived feature implementations with incrementally evaluated model queries
* Provide a flexible interlinking method for fragmented models
* Support declarative definition of high-performance computed features with automatic code generation and validation

==New & Noteworthy==

* Since 0.8.0: The default implementation of query-based features from EMF-IncQuery 0.8.0 uses setting delegates! This means that annotations are generated into the Ecore model for derived features to configure setting delegates and the EMF Code Generator has to be invoked to update the generated model code with the default setting delegate information.
** For more information on setting delegates, see ''Using setting delegates'' section.

==Overview==

Derived features in EMF models represent information (attribute values, references) computed from the rest of the model, such as the number of elements in a given collection or the set of elements satisfying some additional conditions. Such derived features can ease the handling of models significantly, as they appear in the same way as regular features. However, in order to achieve complete transparency for derived features, the developer must ensure that proper change notifications are sent when model modifications cause changes in the value of the derived feature as well. Finally, since the value of the derived feature might be retrieved often, complete recalculation of the value may impact application performance. Therefore, it is better to keep a cached version of the value and update it incrementally based on changes in the model.

Usually, developers who use derived features in EMF have to manually solve each of these challenges for each derived feature they introduce into their model. Furthermore, although the derived features almost always represent the result of a model query (including type constraints, navigation, aggregation), they are implemented as imperative Java code.

In order to help developers in using derived features, VIATRA supports the definition of model queries that provide the results for the derived feature value calculation and includes out-of-the-box change notification and incremental maintenance of results. Additionally, the automatic generation of the Ecore annotations or glue code between the EMF model code and VIATRA offers easy integration into any existing EMF application.

=== Well-behaving structural features ===

The incremental approach of the VIATRA queries relies on change notifications from every object and every feature in the model that is used in the query definitions. Therefore, a regular volatile feature that has no field, therefore there it does not store the current value of the feature and usually does not send proper change notifications (e.g. SET oldValue to newValue). Such features are ignored by VIATRA, unless there is an explicit declaration, that the feature implementation sends proper change notifications at all times. These are called well-behaving structural features.

If your application uses volatile (and often derived) features, you provide proper notifications for them and would like to include them in query definitions, you can explicitly tell VIATRA that the feature is well-behaving. There are two ways to do this:

# extend the ''org.eclipse.viatra.query.runtime.base.wellbehaving.derived.features'' extension point as described [https://github.com/FTSRG/publication-pages/wiki/Using-queries-for-derived-features-(ECMFA12) here]
# register your feature directly into the ''org.eclipse.viatra.query.runtime.base.comprehension.WellbehavingDerivedFeatureRegistry'' using the various ''registerX'' methods. '''Warning''': you must call this method before executing any queries (i.e. before the first ''getMatcher()'' or ''getEngine()'' call), since VIATRA checks the registry when it traverses the model.

== Examples ==

For demonstration, we will use the BPM metamodel from the [https://github.com/IncQueryLabs/org.eclipse.viatra/tree/master/examples/bpm examples repository].

=== Other examples ===

The Simulink model in [https://github.com/FTSRG/massif/ Massif] uses query based features for supporting library blocks, model references, port filtering and many more.

You can find (old) examples using the VIATRA derived features in the following locations:

* [https://github.com/ujhelyiz/EMF-IncQuery-Examples/tree/master/query-driven-soft-interconnections Soft interconnections between models in different resources]

== User documentation ==

VIATRA only provides the back-end for derived features, the developer must define the feature itself in the metamodel first. Once that is complete, the developer creates the query in a regular VIATRA query project in a query definition file and adds a specific annotation with the correct parameters to have the derived feature implementation generated. These steps are detailed in the following:

=== Definition of the derived feature ===
# In the Ecore model (.ecore file), create the desired EAttribute or EReference in the selected EClass and set the name, type and multiplicity information correctly.
# Use the following configuration for the other attributes of the created EStructuralFeature:
#* derived = true (to indicate that the value of the feature is computed from the model)
#* changeable = false (to remove setter methods)
#* transient = true (to avoid persisting the value into file)
#* volatile = true (to remove the field declaration in the object)
# In the Generator model (.genmodel), right-click on the top-level element and select Reload, click Next, Load, and Finish to update the Generator model with the changes done in the Ecore model.
# Right-click on the top-level element and select Generate Model Code to ensure that the getters are properly generated into the EMF model code. You can regenerate the Edit and Editor code as well, though those are not necessary here.

=== Definition of the model query ===
# Create a VIATRA query project and query definition (.vql) file as described in the cheat sheet or this tutorial.
# Make sure that you imported your metamodel into the query definition. Create the VIATRA generator model, if necessary (.vqlgen file).
# Make sure that the project containing the Ecore model or generated code is in the same workspace as the VIATRA query project.
# Create the query corresponding to your derived feature. For example, the tasks corresponding by identifiers to a given job feature would look like this: 
<source lang="java">
package org.eclipse.viatra.examples.bpm.queries.system

import "http://process/1.0"
import "http://system/1.0"

@QueryBasedFeature(feature = "tasks")
pattern JobTaskCorrespondence(Job : Job, Task : Task) = {
 Job.taskIds(Job,TaskId);
 Task.id(Task,TaskId);
}
</source>
# When you save, the VIATRA query builder runs automatically and places the setting delegate annotations in the Ecore model.
# If new query-based feature queries were introduced or the fully qualified name of the pattern for a given feature has changed, the EMF Generator must be invoked. This is needed since the generator uses the setting delegate annotations to create the model code.

Note that the first parameter of the pattern is the source of the derived feature and the second is the target. Although not mandatory, is is good practice to use the (This : EClass, Target) format to ease understanding. The @QueryBasedFeature annotation indicates to the code generator that it should create the setting delegate annotations in the Ecore model.

Saving the query definition initiates the code generation. After it completes, you can open the Ecore model to ensure that the new annotations were correctly created. Note that a well-behaving derived feature extension is also generated into the plugin.xml of the VIATRA Query project to indicate that the given derived feature correctly sends change notifications if the project is loaded.

=== Running the application ===

Once the annotations are generated and the EMF Generator is invoked, you can use the derived features by including the VIATRA Query project into your runtime together with the model project.

=== Annotation parameters ===

The @QueryBasedFeature annotation uses defaults for each possible parameters, which allows developers to avoid using any parameters if the query is correctly written.

In short, parameters are not needed, if the following conditions are satisfied:

* The name of the pattern is the same as the name of the derived feature (comparison uses String.equals())
* The first parameter is the defining EClass and its type is correctly given (e.g. This : Course)
* The second parameter is the target of the derived feature
* The derived feature value is a single EObject or a collection of EObjects

If the derived feature and its query does not satisfy the above conditions, the following parameters can be used in the annotation:

* ''feature ="featureName"'' (default: pattern name) - indicates which derived feature is defined by the pattern
* ''source ="Src"'' (default: first parameter) - indicates which query parameter (using its name) is the source EObject, the inferred type of this parameter indicates which EClass generated code has to be modified
* ''target ="Trg"'' (default: second parameter) - indicates which query parameter (using its name) is the target of the derived feature
* ''kind ="single/many/counter/sum/iteration"'' (default: feature.isMany?many:single) - indicates what kind of calculation should be done on the query results to map them to derived feature values
* ''generateIntoModelCode'' (default: false) - indicates that the legacy code generation should be used instead of setting delegates (see [[VIATRA/Addon/Query_Based_Features/Legacy_Code_Generator|details]])

=== Pitfalls ===

==== Code generation fails for derived feature query ====

Ensure that the .ecore file is available and writeable in the same workspace as the VIATRA query project with the query definitions.

==== Multiple results for a query used in a single (upper bound = 1) feature ====

If you define a query for a single feature that returns multiple results for a given source model element, the value of the derived feature will in most cases be the value from the last match that appeared. However, it is possible to change the values in a way that the feature will have no value, even though it might have exactly one. Therefore, it is important to define the queries for the feature in a way that only one result is possible. You can either make assumptions on your models and use other ways to ensure that there is only one match, or you can explicitly declare in the pattern, that it should only match once for a given source element. Additionally, you can use the Validation framework of VIATRA to create feedback for the user when the query would have multiple results indicating that the model is invalid.

The following is an example for a validated, ensured single feature:

<source lang="java">
@QueryBasedFeature
pattern singleFeature(This : SourceType, Target : TargetType){
 find internalQuery(This, Target);
 1 == count find internalQuery(This, Target);
}

private pattern internalQuery(This : SourceType, Target : TargetType){
 // actual query definition
}

@Constraint(location = "This", severity = "error",
   message="Multiple values for $This.name$.singleFeature!")
pattern singleFeatureInvalid(This : SourceType){
 1 < count find internalQuery(This, _Target);
}
</source>

== Developer documentation ==

The JavaDoc can be found [https://hudson.eclipse.org/viatra/job/viatra-master/lastSuccessfulBuild/artifact/releng/org.eclipse.viatra.docs/target/reference/api/org/eclipse/viatra/addon/querybasedfeatures/runtime/package-summary.html here].

=== Overview of the implementation ===

To support query-backed features captured as derived features, the outputs of the VIATRA query engine need to be integrated into the EMF model access layer at two points: (1) query results are provided in the getter functions of derived features, and (2) query result deltas are processed to generate EMF Notification objects that are passed through the standard EMF API so that application code can process them transparently.

[[Image:Querybased-features-approach.png|600px]]

The application accesses both the model and the query results through the standard EMF model access layer -- hence, no modification of application source code is necessary. In the background, our novel derived feature handlers are attached to the EMF model plugin that integrate the generated query components (pattern matchers).
When an EMF application intends to read a soft link (B1), the current value is provided by the corresponding handler (B2) by simply retrieving the value from the cache of the related query. When the application modifies the EMF model (A1), this change is propagated to the generated query components of VIATRA along notifications (A2), which may update the delta monitors of the handlers (A3). Changes of derived features may in turn trigger further changes in the results sets of other derived features (A4).

=== Using setting delegates ===

Instead of overwriting the generated model code, it is possible to use setting delegates for wrapping query-based features. Setting delegates are the currently recommended way of integrating derived feature computation into EMF models.

In EMF-IncQuery 0.8, the [[VIATRA/Addon/Query_Based_Features/Legacy_Code_Generator|old generator]] was replaced with creating appropriate annotations for using setting delegates. This means that only the Ecore file is modified, but also that the code generation from the genmodel will have to be invoked as well.

To set up setting delegates, the generator automatically puts annotations on the EPackage and EStructuralFeatures:
* on the EPackage, to declare which setting delegates to use:
<source lang="xml">
<eAnnotations source="http://www.eclipse.org/emf/2002/Ecore">
  <details key="settingDelegates" value="org.eclipse.viatra.query.querybasedfeature"/>
</eAnnotations>
</source>
* on the EStructuralFeature which is a query-based feature:
<source lang="xml">
<eAnnotations source="org.eclipse.viatra.query.querybasedfeature">
  <details key="patternFQN" value="querypackage.patternName"/>
</eAnnotations>
</source>

The setting delegate factory is registered by the query-based feature runtime plug-in and EMF will use the factory to create the setting delegate for query-based derived features.

=== Information related to the legacy code generator based approach ===

Moved to new [[VIATRA/Addon/Query_Based_Features/Legacy_Code_Generator|page]]

=== Development plans ===

==== Tracing ====

In order to support debugging query-based features, we should add a "querybasedfeature" option for the (not-yet-existing) global tracing of VIATRA.

The tracing should cover all core functionality:
* initializing query-based features
* computing the initial value of the feature for a given source
* returning results to getter calls
* handling incremental updates (both match set changes and life-cycle notifications)

==== Extending query-based features for subclasses ====

It is possible that a subclass (in a different Ecore model most likely) would want to override the behaviour of a query-based feature. It would be nice to have a way to extend/override the query that is used for computing the query.

# Using query libraries, we could have a query that composes the original pattern with the additional constraints (either with an additional pattern body or just as part of the new body).
# We need to ensure that multiple extensions are possible and they don't cancel each other out. In the case of setting delegates, this is possible, as long as the setting delegate knows about all possible extensions (seems entirely possible).
# The @QueryBasedFeature annotation must include an optional parameter that specifies the extension. This is required to identify cases when a feature would be accidentally overridden.

==== Remaining issues ====

* The setting delegate factory should use the pattern registry to find the required Query Specification to work with. This way, queries defined dynamically can be registered by their creators independently of the query-based feature runtime.
* The "scope" of a feature is important, whether it's the parent EObject, its resorce or the whole resource set. Since setting delegates have to find the proper handler in case of getters, the scope must be taken into consideration.
** The scope should be part of the @QueryBasedFeature annotation, with the default value "resource set".
** We need to handle incorrect scopes (e.g. the parent is not in a resource or resource set): we can either (1) not initialize the handler at all, (2) initialize it with a different scope and recognise when the scope should be corrected, (3) delay initialization, listen to model changes and initialize the handler when the scope becomes correct.
** The scope operations should be logged when tracing is enabled.
* The query-based feature needs a ViatraQueryEngine to access the match set of the query that drives the feature. Since the engine must be created with dynamic mode enabled to use it in certain scenarios, the delegate factory needs a way to accept specific engines to use for given scopes.
** One possibility is to add a putEngineIntoScopeMap method that can be called by clients.