View source for VIATRA/Transformation/EventDrivenVM

{{caution|Old information|This page is not updated anymore; for more up-to-date details look at the language specification at https://www.eclipse.org/viatra/documentation/evm.html instead.}}
= The VIATRA Event-driven Virtual Machine =

== Overview ==

Nowadays, collaboration and scalability challenges in modeling tools are typically addressed with dedicated problem-specific solutions e.g.:
* On-the-fly constraint evaluation engines (to provide scalability for model validation)
* Incremental model transformation tools (to address performance issues of e.g. model synchronization)
* Incremental model comparison algorithms (to support versioning and model merge in collaborative scenarios)
* Design space exploration tools (to optimize models towards a design goal).

The common recurring task in these applications is to capture and process not only the models, but also their changes as a stream of events (operations that affect models). We generalized this approach to provide a common conceptual framework for an '''event-driven virtual machine''' (EVM) architecture. 

The EVM is a rule-based system with a special focus on versatile model transformations, with built-in support for reacting to model and query result changes and user interactions. The EVM integrates various execution schemes into a uniform and flexible architecture, to provide a common framework that even allows for combinations of advanced model transformation scenarios, e.g. by the interleaving of various execution strategies (batch, live/triggered and exploratory) within a single transformation program.

=== Core architecture ===

[[Image:EMFIncQuery_EventDrivenVM_overview.png|thumb|500px|Overview of the Event-driven VM]]

The event-driven virtual machine allows the central management of executable actions defined on event sources and can execute these actions automatically with a predefined schedule.

* An '''activation''' is wrapper of a pattern match with a corresponding rule instance and state.
** Activation '''states''' are: inactive, appeared, fired, updated, disappeared
* A '''rule instance''' manages the activations corresponding to a rule specification in a given VIATRA Query engine.
* A '''rule specification''' defines the life cycle for changing the activation state in response to events and the possible actions that can be executed on an activation in a given state.
** '''Events''' related to a life cycle are: match appears/disappears/updates, activation fires
** State '''transitions''' in a life cycle always have a source state, an event and a target state. There can be only one (source state, event) pair in the life cycle, thus the target state is always deterministic.
** '''Jobs''' are atomic actions that are performed if an activation is fired when in a state defined by the job.
** An activation is '''enabled''' if there is at least one job that is defined for the current state of the activation.
* An '''agenda''' is a collection of rule instances with an added responsibility of ordering the enabled activations of all rule instances related to the same VIATRA Query engine
** The agenda keeps track of the activations of the rule instances by an '''activation notification''' mechanism. Rule instances notify the agenda if one of their activations changed state in response to an event.
* An '''executor''' is responsible for executing enabled activations in the agenda when it is scheduled to do so, and to provide an execution '''context''' to store execution results or other data related to execution.
* A '''scheduler''' is defined to respond to some kind of global event (e.g. transaction commit, user request, or VIATRA Query Base update callback) by scheduling its executor.
* A '''rule engine''' is created for a given VIATRA Query engine and an optional set of rule specifications, and has its own agenda.
* A '''execution schema''' is a special rule engine, that also has a scheduler set up.

== Example code ==

We illustrate the two main usage modes of the EVM with UML models. First, we have to define the preconditions with patterns, then define the rule specifications which can be added to a rule engine or execution schema.

The example project is on the repository: [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/examples/papyrus-uml/org.eclipse.viatra.examples.uml.evm/ UML EVM Example]

=== Precondition pattern definition ===

The code example below shows the '''possibleSuperClass''' and '''onlyInheritedOperations''' patterns of that are based on the UML example, the complete query definition can be found in our repository: [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/examples/papyrus-uml/org.eclipse.viatra.examples.uml.evm/src/org/eclipse/viatra/examples/uml/evm/queries/preconditions.vql preconditions.vql]

<source lang="java">
/* Precondition for add generalization rule */
pattern possibleSuperClass(cl : Class, sup : Class) {
	neg find superClass(cl, _otherSup);
	neg find superClass(_cl2, sup);
}
/* Precondition for create owned operation */
pattern onlyInheritedOperations(cl : Class) {
	find hasOperation(cl, _op);
	neg find ownsOperation(cl, _ownOp);
}
</source>

=== Rule specifications ===

We define two rule specifications, both encapsulated by a method in [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/examples/papyrus-uml/org.eclipse.viatra.examples.uml.evm/src/org/eclipse/viatra/examples/uml/evm/UMLexampleForEVM.java UMLexampleForEVM.java].

The first rule specification uses the  '''possibleSuperClass''' pattern as a precondition and when executed for a given class pair, it creates a new Generalization element to set the class '''sup''' as a superclass for '''cl'''. The life-cycle is the most simple, where neither the updated, nor the disappeared state is used.

<source lang="java">
// the job specifies what to do when an activation is fired in the given state
Job job = Jobs.newStatelessJob(CRUDActivationStateEnum.APPEARED, new PossibleSuperClassProcessor() {
  @Override
  public void process(Class cl, Class sup) {
    System.out.println("Found cl " + cl + " without superclass");
    Generalization generalization = UMLFactory.eINSTANCE.createGeneralization();
    generalization.setGeneral(sup);
    generalization.setSpecific(cl);
  }
});
// the life-cycle determines how events affect the state of activations
DefaultActivationLifeCycle lifecycle = DefaultActivationLifeCycle.DEFAULT_NO_UPDATE_AND_DISAPPEAR;
// the factory is used to initialize the matcher for the precondition
IMatcherFactory<PossibleSuperClassMatcher> factory = PossibleSuperClassMatcher.factory();
// the rule specification is a model-independent definition that can be used to instantiate a rule
RuleSpecification spec = Rules.newSimpleMatcherRuleSpecification(factory, lifecycle, Sets.newHashSet(job));
</source>

The second rule specification is similar, it uses the '''onlyInheritedOperations''' pattern and when executed it creates a new operation with name '''newOp''' and adds it to the class which had no own property before.

<source lang="java">
Job job = Jobs.newStatelessJob(CRUDActivationStateEnum.APPEARED, new OnlyInheritedOperationsProcessor() {
  @Override
  public void process(Class cl) {
    System.out.println("Found class " + cl + " without operation");
    Operation operation = UMLFactory.eINSTANCE.createOperation();
    operation.setName("newOp");
    operation.setClass_(cl);
  }
});
DefaultActivationLifeCycle lifecycle = DefaultActivationLifeCycle.DEFAULT_NO_UPDATE_AND_DISAPPEAR;
IMatcherFactory<OnlyInheritedOperationsMatcher> factory = OnlyInheritedOperationsMatcher.factory();
RuleSpecification spec = Rules.newSimpleMatcherRuleSpecification(factory, lifecycle, Sets.newHashSet(job));
</source>

=== Fire activations manually using a rule engine ===

The first option when using the EVM is creating a rule engine, which manages the set of activations, but will not fire the enabled activations. A rule engine has no context of its own, so the user can create one to use when firing activations. The rule specifications above are returned by the two getter methods, and the '''addRule''' method is used on the rule engine to instantiate the rules.

<source lang="java">
// create rule engine over query engine
RuleEngine ruleEngine = RuleEngines.createViatraQueryRuleEngine(engine);
// create context for execution
Context context = Context.create();
// prepare rule specifications
RuleSpecification createGeneralization = getCreateGeneralizationRule();
RuleSpecification createOperation = getCreateOperationRule();
// add rule specifications to engine
ruleEngine.addRule(createGeneralization);
ruleEngine.addRule(createOperation);
</source>

Once a rule specification is added to the rule engine, the existing activations of a given rule can be retrieved from the rule engine and can fire them manually. Alternatively, the next activation as selected by the conflict resolver (which is a simple hash set without ordering) can be retrieved and fired.

<source lang="java">
// check rule applicability
Set<Activation> createClassesActivations = ruleEngine.getActivations(createGeneralization);
if (!createClassesActivations.isEmpty()) {
    // fire activation of a given rule
    createClassesActivations.iterator().next().fire(context);
}
// check for any applicable rules
while (!ruleEngine.getConflictingActivations().isEmpty()) {
    // fire next activation as long as possible
    ruleEngine.getNextActivation().fire(context);
}
</source>

As long as the rule engine exists, it will keep on managing the activations of added rules. It is possible to remove a single rule, which will remove all its activations from the rule engine, and the rule engine can be disposed when not needed anymore.

<source lang="java">
// rules that are no longer needed can be removed
ruleEngine.removeRule(createGeneralization);
// rule engine manages the activations of the added rules until disposed
ruleEngine.dispose();
</source>

=== Fire activations automatically with an execution schema ===

The second option for using the EVM is to create an execution schema that has a scheduler to fire activations after predefined events and an executor that specifies how to fire activations when scheduled. In the example, we use the feature in VIATRA Query Base that allows us to register a callback on model changes. An execution schema is created using a VIATRA Query engine and a scheduler factory, then the rules are added, in the same way as for the rule engine.

<source lang="java">
// use IQBase update callback for scheduling execution
UpdateCompleteBasedSchedulerFactory schedulerFactory = Schedulers.getIQEngineSchedulerFactory(engine);
// create execution schema over ViatraQueryEngine
ExecutionSchema executionSchema = ExecutionSchemas.createViatraQueryExecutionSchema(engine, schedulerFactory);
// prepare rule specifications
RuleSpecification createGeneralization = getCreateGeneralizationRule();
RuleSpecification createOperation = getCreateOperationRule();
// add rule specifications to engine
executionSchema.addRule(createGeneralization);
executionSchema.addRule(createOperation);
</source>

In the example we simply modify the model by removing a generalization from a random class. When the model modification is handled by VIATRA Query, the callback notifies the scheduler, which starts the executor, which in turn will fire enabled activations as-long-as-possible. The scheduler in the example uses the model update listener of the engine (https://bugs.eclipse.org/bugs/show_bug.cgi?id=398744) to get callbacks on changes.

<source lang="java">
// execution schema waits for a scheduling to fire activations
// we trigger this by removing one generalization at random
SuperClassMatcher.factory().getMatcher(engine).forOneArbitraryMatch(new SuperClassProcessor() {
    @Override
    public void process(Class sub, Class sup) {
        sub.getGeneralizations().remove(0);
    }
});
</source>

Similarly to the rule engine, it is possible to remove a rule from the execution schema or to dispose it when not needed any longer. The main difference between the rule engine and the execution schema is, that once a rule has been added, the activations that are enabled will be executed automatically every time the scheduler is notified. This allows us to implement components that can react to changes incrementally, without requiring additional scaffolding.

<source lang="java">
// rules that are no longer needed can be removed
executionSchema.removeRule(createGeneralization);
// execution schema manages and fires the activations of the added
// rules until disposed
executionSchema.dispose();
</source>

=== Impose ordering between activations of different rules ===

Activations that are enabled are in conflict with each other since firing any of them can cause the other activations to become disabled.
The conflict set of a rule engine is the set of enabled activations of all rules, and users can define a conflict resolver that provides 
an ordering in the conflict set (https://bugs.eclipse.org/bugs/show_bug.cgi?id=403825).

=== Enabling log messages in EVM ===

Due to the event-driven nature of EVM, it is often difficult to debug your program, since the control flow will go through EVM internals and activation life-cycle is handled independently of activation firing.
In order to see exactly what happens inside EVM, you can set the log level of the Log4J logger of rule engines to display DEBUG or even TRACE level messages.
The log will include events, activation state changes, scheduling and executor events, firing and other useful information.

<source lang="java">
// just set the log level of the engine as needed
ruleEngine.getLogger().setLevel(Level.DEBUG);
</source>

== Usage scenarios ==

Both the data binding and validation frameworks of VIATRA use the EVM for handling events.

* Data binding: observable match result collections are created in the createRuleSpecification method of ObservableCollectionHelper (org.eclipse.viatra.databinding.runtime.collection package)
* Validation constraints are created in the constructor of ConstraintAdapter (org.eclipse.viatra.validation.runtime package)

=== Programming against the EVM API ===

The basic usage of the EVM is to react to match set changes easily.

==== Efficiently reacting to pattern match set changes ====

If you want to efficiently react to appearing, changing or disappearing matches, the EVM is a perfect choice.
Just define a rule specification with the correct life-cycle and jobs and create an execution schema as described above.

* If you only want to react to appearance events: use DefaultActivationLifeCycle.DEFAULT_NO_UPDATE_AND_DISAPPEAR and a single APPEARED job.
* If you want to react to both appearance and disappearance: use DefaultActivationLifeCycle.DEFAULT_NO_UPDATE and two jobs, one with APPEARED and the other with DISAPPEARED state.
* If you want to react to changes of match parameters (e.g. an attribute value changes, but the match still exists): use DefaultActivationLifeCycle.DEFAULT and add an additional job with UPDATED state.

=== Roll your own event provider for EVM ===

The EVM core is independent of EMF and VIATRA Query, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=406558
You can create your own event realm and use the EVM core concepts to execute event-driven rules. You can see a small example in http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/examples/evm-proto

== Design decisions and code style ==

These guidelines are derived from the main decision to create a defensive framework to minimize the internal argument checks and idiot-proofing required.

=== User interaction with the framework ===

* Users interact with rules and activations through Façade classes:
** RuleEngine façade provides access to an Agenda and it's rule instances
** ExecutionSchema façade provides access to a Scheduler and through that to the Executor
** These Façade classes can be retrieved through the static methods of EventDrivenVM or by static create methods (for specific implementations)
* Any object that a user can access through the Façade must have only public methods that do not endanger their engine (e.g. Activation.fire(), but not Activation.setState())
* Any collection that a user can access through the Façade must be immutable to avoid modifications (e.g. getActivations)
* Any object that is provided by the user must be copied if it's later modification can cause internal problems (e.g. life cycle for rules)

=== Parameters, input checking and logging ===

* Method parameters cannot be null!
** This is checked by Preconditions.checkNotNull(ref, msg). Return a meaningful message on null.
** Use the ''this.field = checkNotNull(field, msg)'' form in constructors when possible.
** Define delegate methods where optional parameters are allowed.
* All logging is done through the rule base, use the debug level for detailed report messages and error or warning when encountering a real problem (e.g. ViatraQueryException)

=== Default implementations ===

There are a high number of notification mechanisms and event processing, that must have an interface for extendibility, a good default implementation and a clear way of overriding.

* In RuleInstance notification providers and listeners are created in prepareX methods.
* Default life cycles prepared with unmodifiable static instances.
* Update complete provider implementations (IQBase and EMF transaction).
* Scheduler implementations (update complete and timed).
* Job implementations (stateless with a single match processor, and recording for transactional model modification).
** Job implementations can specify their own error handling (https://bugs.eclipse.org/bugs/show_bug.cgi?id=404307)

== Development ideas ==

=== Other future tasks ===

* systematic testing: both unit and component tests on API classes and default implementation

* remove log4j dependency

* add identifier to RuleSpecification (also allow finding rule by id in RuleEngine?)

* support main transformation scenarios
** examples:
*** batch
*** single job appear-disappear
*** semi and fully automated execution
*** ignore initial activations
** part of this will be done in the viatra.emf.runtime plugin

* Completely remove generics from EVM core

* create rule specification builder:
** should this be specific to VIATRA Query or generic? 
<source lang="java">RuleSpecification.builder(eventSourceSpecification).putJob(job1).putJob(job2).putLifecycle(lifecycle).build() // build creates RuleSpecification
 Rules.builder(querySpecification).putAppear(appearProcessor).putDisappear(disappearProcessor).putLifecycle(lifecycle).build()
 Rules.builder(querySpecification).ofAppear(appearProcessor) // only appear job, lifecycle selected based on that unless specified with putLifecycle
</source>

* support retrieval of filtered activations of a rule with a partial match
** also make filters that are reusable between rule specifications

* integrate into upstream components
** query-based features

* ensure that commands stored by RecordingJobs are only kept in Context when necessary or for a short time (e.g. clear at start of scheduled execution?)

* event queueing is problematic (job execution during activation firing can change the state)