Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Xtext/Documentation
Contents
What is Xtext?
The TMF Xtext project provides a domain-specific language (the grammar language) for description of textual programming languages and domain-specific languages. It is tightly integrated with the Eclipse Modeling Framework (EMF) and leverages the Eclipse Platform in order to provide language-specific tool support. In contrast to common parser generators the grammar language is much simpler but is used to derive much more than just a parser and lexer. From a grammar the following is derived:
- incremental, Antlr3-based parser and lexer
- Ecore-based meta models (optional)
- a serializer, used to serialize instances of such meta models back to a parseable textual representation
- an implementation of the EMF Resource interface (based on the parser and the serializer)
- a full-fledged integration of the language into Eclipse IDE
- syntax coloring
- navigation (F3, etc.)
- code completion
- outline views
- code templates
- folding, etc.
The generated artifacts are wired up through a dependency injection framework, which makes it easy to exchange certain functionality in a non-invasive manner. For example if you don't like the default code assistant implementation, you need to come up with an alternative implementation of the corresponding service and configure it via eclipse extension point.
The Grammar Language
At the heart of Xtext there is the grammar language. The grammar language is defined in itself, of course. The grammar can be found here [[1]].
It is a carefully designed DSL for description of textual languages, based on ANTLR's LL(*) algorithm. Mainly one describes the concrete syntax and how to construct an in-memory model (semantic model) from that.
First an example
To get an idea of how it works we'll start by implementing a [simple example] introduced by from Martin Fowler. It's mainly about describing state machines used as the (un)lock mechanism of secret compartments.
One of those state machines could look like this:
events doorClosed D1CL drawOpened D2OP lightOn L1ON doorOpened D1OP panelClosed PNCL end resetEvents doorOpened end commands unlockPanel PNUL lockPanel PNLK lockDoor D1LK unlockDoor D1UL end state idle actions {unlockDoor lockPanel} doorClosed => active end state active drawOpened => waitingForLight lightOn => waitingForDraw end state waitingForLight lightOn => unlockedPanel end state waitingForDraw drawOpened => unlockedPanel end state unlockedPanel actions {unlockPanel lockDoor} panelClosed => idle end
So we have a bunch of declared events, commands and states. Within states there are references to declared actions, which should be executed when entering such a state. Also there are transitions consisting of a reference to an event and a state. Please read Martin's description is it is not clear enough.
In order to implement this language with Xtext you need to write the following grammer:
language SecretCompartments generate secretcompartment "http://www.eclipse.org/secretcompartment" Statemachine : 'events' events+=Event+ 'end' ('resetEvents' resetEvents+=[Event]+ 'end')? 'commands' commands+=Command+ 'end' states+=State+; Event : name=ID code=ID; Command : name=ID code=ID; State : 'state' name=ID ('actions' '{' actions+=[Command]+ '}')? transitions+=Transition* 'end'; Transition : event=[Event] '=>' state=[State];
Language Declaration
The first line
language SecretCompartments
declares the name of the language. Xtext leverages Java's classpath mechanism. this means that the name can be any valid Java qualifier. The file name needs to correspond and have the file extension '*.xtext'. So it needs to be "SecretCompartments.xtext" and must be placed in the default package on the Java's class path.
If you want to place it within a package (e.g. 'foo/SecretCompartment.xtext') the first line must read:
language foo.SecretCompartment
The first line is also used to declare a super language to inherit from. This mechanism is described here.
Rules
Parser Rules
Lexer Rules
Model Construction
Meta Models
Meta-Model Inference
Importing existing Meta Models
Language Inheritance
Xtext support language inheritance. By default (implicitly) each language extends a language called *org.eclipse.xtext.builtin.XtextBuiltin* and is defined as follows:
abstract language org.eclipse.xtext.builtin.XtextBuiltIn_Temp import "http://www.eclipse.org/emf/2002/Ecore" as ecore; lexer ID : "('^')?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*"; lexer INT returns ecore::EInt : "('0'..'9')+"; lexer STRING : " '\"' ( '\\\\' ('b'|'t'|'n'|'f'|'r'|'\\\"'|'\\''|'\\\\') | ~('\\\\'|'\"') )* '\"' | '\\'' ( '\\\\' ('b'|'t'|'n'|'f'|'r'|'\\\"'|'\\''|'\\\\') | ~('\\\\'|'\\'') )* '\\'' "; lexer ML_COMMENT : "'/*' ( options {greedy=false;} : . )* '*/' {$channel=HIDDEN;}"; lexer SL_COMMENT : "'//' ~('\\n'|'\\r')* '\\r'? '\\n' {$channel=HIDDEN;}"; lexer WS : "(' '|'\\t'|'\\r'|'\\n')+ {$channel=HIDDEN;}"; lexer ANY_OTHER : ".";
Just ignore the grammar if you don't yet understand it. It basically provides some commonly used lexer rules which can be used in all grammars.