|
|
| (207 intermediate revisions by 9 users not shown) |
| Line 1: |
Line 1: |
| − | = What is Xtext? =
| + | #REDIRECT [http://www.eclipse.org/Xtext/documentation.html Please view the documentation at our website] |
| − | | + | |
| − | The TMF Xtext project provides a domain-specific language (the grammar language) for description of textual programming languages and domain-specific languages. It is tightly integrated with the Eclipse Modeling Framework (EMF) and leverages the Eclipse Platform in order to provide language-specific tool support.
| + | |
| − | In contrast to common parser generators the grammar language is much simpler but is used to derive much more than just a parser and lexer.
| + | |
| − | From a grammar the following is derived:
| + | |
| − | | + | |
| − | * incremental, Antlr3-based parser and lexer
| + | |
| − | * Ecore-based meta models (optional)
| + | |
| − | * a serializer, used to serialize instances of such meta models back to a parseable textual representation
| + | |
| − | * an implementation of the EMF Resource interface (based on the parser and the serializer)
| + | |
| − | * a full-fledged integration of the language into Eclipse IDE
| + | |
| − | ** syntax coloring
| + | |
| − | ** navigation (F3, etc.)
| + | |
| − | ** code completion
| + | |
| − | ** outline views
| + | |
| − | ** code templates
| + | |
| − | ** folding, etc.
| + | |
| − | | + | |
| − | The generated artifacts are wired up through a dependency injection framework, which makes it easy to exchange certain functionality in a non-invasive manner.
| + | |
| − | For example if you don't like the default code assistant implementation, you need to come up with an alternative implementation of the corresponding service and configure it via eclipse extension point.
| + | |
| − | | + | |
| − | = The Grammar Language =
| + | |
| − | At the heart of Xtext there is the grammar language. The grammar language is defined in itself, of course. The grammar can be found here [[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.tmf/org.eclipse.xtext/plugins/org.eclipse.xtext/src/org/eclipse/xtext/Xtext.xtext?root=Modeling_Project&view=markup]].
| + | |
| − | | + | |
| − | The grammar leanguage is a DSL carefully designed for description of textual languages, based on ANTLR's LL(*) algorithm.
| + | |
| − | The main idea is to let users describe the concrete syntax, and to automatically derive an in-memory model (semantic model) from that.
| + | |
| − | | + | |
| − | == First an example ==
| + | |
| − | To get an idea of how it works we'll start by implementing a [[http://martinfowler.com/dslwip/Intro.html simple example]] introduced by from Martin Fowler.
| + | |
| − | It's mainly about describing state machines used as the (un)lock mechanism of secret compartments.
| + | |
| − | | + | |
| − | One of those state machines could look like this:
| + | |
| − | <source lang="text">
| + | |
| − | events
| + | |
| − | doorClosed D1CL
| + | |
| − | drawOpened D2OP
| + | |
| − | lightOn L1ON
| + | |
| − | doorOpened D1OP
| + | |
| − | panelClosed PNCL
| + | |
| − | end
| + | |
| − |
| + | |
| − | resetEvents
| + | |
| − | doorOpened
| + | |
| − | end
| + | |
| − |
| + | |
| − | commands
| + | |
| − | unlockPanel PNUL
| + | |
| − | lockPanel PNLK
| + | |
| − | lockDoor D1LK
| + | |
| − | unlockDoor D1UL
| + | |
| − | end
| + | |
| − |
| + | |
| − | state idle
| + | |
| − | actions {unlockDoor lockPanel}
| + | |
| − | doorClosed => active
| + | |
| − | end
| + | |
| − |
| + | |
| − | state active
| + | |
| − | drawOpened => waitingForLight
| + | |
| − | lightOn => waitingForDraw
| + | |
| − | end
| + | |
| − |
| + | |
| − | state waitingForLight
| + | |
| − | lightOn => unlockedPanel
| + | |
| − | end
| + | |
| − |
| + | |
| − | state waitingForDraw
| + | |
| − | drawOpened => unlockedPanel
| + | |
| − | end
| + | |
| − |
| + | |
| − | state unlockedPanel
| + | |
| − | actions {unlockPanel lockDoor}
| + | |
| − | panelClosed => idle
| + | |
| − | end
| + | |
| − | </source>
| + | |
| − | So we have a bunch of declared events, commands and states. Within states there are references to declared actions, which should be executed when entering such a state.
| + | |
| − | Also there are transitions consisting of a reference to an event and a state. Please read Martin's description is it is not clear enough.
| + | |
| − | | + | |
| − | In order to implement this language with Xtext you need to write the following grammer:
| + | |
| − | <source lang="text">
| + | |
| − | language SecretCompartments
| + | |
| − |
| + | |
| − | generate secretcompartment "http://www.eclipse.org/secretcompartment"
| + | |
| − |
| + | |
| − | Statemachine :
| + | |
| − | 'events'
| + | |
| − | events+=Event+
| + | |
| − | 'end'
| + | |
| − | ('resetEvents'
| + | |
| − | resetEvents+=[Event]+
| + | |
| − | 'end')?
| + | |
| − | 'commands'
| + | |
| − | commands+=Command+
| + | |
| − | 'end'
| + | |
| − | states+=State+;
| + | |
| − |
| + | |
| − | Event :
| + | |
| − | name=ID code=ID;
| + | |
| − |
| + | |
| − | Command :
| + | |
| − | name=ID code=ID;
| + | |
| − |
| + | |
| − | State :
| + | |
| − | 'state' name=ID
| + | |
| − | ('actions' '{' actions+=[Command]+ '}')?
| + | |
| − | transitions+=Transition*
| + | |
| − | 'end';
| + | |
| − |
| + | |
| − | Transition :
| + | |
| − | event=[Event] '=>' state=[State];
| + | |
| − | </source>
| + | |
| − | | + | |
| − | In the following the different concepts of the grammar language are explained. We refer to this grammar when useful.
| + | |
| − | | + | |
| − | == Language Declaration ==
| + | |
| − | The first line
| + | |
| − | language SecretCompartments
| + | |
| − | | + | |
| − | declares the name of the language.
| + | |
| − | Xtext leverages Java's classpath mechanism. this means that the name can be any valid Java qualifier.
| + | |
| − | The file name needs to correspond and have the file extension '*.xtext'.
| + | |
| − | So it needs to be "SecretCompartments.xtext" and must be placed in the default package on the Java's class path.
| + | |
| − | | + | |
| − | If you want to place it within a package (e.g. 'foo/SecretCompartment.xtext') the first line must read:
| + | |
| − | language foo.SecretCompartment
| + | |
| − | | + | |
| − | The first line can also be used to declare a super language to inherit from. This mechanism is described [[#Language Inheritance| here]].
| + | |
| − | | + | |
| − | == EPackage declarations ==
| + | |
| − | Xtext parsers instantiate Ecore models (aka meta model). An Ecore model basically consists of an EPackage containing EClasses, EDatatypes and EEnums.
| + | |
| − | Xtext can infer Ecore models from a grammar (see [[#Metamodel Inference]]) but it is also possible to instantiate existing Ecore models. You can even mix this and use multiple existing ecore models and infer some others from one grammar.
| + | |
| − | | + | |
| − | === EPackage generation ===
| + | |
| − | The easiest way to get started is to let Xtext infer the meta model from your grammar. This is what is done in the secret compartment example.
| + | |
| − | To do so just state:
| + | |
| − | <code lang="text">
| + | |
| − | generate secretcompartment "http://www.eclipse.org/secretcompartment"
| + | |
| − | </code>
| + | |
| − | | + | |
| − | Which says: generate an EPackage with name secretcompartment and nsURI "http://www.eclipse.org/secretcompartment" (these are the properties needed to create an EPackage).
| + | |
| − | | + | |
| − | === EPackage import ===
| + | |
| − | If you already have create such an EPackage somehow, you could import it:
| + | |
| − | <code lang="text">
| + | |
| − | import "http://www.eclipse.org/secretcompartment"
| + | |
| − | </code>
| + | |
| − | | + | |
| − | === Using multiple packages ===
| + | |
| − | If you want to use multiple EPackages you need to specify aliases like so:
| + | |
| − | <code lang="text">
| + | |
| − | generate secretcompartment "http://www.eclipse.org/secretcompartment"
| + | |
| − | import "http://www.eclipse.org/anotherPackage" as another
| + | |
| − | </code>
| + | |
| − | | + | |
| − | When referring to a type somewhere in the grammar you need to qualify them using that alias (example "another::CoolType"). We'll see later where such type references occur.
| + | |
| − | | + | |
| − | == Rules ==
| + | |
| − | The parsing is based on ANTLR 3, which is a parser generator framework based on an [[http://www.antlr.org/wiki/display/ANTLR3/3.+LL(*)+Parsing+(Excursion) LL(*) algorithm]].
| + | |
| − | Basically parsing can be separated in the following phases.
| + | |
| − | # lexing
| + | |
| − | # parsing
| + | |
| − | # model construction
| + | |
| − | # linking
| + | |
| − | # validation
| + | |
| − | | + | |
| − | === Lexer Rules ===
| + | |
| − | In the first phase a sequence of characters (the text input) is transformed into a sequence of so called tokens.
| + | |
| − | Each token consists of one or more characters and was matched by a particular lexer rule.
| + | |
| − | In the secret compartments example there are no explicitly defined lexer rules, since it uses built-in lexer rules, only (the ID rule).
| + | |
| − | | + | |
| − | That rule is defined in the built-in super language (see [[#Language Inheritance]]) as follows:
| + | |
| − | <code lang="text">
| + | |
| − | lexer ID :
| + | |
| − | "('^')?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*";
| + | |
| − | </code>
| + | |
| − | | + | |
| − | It says that a Token ID starts with an optional '^' character followed by a letter ('a'..'z'|'A'..'Z') or underscore ('_') followed by any number of letters, underscores and numbers ('0'..'9'). Note that this declaration is is a black box where you use Antlr syntax directly. Please ignore the optional '^' for the moment.
| + | |
| − | | + | |
| − | This is the formal definition of lexer rules:
| + | |
| − | | + | |
| − | <code lang="text">
| + | |
| − | LexerRule :
| + | |
| − | 'lexer' name=ID ('returns' type=TypeRef)? ':' body=STRING ';'
| + | |
| − | ;
| + | |
| − | </code>
| + | |
| − | | + | |
| − | ==== Return types ====
| + | |
| − | A lexer rule returns a value, which defaults to a string (type ecore::EString).
| + | |
| − | However, if you want to have a different type you can specify it. For instance, the built-in lexer rule 'INT' is defined like so:
| + | |
| − | | + | |
| − | <code lang="text">
| + | |
| − | lexer INT returns ecore::EInt :
| + | |
| − | "('0'..'9')+";
| + | |
| − | </code>
| + | |
| − | | + | |
| − | This says, that the lexer rule INT returns instances of ecore::EInt. It is possible to define any kind of data type here, which just need to be an instance of ecore::EDataType.
| + | |
| − | In order to tell the parser how to convert the lexed string to a value of the declared data type, you need to provide your own implementation of 'IValueConverterService'.
| + | |
| − | | + | |
| − | Have a look at [[http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.tmf/org.eclipse.xtext/plugins/org.eclipse.xtext/src/org/eclipse/xtext/builtin/conversion/XtextBuiltInConverters.java?root=Modeling_Project&view=markup org/eclipse/xtext/builtin/conversion/XtextBuiltInConverters.java]] to find out how such an implementation looks like.
| + | |
| − | | + | |
| − | The implementation needs to be registered as a service (see [[#Service Framework]]).
| + | |
| − | | + | |
| − | ==== Enum Rules ====
| + | |
| − | TODO
| + | |
| − | ==== String Rules ====
| + | |
| − | TODO
| + | |
| − | | + | |
| − | === Parser Rules ===
| + | |
| − | The parser reads in a sequence of tokens produced by the lexer and walks through the parser rules.
| + | |
| − | | + | |
| − | <code lang="text">
| + | |
| − |
| + | |
| − | </code>
| + | |
| − | | + | |
| − | == Model Construction ==
| + | |
| − | == Meta-Models ==
| + | |
| − | | + | |
| − | The meta-model of a textual language describes the structure of its abstract syntax trees (AST).
| + | |
| − | === Meta-Model Inference ===
| + | |
| − | If you use the [[EPackage _generation]]
| + | |
| − | === Importing existing Meta Models ===
| + | |
| − | | + | |
| − | == Language Inheritance ==
| + | |
| − | Xtext support language inheritance. By default (implicitly) each language extends a language called *org.eclipse.xtext.builtin.XtextBuiltin* and is defined as follows:
| + | |
| − | <source lang="text">
| + | |
| − | abstract language org.eclipse.xtext.builtin.XtextBuiltIn_Temp
| + | |
| − |
| + | |
| − | import "http://www.eclipse.org/emf/2002/Ecore" as ecore;
| + | |
| − |
| + | |
| − |
| + | |
| − | lexer ID :
| + | |
| − | "('^')?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*";
| + | |
| − |
| + | |
| − | lexer INT returns ecore::EInt :
| + | |
| − | "('0'..'9')+";
| + | |
| − |
| + | |
| − | lexer STRING :
| + | |
| − | "
| + | |
| − | '\"' ( '\\\\' ('b'|'t'|'n'|'f'|'r'|'\\\"'|'\\''|'\\\\') | ~('\\\\'|'\"') )* '\"' |
| + | |
| − | '\\'' ( '\\\\' ('b'|'t'|'n'|'f'|'r'|'\\\"'|'\\''|'\\\\') | ~('\\\\'|'\\'') )* '\\''
| + | |
| − | ";
| + | |
| − |
| + | |
| − | lexer ML_COMMENT :
| + | |
| − | "'/*' ( options {greedy=false;} : . )* '*/' {$channel=HIDDEN;}";
| + | |
| − |
| + | |
| − | lexer SL_COMMENT :
| + | |
| − | "'//' ~('\\n'|'\\r')* '\\r'? '\\n' {$channel=HIDDEN;}";
| + | |
| − |
| + | |
| − | lexer WS :
| + | |
| − | "(' '|'\\t'|'\\r'|'\\n')+ {$channel=HIDDEN;}";
| + | |
| − |
| + | |
| − | lexer ANY_OTHER :
| + | |
| − | ".";
| + | |
| − | </source>
| + | |
| − | Just ignore the grammar if you don't yet understand it. It basically provides some commonly used lexer rules which can be used in all grammars.
| + | |
| − | | + | |
| − | = Service Framework =
| + | |
| − | | + | |
| − | = Runtime Architecture =
| + | |
| − | == Value Converters ==
| + | |
| − | == Linking ==
| + | |
| − | | + | |
| − | = UI Architecture =
| + | |
