Skip to main content
Jump to: navigation, search

Difference between revisions of "Xtext/planning 0.8.0"

m (General Cleanups)
(UI Features)
Line 115: Line 115:
*Provide an Eclipse-Builder that runs the user's code generation when the models are modified. The Default-Builder could always run a full build and be customizable to implement an incremental build
*Provide an Eclipse-Builder that runs the user's code generation when the models are modified. The Default-Builder could always run a full build and be customizable to implement an incremental build
*Automatically derive templates for code completion from the grammar
*Automatically derive templates for code completion from the grammar
*Builder that validates model files (obsolete with emf index integration? what about external changes?)
=== Xtext Grammar  ===
=== Xtext Grammar  ===

Revision as of 14:39, 4 August 2009

Here we collect what we want to do in the upcoming development (Helios) for Xtext.

Infrastructure and Organization

Build infrastructure

We need to improve our build infrastructure as well as the general workspace setup. Some requirements:

  • use a dedicated target platform
  • be able to do a local build

It would be quite nice to have a sort of deferred commit: The data is send to the integration server, it compiles everything and executes the test and refuses the commit if something fails. Otherwise the patch is finally committed to the version control system.

Development Infrastructure

We should come up with a minimal set of style guidelines and conventions to minimize the necessary effort to merge changes and understand code. Ideally, we should try to stick to Eclipse defaults as much as possible, as this minimizes configuration efforts. We should use project specific settings wherever applicable to minimize the risk of the I-forgot-to-import-some-settings-scenario. This includes

  • encoding,
  • code style,
  • save participants,
  • compiler warnings,
  • organize imports and
  • pde settings.

Clean ups

General Cleanups

  • delete no longer used projects from repository
  • delete dead resources from workspace (e.g. check files and the like)
  • remove Xtend-based scoping and code completion API
  • unify Guice injection hooks (field, setter, vs. constructor) where possible


  • Review the Scoping API. We should provide a sort of IScopeExtension with methods like #getElement(String):IScopedElement and #getElement(EObject):IScopedElement. This allows more sophisticated implementations of the API.
  • Review the ValueConverter API: the method which turns Objects into strings should have the corresponding part of Node Model available, if such a part exists. This is needed to keep comments within Datatype-Rule-Values and it is helpful to keep the escaping of special characters (e.g. unicode) as it is.
  • Review the Highlighting API: Merge the *Configuration-Interfaces.
  • Review the implementation of the Classpath URI. Try to find a solution, that allows this kind of URI even in other ResourceSet implementations. Maybe consider deprecating the classpath scheme and replace it with a functionally equal implementation based on the index.
  • Review the LabelProvider API. Instead returning a file name of an icon, it would be better to return an Image/ImageDestriptor, since this would allow the usage of overlay icons.

Xtext Grammar

  • The linking of the grammar should be reviewed and updated to reflect the latest changes in the API and to be a better example on how to use the different available APIs. Furthermore there are still some possible links, that are currently plain strings (for example: Assignment.feature is still a string and not a cross-reference to the EStructuralFeature).
  • Rules that do not (always) create objects, when consumed, can lead to confusion (see #283321 and #284428). It might be a good idea to always create an object.

Project structure

Move as much as possible from org.eclipse.xtext into org.eclipse.xtext.xtext (similar to org.eclipse.xtext.xtext.ui). Merge org.eclipse.xtext.ui.core and org.eclipse.xtext.ui.common into org.eclipse.xtext.ui as the difference between both plugins is somewhat indistinct.

Cleanup generator.tests

Observation: It takes an awful amount of time to regenerate all test languages, refresh resources, build java code, synchronize with repository, etc. This large turnaround-time slows down the development process.

  • Create workflow-files for each java-package (that contains .xtext-languages) in /src. We already have that for some packages. This allows to regenerate only the needed languages when working on the tests.
  • GenerateAllTestLanguages.mwe should delegate to the workflows residing in the java-packages. This allows custom generator fragments for each language and avoids having two different generator configurations at the same time for one language.
  • Each language should only have the minimal amount GeneratorFragments activated. The keeps the amount of Java-Files to a minimum and should thereby speed up "refresh resources", "rebuild", "synchronize with rep." etc. Do we still support dynamic EMF? The downside is that a minimal set of fragments would lead to less testing of the fragments themselves.

Third party libs

Update the used libraries:

  • Guice2: Multibinding support, binding overrides, child injectors
  • Google Collections 1.0: We currently use v 0.8 alpha. The API has changed significantly and we should update to 1.0.
  • ANTLR 3.1.x: A bunch of fixes and new features has been incorporated into ANTLR 3.1. Especially the support for grammar imports seems to be interesting.

New Features

Introduce DSLs for some of the stable viewpoints of Xtext

  • should extend one basic expression language (something like [1])
  • Validation/Quickfix
  • EMFatic (incl. behavior)
  • Scoping
  • Indexing

Migrate to upcoming EMF Index changes

  • Query Builder
  • API / SPI separation
  • automatic revalidation if referenced resources have changed
  • refactoring of validation so that
    • in the reconciler no markers (just annotations are set)
    • on save markers are set using a builder
    • on save of referenced resources markers are set using a corresponding index hook

Runtime Features

  • Formatting API that allows to define rules depending on the current state of your model.
  • Formatting API that allows to define rules for grammar elements that are not explicitly represent in the Node Model. There are two ways to implement this: Extend the Node Model to contain all Grammar Elements or to derive the Grammar Elements position from the grammar's state machine.
  • find-Methods for GrammarAccess. Example: ga.findKeywords(";"), ga.getMyRuleAccess().findKeywords(","), ga.getMyRuleAccess().findRuleCallsToThisRule(), ga.findAllKeywordPairs("{", "}")
  • Provide a best practice on how to implement a property sheet that (partially) configures formatting. I don't think its a good idea to expose the formating rules on the property sheet. They are to fine grained/technical.
  • Allow tracing / impact analysis to implement partial linking, validation etc.
  • Introduce an official API that allows to define default values of the semantic model. (maybe through above's EMFatic DSL) (moritz: why is the common post processing hook for the semantic model not sufficient? Sebastian: Only very simple usecases require static default values. In many other scenarios we'll need defaults based on the state of the model.)
  • Derive an EMF Validator from the grammar which checks if a model complies with all the implicit constrains of the grammar. Currently I (moritz) am aware of three kind of constraints:
    • Quantities: QRule: (a+=ID b+=INT); implies that a.size() == b.size()
    • Orders: X: Y | Z; ORule: a+=Y a+=Z; a is of type X, but the grammar implies that it may only contain instances of Y and Z and(!) that all instance of Y must be positioned before Z's instance in list a
    • Values: all values must comply with their TerminalRules/DatatypeRules/ValueConverters
  • Currently, the EcoreGeneratorFragment is inflated with configuration flags to customize the .genmodel. However, its not the EcoreGeneratorFragments role to be a facade for the genmodel. It would be better to a have a common post processing hook (via Xtend?) as we already have it for the semantic model. This hook, however, would only needed to be triggered by the EcoreGeneratorFragment and not by the editor. See also [2].

UI Features

  • (delete me if this is already solved) Update the editor if the model is changed. If single EAttributes are modified, only their values need to be updated in the editor. If EReference are modified, the ParseTreeConstructor for the EObject and all of its children need to be executed. If an EAttributes contributes to an EObjects referenceable name, all cross references pointing to this EObject need to be found and updated. Since each grammar implies constraints, not all models are serializeable. Therefore, the model can not be serialized after each modification, but only after "some kind of transaction" has finished.
  • AutoEdit: This would greatly improve the user experience and help to minimize the on-type-changes in the semantic model. Without autoedit, the model becomes totally corrupt when the user tries to write a STRING literal, because everything behind the opening quotes will be consumend as a STRING. In almost any case this is not the intention of the user. Autoedit would insert a matching closing quote after to opening quote and the model will remain much more stable. Btw: Autoedit is not the right term here. JDT implemented this with a VerifyKeyListener (see org.eclipse.jdt.internal.ui.javaeditor.CompilationUnitEditor.BracketInserter).
  • Quick fixes, QuickAssist-API
  • Advanced Navigation / Hyperlinking API: Find References, allow to define references by convention, etc.
  • a general API that allows to define arbitrary views for model elements (comparable to viewpoints in the Outline)
  • View points in the Outline
  • Hook for tooltips: users might want to display additional information when the mouse hovers over a reference, e.g. they might want to see the rafeerenced element's documentation. We should provide a hook for feeding this information.
  • Wizards
    • Wizard-Framework
    • Enhance "New Xtext Project" wizard: let user choose among various examples (see PDE New Project wizard to see this in action)
    • Add "New DSL Project" wizard to generated DSL, supply hooks for customization
  • Refactoring
    • rename refactoring
    • delete (model element / model file) refactoring - notifies the user if there are any references to the to-be-deleted element / file
  • Provide an Eclipse-Builder that runs the user's code generation when the models are modified. The Default-Builder could always run a full build and be customizable to implement an incremental build
  • Automatically derive templates for code completion from the grammar
  • Builder that validates model files (obsolete with emf index integration? what about external changes?)

Xtext Grammar

  • Terminal fragments: Reusable elements in terminal rules, that are not globally visible (see ANTLR Fragments)
  • Aspects in production rules: Reuse different aspects of the grammar across various parser rules. One may think of infered EObjects that implement different interfaces such as Named, Documentable or FooContainer or BarReference. (You may call it a fragment).
  • Allow a notation to call the super-definition of an overridden parser rule.
  • Allow to define the default of an EEnum (even with an empty literal representation).
  • Try to generate named opposite references for eContainer.
  • add support for semantic predicates
  • add support for syntactic predicates
  • allow mixin of multiple grammars
  • allow names for arbitrary grammar elements: this would make the GrammarAccess more robust with regards to grammar modifications.
  • Consider introducing annotations. Potential use cases that come to mind (I suppose the user will come up with many more!): @Override for rules, default values for attributes, formatting. With different retention policies the annotation processors could hook into various stages: model inference (linking), generation (using accompanying fragment).

Xtext Grammar Editor

Debug-support for to Java compiled languages

If one had the trace from the Java Code back to the model, it might be possible to hook JDT's debugger in order to show the model instead of the java code during debugging. Also it would be nice if we could come up with suitable abstractions over what is shown in the variables view then.

Generator Features

  • Consider making the Xtext generator framework (i.e. generator fragments) more sophisticated by introducing some kind of GeneratorContext (subsuming the Grammar and XpandExecutionContext). This context could contain:
    • Some general information (e.g. names and paths to runtime and UI project and file extensions used by DSL).
    • Log of already executed fragments, which would allow a fragment to verify that a certain other fragment has or has not been executed.
    • Information provided (written to the context) by the executed fragments (e.g. GenModels generated / used by the EcoreGeneratorFragment), which subsequent fragments can use. I think the EcoreGeneratorFragment is a particularly good example as fragments will often want to generate code relying on the GenModel generated code.

Back to the top