Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Gymnast"

(POJO-based CSTs)
(Where to download Gymnast from)
 
(6 intermediate revisions by 3 users not shown)
Line 3: Line 3:
 
Gymnast is a framework for jumpstarting text editors for custom Domain Specific Languages, written by Chris Daly, and summarized in [http://www.sts.tu-harburg.de/%7Emi.garcia/SoC2007/GymnastSlides.pdf these] slides.  
 
Gymnast is a framework for jumpstarting text editors for custom Domain Specific Languages, written by Chris Daly, and summarized in [http://www.sts.tu-harburg.de/%7Emi.garcia/SoC2007/GymnastSlides.pdf these] slides.  
  
Gymnast is being distributed along with [[Emfatic]], as their developer communities overlap (at least, so far).
+
Gymnast is being distributed along with [[Emfatic]], as their developer communities overlap (at least, so far).  
 +
 
 +
In the past, many improvements has been included in Gymnast. For example, JavaCC and Antlr lexer files can be autogenerated now instead of manually creating and adding them to the project. This article reflects those changes made to the Gymnast framework. Please refer to [[Grammar2Ecore]] for details on the generation of Java classes from grammar.
 +
 
 +
== Where to download Gymnast from ==
 +
 
 +
[[Image:WhereToDownloadGymnastFrom.png]]
  
 
== Overview ==
 
== Overview ==
  
Gymnast comprises two main parts (Runtime and Generator), each of which can in
+
Gymnast comprises two main parts (Runtime and Generator), each of which can in turn be divided into Core and User Interface. This section is all about using the Gymnast Generator to:
turn be divided into Core and User Interface. This section is all about using the
+
Gymnast Generator to:
+
  
 
<ol>
 
<ol>
 
   <li>create <code>.ast</code> files</li>
 
   <li>create <code>.ast</code> files</li>
 
   <li>configure options in that file</li>
 
   <li>configure options in that file</li>
   <li>manually add a lexer file, and</li>
+
   <li>autogenerate lexer files, and</li>
 
   <li>manually add the parser's <code>.jar</code> to the (Build path or Plugin dependencies)</li>
 
   <li>manually add the parser's <code>.jar</code> to the (Build path or Plugin dependencies)</li>
 
</ol>
 
</ol>
Line 20: Line 24:
 
so that POJO-style CST classes can be generated and compiled. More details are discussed in the technical report  
 
so that POJO-style CST classes can be generated and compiled. More details are discussed in the technical report  
 
[http://www.sts.tu-harburg.de/%7Emi.garcia/SoC2007/draftreport.pdf Generation of Eclipse-based IDEs for Custom DSLs]
 
[http://www.sts.tu-harburg.de/%7Emi.garcia/SoC2007/draftreport.pdf Generation of Eclipse-based IDEs for Custom DSLs]
 
 
 
  
 
Let's say we want to reproduce the steps to generate POJO CST classes for Emfatic.
 
Let's say we want to reproduce the steps to generate POJO CST classes for Emfatic.
An Eclipse project should be created and an .ast and a lexer file added, to
+
* An Eclipse project should be created and a grammar file (emfatic.ast) is added, to look as shown in Figure 1.  
look as shown in Figure 1(a) (more on that lexer file later).  
+
<table border="0" cellpadding="8">
 
+
<tr>
 +
<td>[[Image:ExtendedGymnast1.PNG]]</td>
 +
</tr>
 +
<caption align="bottom"><b>Figure 1</b> Before Gymnast Generator </caption>
 +
</table>
  
 +
* Dependent Plug-ins are added in the MANIFEST.MF file. Hence right clicking on the grammar file shows options like 'Generate .genmodel and Java', 'Generate JavaCC lexer file', 'Generate Antlrv2 lexer file' etc. Refer Figure 2.
 
<table border="0" cellpadding="8">
 
<table border="0" cellpadding="8">
 
<tr>
 
<tr>
<td>[[Image:GymnastHowToUseGymnast1.PNG]]</td>
+
<td>[[Image:ExtendedGymnast2.jpg]]</td>
 
</tr>
 
</tr>
<caption align="bottom"><b>Figure 1(a)</b> Before Gymnast Generator</caption>
+
<caption align="bottom"><b>Figure 2 </b> Context menu for grammar </caption>
 +
</table>
  
 +
====  Generation of lexer files ====
 +
The grammar file can be checked for well-formedness by clicking 'Check Well-formedness' from the context menu. After checking for the well-formedness of the grammar, JavaCC and Antlr lexer files can be auto generated by clicking 'Generate JavaCC lexer file' and 'Generate Antlrv2 lexer file' from the context menu of the grammar file. This autogeneration of lexer files is one of the enhancements made to Gymnast. In this example, emfaticLexer.jj and emfaticLexer.g are JavaCC and Antlr lexer files respectively (Figure 3).
 +
<table border="0" cellpadding="8">
 +
<tr>
 +
<td>[[Image:ExtendedGymnast3.jpg]]</td>
 +
</tr>
 +
<caption align="bottom"><b>Figure 3 </b> Generated lexer files </caption>
 
</table>
 
</table>
  
 
+
==== Generation of AST and Parser ====
After the generator is run (by
+
After the generator is run (by rightclicking on <code>emfatic.ast</code> and choosing "Generate AST") the result looks as in Figure
rightclicking on <code>emfatic.ast</code> and choosing "Generate AST") the result looks as in Figure
+
4.  
1(b).  
+
 
+
  
 
<table border="0" cellpadding="8">
 
<table border="0" cellpadding="8">
Line 47: Line 59:
 
<td>[[Image:GymnastHowToUseGymnast2.PNG]]</td>
 
<td>[[Image:GymnastHowToUseGymnast2.PNG]]</td>
 
</tr>
 
</tr>
<caption align="bottom"><b>Figure 1(b)</b> After Gymnast Generator</caption>
+
<caption align="bottom"><b>Figure 4</b> After Gymnast Generator</caption>
 
+
 
</table>
 
</table>
  
 +
Not shown here is the added plugin dependency to <code>org.antlr</code>, which can be installed as described in the next section.
  
 +
Gymnast reports its operation on the console:
  
Not shown here is the added plugin dependency to <code>org.antlr</code>, which can be
+
<table border="0" cellpadding="8">
installed as described next.
+
<tr>
 +
<td>[[Image:GymnastConsoleOutput.PNG]]</td>
 +
</tr>
 +
<caption align="bottom"><b>Figure 5</b> Console output of Gymnast Generator</caption>
 +
</table>
  
 
== What if you target ANTLR instead of JavaCC ==
 
== What if you target ANTLR instead of JavaCC ==
 
  
 
In that case, the code generated for the parsing infrastructure will depend on the ANTLR parser generator framework.   
 
In that case, the code generated for the parsing infrastructure will depend on the ANTLR parser generator framework.   
The easiest way to install a plugin with ANTLR is to get it from the update site  
+
The easiest way to install a plugin with ANTLR is to get it from the update site on SourceForge by following these steps:
on SourceForge by following these steps:
+
 
<p/>
 
<p/>
 
<ol>
 
<ol>
 
<li>In Eclipse, go to <i>Help -> Software Updates -> Find and Install....</i></li>
 
<li>In Eclipse, go to <i>Help -> Software Updates -> Find and Install....</i></li>
 
 
<li>Select <i>Search for new features to install</i> and click on Next.</li>
 
<li>Select <i>Search for new features to install</i> and click on Next.</li>
 
<li>Press the <i>New Remote Site...</i> button and enter the following values:</li>
 
<li>Press the <i>New Remote Site...</i> button and enter the following values:</li>
Line 80: Line 94:
 
</ol>
 
</ol>
  
== Choosing target parser, and providing the parser-dependent lexer file  ==
+
== Choosing target parser==
 
+
The choice of target parser is made with the option <code>parserGenerator = ". . ."</code>
+
(Figure 2). Unlike ANTLR, JavaCC requires no runtime library: all the code it
+
needs gets generated.
+
 
+
 
+
The lexer file (<code>emfaticLexer.g</code> in the ANTLR example) is actually constant for all
+
<code>.ast</code>s we care to write. It can be copied from <code>org.eclipse.gymnast.generator.core.parser</code>
+
(the file ending in <code>.g</code> is the ANTLR one, ending in <code>.jj</code> the JavaCC one) The
+
only thing that needs updating are its filename (to match the name of the DSL in
+
question) and, in the case of ANTLR, the first line in the lexer file (because such
+
heading also contains the name of the DSL).
+
  
 +
The choice of target parser is made with the option <code>parserGenerator = ". . ."</code> (Figure 6). Unlike ANTLR, JavaCC requires no runtime library: all the code it needs gets generated.
  
 
<table border="0" cellpadding="8">
 
<table border="0" cellpadding="8">
Line 99: Line 102:
 
<td>[[Image:GymnastantlrOption.PNG]]</td>
 
<td>[[Image:GymnastantlrOption.PNG]]</td>
 
</tr>
 
</tr>
<caption align="bottom"><b>Figure 2</b> How to choose a parser</caption>
+
        <caption align="bottom"><b>Figure 6</b> How to choose a parser</caption>
 
+
 
</table>
 
</table>
  
 
== Exploring the generated artifacts ==
 
== Exploring the generated artifacts ==
  
The generated grammar files (for the ANTLR example the one highlighted in
+
The generated grammar files (for the ANTLR example the one highlighted in Figure 7) are not supposed to be messed up by the user, however in order to explore them the following Eclipse plug-ins prove useful:
Figure 3) are not supposed to be messed up by the user, however in order to explore
+
them the following Eclipse plugins prove useful:
+
  
 
<ul>
 
<ul>
Line 126: Line 126:
 
<td>[[Image:GymnastGeneratedGrammarFile.PNG]]</td>
 
<td>[[Image:GymnastGeneratedGrammarFile.PNG]]</td>
 
</tr>
 
</tr>
<caption align="bottom"><b>Figure 3</b> Generated artifacts</caption>
+
<caption align="bottom"><b>Figure 7</b> Generated artifacts</caption>
 
</table>
 
</table>
 
  
 
== POJO-based CSTs ==
 
== POJO-based CSTs ==
Line 139: Line 138:
 
</ul>
 
</ul>
  
 +
The input-output relationship and the resulting code patterns are described by Gymnast's author, Chris J Daly, in an Eclipse Tech Panel Exchange presentation, available [http://www.sts.tu-harburg.de/~mi.garcia/SoC2007/GymnastSlides.pdf here].
  
The input-output relationship and the resulting code patterns are described by
+
The parser-generator-independent CST classes generated by Gymnast Generator implement <code>org.eclipse.gymnast.runtime.core.ast.ASTNode</code> (Figure 8).  
Gymnast's author, Chris J Daly, in an Eclipse Tech Panel Exchange presentation,
+
available [http://www.sts.tu-harburg.de/~mi.garcia/SoC2007/GymnastSlides.pdf here].
+
 
+
 
+
The parser-generator-independent CST classes generated by Gymnast Generator
+
implement <code>org.eclipse.gymnast.runtime.core.ast.ASTNode</code> (Figure 4(a)).  
+
  
 
<p></p>
 
<p></p>
Line 154: Line 148:
 
<td>[[Image:GymnastASTNode.PNG]]</td>
 
<td>[[Image:GymnastASTNode.PNG]]</td>
 
</tr>
 
</tr>
<caption align="bottom"><b>Figure 4(a) </b><code>ASTNode</code></caption>
+
<caption align="bottom"><b>Figure 8 </b><code>ASTNode</code></caption>
 
</table>
 
</table>
  
  
 
A sample
 
A sample
POJO-style CST class is depicted in Figure 4(b), corresponding to the following
+
POJO-style CST class is depicted in Figure 9, corresponding to the following grammar production.
grammar production.
+
  
 
<code><pre>  
 
<code><pre>  
Line 176: Line 169:
 
<td>[[Image:GymnastClassDecl.PNG]]</td>
 
<td>[[Image:GymnastClassDecl.PNG]]</td>
 
</tr>
 
</tr>
<caption align="bottom"><b>Figure 4(b) </b><code>ClassDecl</code></caption>
+
<caption align="bottom"><b>Figure 9 </b><code>ClassDecl</code></caption>
 
</table>
 
</table>
  

Latest revision as of 07:34, 26 May 2008

How to use Gymnast Generator

Gymnast is a framework for jumpstarting text editors for custom Domain Specific Languages, written by Chris Daly, and summarized in these slides.

Gymnast is being distributed along with Emfatic, as their developer communities overlap (at least, so far).

In the past, many improvements has been included in Gymnast. For example, JavaCC and Antlr lexer files can be autogenerated now instead of manually creating and adding them to the project. This article reflects those changes made to the Gymnast framework. Please refer to Grammar2Ecore for details on the generation of Java classes from grammar.

Where to download Gymnast from

WhereToDownloadGymnastFrom.png

Overview

Gymnast comprises two main parts (Runtime and Generator), each of which can in turn be divided into Core and User Interface. This section is all about using the Gymnast Generator to:

  1. create .ast files
  2. configure options in that file
  3. autogenerate lexer files, and
  4. manually add the parser's .jar to the (Build path or Plugin dependencies)

so that POJO-style CST classes can be generated and compiled. More details are discussed in the technical report Generation of Eclipse-based IDEs for Custom DSLs

Let's say we want to reproduce the steps to generate POJO CST classes for Emfatic.

  • An Eclipse project should be created and a grammar file (emfatic.ast) is added, to look as shown in Figure 1.
ExtendedGymnast1.PNG
Figure 1 Before Gymnast Generator
  • Dependent Plug-ins are added in the MANIFEST.MF file. Hence right clicking on the grammar file shows options like 'Generate .genmodel and Java', 'Generate JavaCC lexer file', 'Generate Antlrv2 lexer file' etc. Refer Figure 2.
ExtendedGymnast2.jpg
Figure 2 Context menu for grammar

Generation of lexer files

The grammar file can be checked for well-formedness by clicking 'Check Well-formedness' from the context menu. After checking for the well-formedness of the grammar, JavaCC and Antlr lexer files can be auto generated by clicking 'Generate JavaCC lexer file' and 'Generate Antlrv2 lexer file' from the context menu of the grammar file. This autogeneration of lexer files is one of the enhancements made to Gymnast. In this example, emfaticLexer.jj and emfaticLexer.g are JavaCC and Antlr lexer files respectively (Figure 3).

ExtendedGymnast3.jpg
Figure 3 Generated lexer files

Generation of AST and Parser

After the generator is run (by rightclicking on emfatic.ast and choosing "Generate AST") the result looks as in Figure 4.

GymnastHowToUseGymnast2.PNG
Figure 4 After Gymnast Generator

Not shown here is the added plugin dependency to org.antlr, which can be installed as described in the next section.

Gymnast reports its operation on the console:

GymnastConsoleOutput.PNG
Figure 5 Console output of Gymnast Generator

What if you target ANTLR instead of JavaCC

In that case, the code generated for the parsing infrastructure will depend on the ANTLR parser generator framework. The easiest way to install a plugin with ANTLR is to get it from the update site on SourceForge by following these steps: <p/>

  1. In Eclipse, go to Help -> Software Updates -> Find and Install....
  2. Select Search for new features to install and click on Next.
  3. Press the New Remote Site... button and enter the following values:
  4. Now check the new ANTLR checkbox and click on Next.
  5. Select version 3.0.5 of the feature org.antlr.ui and click on Next.
  6. Accept the licensing terms and click on Next and Finish.
  7. Click on Install when you see a pop-up message warning about unsigned features.
  8. When prompted, click on Yes to restart the workbench.

Choosing target parser

The choice of target parser is made with the option parserGenerator = ". . ." (Figure 6). Unlike ANTLR, JavaCC requires no runtime library: all the code it needs gets generated.

GymnastantlrOption.PNG
Figure 6 How to choose a parser

Exploring the generated artifacts

The generated grammar files (for the ANTLR example the one highlighted in Figure 7) are not supposed to be messed up by the user, however in order to explore them the following Eclipse plug-ins prove useful:

GymnastGeneratedGrammarFile.PNG
Figure 7 Generated artifacts

POJO-based CSTs

The generated POJO-style CST classes:

  • exhibit useful patterns for later processing (visiting in particular)
  • are independent from the chosen parser generator

The input-output relationship and the resulting code patterns are described by Gymnast's author, Chris J Daly, in an Eclipse Tech Panel Exchange presentation, available here.

The parser-generator-independent CST classes generated by Gymnast Generator implement org.eclipse.gymnast.runtime.core.ast.ASTNode (Figure 8).

GymnastASTNode.PNG
Figure 8 ASTNode


A sample POJO-style CST class is depicted in Figure 9, corresponding to the following grammar production.

 
sequence classDecl : (abstractModifier)?
                     classKind name=ID
                     (typeParamsInfo=typeParamsInfo)?
                     ("extends" superTypes=commaListBoundExceptWild)?
                     (COLON instClassName=boundExceptWildcard)?
                     LCURLY classMemberDecls RCURLY ;


GymnastClassDecl.PNG
Figure 9 ClassDecl

Organization

Initial committers

Back to the top