Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "DLTK Core Architecture"

(Improve wording (no addition to content))
Line 2: Line 2:
 
== Build paths ==
 
== Build paths ==
  
−
Similar to java class paths, DLTK has a concept of build paths.  
+
Similar to Java's class paths, DLTK has the concept of build paths.  
  
−
Build paths is a set of source folders, library containers and references to another projects. This paths are used as for model building, so for launching.  
+
A build path is the set of source folders, library containers and references to other projects. The build path is used for model building and launching.  
  
−
Paths is stored in .buildpath file from project's root folder. DLTK automatically parses it when required. You can get project build paths via IScriptProject.getRawBuildpath() method. To set buildpath to project you should use setRawBuildpath() method. Elements of IBuildpathEntry may be created with DLTKCore.new*Entry(...) methods.  
+
The build path is stored in the file .buildpath relative to the project's root folder. DLTK automatically reads it when required. You can get the current project's build path as an array of <code lang="java">IBuildpathEntry</code>s via the <code lang="java">IScriptProject.getRawBuildpath()</code> method. To change the build path for a project use the <code lang="java">setRawBuildpath()</code> method. New elements of <code lang="java">IBuildpathEntry</code> may be created with the <code lang="java">DLTKCore.new*Entry(...)</code> methods.  
  
 
== Model ==
 
== Model ==
  
−
Model is a key thing in DLTK you should know about. In fact, as you probably know, DLTK model had been cloned from JDT model. So, if you are familiar with JDT model, you will understand everything here quickly.
+
The DLTK model is central to understanding DLTK. The DLTK model is based on the JDT model, so if you are familiar with that then you will understand everything here quickly.
−
As JDT, DLTK uses an in-memory object model to represent the workspace structure from projects level to source file internals. This structure is derived from the project's build path. The model is hierarchical.
+
 
−
The following table summarizes the different kinds of model elements. All elements classes support the IModelElement interface.
+
Like JDT, DLTK uses an in-memory, hierarchical object model to represent the workspace structure from the project level down to source file internals. This structure is derived from the project's build path.
 +
 
 +
The following table summarizes the different kinds of model elements. All elements classes support the <code lang="java">IModelElement</code> interface.
  
 
<table border=1>
 
<table border=1>
Line 21: Line 23:
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IScriptModel</td>
+
<td><code lang="java">IScriptModel</code></td>
−
<td>IJavaModel</td>
+
<td><code lang="java">IJavaModel</code></td>
 
<td>Represents the root model element, corresponding to the
 
<td>Represents the root model element, corresponding to the
 
workspace. The parent of all projects with the script natures. It also
 
workspace. The parent of all projects with the script natures. It also
Line 28: Line 30:
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IScriptProject</td>
+
<td><code lang="java">IScriptProject</code></td>
−
<td>IJavaProject</td>
+
<td><code lang="java">IJavaProject</code></td>
 
<td>Represents a script project in the workspace. (Child of
 
<td>Represents a script project in the workspace. (Child of
−
IScriptModel)</td>
+
<code lang="java">IScriptModel</code>)</td>
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IProjectFragment</td>
+
<td><code lang="java">IProjectFragment</code></td>
−
<td>IPackageFragmentRoot</td>
+
<td><code lang="java">IPackageFragmentRoot</code></td>
 
<td>Represents a project fragment, and maps the contents to an
 
<td>Represents a project fragment, and maps the contents to an
 
underlying resource which is either a folder, JAR, or ZIP file. (Child
 
underlying resource which is either a folder, JAR, or ZIP file. (Child
−
of IScriptProject)</td>
+
of <code lang="java">IScriptProject</code>)</td>
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IScriptFolder</td>
+
<td><code lang="java">IScriptFolder</code></td>
−
<td>IPackageFragment</td>
+
<td><code lang="java">IPackageFragment</code></td>
 
<td>Represents a folder containing script files inside. (Child of
 
<td>Represents a folder containing script files inside. (Child of
−
IProjectFragment )</td>
+
<code lang="java">IProjectFragment</code>)</td>
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>ISourceModule</td>
+
<td><code lang="java">ISourceModule</code></td>
−
<td>ICompilationUnit</td>
+
<td><code lang="java">ICompilationUnit</code></td>
−
<td>Represents a source file. (Child of IScriptFolder)</td>
+
<td>Represents a source file. (Child of <code lang="java">IScriptFolder</code>)</td>
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IPackageDeclaration</td>
+
<td><code lang="java">IPackageDeclaration</code></td>
−
<td>IPackageDeclaration</td>
+
<td><code lang="java">IPackageDeclaration</code></td>
 
<td>Represents a package declaration in a source module. (Child
 
<td>Represents a package declaration in a source module. (Child
−
of ISourceModule )</td>
+
of <code lang="java">ISourceModule</code>)</td>
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IType</td>
+
<td><code lang="java">IType</code></td>
−
<td>IType</td>
+
<td><code lang="java">IType</code></td>
 
<td>Represents either a class/module/namespace inside a source
 
<td>Represents either a class/module/namespace inside a source
 
file.</td>
 
file.</td>
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IField</td>
+
<td><code lang="java">IField</code></td>
−
<td>IField</td>
+
<td><code lang="java">IField</code></td>
−
<td>Represents a field inside a type. (Child of IType)</td>
+
<td>Represents a field inside a type. (Child of IType</code>)</td>
 
</tr>
 
</tr>
 
<tr>
 
<tr>
−
<td>IMethod</td>
+
<td><code lang="java">IMethod</code></td>
−
<td>IMethod</td>
+
<td><code lang="java">IMethod</code></td>
 
<td>Represents a method or constructor inside a type. (Child of
 
<td>Represents a method or constructor inside a type. (Child of
−
IType)</td>
+
IType</code>)</td>
 
</tr>
 
</tr>
 
</table>
 
</table>
  
−
You can use DLTKCore.create(...) methods to access model. These methods allow to create appropriate model element from file, resource or project.
+
You should use the <code lang="java">DLTKCore.create(...)</code> methods to build out a model. These methods make it easy for the DLTK user to create the appropriate model element from a file, resource or project.
  
 
== Model building ==
 
== Model building ==
  
−
DLTK Model elements from workspace level down to source modules level are built automatically. All things, that you should do for that:
+
DLTK automatically provides model elements from the workspace level down to the source modules level. To extend the model, the DLTK user should:
−
* Contribute IDLTKLanguageToolkit interface implementation via org.eclipse.dltk.core.language extension point.
+
* contribute a <code lang="java">IDLTKLanguageToolkit</code> interface implementation via the <code>org.eclipse.dltk.core.language</code> extension point,
−
* Set correct nature as in extension point attribute, so in getNatureId() method.
+
* contribute the language-specific nature as an extension point attribute, and return that from the <code lang="java">getNatureId()</code> method,
−
* Methods validateSourceModule() and validateSourcePackage() should return OK if source module/package is a real source module, or anything else if it's a regular file/folder.
+
* implement methods <code lang="java">validateSourceModule()</code> and <code lang="java">validateSourcePackage()</code> to return OK only for source modules or packages that are real source modules.
  
−
After that, projects, that have given nature will be considered as script project and model for them will be built accordingly to internal structure, validate...() methods and build paths.
+
User projects that have the right nature will then be considered as a script project and the DLTK model for them will be built accordingly to internal structure, results from <code lang="java">validate...()</code> methods and from build paths.
  
−
For building source module's internal model there is another mechanism called source element parsers. They are contributed via org.eclipse.dltk.core.sourceElementParsers ext. point and should implement ISourceElementParser.  
+
For building a source module's internal model elements, there is another mechanism called source element parsers. These are contributed via the <code>org.eclipse.dltk.core.sourceElementParsers</code> extension point and should implement <code lang="java">ISourceElementParser</code>.  
  
−
Main task of source element parser is to parse source file and report internal model elements info to given ISourceElementRequestor.
+
The main task of the source element parser is to parse source files and report internal model element information to the given <code lang="java">ISourceElementRequestor</code>.
  
 
== Search engine ==
 
== Search engine ==
Line 95: Line 97:
 
===Indexes===
 
===Indexes===
  
−
Indexes is a main facility which search engine uses. Index is a set of documents and keys associated with them. There are possible several different indexes (for type names, for methods, ...).  
+
The platform's search engine uses indexes. An index is a set of documents and the keys associated with them. There are possible several different indexes (for type names, methods, ...).  
  
−
DLTK automatically index all source files in a separate thread. It uses standard source element parser with a requester set to SourceIndexerRequestor object. Source element parser doesn't know anything about search and just reports model elements info. Task of SourceIndexerRequestor is to report appropriate index keys to given SourceIndexer. User may extend SourceIndexerRequestor if it's required.  
+
DLTK automatically builds an index for all script source files in a separate thread. The DLTK provides a standard source element parser with a requester set to it's own <code lang="java">SourceIndexerRequestor</code> object. The source element parser doesn't know anything about search and just reports model elements info. The task of the <code lang="java">SourceIndexerRequestor</code> is to find and report appropriate index keys to the platform's <code lang="java">SourceIndexer</code>. The DLTK user may extend the DLTK's <code lang="java">SourceIndexerRequestor</code> as required.  
  
 
===Search===
 
===Search===
  
−
Before using the search engine user should prepare a special object: search pattern. It's a SearchPattern class object. It may be a TypeDeclarationPattern or may be MethodPattern. Static method SearchPattern.createPattern() may be used for that purpose.
+
Before using the search engine, the user should prepare DLTK <code lang="java">SearchPattern</code> objects. These may be a <code lang="java">TypeDeclarationPattern</code> or a <code lang="java">MethodPattern</a>. These can be created using the static method <code lang="java">SearchPattern.createPattern()</code>.
−
After that user should specify a search scope(project, workspace,...). It may be created with a SearchEngine.createSearchScope() method.  
+
 
−
And final item is a search requestor. It's a SearchRequestor object, that will receive all successfull search matches.  
+
After that, the user should specify a search scope(project, workspace,...). The scope may be created with the <code lang="java">SearchEngine.createSearchScope()</code> method.
−
After that SearchEngine::search() may be called. Here is an example:
+
 
−
<code><pre>
+
The final item required is a <code lang="java">SearchRequestor</code> object that will receive all successful search matches.  
−
SearchRequestor requestor = new SearchRequestor() {
+
When that is ready, DLTK's <code lang="java">SearchEngine.search()</code> may be called. Here is an example:
 +
<source lang="java">
 +
SearchRequestor requestor = new SearchRequestor() {
 +
    public void acceptSearchMatch(SearchMatch match)
 +
        throws CoreException {
 +
            // process match
 +
        }
 +
    };
  
−
public void acceptSearchMatch(SearchMatch match)
+
SearchPattern pattern = SearchPattern.createPattern(namePattern,
−
throws CoreException {
+
    IDLTKSearchConstants.METHOD, IDLTKSearchConstants.DECLARATIONS,
−
// process match
+
    SearchPattern.R_PATTERN_MATCH | SearchPattern.R_EXACT_MATCH);
−
}
+
IDLTKSearchScope scope = SearchEngine.createWorkspaceScope(RubyLanguageToolkit
 +
    .getDefault());
  
−
};
+
try {
−
SearchPattern pattern = SearchPattern.createPattern(namePattern,
+
    SearchEngine engine = new SearchEngine();
−
IDLTKSearchConstants.METHOD, IDLTKSearchConstants.DECLARATIONS,
+
    engine.search(pattern, new SearchParticipant[] { SearchEngine
−
SearchPattern.R_PATTERN_MATCH | SearchPattern.R_EXACT_MATCH);
+
        .getDefaultSearchParticipant() }, scope, requestor, null);
−
IDLTKSearchScope scope = SearchEngine.createWorkspaceScope(RubyLanguageToolkit
+
} catch (CoreException e) {
−
.getDefault());
+
    if (DLTKCore.DEBUG)
−
try {
+
        e.printStackTrace();
−
SearchEngine engine = new SearchEngine();
+
}
−
engine.search(pattern, new SearchParticipant[] { SearchEngine
+
</source>
−
.getDefaultSearchParticipant() }, scope, requestor, null);
+
−
} catch (CoreException e) {
+
−
if (DLTKCore.DEBUG)
+
−
e.printStackTrace();
+
−
}
+
  
−
</pre></code>
+
===How search works===
  
−
===How search works?===
+
Extending the search engine requires understanding of how the search engine works. When <code lang="java">SearchEngine.search()</code> is called, a special <code lang="java">PatternSearchJob</code> is created containing all the indexes being enumerated. As result the list of documents containing matching keys will be found. Next, each document is reparsed with a <code lang="java">MatchLocator</code> and appropriate <code lang="java">SearchMatch</code> objects are reported.
  
−
Extending a search engine requires understanding of how search engine works. After you had called search() method, a special PatternSearchJob are being created. Inside it all indexes are being enumerated. As result, list of documents, containing a matching key will be received. After that, using a MatchLocator class each document will be reparsed and appropriate SearchMatch objects reported. User given MatchLocatorParser are being used for reparsing. It knows a MatchLocator and PossibleMatch objects and while parsing should call match(...) method from the locator.  
+
The DLTK user should provide an implementation of <code lang="java">MatchLocatorParser</code> to use for reparsing. This object will receive a <code lang="java">MatchLocator</code> and a <code lang="java">PossibleMatch</code> (indicating a candidate source file) and while parsing should invoke the <code lang="java">match(...)</code> method on the <code lang="java">MatchLocator</code> with the matches it determines.
  
 
==Runtime model ("mixin" model)==
 
==Runtime model ("mixin" model)==

Revision as of 18:19, 13 September 2010

Build paths

Similar to Java's class paths, DLTK has the concept of build paths.

A build path is the set of source folders, library containers and references to other projects. The build path is used for model building and launching.

The build path is stored in the file .buildpath relative to the project's root folder. DLTK automatically reads it when required. You can get the current project's build path as an array of IBuildpathEntrys via the IScriptProject.getRawBuildpath() method. To change the build path for a project use the setRawBuildpath() method. New elements of IBuildpathEntry may be created with the DLTKCore.new*Entry(...) methods.

Model

The DLTK model is central to understanding DLTK. The DLTK model is based on the JDT model, so if you are familiar with that then you will understand everything here quickly.

Like JDT, DLTK uses an in-memory, hierarchical object model to represent the workspace structure from the project level down to source file internals. This structure is derived from the project's build path.

The following table summarizes the different kinds of model elements. All elements classes support the IModelElement interface.

<tr> <td>IMethod</td> <td>IMethod</td> <td>Represents a method or constructor inside a type. (Child of IType</code>)</td> </tr> </table> You should use the DLTKCore.create(...) methods to build out a model. These methods make it easy for the DLTK user to create the appropriate model element from a file, resource or project.

Model building

DLTK automatically provides model elements from the workspace level down to the source modules level. To extend the model, the DLTK user should:

  • contribute a IDLTKLanguageToolkit interface implementation via the org.eclipse.dltk.core.language extension point,
  • contribute the language-specific nature as an extension point attribute, and return that from the getNatureId() method,
  • implement methods validateSourceModule() and validateSourcePackage() to return OK only for source modules or packages that are real source modules.

User projects that have the right nature will then be considered as a script project and the DLTK model for them will be built accordingly to internal structure, results from validate...() methods and from build paths.

For building a source module's internal model elements, there is another mechanism called source element parsers. These are contributed via the org.eclipse.dltk.core.sourceElementParsers extension point and should implement ISourceElementParser.

The main task of the source element parser is to parse source files and report internal model element information to the given ISourceElementRequestor.

Search engine

Indexes

The platform's search engine uses indexes. An index is a set of documents and the keys associated with them. There are possible several different indexes (for type names, methods, ...).

DLTK automatically builds an index for all script source files in a separate thread. The DLTK provides a standard source element parser with a requester set to it's own SourceIndexerRequestor object. The source element parser doesn't know anything about search and just reports model elements info. The task of the SourceIndexerRequestor is to find and report appropriate index keys to the platform's SourceIndexer. The DLTK user may extend the DLTK's SourceIndexerRequestor as required.

Search

Before using the search engine, the user should prepare DLTK SearchPattern objects. These may be a TypeDeclarationPattern or a MethodPattern</a>. These can be created using the static method <code lang="java">SearchPattern.createPattern().

After that, the user should specify a search scope(project, workspace,...). The scope may be created with the SearchEngine.createSearchScope() method.

The final item required is a SearchRequestor object that will receive all successful search matches. When that is ready, DLTK's SearchEngine.search() may be called. Here is an example: 

SearchRequestor requestor = new SearchRequestor() {
    public void acceptSearchMatch(SearchMatch match)
        throws CoreException {
            // process match
        }
    };
 
SearchPattern pattern = SearchPattern.createPattern(namePattern,
    IDLTKSearchConstants.METHOD, IDLTKSearchConstants.DECLARATIONS,
    SearchPattern.R_PATTERN_MATCH | SearchPattern.R_EXACT_MATCH);
IDLTKSearchScope scope = SearchEngine.createWorkspaceScope(RubyLanguageToolkit
    .getDefault());
 
try {
    SearchEngine engine = new SearchEngine();
    engine.search(pattern, new SearchParticipant[] { SearchEngine
        .getDefaultSearchParticipant() }, scope, requestor, null);
} catch (CoreException e) {
    if (DLTKCore.DEBUG)
        e.printStackTrace();
}

How search works

Extending the search engine requires understanding of how the search engine works. When SearchEngine.search() is called, a special PatternSearchJob is created containing all the indexes being enumerated. As result the list of documents containing matching keys will be found. Next, each document is reparsed with a MatchLocator and appropriate SearchMatch objects are reported.

The DLTK user should provide an implementation of MatchLocatorParser to use for reparsing. This object will receive a MatchLocator and a PossibleMatch (indicating a candidate source file) and while parsing should invoke the match(...) method on the MatchLocator with the matches it determines.

Runtime model ("mixin" model)

DLTK has a very simple, but really powerful structure for managing runtime source files model. If model elements may be constructed from several source files(that's why model is called "mixin") or modified during execution, this approach can help.

In fact, mixin-model is built on top of the standard indexes facility. Mixin-parser(IMixinParser implementation, ext. point org.eclipse.dltk.core.mixin) reports keys(String objects) while parsing. One key may be reported many times from many places. To each key some Object may be attached. And... that's all. DLTK worries about everything else.

Lets consider the following example for Ruby. 

file1.rb

class Foo # key "Foo" reported, IType object attached
end

class Foo # key "Foo" reported, IType object attached
	def doo # key "Foo{doo" reported, IMethod object attached
	end
end

file2.rb

class Foo # key "Foo" reported, IType object attached
	def doo2 # key "Foo{doo2" reported, IMethod object attached
	end
end

Now, if we'll ask model for Foo key(MixinModel.get() method), we'll fetch IMixinElement with information, that this key has been reported from file1.rb and file2.rb and we'll be able to get every IType object attached. More. "{" is a standard delimeter, so get can call getChilds() and fetch information about "Foo{doo" and "Foo{doo2" keys.

Type inference

DLTK has a language independent engine for building type inference systems. It uses demand-driven analysis with subgoal pruning algorithm. Key abstractions here:

  • Goal. It may be type of some ASTNode in source file, incoming/outcoming data flow, whatever. Goals are unique. So, two object of IGoal having a common ASTNode object is equal.
  • Goal evaluator. Strategy, that knows how to evaluate goal. While evaluating it may produce another helper goals (sub-goals), may wait for their results and only after that produce it's own result.
  • Goal evaluator factory. This factory constructs a GoalEvaluator objects for given IGoals.
  • Pruner. Pruner is a straregy that are able to cancel some evaluators as non-important. Also using pruner is possible to implement time limits for evaluations.

// TODO

Launching

Interpreter management

Each interpreter installation in system is stored inside a IInterpreterInstall object. Such object knows interpreter name, executable path, arguments, library paths and also installation type and IInterpreterRunner object. Installation type is represented via IInterpreterInstallType object. Installation type knows about all installations with such type, knows how to fetch default library locations and able to validate installations.

For each language, DLTK stores a separate set of IInterpreterInstalls. One of them should be marked as "default".

The key class for accessing installation is a ScriptRuntime.

However, ScriptRuntime allows only to fetch installs, not to modify them. In fact, there is no beautiful way for that. So here we'll describe relatively low-level way to do that.

All information about interpreter installs are stored in plugin preferences in XML format. XML data can be read/created using an InterpreterDefinitionsContainer. This class represent a set of interpreter installs and allows to read them from XML or store to XML.

For setting set of interpreters in preferences, there are exists an IntepreterUpdater class. It takes an InterpreterDefinitionsContainer object and stores in plugin preferences. After that ScriptRuntime will be able to read new values.

Launching a script

DLTK launching engine is fully based on standard Eclipse launching, so there are not a lot of new things are here.

Cause each InterpreterInstall has an IInterpreterRunner object, launch process looks like following:

  • get selected interpreter install info from launch configuration (it should be put there before)
  • get runner
  • launch it

In fact, all this stuff is already implemented. First, there is a AbstractScriptLaunchConfigurationDelegate class. It requires only two methods from user: getLanguageId() and createInterpreterConfig(). InterpreterConfig is a simple structure containing information for launch in a low-level form.

Also, for IInterpreterRunner exists a AbstractInterpreterRunner class, that requires from user only a launching plugin id and created process type.

For launching scripts programmatically there is a ScriptLaunchUtil class. It contains lots of methods for launching scripts.

DLTK AST

Element JDT-analog Description
IScriptModel IJavaModel Represents the root model element, corresponding to the

workspace. The parent of all projects with the script natures. It also

gives you access to the projects without the script nature.
IScriptProject IJavaProject Represents a script project in the workspace. (Child of IScriptModel)
IProjectFragment IPackageFragmentRoot Represents a project fragment, and maps the contents to an

underlying resource which is either a folder, JAR, or ZIP file. (Child

of IScriptProject)
IScriptFolder IPackageFragment Represents a folder containing script files inside. (Child of IProjectFragment)
ISourceModule ICompilationUnit Represents a source file. (Child of IScriptFolder)
IPackageDeclaration IPackageDeclaration Represents a package declaration in a source module. (Child of ISourceModule)
IType IType Represents either a class/module/namespace inside a source file.
IField IField Represents a field inside a type. (Child of IType</code>)
ASTNote Superclass of all the ast nodes.
ASTListNode Represents list of nodes.
ModuleDeclaration Top-level node for a source file.
TypeDeclaration Declaration of class/module/namespace.
MethodDeclaration Declaration of procedure or method.


Other classes you can find in org.eclipse.dltk.ast.* packages. Usage of DLTK AST is not mandatory, but some DLTK features like folding may rely on it and greatly simplify implementation.

Retrieved from "https://wiki.eclipse.org/index.php?title=DLTK_Core_Architecture&oldid=219709"

Copyright © Eclipse Foundation, Inc. All Rights Reserved.