Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "JSDT/JSDT Code Analytics"

m (Psuzzi.gmail.com moved page JSDT/JSDTCodeAnalytics to JSDT/JSDT Code Analytics)
(3 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
</div>
 
</div>
  
This document is to present the current status of the code analysis in JSDT and to discuss possible improvements.
+
This document presents the status of the code analysis in JSDT and discusses possible improvements.  
Please, feel free to edit this document and contribute to the discussion.
+
The more participation, the higher chances we have to go in the right direction.  
+
  
= JSDT Code Analytics =
+
Please, feel free to edit this document and to contribute to the discussion. 
 +
 
 +
== Current Status ==
 +
 
 +
We're using Closure Compiler because of its good performances and architecture ([http://eclipse.1072660.n5.nabble.com/About-parsing-and-conversion-times-td181113.html See discussion] [https://dev.eclipse.org/mhonarc/lists/wtp-dev/msg09853.html #]).
 +
 
 +
There are a lot of resources about CC: [https://github.com/google/closure-compiler/wiki/Tutorials tutorials],
 +
[https://github.com/google/closure-compiler/wiki/FAQ FAQs],
 +
[https://github.com/google/closure-compiler/wiki/Design-Documents design documents].
  
 
JSDT 2.0 is missing the content outline and content proposal functionality.
 
JSDT 2.0 is missing the content outline and content proposal functionality.
Line 25: Line 31:
 
* with the output tree, and the current position in code, we should feed an inference engine, to give content assist proposals.
 
* with the output tree, and the current position in code, we should feed an inference engine, to give content assist proposals.
  
== Current Status ==
 
 
We're using Closure Compiler because of its good performances and architecture ([http://eclipse.1072660.n5.nabble.com/About-parsing-and-conversion-times-td181113.html See discussion] [https://dev.eclipse.org/mhonarc/lists/wtp-dev/msg09853.html #]).
 
 
There are a lot of resources about CC: [https://github.com/google/closure-compiler/wiki/Tutorials tutorials],
 
[https://github.com/google/closure-compiler/wiki/FAQ FAQs],
 
[https://github.com/google/closure-compiler/wiki/Design-Documents design documents].
 
 
== Idea 1 -
 
We know it is possible to use Closure Compiler to
 
We know there are IDEs using Closure Compiler to provide
 
 
 
Closure Compiler provides
 
We could extend Closure Compiler
 
  
 
== Ideas ==
 
== Ideas ==
Line 51: Line 42:
 
Below, there is a list of possible ideas to use a tolerant parser to produce a partial tree that wee could use both for the outline tree and for the content assist.  
 
Below, there is a list of possible ideas to use a tolerant parser to produce a partial tree that wee could use both for the outline tree and for the content assist.  
  
Please, remember that any comment will be useful!
+
Please, comment below if you think you can help!
  
== Closure Compiler ==  
+
== Improve Closure Compiler ==  
  
 
Closure Compiler is a tolerant javascript-to-javascript written in java. The current version supports ES6, and we're using it for JSDT parsing.
 
Closure Compiler is a tolerant javascript-to-javascript written in java. The current version supports ES6, and we're using it for JSDT parsing.
Line 59: Line 50:
 
After CC parsing, we convert its ParseTree into a jsdt.dom.AST, with the class ClosureCompilerASTConverter.  
 
After CC parsing, we convert its ParseTree into a jsdt.dom.AST, with the class ClosureCompilerASTConverter.  
  
We know it is possible to generate a content treee with it (i.e. [https://www.youtube.com/user/DigiAreaInc this], [https://www.youtube.com/watch?v=GLzEn0Ovsgw this])
+
We know there is at least one Eclipse-based IDE using CC to generate its content outline tree, and the result looks good (i.e. [https://www.youtube.com/user/DigiAreaInc this], [https://www.youtube.com/watch?v=GLzEn0Ovsgw this]).
 +
 
 +
I think we could use the Closure Compiler's Parse Tree to generate a tree which is reusable from the outline tree and that can be used as input for the inference engine.
 +
 
 +
If reusing the parse tree is not an option, we could check if we [https://github.com/google/closure-compiler/wiki/Writing-Compiler-Pass writing a compiler pass] can be useful to generate a content tree.
  
I think we could tweak the CC parsing, and make an extra pass just for generating a tree that we can use for the content outline, and as input to the inference engine, to provide content assist.
+
Also, we could kindly ask the [https://groups.google.com/forum/#!forum/closure-compiler-discuss Closure Compiler forum] to check which is the suggested direction.
  
 +
== Improve with Tern ==
  
== Tern ==
+
[http://ternjs.net/doc/manual.html Tern] is a code-analysis engine for JavaScript written in javascript.
 +
It uses the [http://marijnhaverbeke.nl/blog/acorn.html Acorn parser] to provide [http://marijnhaverbeke.nl/blog/tern.html javascript type inference].
 +
It is a good model for the functionalities we want to improve ([http://ternjs.net/doc/demo/index.html#simple demo]), and it is currently included in JBossTools.
 +
Its downside is that it requires loading all the source files into an [http://ternjs.net/doc/manual.html#server http server], which makes the loading slow.
  
Tern.js is a code-analysis engine for JavaScript written in javascript. It is a good model for the functionalities we want to improve.
+
Ideally, we could improve the communication times by using file communication instead of http communication.  
As a downside, it requires loading all the source files, which should be sent POST to an http server.  
+
However, even in this case, we'll still have the problem of communicating data between Tern (JavaScript) and JSDT (Java).  
  
== Js Program ==
+
== Improve with Node.js Program ==
  
 
We could write a .js program which loads all the available source files, and then outputs a json file with all the information needed to build a content tree and store it on disk. Then, we could use the .json file on disk to infer the suggestions.
 
We could write a .js program which loads all the available source files, and then outputs a json file with all the information needed to build a content tree and store it on disk. Then, we could use the .json file on disk to infer the suggestions.

Revision as of 09:30, 19 April 2017

This document presents the status of the code analysis in JSDT and discusses possible improvements.

Please, feel free to edit this document and to contribute to the discussion.

Current Status

We're using Closure Compiler because of its good performances and architecture (See discussion #).

There are a lot of resources about CC: tutorials, FAQs, design documents.

JSDT 2.0 is missing the content outline and content proposal functionality.

  • In JSDT 1.0 we used to parse all the .js files with Rhino and to store the AST in memory. With the full AST in memory, it was easy to generate a content outline tree and then to use an inference engine to provide the content assist.
  • In JSDT 2.0 we introduced tolerant parsing, but we parse only the .js files which are open in the editor. Also, we do not load the full AST in memory, and we do not generate a full content outline tree and the content assist is not good enough.

Despite JSDT 2.0 is modern and fast, we should fix content outline and content assist for making users happy, (ie Bug 510677#c3).

Improve JSDT 2.0

How can we improve JSDT by restoring the content outline and the content proposal?

  • We need to parse all the .js sources with a fast parser and produce an output tree.
  • with the output tree, we should build the content outline: the JavaScript object hierarchy
  • with the output tree, and the current position in code, we should feed an inference engine, to give content assist proposals.


Ideas

Possible ideas to restore content outline and content assist:

  • use Closure Compiler to parse all files and generate an in-memory content tree; then provide content assiwt with an inference engine.
  • use Tern.js to read the whole source code, generate the content tree and provide the content assist.
  • write a Node.js program that loads all the .js files, and generates a content tree in json format, to be stored in as Eclipse project file. Then, provide content assist with an inference engine.

Below, there is a list of possible ideas to use a tolerant parser to produce a partial tree that wee could use both for the outline tree and for the content assist.

Please, comment below if you think you can help!

Improve Closure Compiler

Closure Compiler is a tolerant javascript-to-javascript written in java. The current version supports ES6, and we're using it for JSDT parsing.

After CC parsing, we convert its ParseTree into a jsdt.dom.AST, with the class ClosureCompilerASTConverter.

We know there is at least one Eclipse-based IDE using CC to generate its content outline tree, and the result looks good (i.e. this, this).

I think we could use the Closure Compiler's Parse Tree to generate a tree which is reusable from the outline tree and that can be used as input for the inference engine.

If reusing the parse tree is not an option, we could check if we writing a compiler pass can be useful to generate a content tree.

Also, we could kindly ask the Closure Compiler forum to check which is the suggested direction.

Improve with Tern

Tern is a code-analysis engine for JavaScript written in javascript. It uses the Acorn parser to provide javascript type inference. It is a good model for the functionalities we want to improve (demo), and it is currently included in JBossTools. Its downside is that it requires loading all the source files into an http server, which makes the loading slow.

Ideally, we could improve the communication times by using file communication instead of http communication. However, even in this case, we'll still have the problem of communicating data between Tern (JavaScript) and JSDT (Java).

Improve with Node.js Program

We could write a .js program which loads all the available source files, and then outputs a json file with all the information needed to build a content tree and store it on disk. Then, we could use the .json file on disk to infer the suggestions.

Back to the top