Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "JSDT/JSDT Code Analytics"
(Created page with "= JSDT Code Analytics = JSDT 2.0 is missing the content outline and content proposal functionality. In JSDT 1.0 we used to parse all the .js files with Rhino and to store th...") |
(→Improve Closure Compiler) |
||
| (14 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | = | + | <div style="float:right"> |
| + | __TOC__ | ||
| + | </div> | ||
| − | + | This document presents the status of the code analysis in JSDT and discusses possible improvements. | |
| − | + | Please, feel free to edit this document and to contribute to the discussion. | |
| − | + | ||
| − | + | == Current Status == | |
| − | + | ||
| − | + | We're using Closure Compiler because of its good performances and architecture ([http://eclipse.1072660.n5.nabble.com/About-parsing-and-conversion-times-td181113.html See discussion] [https://dev.eclipse.org/mhonarc/lists/wtp-dev/msg09853.html #]). | |
| − | ( | + | |
| − | + | There are a lot of resources about CC: [https://github.com/google/closure-compiler/wiki/Tutorials tutorials], | |
| + | [https://github.com/google/closure-compiler/wiki/FAQ FAQs], | ||
| + | [https://github.com/google/closure-compiler/wiki/Design-Documents design documents]. | ||
| − | + | JSDT 2.0 is missing the content outline and content proposal functionality. | |
| − | + | ||
| − | + | ||
| − | + | * In '''JSDT 1.0''' we used to parse all the .js files with Rhino and to store the AST in memory. With the full AST in memory, it was easy to generate a content outline tree and then to use an inference engine to provide the content assist. | |
| + | * In '''JSDT 2.0''' we introduced tolerant parsing, but we parse only the .js files which are open in the editor. Also, we do not load the full AST in memory, and we do not generate a full content outline tree and the content assist is not good enough. | ||
| − | + | Despite JSDT 2.0 is modern and fast, we should fix content outline and content assist for making users happy, | |
| − | + | (ie [https://bugs.eclipse.org/bugs/show_bug.cgi?id=510677#c3 Bug 510677#c3]). | |
| − | + | ||
| − | + | ||
| − | + | == Improve JSDT 2.0 == | |
| − | + | How can we improve JSDT by restoring the content outline and the content proposal? | |
| − | + | ||
| − | + | ||
| − | + | * We need to parse all the .js sources with a fast parser and produce an output tree. | |
| − | + | * with the output tree, we should build the content outline: the JavaScript object hierarchy | |
| − | We | + | * with the output tree, and the current position in code, we should feed an inference engine, to give content assist proposals. |
| − | + | ||
| − | + | ||
| − | + | ||
| − | |||
| − | + | == Ideas == | |
| − | + | ||
| − | + | Possible ideas to restore content outline and content assist: | |
| − | + | ||
| + | * use Closure Compiler to parse all files and generate an in-memory content tree; then provide content assist with an inference engine. | ||
| + | * use Tern.js to read the whole source code, generate the content tree and provide the content assist. | ||
| + | * write a Node.js program that loads all the .js files, and generates a content tree in json format, to be stored in as Eclipse project file. Then, provide content assist with an inference engine. | ||
| + | Below, there is a list of possible ideas to use a tolerant parser to produce a partial tree that wee could use both for the outline tree and for the content assist. | ||
| − | + | Please, comment below if you think you can help! | |
| + | == Use Closure Compiler AST == | ||
| − | + | Closure Compiler is a tolerant javascript-to-javascript written in java. The current version supports ES6, and we're using it for JSDT parsing. | |
| + | After CC parsing, we convert its ParseTree into a jsdt.dom.AST, with the class ClosureCompilerASTConverter. | ||
| − | + | We know there is at least one Eclipse-based IDE using CC to generate its content outline tree, and the result looks good (i.e. [https://www.youtube.com/user/DigiAreaInc this], [https://www.youtube.com/watch?v=GLzEn0Ovsgw this]). | |
| + | We could use the Closure Compiler's Parse Tree to generate a tree which is reusable from the outline tree and that can be used as input for the inference engine. | ||
| − | + | If reusing the parse tree is not an option, we could check if we [https://github.com/google/closure-compiler/wiki/Writing-Compiler-Pass writing a compiler pass] can be useful to generate a content tree. | |
| − | + | Also, we could kindly ask the [https://groups.google.com/forum/#!forum/closure-compiler-discuss Closure Compiler forum] to check which is the suggested direction. | |
| + | == Improve with Tern == | ||
| − | + | [http://ternjs.net/doc/manual.html Tern] is a code-analysis engine for JavaScript written in javascript. | |
| − | + | It uses the [http://marijnhaverbeke.nl/blog/acorn.html Acorn parser] to provide [http://marijnhaverbeke.nl/blog/tern.html javascript type inference]. | |
| − | + | It is a good model for the functionalities we want to improve ([http://ternjs.net/doc/demo/index.html#simple demo]), and it is currently included in JBossTools. | |
| − | + | Its downside is that it requires loading all the source files into an [http://ternjs.net/doc/manual.html#server http server], which makes the loading slow. | |
| − | + | ||
| − | + | Ideally, we could improve the communication times by using file communication instead of http communication. | |
| + | However, even in this case, we'll still have the problem of communicating data between Tern (JavaScript) and JSDT (Java). | ||
| − | + | == Improve with Node.js Program == | |
| − | + | We could write a .js program which loads all the available source files, and then outputs a json file with all the information needed to build a content tree and store it on disk. Then, we could use the .json file on disk to infer the suggestions. | |
Latest revision as of 02:20, 23 May 2017
This document presents the status of the code analysis in JSDT and discusses possible improvements.
Please, feel free to edit this document and to contribute to the discussion.
Current Status
We're using Closure Compiler because of its good performances and architecture (See discussion #).
There are a lot of resources about CC: tutorials, FAQs, design documents.
JSDT 2.0 is missing the content outline and content proposal functionality.
- In JSDT 1.0 we used to parse all the .js files with Rhino and to store the AST in memory. With the full AST in memory, it was easy to generate a content outline tree and then to use an inference engine to provide the content assist.
- In JSDT 2.0 we introduced tolerant parsing, but we parse only the .js files which are open in the editor. Also, we do not load the full AST in memory, and we do not generate a full content outline tree and the content assist is not good enough.
Despite JSDT 2.0 is modern and fast, we should fix content outline and content assist for making users happy, (ie Bug 510677#c3).
Improve JSDT 2.0
How can we improve JSDT by restoring the content outline and the content proposal?
- We need to parse all the .js sources with a fast parser and produce an output tree.
- with the output tree, we should build the content outline: the JavaScript object hierarchy
- with the output tree, and the current position in code, we should feed an inference engine, to give content assist proposals.
Ideas
Possible ideas to restore content outline and content assist:
- use Closure Compiler to parse all files and generate an in-memory content tree; then provide content assist with an inference engine.
- use Tern.js to read the whole source code, generate the content tree and provide the content assist.
- write a Node.js program that loads all the .js files, and generates a content tree in json format, to be stored in as Eclipse project file. Then, provide content assist with an inference engine.
Below, there is a list of possible ideas to use a tolerant parser to produce a partial tree that wee could use both for the outline tree and for the content assist.
Please, comment below if you think you can help!
Use Closure Compiler AST
Closure Compiler is a tolerant javascript-to-javascript written in java. The current version supports ES6, and we're using it for JSDT parsing.
After CC parsing, we convert its ParseTree into a jsdt.dom.AST, with the class ClosureCompilerASTConverter.
We know there is at least one Eclipse-based IDE using CC to generate its content outline tree, and the result looks good (i.e. this, this).
We could use the Closure Compiler's Parse Tree to generate a tree which is reusable from the outline tree and that can be used as input for the inference engine.
If reusing the parse tree is not an option, we could check if we writing a compiler pass can be useful to generate a content tree.
Also, we could kindly ask the Closure Compiler forum to check which is the suggested direction.
Improve with Tern
Tern is a code-analysis engine for JavaScript written in javascript. It uses the Acorn parser to provide javascript type inference. It is a good model for the functionalities we want to improve (demo), and it is currently included in JBossTools. Its downside is that it requires loading all the source files into an http server, which makes the loading slow.
Ideally, we could improve the communication times by using file communication instead of http communication. However, even in this case, we'll still have the problem of communicating data between Tern (JavaScript) and JSDT (Java).
Improve with Node.js Program
We could write a .js program which loads all the available source files, and then outputs a json file with all the information needed to build a content tree and store it on disk. Then, we could use the .json file on disk to infer the suggestions.