DLTK 1.0 Planning

Proposed Themes

Following themes are proposed for inclusing into DLTK 1.0 development plan. Themes do not overlap, however Project Management and Configuration Theme and New TCL Parser and Builder Theme implies API changes, and ideally shall be done early in the long-term development plan.

Project Management and Configuration Theme

Improvements of project configuration and manangement infrastructure (which is partially discussed here: DLTK Interpreters, User libraries, and Project management) looks to be one of potential candidates for DLTK 1.0.

If we will decide to go with this work, it looks reasonable to switch to EMF-based interpreters and project management approach, which was stated as one of DLTK 1.0 goals: DLTK 1.0 - Goals and High Demands#DLTK Application management. Reason for this switch in increasing complexity of current project management, and initial ignorance of potential configuration flexibilities comparing to JDT's focus on well-defined Java platform binaries, libraries, etc.

Another good effect from switching to EMF-based Project configurations is simplifications of the APIs for DLTK adopters who want to manage DLTK projects from own code via DLTK APIs.

Related Plan Items

• tclchecker preferences to support pco files (TCL)
• Man pages and other documentations must be bound to the code (TCL)
• Teapot support (TCL)
• RubyGems support (Ruby)

New TCL Parser and Builder Theme

Current parser infrastructure limitations

On DLTK TCL 0.95 parsing infrastucture we have following problem that might be resolved to done well all the advanced features:

TCL everything-is-a-string concept is cool but it makes difficult any tooling support (particulary ones dealing with code analysis). TCL code is also string so to parse TCL source properly any tool must know parameter types at least to follow execution flow in the source. Consider TCL statement

 if a b c

Tool must know that for the case above _b_ and _c_ strings are source code which may be executed and shall be handled appropriately by the tool. Real life is much complex and written above is true only if _c_ string is not equal to _else_ value... This rules (proc metadata) can not be derived (inferred) from program source, and the only way is to supply this rules externally.

Current parser has hard-coded metadata for widely used control-flow TCL procs, such as _if_, _switch_, _while_, etc. This allow DLTK to walk through large part of user program, but this is not enough to cover user programs well. Consider user have some proc _x_, having argument a, which assumed to be executed by the command. Code inlined in first argument of statement like:

 x { set a 5 }

will be ignored by the tool because tool do not know _x_ proc semantic... Such ignorance may led to many inconsitences e.g. tool will not know about variables, procs, package imports... everything that happen inside such code.

Obviously dream-tool may _infer_ parameter types for user procs using project source code. This looks to be extremely hard for TCL but possible, consider

 proc x {exp code} {
   if [expr $exp] 
     {return [eval $code]}
 }

code analyzer may understand that proc x assume code to be supplied in _code_ argument. However:

1. Tool can't infer anything for built-in commands, due to lack of information. And metadata shall be supplied.
2. To implement argument type inference tool must be aware of commands, which analyzer met during type inference.

So implementation of command/proc metadata is important for any tool, which analyze code in any form and for any needs (like navigation, search, and code assistance).

Advantages for the End User

• Increased quality of existing functionality. Now many pieces of TCL code is missed from analysis. As I mentioned above DLTK have only hard-coded subset of built-in (mostly control flow) commands as well as some ITcl and XOTcl related rules. Rich metadata for TCL libraries/extensions will improve quality of existing features like code navigation, search, assist.

• TCL Code Checks. During meta-data driven TCL code parsing all the code will be matched against command definitions (metadata), which is necessarily to build correct AST. During this matching process all relevant errors in code will be identified (like _wrong number of parameters_, see [dltk:TCL Checks Table#Checks based on command definition]. These error reporting will cover about 20 of TclPro checks, which will be performed in milliseconds during source parsing process (which is necessariliy anyway). _We're not going to replace TCLChecker_, which is a great software. This advantage is what end-user getting for free (as a side effect of metadata being available).

• Advanced Code Assist. Starting from good [Argument Hinting (DLTK-223)|DLTK-223@bugs], which will be always avaliable with metadata (including when no man pages supplied), metadata-driven arguments hinting can give more information to the user comparing to man page synopsis, which is not always complete in terms of options enumeration, as well as do not include argument type information (this information is often present in free form as a part of command description and can't be automatically extracted); support for things like Hemang mentioned in [Add support for OO style commands|DLTK-629@bugs]; and more.

• Offline Indexes. Technically new AST will be implemented with Eclipse EMF framework, and as a result easily persisted and organized in offline indexes. This would drastically improve build performance for read-only folders (e.g. interpreter and external static libraries).

Features directly depends on this theme

• TCL indenter (formatter) require knowledge about attribute types, so this feature could be implemented as a part (or after) this theme.

Foundation for Future

Obviously _Refactoring Support_ is a peak of evolution of modern development tools. Java IDEs reached this peak at early 2000yy. IDEs for dynamic lanugages still lack of this support, which is highest demand among any IDE user. Also it looks like TCL users is not an exception and some already requested this feature, so proposed parser improvement is an important and huge step during climb on top.

Powerful Console Theme

As a Java oriented IDE (initially) Eclipse lack of powerful Console support. But developers using interpreted languages like TCL expect good interactive (console, shell) support from tooling. Current Eclipse Console is a simple stream-based output to the the window, with some coloring and pattern matching features, which is not enough for good interactive console with features like:

Included Plan Items

• Console Syntax highlighting
• Console Code assist
• Console showing API Documentation
• Show variable values on hover
• Proper OS streams handling
• Console Available for local and remote processes