JDT Core Programmer Guide/Completion

In this page we collect our understanding of how code completion is implemented, in particular the beast that is CompletionParser

Source code range

Various parties influence the source range that is analysed (scanned & parsed):

• Parser.parseBlockStatements() starts with scanner.resetTo(md.bodyStart, bodyEnd(md))
  • CompletionParser implements bodyEnd() as answering the cursorLocation
• Since stopping at the cursorLocation is unacceptable in some situations (notably involving lambdas), AssistParser.#fallBackToSpringForward() may set eofPosition to one of (a) actual source end or (b) real body end of the method (see below).

It is unclear, how much is gained by stopping to parse in the middle of code:

• does it help to cope with broken code beyond the cursor location?
• does it improve performance? (I doubt it has significant impact. --SH)
• in bug 473654 experiments to avoid setting eof to cursorLocation caused regressions, so some code must meanwhile depend on this strategy.
• it definitely helps to avoid parsing incomplete code in a way that overruns method boundaries (as was observed in bug 473654#c26).

Recovery

This is the story of Parser.currentElement.

Recovery starts when a syntax error is encountered, the chain resumeOnSyntaxError() -> buildInitialRecoveryState() sets things in motion:

• buildInitialRecoveryState() essentially converts existing ASTNodes in Parser.astStack into RecoveredElements.
  • This step maps the nested structure indicated by astStack into a proper tree of recovered elements.
    • More elements will be added later using RecoveredElement.add(*)
    • currentElement will always point to a current leaf in the tree, the rest being reachable via the parent chain.
  • Expressions are explicitly omitted.
    • An exception is implemented for block lambdas, so that an empty block is inserted into the tree of recovered elements.
    • If a lambda appears, e.g., in the RHS of a local declaration, the block cannot be made a child of the local, so the hierarchy has to be tweaked:
      • The local declaration is considered as complete
      • The enclosing block is resumed
      • The new block (body of the lambda) is inserted as a sibling to the local declaration.

The last expression may, however, be processed by updateRecoveryState() (right after resumeOnSyntaxError()):

• relevant things may happen below attachOrphanCompletionNode() -> buildMoreCompletionContext().
  • buildMoreCompletionEnclosingContext() has special treatment for code nested in an if statement and specifically for instanceof.

to be continued.

Lambda Specifics

Completion involving lambdas has essentially been implemented via bug 422468 and bug 423987.

fallBackToSpringForward()

This method will essentially try two strategies:

1. Can we close a pending lambda, either by the next real token, or by one of a fixed set of RECOVERY_TOKENS?
2. If not, can we just jump to the next 'thing'?

For strategy (1) we first check if the eofPosition needs to be pushed past the cursor location, allowing to scan at least up-to the end of the current method body.

For strategy (2) we keep a stack (since bug 535743) of snapshots, represented by instances of CompletionParser, which mainly hold the various stack pointers, plus various assist-specific fields.

• The snapshots are created / updated by calling commit() when various elements are consumed which could contain a lambda expression (see triggerRecoveryUponLambdaClosure()).
• In one arm, fallBackToSpringForward() will copy the state of the latest / matching snapshot into the current parser. After such copyState() the parser will be told, to RESUME (rather than RESTART) in order to keep the parsing state reached thus far.

actFromTokenOrSynthetic()

Via bug 473654 another strategy for closing a pending lambda has been added:

Method actFromTokenOrSynthetic() may synthesize an arbitrary sequence of RECOVERY_TOKENS as long as they can be accepted by the parser. Here we are feeding the tokens right into the parser as if they were coming from the scanner.