Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

JDT Core/Null Analysis/External Annotations

< JDT Core‎ | Null Analysis
Revision as of 13:24, 27 January 2015 by Stephan.herrmann.berlin.de (Talk | contribs) (Configuration in the IDE)

Note.png
Note
This page describes unreleased work in progress. See bug 331651 as the umbrella bug for this work area


Attaching External Annotations to libraries

Annotation-based null analysis is incomplete as long as libraries consumed by a project have no null annotations in their API. To fill this gap, it should be possible to attach "external annotations" to a library after the fact, i.e., without touching and re-packaging the library itself.

Configuration in the IDE

JDT should understand a new classpath attribute called "annotationpath", which - similar to attaching sources and javadoc - attaches a collection of external annotations to a particular classpath entry pointing to a library. The value of the new attribute can have any of these four shapes:

  • workspace absolute: starts with '/' and is interpreted relative to the workspace
  • project relative: does not start with '/' and is interpreted relative to the enclosing project
  • variable relative: starts with the name of a defined classpath variable
  • file system absolute: if a path starting with '/' cannot be interpreted relative to the workspace, it is interpreted as an absolute path in the local file system.

Status: all four shapes are support by WIP implementation Ok green.gif

Several packagings of such a collection of external annotations have been considered:

directory tree
this packaging is similar to how Java sources are stored in a directory tree, where each directory represents a package and each contained file represents one primary type (class, interface, enum)
Status: implemented Ok green.gif
zip
this packaging simply stores a directory tree in one compressed archive file, inside of which the same structure as above is preserved
Status: implemented Ok green.gif
big file
if annotations are sparse, instead of many tiny files it might be convenient to combine annotations for many types into one file. This could be one file per library or one file per package
Status: not implemented
combined
this packaging is a combination of "zip" and "directory tree". It is proposed in order to consume existing distributions of external annotations in zip format, while still being able to incrementally add more annotations as required by the consuming project. This packaging is inspired by Eiffel's melting ice technology. For external annotations to melt would mean that annotations for a particular type would be extracted from the zip archive and stored as an editable file in the corresponding directory tree. Changes are then made to this file, which will override the content of the zip archive.
Status: not implemented

Individual annotation files use the file extension ".eea" for "Eclipse External Annotations". In zipped format either of ".zip" or ".jar" should be accepted.

Details are tracked in these bugs

JDT/Core
bug 440477 [null] Infrastructure for feeding external annotations into compilation
JDT/UI
bug 440815 External annotations need UI to be attached to library jars on the classpath

Headless consumption

It should also be possible to consume external annotations when compiling outside the IDE or under the control of a build system like ant, maven or gradle. For this reason the batch compiler must be configured in situations where no Eclipse project is available as the context for compilation.

one dedicated path
Using the command line option "-annotationpath location" the batch compiler can be configured to read external annotations from one specified locations, which requires to merge annotations for all relevant libraries into one package (see the packagings discussed above)
Status: implemented Ok green.gif
from classpath
Alternatively, the batch compiler could be configured to scan the classpath used for compilation in order to search external annotations for any type it is reading from a library. This option may be less efficient than other approaches, but it provides the easiest means for integrating external annotations into existing concepts of dependency management - external annotation packages would be just additional artifacts to put on the classpath for compilation. This option would be enabled by an argument-less switch on the command line, say "-externalAnnotationsFromClasspath"
Status: not implemented

Details are tracked in this bug:

JDT/Core
bug 440687 [compiler][batch][null] improve command line option for external annotations


File format

On the one hand, the file format could remain a private implementation detail of the compiler, if the IDE just provides the necessary operations for manipulating external annotations (like, e.g., quick assists). This would make the binary .class file format a natural choice for implementation, as it allows to store all required information and can already be interpreted by the compiler.

On the other hand, annotation files should be amenable to storing, comparing and merging using any version control system. This advocates the use of a textual format.

Additionally, a publicly defined textual file format will allow other tools to operate on annotation files (e.g., conversion from/to other formats, automatic merging etc.), and seasoned users could even directly edit annotation files (although no dedicated editors are planned).

The design is strongly influenced by the need for a low-footprint implementation: files must be reasonably small (speaking against any xml formats) and implementation of file reading must be small and efficient.

The specification of this file format is split into the aspects layout and encoding.

File layout

External annotations for a particular type consist of a type header and entries for individual members.

The basic layout is line-based, i.e., each line represents one element and linebreaks within an element are not allowed. Each line can be either of empty, type header, member name, original signature, or annotated signature.

type header
starts with one of the keywords class, interface or enum followed by the qualified name of the type.
member name
the simple name of a field or method
original signature
directly follows a type header or member name; starts with a single blank followed by the original signature of the preceding element using the encoding discussed below
annotated signature
optionally follows the originally signature, and shares the same format, except that this will contain the actually information about annotations

To provide for maximum stability, members of a type should be alphabetically sorted.

It may become relevant to include meta data (version of a library, origin of an annotation etc.). While no format for this has been defined as of yet, it might be prudent to specify that any of the line formats may contain meta data, which are not to be interpreted by the compiler. To allow for such future extensions, we may define that the each line may contain arbitrary trailing content separated from what has been specified above by any amount of white space (blanks and tabs).

Status: implemented Ok green.gif, except for the skipping of meta data.

Textual encoding of signatures

The current WIP implementation is based on signatures as defined in JVMS 4.7.9.1. This format is used unaltered for "original signatures", and for "annotated signatures" the following changes are applied to the grammar from JVMS (additions in bold face):

ClassTypeSignature:
L [Annot] [PackageSpecifier] SimpleClassTypeSignature {ClassTypeSignatureSuffix} ;
TypeVariableSignature:
T [Annot] Identifier ;
ArrayTypeSignature:
[ [Annot] JavaTypeSignature
TypeParameter:
[Annot] Identifier ClassBound {InterfaceBound}
Annot
0
1

This basically means, that after any of the tokens "L", "T", or "[" an optional "0" or "1" can occur, where "0" represents the nullable annotation, and "1" represents the nonnull annotation. Thus this format is independent of the concrete configured annotations to be used by the JDT compiler, and thus will not create a conflict, when external annotations are contributed from different sources.

Note that in this format, nullness of an array dimension follows the corresponding "[" token, whereas in Java source syntax it precedes that token, i.e., "@NonNull String @NonNull[] @Nullable[]" translated to "[1[0L1java/lang/String;". Curiously, the class file signature better represents the order of annotations from outer to inner.

Status: implemented Ok green.gif, with these exceptions:

  • No support yet for annotations on type parameters (declarations). This part is slightly trickier, because it lacks a leading toke (L/T/[), but AFAICT the grammar is still unambiguous - to be demonstrated soon.
  • No validation of files against the specified format as well as against the host library.

As an alternative, it has been proposed to use an encoding similar to Java source code (but without argument names in method signatures). This format would be somewhat easier to edit manually, but more difficult to match to signatures found in the libraries (jar).

Type Annotations (Java 8)?

The work has been started targeting type annotations from Java 8, because that's the long term goal and provides the greater challenge. The WIP implementation demonstrates that a subset of these annotations can be consumed in projects at 1.7-. Technically, annotations are written as if they were type annotations on a top-level type of a field/method parameter/method return, but internally these are then interpreted as declaration annotations of the corresponding field/method parameter/method.

Ergo: the approach is compatible with both styles of annotations, 1.7- and TYPE_USE at 1.8+.

Example

Declaring that "V Map.get(Object key)" may return null (independent of constraints on V) would be written like this:

interface java/util/Map
 <KV>
get
 (Ljava/lang/Object;)TV;
 (Ljava/lang/Object;)T0V;

Support for working with external annotations

Some support may be provided by JDT/UI, but the design should be open for contributions in separate plug-ins.

Quick fix

Existing quick fixes that are offered on compiler errors/warnings regarding null problems should be extended so that they produce external annotations instead of inserting annotations into Java source code.

Status:

bug 458186 [null][quick fix] Enhance existing quick fixes to insert null annotations into external annotation files
Ok green.gif proof-of-concept implementation exists in a private branch.

Quick assist

Not all use cases of (manually) creating external annotations can be associated to a compiler error/warning. For these use cases a quick assist should be offered when looking at the (attached source code of) a library.

Status:

bug 458200 [null][quick assist] Quick assists for adding external null annotations to library classes
Progress.gif depends on bug 458201 for which a draft implementation exists

Additional operations

The following operations can be contributed by a separate plug-in:

  • Creating an annotation template file from any class file. The template would contain only original signatures, users could then copy any signature to create an annotated signature, i.e., manually creating external annotations would consist of:
    • create a template for the type
    • copy the signature of a member
    • insert 0's and 1's representing annotations
  • Import/export from/to other formats. Given that existing tools already formats for external annotations, a new import wizard could support to import and convert a file from such format into the Eclipse format. Correspondingly for export.
  • Inferring null annotations. Tools exist that analyse the class or source files of a library, and infer null annotations. A UI operation should support invoking such a tool and converting the output into the Eclipse format.
  • Integrations like m2e could recognize project dependencies as containing external annotations and set up the project's classpath in Eclipse to point to the external annotation attachment.
  • ...

All this should be possible based just on the following API:

  • Definition of the file format
  • Specification of the extra classpath attribute


Collecting annotations

To become really effective the technical approach should be accompanied with a community effort to create and collect external null annotations for all relevant libraries.

Manually maintain like source code

The simplest approach for this community effort would be to create a forest of source repositories holding the external annotation files, either one library per repo, or "big" repos comprising multiple libraries. No new technology is needed for this.

From a technical p.o.v. even packaging external annotations as zip, uploading to some central server and discovery of these artifacts using the build technology of your choice are all straight-forward.

Issues to address here include: defining the rules and conventions for the above, and the questions whether enough people are willing to invest the effort for such manual maintenance of an external annotation packge.

Facilitate crowd-sourcing

To lower the barrier of contributing to a global collection of external annotations, techniques from crowd-sourcing could be integrated into the typical workflows, to automatically collect the relevant data directly from members of the crowd. This approach consists of a phase of collection and a phase of discovery.

Collecting external annotations could be directly hooked into some of the operations mentioned above. Interesting events to observe for this purpose include:

  • changes of annotation files in the workspace can be observed using an IResourceChangeListener. Based on the definition of the file format, such a listener can recognize relevant changes and further process / upload as desired.
  • adding an external annotation to a particular signature.
bug 457792 has been filed to investigate, what should be provided by JDT to allow other plugins to receive the relevant events / information.

From this it appears, that by resolving the mentioned bug 457792 the design should be sufficiently open for implementing a collector as a separate plug-in.

Discovering external annotations: External plug-ins are free to analyse each project's classpath (perhaps triggered by a classpath change event), and to consult any well-known server(s) whether external annotations for referenced libraries (version) are available. If so, after download the project can simply be updated by adding the necessary extra classpath attribute to point to the external annotation attachment.

No additional API in JDT seems to be necessary for this purpose.

Details are discussed in these bugs:

bug 449653 Repository for external null annotations
bug 457792 [null] extension point to enable crowd-sourcing of external annotations
here "extension point" is said as a placeholder for any kind of API addition

Refereneces

Formats
  • Annotation File Format, extension .jaif, by the group of Michael Ernst.
  • IntelliJ have their own XML format for external annotations, but I couldn't find a specification of that format, anybody?
Tools
Nit: Nullability Inference Tool
JastAddJ NonNullInference
KAnnotator

Back to the top