Difference between revisions of "JDT Core/Null Analysis"

From Eclipsepedia

Jump to: navigation, search
(Degree of Annotating)
(link to status section for WIP)
(45 intermediate revisions by one user not shown)
Line 1: Line 1:
This page discusses a proposed improvement in the static null analysis of the JDT compiler.
+
<css>
 +
div.b {
 +
    padding:1px;
 +
    background-color:#404040;
 +
}
 +
div.hover {
 +
    background-color:#ffffde;
 +
}
 +
div.fix {
 +
    background-color:#ffffff;
 +
}
 +
</css>
 +
This page describes ongoing work on improving the static null analysis of the JDT compiler.
  
See also {{bug|186342}}.
+
The initial master bug for this work was  {{FixedBug|186342}}, this part has been released for Eclipse Juno (JDT 3.8).
  
  
Line 9: Line 21:
 
null checks etc.
 
null checks etc.
  
However, the current analysis is restricted to flow analysis '''within one method'''.
+
However, the analysis in JDT &le; 3.7 is restricted to flow analysis '''within one method'''.
 
No assumptions can be made about
 
No assumptions can be made about
 
* arguments flowing into a method
 
* arguments flowing into a method
Line 19: Line 31:
 
* explicit contracts via an extended type system or annotations
 
* explicit contracts via an extended type system or annotations
  
The second option is well explored in research and some existing tools (like FindBugs)
+
The second option is well explored in research and some existing tools (like the Checker Framework, JML, FindBugs)
 
already introduce specific annotations to this end.
 
already introduce specific annotations to this end.
  
One could argue that advanced analysis should be left to specialized tools (like FindBugs)
+
One could argue that advanced analysis should be left to specialized tools  
 
but having something like this in the JDT compiler should show two benefits:
 
but having something like this in the JDT compiler should show two benefits:
 
* feedback is more immediate and it is available for all JDT users without installing more software
 
* feedback is more immediate and it is available for all JDT users without installing more software
* analysis might be more precise than existing tools, because the actual flow analysis in the JDT compiler is already pretty strong (unproven claim).
+
* analysis might be more precise than some existing tools provide, because the actual flow analysis in the JDT compiler is already pretty strong (unproven claim).
  
==Design Space for Inter-Procedural Null Analysis==
 
This section discusses different options how tools for inter-procedural analysis could be designed. This write-up was made in preparation of designing a concrete strategy for the JDT.
 
  
===Degree of Annotating===
+
[[Image:Video.png]] See also the recording of this [http://eclipsecon.org/europe2011/ ECE 2011] [http://eclipsecon.org/europe2011/sessions/bye-bye-npe session]: '''[http://www.fosslc.org/drupal/content/bye-bye-npe Bye, bye, NPE]'''
A '''radical approach''' would suggest that '''every reference type''' in the program must explicitly exclude or include the value <code>null</code>. E.g., <code>String</code> would not be legal type in any declaration (local variable, field, method signature) but only <code>@NonNull String</code> and <code>@MaybeNull String</code> are.
+
  
This radical approach has two problems:
+
==Actual Strategy in the JDT==
* it introduces vast efforts for annotating every type reference in the program
+
* it is very difficult to apply to intermediate variables within a method body with branches, loops etc.
+
E.g., the JDT compiler has no problem seeing that this is safe:
+
<source lang="java">
+
Foo foo2 = null;
+
if ((foo != null) && foo.isOK())
+
  foo2 = foo;
+
else
+
  foo2 = new Foo();
+
foo2.bar();
+
</source>
+
In the radical approach at least two more variable were necessary: at each point where analysis finds that a value cannot be null, a new variable with a differently annotated type would be needed. By contrast, the compiler can manage these intermediate states implicitly.
+
  
Thus it seems better feasible not to strive for full proofs of the absence of runtime errors, but to focus on gradually feeding more information into the analysis in order to just detect ''more'' (instead of all) potential runtime errors already during compilation. The radical approach can be weakened in two ways:
+
{{note|Work in progress|For Eclipse Juno (JDT 3.8) we released a first version of this feature. The implemented features are complete and tested, but further improvements are still in the pipe line (see [[#Status]]).}}
* make annotations optional
+
* limit the program locations where annotations should occur (note, how this actually relates to [http://jcp.org/en/jsr/summary?id=308 JSR 308]).
+
  
In the vein of design by contract the following locations are relevant
+
By default the JDT does not support inter-procedural null analysis, however, starting with 3.8 the JDT can be configured to use annotations for extended null checking.
* method parameters (= method precondition)
+
* method return value (= method postcondition)
+
* fields (= invariant)
+
To truly reflect design by contract one might want to weaken the rules to
+
* exclude non-API methods from contracts
+
*: '''but''' ''inter-procedural analysis is also relevant within a class, so annotating even private methods is useful, too''.
+
* accept inconsistent field states in the middle of a method body (only the first read and the last write within each method would be checked)
+
*: '''but''' ''such intermediate inconsistent states might be observed by concurrent executions and would make the analysis unreliable.''
+
  
===Syntax===
+
Up-to-date documentation for the annotation-based null analysis and its new configuration options can be found in the Eclipse help (Eclipse 3.8 and greater):
Generally annotations could happen in two ways:
+
* '''Java development user guide'''
 +
** '''Reference > Preferences > Java > Compiler > Errors/Warnings'''
 +
**: scroll down to '''Null analysis''' -- [http://help.eclipse.org/juno/topic/org.eclipse.jdt.doc.user/reference/preferences/java/compiler/ref-preferences-errors-warnings.htm read online]
 +
** '''Tasks > Improving Java code quality > Using null annotations'''
 +
**: [http://help.eclipse.org/juno/topic/org.eclipse.jdt.doc.user/tasks/task-using_null_annotations.htm?cp=1_3_9_0 read online]
  
Extended '''Javadoc''', like:
+
===Specifying nullness===
<source lang="java">
+
/**
+
* @param @nonnull input ...
+
* @return @maybenull ...
+
*/
+
public String foo(String input) { ... }
+
</source>
+
  
Alternatively, '''Java 5 annotations''' can be used to say the same:
+
Null annotations in method signatures can be interpreted as [[/Null Contracts|null contracts]], however, a more general approach considers null annotations as an extension of the type system. Eventually - that is once JSR 308 can be used - all type references should either include or exclude null, which allows for complete checking of any possible dereferencing of null. In other words, a fully annotated program which passes the type checker will never raise an NPE at runtime.
<source lang="java">
+
public @maybenull String foo(@nonnull String input) { ... }
+
</source>
+
  
This is where [http://www.jcp.org/en/jsr/detail?id=305 JSR 305] comes into focus which covers the issue of standardizing "Annotations for Software Defect Detection"
+
To achieve this guarantee two annotations are used. The specific annotations types can be selected as a preference, but the following defaults are provided:
 +
* [http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/tree/org.eclipse.jdt.annotation/src/org/eclipse/jdt/annotation/NonNull.java org.eclipse.jdt.annotation.NonNull]
 +
* [http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/tree/org.eclipse.jdt.annotation/src/org/eclipse/jdt/annotation/Nullable.java org.eclipse.jdt.annotation.Nullable]
  
After a first analysis I see two problems with JSR 305:
+
For any variable who's type is annotated with @NonNull (or the configured equivalent) the following rules apply:
* It has stalled, no official documents produced in 4 years, I couldn't find a proof of any activity during the last 2 years. JSR is marked as inactive, may soon be withdrawn.
+
* It is illegal to bind null or a value that can be null to the variable. (For fields and local variables this applies to initialization and assignments, for method argument binding a value means to pass an actual argument in a method call).
* It is far more generic than what we need for this specific issue: everything is built upon a meta annotation <code>@TypeQualifier</code>, it invests in supporting four states:
+
* It is legal and safe to dereference such a variable for accessing a field or a method of the bound object.
** unspecified (no annotation)
+
** @UnknownNullness (same interpretation as unspecified)
+
** @Nonnull
+
** @NullFeasible
+
: The rationale for @UnknownNullness is for discarding an inherited specification. I wasn't convinced that any contract could be specialized to "no contract" - when specializing an inherited contract you should be explicit what the new contract is (which must be conform to the inherited contract).
+
See also these [http://www.cs.umd.edu/~pugh/JSR-305.pdf slides (May 2008)] by William Pugh.
+
  
The main issue with both syntaxes is the lack of standardization.
+
For any variable who's type is annotated with @Nullable (or the configured equivalent) the following rules apply:
 +
* It is legal to bind null or a value that can be null to the variable (see details above).
 +
* It is illegal to dereference such a variable for either field or method access.
  
====Retention====
+
The above rules imply that the value from a @NonNull variable can be bound to a variable annotated with @Nullable, but the opposite direction is generally illegal.
Annotations have the advantage that a CLASS retention (or perhaps even RUNTIME) would support compiling against contracts in class files, which is not easily possible with the Javadoc based approach.
+
Only after an explicit null check can a @Nullable variable be treated as being @NonNull for the sake of binding to another @NonNull variable or for dereferencing.
  
===Standard vs. Configuration===
+
For interaction with inheritance see [[/Null Contracts#Null Contract Inheritance|Null Contract Inheritance]].
Once the JDT compiler officially supports any specific syntax this creates a de-facto
+
standard which might conflict with existing and future standards (any code written against
+
the de-facto standard might be incompatible with future tools).
+
  
Possible solutions:
+
===Usage===
# Wait for the standard annotations (which may be waiting for ever)
+
In order to try the new analysis against any existing Java project the following steps should help:
# Make the concrete syntax configurable
+
#* select between Javadoc and annotation styles
+
#* select the exact annotation classes to use (cf. {{bug|186342#c12}}).
+
#: Do ''not'' provide a default, as not to define anything standardish
+
# Provide implementation as a separate "use on your own risk" plug-in
+
  
(1) doesn's look attractive to me. (2) is kind-of a workaround, carefully giving the message: we are not defining a standard, if you use this you should be prepared to change your annotations once a standard is created (automatic migration to a new standard shouldn't be so hard OTOH). (3) might be used to side-step the whole issue by saying: we're only doing technical exploration, but interested folks may still download and use this as early adopters. Technically, I would implement this using [[:Category:OTEquinox|OT/Equinox]] :).
+
* Open the compiler preferences for your project:
This would be an intermediate solution (still need to download additional stuff - but works as integral part of the incremental compiler) - shouldn't be difficult to migrate into the JDT/Core once we know more about standardization.
+
:* Ensure compliance is 1.5 or higher
 +
:* Find the section '''Null analysis''' and select '''Enable annotation-based null analysis'''
 +
::[[Image:annotation-based-null-analysis.png]]
 +
:* You will be prompted to update the severity of some null-related problems, this is recommended.
 +
* Apply any of the annotations <code>@NonNull</code>, <code>@Nullable</code> or <code>@NonNullByDefault</code> in your code.
 +
** The annotation will be unresolvable at first, but a quick fix is offered to update the project setup:
 +
*: (see also: [http://help.eclipse.org/juno/topic/org.eclipse.jdt.doc.user/tasks/task-using_null_annotations.htm?cp=1_3_9_0_2#buildpath_setup Help: buildpath setup])
 +
*:* '''Copy library with default annotations to build path''' (plain Java projects), ''or'':
 +
*:* '''Add library with default annotations to build path''' (Plug-in projects).
 +
* Define <code>@NonNull</code> as the default at the granularity of your choice (package/type):
 +
** '''package''': add a file <code>package-info.java</code> with contents like this:
 +
**: <code>@NonNullByDefault package org.my.pack.age;</code>
 +
** '''type''': add <code>@NonNullByDefault</code> to the type declaration.
 +
* At this point you should see plenty of new errors and warnings
  
My irrational hopes are, that once users find out how great this is, someone will step forward and declare a standard. It would be great if people have something to play with and actually see the difference.
+
{{tip|Hints:|Setting the default to <code>@NonNull</code> is the recommended option for new projects, but for existing projects this could require a major clean-up in terms of inserting explicit <code>@Nullable</code> annotations in many locations. Experience shows, that this is a non-trivial task since in existing code the original intention, which parameter/return value should be allowed to be null, is usually blurred. Here starting with no default but adding individual annotations where the intention is clear will cause less disruption. This incremental approach should be seen as a long-term yet low effort task, not only for getting rid of NPEs but also for sorting the responsibilities in the code, which may have eroded over time.<br>For more hints on adpoting these annotations see [[JDT Core/Null Analysis/Adopting Null Annotations|Adoption Null Annotations]].}}
  
===Semantical details===
+
====Cleaning up====
I see these issues worth discussing:
+
When applying the new analysis to a big existing project, the sheer number of new problems may look intimidating but that's where quick fixes will come to the rescue.
# Do we need more than @Nonnull and @MaybeNull?
+
Currently the following problems offer a quickfix:
# How exactly do annotations interact with inheritance (should mainly just apply the rules of design by contract, actually)
+
# What are the defaults?
+
# How do field specifications interact with concurrency?
+
  
Regarding (3) I believe in [http://users.encs.concordia.ca/~chalin/papers/TR-2006-003.v3s-pub.pdf "Non-null References by Default in Java: Alleviating the Nullity Annotation Burden"].
+
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Null type mismatch: required '@NonNull Foo' but the provided value is null</div></div>
OTOH, for perfect freedom one could make the default configurable in a hierarchical way
+
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Null type mismatch: required '@NonNull Foo' but the provided value is specified as @Nullable</div></div> ''not in JDT 3.8.0 - see {{bug|337977#c19}}
(see also [http://www.cs.umd.edu/~pugh/JSR-305.pdf Pugh pp. 43 ff]):
+
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Null type mismatch: required '@NonNull Foo' but the provided value is inferred as @Nullable</div></div>
* per project, package, class, method
+
* <div class="b"><div class="hover">[[Image:Quickfix_warning_obj.gif]] Null type safety: The expression of type Foo needs unchecked conversion to conform to '@NonNull Foo'</div></div>
* all public, protected, default members
+
*: '''Fixable for these locations: return statements''':
* all fields / method parameters / method return values (also for convenience: signatures (param|return) / all)
+
*: Note that the mentioned @NonNull declaration may be implicit via an applicable default
 +
*: In cases 3) and 4) use only with care: the compiler has no clear indication if @Nullable was actually intended or not
 +
*: The fix is:
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Declare method return as @Nullable</div></div>
 +
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Null comparison always yields false: The variable x is specified as @NonNull</div></div>
 +
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Redundant null check: The variable x is specified as @NonNull</div></div>
 +
*: '''Fixable for these locations: null check for a method parameter'''
 +
*: The fix is:
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Declare method parameter as @Nullable</div></div>
 +
*: Otherwise a null check may indeed be unnecessary and should be deleted.
 +
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] The return type is incompatible with the @NonNull return from SuperFoo.foo()</div></div>
 +
*: '''Location: declaration of an overriding method'''
 +
*: Note again that the mentioned @NonNull declaration may be due to a default.
 +
*: Possible fixes are:
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Change return type of foo(..) to '@NonNull'</div></div>
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Change return type of overridden foo(..) to '@Nullable'</div></div>
 +
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Illegal redefinition of parameter a, inherited method from SuperFoo declares this parameter as @Nullable</div></div>
 +
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Illegal redefinition of parameter a, inherited method from SuperFoo does not constrain this parameter.</div></div>
 +
*: '''Location: Parameter declaration of an overriding method'''
 +
*: The second form occurs when no null default applies at the scope of the super method.
 +
*: Possible fixes are:
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Change parameter type to '@Nullable'</div></div>
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Change parameter type in overridden 'foo(..)' to '@NonNull'</div></div>
 +
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Missing non-null annotation: inherited method from SuperClass declares this parameter as @NonNull</div></div>
 +
* <div class="b"><div class="hover">[[Image:Quickfix_error_obj.gif]] Missing nullable annotation: inherited method from SuperClass declares this parameter as @Nullable</div></div>
 +
*: '''Location: Parameter declaration of an overriding method'''
 +
*: Quick fix is either of:
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Change parameter type to @NonNull</div></div>
 +
*: <div class="b"><div class="fix">[[Image:Correction_change.gif]] Change parameter type to @Nullable</div></div>
  
{{bug|186342#c14}} points out that finding a good default is difficult because different not-annotated '''library functions''' have different contracts (e.g: HashMap.get() -> @MaybeNull, JTable.getSelectedColumns() -> @NonNull). Thus either default would produce a lot of false positives. For solving this issue one might use more fine grained control also over 3rd-party code, like a '''nullity-profile''', with externalized defaults and exceptions / contracts.
+
These quick fixes can be applied...
 +
* individually (Ctrl-1)
 +
* all occurrences per file (via the hover)
 +
* all occurrences (via context menu in the Problems view)
 +
Note, that some quick fixes require to modify another compilation unit (file) than the one
 +
where the problem was observed. For these quickfixes the current implementation doesn't
 +
support fixing several equal issues in bulk (for the technical background see
 +
{{bug|337977}}).
  
Actually, a nullity-profile could also be used to map existing annotations within 3rd-party code to a different annotation class.
+
===Defaults at different levels===
 +
If no null annotations are used, the compiler uses the original Java semantics,
 +
where the following is legal for all variables of reference types:
 +
* assign <code>null</code>, ''and''
 +
* dereference without check.
  
==Actual Stragegy in the JDT==
+
To generally avoid these weak semantics you may want to declare that by default all types should be considered as nonnull.
  
By default the JDT does not support inter-procedural null analysis, however, starting with {{bug|186342}} the JDT can be configured to use annotations for null contracts across method boundaries.
+
This is done using the annotation 'NonNullByDefault'. The qualified type
 +
name of this annotation can be configured using the preference "'NonNullByDefault' annotation".
 +
The built-in value for these preference is <code>org.eclipse.jdt.annotation.NonNullByDefault</code>.
  
===Configuring Null Annotations for the JDT===
+
* This annotation takes an optional boolean parameter; when set to false this causes the annotation to ''cancel'' a default that may possible apply at the current location. This is useful when, e.g., sub-classing a legacy class without null annotation, where the sub-class sits in a place that would otherwise apply non-null as the default, which would make all overrides incompatible with inherited methods.
By default the JDT does not recognize any null annotations but it can be configured to do so.
+
For this purpose three independent options exist:
+
# specify the names of annotation types to be used for marking nullable vs. nonnull types.
+
# emulate null annotation types that do not exist on the build path
+
# implicitly import null annotation types in every Java file so that null annotations can directly be used by their simple name.
+
  
If at least one of these options is defined / enabled, the compiler will start analysing null contracts.
+
This annotation can be applied to any package, Java type or method and affects all
 
+
method returns and parameters with undefined null status within their scope. (More locations will be supported in the future, but local variables are intentially unaffected by any default).
If null contracts are enabled ..
+
# ... without specifying annotation type names then the following built-in defaults will be used:
+
#* nullable = <code>org.eclipse.jdt.annotation.Nullable</code>
+
#* nonnull = <code>org.eclipse.jdt.annotation.NonNull</code>
+
# ... without enabling annotation type emulation then the specified annotation types have to be provided on the build path (either as source or binary files).
+
# ... without enabling implicit annotation type imports then annotation types have to be imported or referenced by their fully qualified name.
+
 
+
===Null Contracts===
+
Once properly configured the JDT compiler supports specification and checking of null contracts. Each part of a null contract implies an obligation for one side and a guarantee for the other side.
+
 
+
====Method Parameters====
+
 
+
When a method parameter is specified as '''nullable''' this defines the <font color="red">obligation</font> for the '''method implementation''' to cope with null values without throwing NPE. '''Clients''' of such a method enjoy the <font color="green">guarantee</font> that it is safe to call the method with a null value for the given parameter.
+
 
+
When a method parameter is specified as '''nonnull''' all '''clients''' are <font color="red">obliged</font> to ensure that null will never be passed as the value for this parameter. Thus the '''method implementation''' may rely on the <font color="green">guarantee</font> that null will never occur as the value for this parameter.
+
 
+
====Method Returns====
+
The situation is reversed for method returns. All four cases are summarized by the following table:
+
 
+
{|border=1
+
! !! caller !! method implementation
+
|-
+
| nullabel parameter
+
| <font color="green">may safely pass null without checking</font>
+
| <font color="red">must check before dereference</font>
+
|-
+
| nonnull parameter 
+
| <font color="red">must not pass null  </font>
+
| <font color="green">may use without checks</font>
+
|-
+
| nullable return   
+
| <font color="red">must check before dereference</font>
+
| <font color="green">can safely pass null</font>
+
|-
+
| nonnull return   
+
| <font color="green">may use without check</font>
+
| <font color="red">must not return null</font>
+
|-
+
|}
+
 
+
====Local Variables====
+
Null contracts can also be defined for local variables although this doesn't improve the
+
analysis by the compiler, because local variables can be fully analyzed without annotations, too. Here the main advantage of null annotations is in documenting intentions.
+
 
+
The following is an example of a program where all sides adhere to their respective part of the contract:
+
<source lang="java">
+
public class Supplier {
+
    // this method requires much but delivers little
+
    @Nullable String weakService (@NonNull String input, boolean selector) {
+
        if (selector)
+
            return input;
+
        else
+
            return null;
+
    }
+
    // this method requires nothing and delivers value
+
    @NonNull String strongService (@Nullable String input) {
+
        if (input == null)
+
          return "";
+
        else
+
          return input.toUpperCase();
+
    }
+
}
+
public class Client {
+
    void main(boolean selector) {
+
        Supplier supplier = new Supplier();
+
        @Nullable String value = supplier.weakService("OK", selector);
+
        @NonNull String result = supplier.strongService(value);
+
        System.out.println(result.toLowerCase());
+
    }
+
}
+
</source>
+
Notes:
+
* Althoug we know that toUpperCase() will never return null, the compiler does not know as long as <code>java.lang.String</code> does not specify null contracts. Therefor the compiler has to raise a warning that it has "insufficient nullness information" to guarantee contract adherence.
+
* Null annotations for the local variables are redundant here, as the nullness information can be fully derived from the variable's initialization (and no further assignments exist).
+
  
===Null Contract Inheritance===
+
==Status==
A method that overrides or implements a corresponding method of a super type (class or interface) inherits the full null contract. Its implementation will thus be checked against
+
this inherited contract. For the sake of safe polymorphic calls,
+
null contracts must be inherited without modification, ''or'' be redefined in the following ways:
+
* A nonnull method parameter may be '''relaxed''' to a nullable parameter. The additional checks have to be performed in the body of the overriding method. Callers of the super type must still pass nonnull, while callers which are aware of the sub type may pass null.
+
* A nullable method return may be '''tightened''' to a nonnull return. The additional checks must again be performed in the body of the overriding methods. Callers of the super type still have to check for null, whereas callers which are aware of the sub type may safely assume nonnull return values.
+
  
Any overrides that attempt to change a null contract in the opposite directions will raise a compile time error.
+
===Done===
 +
At the current point the following bugs are resolved - functionality is released with Eclipse Juno:
 +
* {{FixedBug|186342}}.- [[Image:Ok_green.gif]] [compiler][null] Using annotations for null checking
 +
* {{FixedBug|334455}} - [[Image:Ok_green.gif]] UI for new preferences regarding null annotations (plus a dup: {{FixedBug|364815}}).
 +
* {{FixedBug|334457}} - [[Image:Ok_green.gif]] [compiler][null] check compatibility of inherited null contracts
 +
* {{FixedBug|331647}} - [[Image:Ok_green.gif]] [compiler][null] support flexible default mechanism for null-annotations
 +
* {{FixedBug|365208}} - [[Image:Ok_green.gif]] [compiler][batch] command line options for annotation based null analysis
  
This explicitly implies that callers only need to inspect the null contract of the statically declared type of a call target to safely assume that all runtime call targets will all adhere (at least) to the contract as specified in the statically declared type, even if the runtime type of the call target is any sub type of the declared type.
+
===Current Beta===
 +
A [[/Beta|Beta Release]] is offered for early adopters, which adds experimental support for null annotations for fields.
  
 
===Future===
 
===Future===
The following bugzillas address future improvements of the above strategy (order roughly by priority):
+
The following bugzillas address future improvements of the above strategy:
* {{bug|334455}} - UI for new preferences regarding null annotations
+
* {{bug|337977}} - [[Image:Progress.gif]] [quick fix] Add quickfixes for null annotations - partly released for 3.8
* {{bug|334457}} - [compiler][null] check compatibility of inherited null contracts
+
* {{bug|331649}} - [[Image:Progress.gif]] [compiler][null] consider null annotations for fields
* {{bug|331647}} - [compiler][null] support flexible default mechanism for null-annotations
+
* {{bug|331651}} - [[Image:Glass.gif]] [compiler][null] Support nullity profiles for libraries
* {{bug|331651}} - [compiler][null] Support nullity profiles for libraries
+
* {{bug|331649}} - [compiler][null] consider null annotations for fields
+
  
 
[[Category:JDT]]
 
[[Category:JDT]]

Revision as of 13:52, 1 July 2012

This page describes ongoing work on improving the static null analysis of the JDT compiler.

The initial master bug for this work was bug 186342, this part has been released for Eclipse Juno (JDT 3.8).


Contents

Introduction

The static analysis of the JDT compiler detects many potential programming problems related to the null-ness of variables: dereferencing a null value (-> NPE), redundant null checks etc.

However, the analysis in JDT ≤ 3.7 is restricted to flow analysis within one method. No assumptions can be made about

  • arguments flowing into a method
  • return values from method calls and
  • field reads.

In order to include these elements in the analysis one could either

  • use whole program analysis (very expensive - not feasible for a (incremental) compiler)
  • explicit contracts via an extended type system or annotations

The second option is well explored in research and some existing tools (like the Checker Framework, JML, FindBugs) already introduce specific annotations to this end.

One could argue that advanced analysis should be left to specialized tools but having something like this in the JDT compiler should show two benefits:

  • feedback is more immediate and it is available for all JDT users without installing more software
  • analysis might be more precise than some existing tools provide, because the actual flow analysis in the JDT compiler is already pretty strong (unproven claim).


Video.png See also the recording of this ECE 2011 session: Bye, bye, NPE

Actual Strategy in the JDT

Note.png
Work in progress
For Eclipse Juno (JDT 3.8) we released a first version of this feature. The implemented features are complete and tested, but further improvements are still in the pipe line (see #Status).


By default the JDT does not support inter-procedural null analysis, however, starting with 3.8 the JDT can be configured to use annotations for extended null checking.

Up-to-date documentation for the annotation-based null analysis and its new configuration options can be found in the Eclipse help (Eclipse 3.8 and greater):

  • Java development user guide
    • Reference > Preferences > Java > Compiler > Errors/Warnings
      scroll down to Null analysis -- read online
    • Tasks > Improving Java code quality > Using null annotations
      read online

Specifying nullness

Null annotations in method signatures can be interpreted as null contracts, however, a more general approach considers null annotations as an extension of the type system. Eventually - that is once JSR 308 can be used - all type references should either include or exclude null, which allows for complete checking of any possible dereferencing of null. In other words, a fully annotated program which passes the type checker will never raise an NPE at runtime.

To achieve this guarantee two annotations are used. The specific annotations types can be selected as a preference, but the following defaults are provided:

For any variable who's type is annotated with @NonNull (or the configured equivalent) the following rules apply:

  • It is illegal to bind null or a value that can be null to the variable. (For fields and local variables this applies to initialization and assignments, for method argument binding a value means to pass an actual argument in a method call).
  • It is legal and safe to dereference such a variable for accessing a field or a method of the bound object.

For any variable who's type is annotated with @Nullable (or the configured equivalent) the following rules apply:

  • It is legal to bind null or a value that can be null to the variable (see details above).
  • It is illegal to dereference such a variable for either field or method access.

The above rules imply that the value from a @NonNull variable can be bound to a variable annotated with @Nullable, but the opposite direction is generally illegal. Only after an explicit null check can a @Nullable variable be treated as being @NonNull for the sake of binding to another @NonNull variable or for dereferencing.

For interaction with inheritance see Null Contract Inheritance.

Usage

In order to try the new analysis against any existing Java project the following steps should help:

  • Open the compiler preferences for your project:
  • Ensure compliance is 1.5 or higher
  • Find the section Null analysis and select Enable annotation-based null analysis
Annotation-based-null-analysis.png
  • You will be prompted to update the severity of some null-related problems, this is recommended.
  • Apply any of the annotations @NonNull, @Nullable or @NonNullByDefault in your code.
    • The annotation will be unresolvable at first, but a quick fix is offered to update the project setup:
    (see also: Help: buildpath setup)
    • Copy library with default annotations to build path (plain Java projects), or:
    • Add library with default annotations to build path (Plug-in projects).
  • Define @NonNull as the default at the granularity of your choice (package/type):
    • package: add a file package-info.java with contents like this:
      @NonNullByDefault package org.my.pack.age;
    • type: add @NonNullByDefault to the type declaration.
  • At this point you should see plenty of new errors and warnings
Idea.png
Hints:
Setting the default to @NonNull is the recommended option for new projects, but for existing projects this could require a major clean-up in terms of inserting explicit @Nullable annotations in many locations. Experience shows, that this is a non-trivial task since in existing code the original intention, which parameter/return value should be allowed to be null, is usually blurred. Here starting with no default but adding individual annotations where the intention is clear will cause less disruption. This incremental approach should be seen as a long-term yet low effort task, not only for getting rid of NPEs but also for sorting the responsibilities in the code, which may have eroded over time.
For more hints on adpoting these annotations see Adoption Null Annotations.


Cleaning up

When applying the new analysis to a big existing project, the sheer number of new problems may look intimidating but that's where quick fixes will come to the rescue. Currently the following problems offer a quickfix:

  • Quickfix error obj.gif Null type mismatch: required '@NonNull Foo' but the provided value is null
  • Quickfix error obj.gif Null type mismatch: required '@NonNull Foo' but the provided value is specified as @Nullable
    not in JDT 3.8.0 - see bug 337977#c19
  • Quickfix error obj.gif Null type mismatch: required '@NonNull Foo' but the provided value is inferred as @Nullable
  • Quickfix warning obj.gif Null type safety: The expression of type Foo needs unchecked conversion to conform to '@NonNull Foo'
    Fixable for these locations: return statements:
    Note that the mentioned @NonNull declaration may be implicit via an applicable default
    In cases 3) and 4) use only with care: the compiler has no clear indication if @Nullable was actually intended or not
    The fix is:
    Correction change.gif Declare method return as @Nullable
  • Quickfix error obj.gif Null comparison always yields false: The variable x is specified as @NonNull
  • Quickfix error obj.gif Redundant null check: The variable x is specified as @NonNull
    Fixable for these locations: null check for a method parameter
    The fix is:
    Correction change.gif Declare method parameter as @Nullable
    Otherwise a null check may indeed be unnecessary and should be deleted.
  • Quickfix error obj.gif The return type is incompatible with the @NonNull return from SuperFoo.foo()
    Location: declaration of an overriding method
    Note again that the mentioned @NonNull declaration may be due to a default.
    Possible fixes are:
    Correction change.gif Change return type of foo(..) to '@NonNull'
    Correction change.gif Change return type of overridden foo(..) to '@Nullable'
  • Quickfix error obj.gif Illegal redefinition of parameter a, inherited method from SuperFoo declares this parameter as @Nullable
  • Quickfix error obj.gif Illegal redefinition of parameter a, inherited method from SuperFoo does not constrain this parameter.
    Location: Parameter declaration of an overriding method
    The second form occurs when no null default applies at the scope of the super method.
    Possible fixes are:
    Correction change.gif Change parameter type to '@Nullable'
    Correction change.gif Change parameter type in overridden 'foo(..)' to '@NonNull'
  • Quickfix error obj.gif Missing non-null annotation: inherited method from SuperClass declares this parameter as @NonNull
  • Quickfix error obj.gif Missing nullable annotation: inherited method from SuperClass declares this parameter as @Nullable
    Location: Parameter declaration of an overriding method
    Quick fix is either of:
    Correction change.gif Change parameter type to @NonNull
    Correction change.gif Change parameter type to @Nullable

These quick fixes can be applied...

  • individually (Ctrl-1)
  • all occurrences per file (via the hover)
  • all occurrences (via context menu in the Problems view)

Note, that some quick fixes require to modify another compilation unit (file) than the one where the problem was observed. For these quickfixes the current implementation doesn't support fixing several equal issues in bulk (for the technical background see bug 337977).

Defaults at different levels

If no null annotations are used, the compiler uses the original Java semantics, where the following is legal for all variables of reference types:

  • assign null, and
  • dereference without check.

To generally avoid these weak semantics you may want to declare that by default all types should be considered as nonnull.

This is done using the annotation 'NonNullByDefault'. The qualified type name of this annotation can be configured using the preference "'NonNullByDefault' annotation". The built-in value for these preference is org.eclipse.jdt.annotation.NonNullByDefault.

  • This annotation takes an optional boolean parameter; when set to false this causes the annotation to cancel a default that may possible apply at the current location. This is useful when, e.g., sub-classing a legacy class without null annotation, where the sub-class sits in a place that would otherwise apply non-null as the default, which would make all overrides incompatible with inherited methods.

This annotation can be applied to any package, Java type or method and affects all method returns and parameters with undefined null status within their scope. (More locations will be supported in the future, but local variables are intentially unaffected by any default).

Status

Done

At the current point the following bugs are resolved - functionality is released with Eclipse Juno:

  • bug 186342.- Ok green.gif [compiler][null] Using annotations for null checking
  • bug 334455 - Ok green.gif UI for new preferences regarding null annotations (plus a dup: bug 364815).
  • bug 334457 - Ok green.gif [compiler][null] check compatibility of inherited null contracts
  • bug 331647 - Ok green.gif [compiler][null] support flexible default mechanism for null-annotations
  • bug 365208 - Ok green.gif [compiler][batch] command line options for annotation based null analysis

Current Beta

A Beta Release is offered for early adopters, which adds experimental support for null annotations for fields.

Future

The following bugzillas address future improvements of the above strategy:

  • bug 337977 - Progress.gif [quick fix] Add quickfixes for null annotations - partly released for 3.8
  • bug 331649 - Progress.gif [compiler][null] consider null annotations for fields
  • bug 331651 - Glass.gif [compiler][null] Support nullity profiles for libraries