Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "JDT Core Programmer Guide/ECJ"
(→Compilation phases) |
(→What IS the Compiler / ECJ?: - updated after ecj refactoring via issue #181) |
||
Line 4: | Line 4: | ||
Strange enough this question does not have a single true answer. | Strange enough this question does not have a single true answer. | ||
− | ===Project organization=== | + | ===Project organization before 4.27 === |
+ | |||
+ | {{tip|Project refactoring|As of https://github.com/eclipse-jdt/eclipse.jdt.core/issues/181 the project structure has been refactored. This section describes the state '''before''' this refactoring.}} | ||
+ | |||
The following locations contribute to the compiler: | The following locations contribute to the compiler: | ||
* org.eclipse.jdt.core/compiler | * org.eclipse.jdt.core/compiler | ||
Line 16: | Line 19: | ||
* During production builds class files from different projects need to be merged into the single '''ecj.jar''' (this jar file is created as org.eclipse.jdt.core-*-SNAPSHOT-batch-compiler.jar and renamed to ecj.jar afterwards). Search for "batch-compiler" in pom files of the projects mentioned above, to see how the compiler is assembled. | * During production builds class files from different projects need to be merged into the single '''ecj.jar''' (this jar file is created as org.eclipse.jdt.core-*-SNAPSHOT-batch-compiler.jar and renamed to ecj.jar afterwards). Search for "batch-compiler" in pom files of the projects mentioned above, to see how the compiler is assembled. | ||
* Additionally, an ant script exists, org.eclipse.jdt.core/scripts/export-ecj.xml, that should allow manually creating ecj.jar from within Eclipse. This script is also executed when building org.eclipse.jdt.core using PDE/Build, ''probably'' happening also when interactively exporting org.eclipse.jdt.core as a deployable plug-in using the export wizard. | * Additionally, an ant script exists, org.eclipse.jdt.core/scripts/export-ecj.xml, that should allow manually creating ecj.jar from within Eclipse. This script is also executed when building org.eclipse.jdt.core using PDE/Build, ''probably'' happening also when interactively exporting org.eclipse.jdt.core as a deployable plug-in using the export wizard. | ||
+ | |||
+ | ===Project organization since 4.27 M1 === | ||
+ | |||
+ | {{tip|Removed projects:|With the [https://github.com/eclipse-jdt/eclipse.jdt.core/issues/181 ecj project refactoring] the following projects have been removed: | ||
+ | * org.eclipse.jdt.compiler.tool - ''fragment merged into org.eclipse.jdt.core.compiler.batch'' | ||
+ | * org.eclipse.jdt.compiler.apt - ''fragment merged into org.eclipse.jdt.core.compiler.batch'' | ||
+ | * org.eclipse.jdt.core.ecj.validation - ''no longer relevant''}} | ||
+ | |||
+ | |||
+ | The compiler resides in plug-in <code>org.eclipse.jdt.core.compiler.batch</code>. This plug-in has no dependencies and can thus be used in standalone mode. When deployed as a jar file, this plug-in serves as the "batch compiler" and is identical to the file '''ecj.jar'''. | ||
+ | |||
+ | For use in the IDE (or other OSGi-applications), clients may still refer to the compiler via <code>org.eclipse.jdt.core</code> which re-exports all relevant packages from <code>org.eclipse.jdt.core.compiler.batch</code>. | ||
===The ant adapter=== | ===The ant adapter=== | ||
− | For using ecj with ant, '''jdtCompilerAdapter.jar''' is created from <code>org.eclipse.jdt.core | + | For using ecj with ant, '''jdtCompilerAdapter.jar''' is created from selected classes inside <code>org.eclipse.jdt.core.compiler.batch</code>: |
+ | * org.eclipse.jdt.internal.antadapter.* | ||
+ | * org.eclipse.jdt.core.JDTCompilerAdapter | ||
+ | |||
+ | The same class files are also added to ecj.jar. | ||
+ | |||
+ | '''jdtCompilerAdapter.jar''' is created in project <code>org.eclipse.jdt.core</code> (!). The reason for doing this "remotely" (i.e., not in ..compiler.batch) is in the fact that ecj.jar (=org.eclipse.jdt.core.compiler.batch.jar) should ''not'' contain jdtCompilerAdapter.jar, but actually <code>org.eclipse.jdt.core</code> needs it for registration with extension point <code>org.eclipse.ant.core.extraClasspathEntries</code>. | ||
+ | |||
+ | In PDE/Build scenarios (e.g., when interactively exporting the plug-in from the IDE) creation of jdtCompilerAdapter.jar happens by a cooperation between <code>customBuildCallbacks.xml</code> and <code>scripts/export-ecj.jar</code>. During maven builds, this is done ''how?''[[Image:Questionmark.gif]] | ||
+ | |||
+ | Documentation about peculiarities of this jar file are described in some more detail in https://github.com/eclipse-jdt/eclipse.jdt.core/blob/master/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/core/README.md | ||
===Interfacing with other components=== | ===Interfacing with other components=== |
Revision as of 11:09, 5 January 2023
Contents
A Hitchhiker's Guide to ECJ
What IS the Compiler / ECJ?
Strange enough this question does not have a single true answer.
Project organization before 4.27
The following locations contribute to the compiler:
- org.eclipse.jdt.core/compiler
- org.eclipse.jdt.core/batch
- org.eclipse.jdt.compiler.tool
- org.eclipse.jdt.compiler.apt
Since the compiler does not directly correspond to any project / plug-in the following measures are relevant:
- Classes in source folders
compiler
andbatch
are not allowed to access classes in other source folders of org.eclipse.jdt.core. To avoid any violations, a secondary project has been created: org.eclipse.jdt.core.ecj.validation. This project should be imported into the workspace before working on the compiler. It contains only links to the two mentioned source folders and will signal errors, if any class outside this scope is used. The project is not intended for editing. - During production builds class files from different projects need to be merged into the single ecj.jar (this jar file is created as org.eclipse.jdt.core-*-SNAPSHOT-batch-compiler.jar and renamed to ecj.jar afterwards). Search for "batch-compiler" in pom files of the projects mentioned above, to see how the compiler is assembled.
- Additionally, an ant script exists, org.eclipse.jdt.core/scripts/export-ecj.xml, that should allow manually creating ecj.jar from within Eclipse. This script is also executed when building org.eclipse.jdt.core using PDE/Build, probably happening also when interactively exporting org.eclipse.jdt.core as a deployable plug-in using the export wizard.
Project organization since 4.27 M1
The compiler resides in plug-in org.eclipse.jdt.core.compiler.batch
. This plug-in has no dependencies and can thus be used in standalone mode. When deployed as a jar file, this plug-in serves as the "batch compiler" and is identical to the file ecj.jar.
For use in the IDE (or other OSGi-applications), clients may still refer to the compiler via org.eclipse.jdt.core
which re-exports all relevant packages from org.eclipse.jdt.core.compiler.batch
.
The ant adapter
For using ecj with ant, jdtCompilerAdapter.jar is created from selected classes inside org.eclipse.jdt.core.compiler.batch
:
- org.eclipse.jdt.internal.antadapter.*
- org.eclipse.jdt.core.JDTCompilerAdapter
The same class files are also added to ecj.jar.
jdtCompilerAdapter.jar is created in project org.eclipse.jdt.core
(!). The reason for doing this "remotely" (i.e., not in ..compiler.batch) is in the fact that ecj.jar (=org.eclipse.jdt.core.compiler.batch.jar) should not contain jdtCompilerAdapter.jar, but actually org.eclipse.jdt.core
needs it for registration with extension point org.eclipse.ant.core.extraClasspathEntries
.
In PDE/Build scenarios (e.g., when interactively exporting the plug-in from the IDE) creation of jdtCompilerAdapter.jar happens by a cooperation between customBuildCallbacks.xml
and scripts/export-ecj.jar
. During maven builds, this is done how?
Documentation about peculiarities of this jar file are described in some more detail in https://github.com/eclipse-jdt/eclipse.jdt.core/blob/master/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/core/README.md
Interfacing with other components
- Name Environments: To interface with its environment, the compiler needs an instance of
org.eclipse.jdt.internal.compiler.env.INameEnvironment
. During batch compilation, classorg.eclipse.jdt.internal.compiler.batch.FileSystem
is used. But using different implementations of this interface other components like the builder can provide required classes into the compiler. - IBinaryType: Different use cases use different implementations to represent existing .class files to which Java sources being compiled can refer.
- ITypeRequestor: Whenever a new type is found from the name environment, it is first passed to methods of the type requestor. Normally, the
Compiler
itself acts as the type requestor, which will add a representation of the discovered type to the internal data structures of the compiler, but code assist, type hierarchy, search and indexing each have their own implementation of these hooks into the compiler.
Variants of the compiler
The central class is org.eclipse.jdt.internal.compiler.Compiler
, which is used as-is in some use cases, but also a few subclasses exist, which are variants of the compiler, with purposes different from generating .class
files. In other use cases, not the Compiler class, but org.eclipse.jdt.internal.compiler.parser.Parser
is subclassed to achieve different functionality. The latter strategy is used notably for code select and code complete functionality.
Phases
As is standard in compiler technology, ecj operates on Java files in several phases, which are roughly outlined as:
- Scan and parse, i.e., transform a character stream first into a stream of tokens, then into the abstract syntax tree (AST)
- Build and connect type bindings, i.e., overlay the syntactic tree structure (AST) with a semantic graph of bindings.
- Verify methods: analyse inheritance, overriding and overloading of methods
- Resolve: interpret identifiers and link them to the bindings which they represent using various lookups.
- Analyse: perform flow analysis in order to detect errors like variables read before assigned, final variables re-assigned, and also analysis of (potential) null pointers and resource leaks. This phase may also detect a few more errors that need the AST to be fully resolved.
- Generate: Allocate positions to variables (as used in load and store operations of the byte code), then generate the byte code, in the steps shown below. Note that still during code generation some errors may be detected and reported.
- Generate the general class file structure with relevant byte code attributes
- Generate the
Code
attributes containing the actual byte code instructions for methods, constructors and initializers.
Looking at class Compiler
the phases are written slightly differently:
-
beginToCompiler
/internalBeginToCompile
:-
parse
ordietParse
-
buildTypeBindings
-
completeTypeBindings
, here bindings are linked / connected with each other, which requires all bindings to already exist.
-
- optionally:
processAnnotations
-
processCompilationUnits
/process
-- at this point a separate compilation thread may be spawned:ProcessTaskManager
-
getMethodBodies
: the initial parse may have skipped method bodies, parse them now, perhaps only selectively -
faultInTypes
: ensure that all bindings are properly created and initialized -
verifyMethods
-
resolve
-
analyseCode
-
generateCode
-
finalizeProblems
: before errors and warnings are actually reported to the user, they are filtered by any@SuppressWarnings
annotations found in the source.
-
Processing order
- Sequentiel phases vs. demand-driven computations
In addition to the sequential process outlined by the phases above, some computations will be triggered on demand.
Existing tricks to fine-tune order of processing steps:
-
ClassScope.deferredBoundChecks
:
- Inside
ParameterizedQualifiedTypeReference.internalResolveLeafType
we normally perform aboundCheck()
. However, in some situations, notably duringScope.connectTypeVariables()
we may not be ready yet to perform that check. If argumentboundCheck
is true, the check is deferred, adding an element to the listdeferredBoundChecks
, which will be processed viaClassScope.checkParameterizedTypeBounds()
- Similarly
MemberValuePair.resolveTypeExpecting(..)
may add runnables to the same listdeferredBoundChecks
. This accounts for the fact that resolving annotations may happen at particularly unexpected points in time.
-
LookupEnvironment.deferredEnumMethods
:
- During
scanMethodForNullAnnotation()
we want to mark the generated enum methods "valueOf" and "values" as returning a nonnull type. To do so we want to add the configured nonnull annotation, but null annotations may not yet be initialized, in particular because "@NonNullByDefault" depends on an enum, whose "valueOf" and "values" methods should be marked as returning nonnull. To cut this circular dependency, we check if null annotations have been initialized, and if not we add the enum method todeferredEnumMethods
for processing fromusesNullTypeAnnotations()
.
-
LocalDeclaration.duplicateCheckObligation
:
- This paragraph is outdated!
- To handle a specific scoping issue of instanceof pattern variables, the current implementation admits possibly-duplicate variables during
LocalDeclaration.resolve()
, because at that point we don't yet have the necessary flow information. Only later duringanalyseCode
we process that deferredduplicateCheckObligation
.
Note, that to be true to the spec, more situations need flow information right during resolving, so the entire design of separating phases resolve and analyseCode is at stake, see bug 562824 .
Subpages
Compilation phases
Data structures
Java concepts
Some Java concepts pose specific challenges for the compiler