CDT/Build/Doug

Doug's Proposal for a new CDT Build System

Overview

• Use org.eclipse.core.resources IBuildConfiguration to model build configs
  • Allows for config specific references and references to configs for finer grain build dependencies.
• Build configurations are bags of attributes
  • Adapt from IBuildConfiguration to ICBuildConfiguration which implements the bag
  • Bag stored per configuration as project properties (i.e. in .settings).
• Builders implemented as real IncrementalProjectBuilders
  • Attribute in ICBuildConfiguration to know whether a given builder is enabled for a given config.
    • Allows for different builders to work on the same project, e.g. CMake and Xcode builders
  • Project Property page for editing builder settings that are stored in the attribute bag.
  • Builders can have a role in scanner discovery.
    • They know the include paths and macros used to generate the makefiles and some have the ability to report what they are.
• Build configurations also have Toolchain
  • Contributes to build environment vars, e.g. PATH to find the tools to execute and other necessary env vars.
  • Scanner Discovery settings to find built-ins and parse build output
  • Error parsers to create error markers for compile and link errors
  • Binary parser to detect whether files are binaries and can be launched
  • Toolchains can be defined in extension, or in toolchain files (local or shared a la .launch files).
• Relationship with Launch Configurations
  • One of the key tenants of the Launch Bar is that we should be able to determine what tools to use to build for and launch a given launch configuration.
  • Toolchains define a target os/arch for this purpose
  • Toolchains also include all tools needed for a given launch config, including debuggers, code coverage tools, and other run-time analysis tools.
  • As such, a given launch configuration (and launch mode?) map to a given build configuration.
• Anticipated Builders
  • CMake
  • qmake
  • autotools
  • plain make, or whatever build command the user wants
  • Arduino builder that generates makefiles based on Arduino SDK metadata
  • CDT generated makefiles (backwards compat, deprecated?)
  • CDT internal (backwards compat, deprecated?)
• Backwards compat issues
  • How to map IBuildConfiguration to IConfiguration
    • Use the default IBuildConfiguration that exists already to signal using old configs?
      • Actually, that would let us build the new system without breaking old ones
    • Project conversion?
      • Pretty ugly on startup. Would need to make sure things aren't too broken.
      • Prompt the user to convert?

Design

From the experiment carried out with the minimal project as discussed in the Notes below, it should be possible to introduce a new build system that is totally independent of the old one. That would allow us to support both project types.

Essentially all we need to do for new projects is to not create the ICProjectDescriptor or any ICConfigurationDescriptors. Nor do we register anything in the IProjectDescriptor other than the cnature and the ccnature for C++ projects. We need to create our own builder and have the scanner info provider registered against it and we're good to go. And we need to create any IBuildConfiguration's we need.

The rest of this section discusses the design of the new things. (Note: work in progress, more details will be continually added)

Build Configurations

• build config settings stored in Project scoped preferences
• per resource config settings would allow for things like file/folder exclusions per config.

Builder

• provides an IncrementalProjectBuilder for the project. Full control over what happens at build time.
• expect a different builder for each build system.
  • For example, I am starting with the Arduino builder which generates Makefiles from the board metadata we get from Arduino, adds in the list of source files for the given build config (allowing for per config exclusions) to the Makefile. Then calls make.
  • I'd expect CMake and qmake and a standard "make" to each have their own builders.
• probably need a super class to do common things such as get the build console which would need the list of error parsers (see section on build console).
• multiple builders can be supported by a single project if the builders have 'enabled' config settings. They can then ignore builds when the config setting for the current config is disabled.
  • I've seen a few projects that have CMake files and regular Makefiles and the user picks based on their current environment and sometimes toolchain.

Build Console

• Going to take the opportunity and experiment with the pattern matching supported by the TextConsole.
• The builder fetches the console and tells it what error parsers it wants to use and the process to grab output from.
• The Error parser provides the info needed by IPatternMatchListener methods. It also provides a method to take the matching text from the console and produce a IMarker for it. Error parsers need to make sure they don't produce more than one marker for the same error.
• The console is provided through a service to the builder. The console is in the UI plug-in and the builder is in the Core plug-in.
• The console provider takes care of creating the IConsole, likely a MessageConsole but maybe just an IOConsole. When requested to monitor a build, it creates the IPatternMatchListeners for the requested ErrorParsers and adds them to the console and sets up the output and error streams to output to the Console. When the build is finished, the pattern match listeners are removed. (We may want to find a way to make that more optimal so listeners aren't created when being reused).

Scanner Info Provider

• Today, we can register an IScannerInfoProvider against a project builder. That overrides the CDT default.
• Optimally, scanner info providers can do different things for different builders. Some builders can provide scanner info themselves. Others require build output scanners to figure things out.
• Output scanners are probably common enough we would want a framework for them. That way GCC output scanners can be reused by different builders.

New Project Wizard

• Inherent in a new build system is the need for a new new project wizard. The current one assumes the old build system.
• Keep it simple, template based.
• Template tags allow us to build filters.
• Templates generate source files and set up the default build settings.
• Totally free form. Templates can use whatever generators they want (e.g. Freemarker used currently by the Arduino projects).
• Templates also provide wizard pages to gather the data they need. We can use the wizard transition to go from the common wizard to the template wizard. This is done in a few places, for example, the Launch Bar new configuration wizard. The Template wizard can then also be registered on it's own if desirable.
  • Common pages provided for common things, like the main page, language selection, etc, but they are optional.
  • Extension point to provide templates with descriptions, icons, etc. Don't want to load the template plug-in unless the user chooses to use the template.
• Ensure we separate UI and Core so that we can programatically create projects from the core.

Notes

Qt qmake

Doing qt just as I had done Arduino. The big difference with Qt is that I'm force into considering different types of toolchains. For Arduino, everything is gcc based. For Qt, we also need to support Visual C++. And the QNX toolchain is slightly different from the GCC one, or at least it's wrapper is. Also note the Clang is so compatible with GCC, it might as well be serviced with GCC.

This leads to my first considerations for the new build model, as simple as I can make it.

• IBuildConfiguration is at the root. It's very simple, just the project and a name and provided by org.eclipse.core.resources.
• CBuildConfiguration abstract class associates with IBuildConfiguration and adds support for storing settings in the Project-scoped preferences per config.
• QtBuildConfiguration, ArduinoBuildConfiguration, etc. are concrete subclasses of CBuildConfiguration and are the adapters from IBuildConfiguration. Each has it's own IncrementalProjectBuilder and nature that manage how the project is built. The builder is the one that does the adapting and most build functionality is provided by this class.
• CToolchain is the abstract parent class for toolchains. Toolchains are adapted from CBuildConfiguration and store any per config settings there. There is one toolchain per config. These objects provide things like scanner info, error parsers, etc.
• GccToolchain, VisualCppToolchain, etc. implement the IToolChain interface to provide per toolchain functionality. There is an Extension that maps toolchain id, which is stored in the config settings, with the adapter class.
• Each build system provides it's own mechanism to figure out the toolchain for a config. Either it asks the user, or provides an automated system. For example Qt can map the QMAKE_XSPEC to a toolchain id, e.g. macx-clang to the gcc toolchain.

Note that the C in the class names mean CDT to help differentiate from similar classes in the Platform.

Right now the assumption is that there is only one build system per project, i.e. Qt, Arduino, CMake, etc. The new project generator sets that up. Adding build systems to a project could be possible but will involve figuring out how to enable/disable builders per config.

Also, the assumption is that Toolchains are independent of the build system. Hopefully that's true. If not we'll have to introduce another adapter layer to adapt Toolchains to build system specific Toolchains.

Minimal project

Assuming we want to start from scratch (and then add back in the old stuff with a backwards compat layer), I'm trying the Arduino Projects and see if I can make them work. It's a great test bed and that's what it's there for.

Here are issues I've uncovered and things that seem to work

• Project only contains the C and C++ nature and the Arduino nature which has a project builder.
• How far can we get without using the ICProjectDescriptor? All that information should be in persistent properties.
• Ran into a check in the AbstractPage class that disables property pages that inherit from it if the project doesn't have a project descriptor, or any configurations (isCDTPrj()).
• All other pages worked and the indexer started up.
• Overriding isCDTPrj to return true if the project has the C nature and adding checks to get rid of NPEs, we come up in a pretty sane state.
• Except for the Preprocessor Include page. It's organized by configuration. It would have to change to map to core.resources build config, or we need to keep the list of ICConfigurationDescriptors in sync with them.
• To start, I'll set up a sync so we can have both.
• IConsoleParser gives us the hook to add build output parsers for error parsing and language settings providers.
  • The abstract build output parsers should probably move to the CDT core, the error parsers are already there.
• Why do we have our own problem marker framework? Why don't the error parsers directly create the markers?
  • Only advantage I can see is that the framework checks for duplicates, which happens a lot with shared header files.
  • But that could be done with a shared method in the error parser parent class.
• Why do we have the whole descriptor framework (.cproject)?
  • What's so bad about persistent properties in the IResource framework? (update: this is mixed up, see below).
  • You can do hierarchical properties with property key's. Doing that for Arduino's property files.
• Why does CDT have it's own BuildConsole. Why not use the TextConsole?
  • It gives us pattern matching and hyperlinking for free.
  • Or is it too slow?
• Language Settings Provider
  • It was a good effort but the complexity of the Language Settings Provider is pretty scary.
    • The layers are pretty deep
  • It really depends on the legacy PathEntry framework and the ICProjectDescriptor framework
  • Can we build something simpler, that doesn't depend on either?
  • As far a I can tell, this is the only dependency left on the project descriptors which we could now get rid of.
• Build Config settings
  • I had the IResource persistent properties mixed up. They are stored across restarts but not shared in the project.
  • What I really want is ProjectScope preferences which are the things stored in the .settings folder.
  • Need to figure out how to do resource-level settings, exclusions in particular.
• The ability to set up source and output roots is missing in this set up.
  • For Arduino, doing it programmatically since I own the project structure.
  • Might need to clone the existing one which seems very tied to the old configurations
  • Probably need a new one or bring the very old one back
• Lots of fun doing the scanner discovery which is now complete for Arduino.
  • One issue is selecting which build config to use to gather the scanner info.
  • Caching the built-in calculation is important. The scanner info provider is called by the CReconciler!
    • For Arduino, we only have project level settings so caching it in the arduino build config adapter.
  • BTW, there's no build output scanning for Arduino. Everything is based on the selected board type.
    • Which makes scanner info available even without doing a build, the cat's meow!
• Integration with launch
  • The Arduino metadata files define the upload tool as well as the build tools.
  • The Arduino remote connection has attributes for the Arduino board.
  • So does the Arduino build configuration. Find the match and get the upload command.
  • And where the match doesn't exist create a new build configuration for it.
  • Starts to make me wonder how visible we need build configurations.
    • This may all change in a more free form project configuration.
• Error parsing/hyperlinks
  • Got this working using the TextConsole pattern listeners
  • The Matcher job lets the hyperlink/marker creation happen off the critical path of getting build output to the console.
  • Some of problems though
    • Not efficient when dealing with multiple toolchains
      • Need to add and remove listeners before each build
      • The pattern's aren't precompiled
    • Pattern match listener interface is in UI, error parsers are usually core things
      • Hyperlink is UI but the marker generator is core
    • The matcher seems to follow the build by quite a lot, i.e. noticeable delay before hyperlinks show up
  • Are these bad enough problems to warrant our own console
    • If we do our own, follow this architecture
    • But provide mechanisms to alert the console when a build is over
    • And precompile the patterns and clean up the API