Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Eclipse b3/proposals/Build File Syntax

< Eclipse b3
Revision as of 19:32, 6 October 2009 by Henrik.lindberg.puppet.com (Talk | contribs) (Simple Example)

Introduction

This document contains ideas for an xtext based syntax describing build meta data.

This text includes some explanations of the b3 models as this is needed to understand the proposed syntax.

The idea is that a b3 build file is used as input to b3 commands. Commands in b3 are in practice just two basic commands - either "import" (to populate the workspace), or "invoke" (to get the value (or side effect) of a build unit part (e.g. MyProduct#productZip). (Although, there may be other commands that can be issued that b3 handle, like querying, debugging, dumping, logging, etc. - but in principle, there is just "import" and "invoke").

It is possible to do both import and return a result at the same time. The only difference is that the "import" does not require specification of a particular part, and that the typical use of "invoke" is to perform it against what has already been imported to (or preexisted in) the workspace, thus not requiring a resolver specification other than the default (workspace and target platform).

Context

The b3 build file can be thought of as a configuration of the context in which commands to b3 execute. The b3 context contains:

  • units (the build units in the context)
  • resolutions (the resolution of each required capability)
  • advice (the set of advice to apply in the context)
  • requests (the input request(s), and requests originating from required capabilities)
  • resolver (the network of resolvers to use to resolve requests/requirements).

Each request sets up a new context (it may inherit from the default context), and it is (optionally) configured with a build file.

Requests

Requests, as you have already seen are the original "import statements" and requests made by "required capabilities" in the resolution being performed (as part of an import or an invocation). The requests can be advised, but are otherwise not terribly interesting :). Note that it is possible ro request something in any namespace, but that the resolution will always resolve this into some build unit providing the capability (analogous to how p2 does this).

Resolver

A resolver resolves a request into a resolution. It can handle the entire process of connecting to a repository, reading the information, and translating it to the common build model (i.e. translate it into build units), or it can be a composition of a connector and a meta data translator. Resolvers can be composed into a network of resolvers to facilitate resolution from multiple repositories.

Resolutions

Resolutions are the result of resolving - they tie the resulting build units to the requirements that caused them to appear in the context. Interesting, advanced things can be done with this as shown in the advise section, but mainly this is a mechanism to tie everything together and enable the invocation of actions that traverse the resulting model.

Simple Example

This page describes a lot of syntax. Most of it is used for advanced situations. A typically build file for typical project using eclipse as runtime requires a minimum of specification. So here is a simple example.

requests {
    org.myorg.myproj.root, eclipse.feature;
}
resolver {
    platform:/plugin:/;
    platform:/resource:/;
    p2:http://www.someplace.good/updates-3.5;
    svn:svn+ssh://org.myorg.repo/productx;
}
action buildAll {
    org.eclipse.myproj.site#site.p2;
    org.eclipse.myproj.packaging;
}
advice {
    /requests[name=org.myorg.*]/options {
        source=true;
        mutable=true;
        }
    /resolutions[name=org.eclipse.*]/options {
        location = platform:/plugin:/;
        }
}

This build file specifies that org.eclipse.myproj.root is a wanted eclipse.feature. The latest version is wanted. The resolution takes place by looking at what is available in the target platform and workspace, in a subversion repository and some p2 repositories. All dependencies are resolved, and everything from myorg is requested as mutable source and stored in the workspace, and everything from eclipse ends up in the target platform.

Build File

The build file is a file with the suffix ".b3".

Build file syntax
build: import-statements general-statements ;

import-statements
    :
    | import-statements import
    ;

import: 'import' package-name ;

general-statements
    : 
    | general-statements general-statement
    ;

general-statement
    : resolver-statement;
    | advice-statement;
    | properties-statement;
    | request-statement;
    | action-statement;
    | synchronized-statement;
    ;

Comments

The build file supports the same type of comments as java, '//' for comments to the end of line, and '/*' and '*/' for block comments.

Import Statements

Import statements makes it possible to use short form names in the rest of the build file. Names of resolvers, etc. should be fully qualified to reduce naming collisions. Imports reduces typing, and increases readability. The syntax should be the same as java package imports.

import org.eclipse.b3.resolvers.*; 

Example of import statement

Resolver Statement

A context has a resolver network consisting of resolver nodes. Two special nodes "first" and "best" are used to specify how the search is conducted - resolvers in a "first" list results in the first found resolution, and resolvers in a "best" list results in the resolution with the highest score (compared to the request).

A b3 build file makes it possible to specify the resolution network with a simple syntax.

resolver-statement
    : 'resolver' '{' resolver-config '}'
    ;
resolver-config
    : resolver-class-ref optional-advice ';'
    ;
resolver-class-ref 
    : class-reference   // name of a resolver class
    | uri                          // a uri
    | quoted-uri           // a "uri" - needed when uri contains special characters such as ';'
    ;
optional-advice
    :
    | compound-advice // shown elsewhere
    ;

The specified resolution node itself is a "first" type of node.

import org.eclipse.b3.resolvers.*; // import names of resolvers to reduce typing

resolver {
    TargetResolver;
    WorkspaceResolver;
    }

This specifies that an attempt is made to resolve against the target platform, and if that fails against the Workspace. No options are specified for the resolvers. Resolver class names are used to specify the resolvers to use.

What if I want to resolver first from Target platform, but not for certain tool bundles that, if I have them in the workspace, I would like to get those instead of the installed. This looks like this:

resolver {
    WorkspaceResolver {
        namePattern="org.myorg.mytool.*";
        };
    TargetResolver;
    WorkspaceResolver;
    }

Here is an example using best:

resolver {
    TargetPlatformResolver;
    WorkspaceResolver;
    Best {
        P2Resolver {
            location = "http://www.someplace.good";
            }
        SVNResolver {
            location = "svn+ssh://someplace.else";
            ...;
            };
        }
    }

There are several settable properties in the different resolvers. (Compare to Buckminster's Readers in the RMAP). The examples above shows examples of compound-advice statements.

Short form, using URI schemes

Since the specification of a resolver typically just consists of a location (a URI), the full syntax can be shortened. The use of a URI instead of a class reference specifies both the resolver class and the location. The URI may have to be quoted when it contains special characters (such as ';' or '{' and '}'. Using property expansion in these strings will always require quoting. Note that the short form can be combined with additional advice.

resolver {
  platform:/plugin/xxx;
  platform:/resource://xxx;
  Best {
    p2:http://www.someplace.good;
    svn:svn+ssh://someplace.else;
    "p2:http://www.someplace.good/updates-${eclipse.release}";
    svn:svn+ssh://someplace.different {
         someProperty="someValue";
         }
  };
}

The 'platform' scheme is already recognized by Eclipse giving the host part a very special meaning (plugin basically means TP, resource means workspace, etc. Mapping of URI scheme to resolver class is done using extensions.

Redirection

Redirection is performed by using a redirection resolver (one is provided by b3). The location of this resolver is interpreted as a file or URI references to another build file. And the resolver specified in that build file is used. This solution gets rid of what would otherwise be a special case. The redirect could be registered as a resolver URI scheme with b3 to make the syntax very compact:

redirect:http://someorg.org/somebuildfile.b3;

Advice Statement

Advice sets values in the model in a fashion similar to how CSS is used with HTML. This proposal uses syntax specified by XPath. An XPath is basically a predicate that results in a set of matched nodes. The specified b3 advice acts on this set of nodes.

advice-statement :
    | 'advice' optional-name compound-advice
    | 'advice' 'using' file-or-uri-reference optional-name-list ';'
    ;

optional-name : identifier ;

optional-name-list :
    |
    | '{' name-list '}'
    ;

name-list :
    | identifier 
    | name-list ',' identifier 
    ;

compound-advice :
    | '{' advice-list '}'
    ;

advice-list :
    | 
    | advice-list advice;
    ;

advice :
    | query '=' advice-expr ';'
    | query compound-advice
    ;

query :
    | XPATHEXPR
    ;

advice-expr
    : numericValue
    | booleanValue
    | 'new' classname optional-advice
    | string-expr
    ;

string-expr 
    : quoted-string
    | identifier '(' arguments ')'  // i.e. string functions - format, toUpper, toLower, split, replace, propertyRef,...
    ;

arguments
    : 
    | string-expr
    | arguments ',' string-expr
    ;

optional-advice :
    | 
    | compound-advice;
    ;

The query is relative to the context - which is analogous to the "document" in XML/HTML. At the root of the context there are:

  • units (the build units in the context)
  • resolutions (the resolution of each required capability)
  • advice (the set of advice to apply in the context)
  • requests (the input request(s), and requests originating from required capabilities)
  • resolver (the network of resolvers to use to resolve requests/requirements).

It is unclear if it should be allowed to apply advice on all of these (i.e. advice on advice). The idea is to primarily advice on requests, and resolutions, and in some cases on units.

So here are some examples:

/requests[name=org.myorg.projx.*]/options/source = true;

All requests to org.myorg.projx.* should be requests for source

/requests[name=org.myorg.projx.*]/options/mutable = true;

All requests to org.myorg.projx.* should be requests for mutable

/requests[name=org.myorg.projx.*]/options {
    source = true;
    mutable = true;
    }

shorthand for the two pieces of advice above

/requests[name=org.myorg.projx.*, namespace="eclipse.feature"]/options/ {
    source = true;
    mutable = true;
    }

if I only need mutable source for the eclipse.features

Options include: source, mutable, branchTagPath, timestamp, revision, resolverFilter, filterGroups, overlayPath, includeParts, excludeParts, and prune.

Examples - Advice on resolutions

Place all of my organization's "core" components in the target platform (the rest goes to workspace by default).

/resolutions[name=org.myorg.core.*] /options {
    materializer = org.eclipse.b3.targetPlatformMaterializer;
    location = /where/it/should/be/stored;
    conflictResolution = update;
}

It is possible to make very advanced advice:

/resolutions/[name=org.myorg.*]/subResolutions/resolution[namespace="org.myorg.annotations"]/options {
    materializer = org.eclipse.b3.workspaceMaterialzer;
    location = /where/they/are/stored;
}

This would store all build units that provide a capability in the namespace org.myorg.annotations in a particular location on disk and link them to the workspace.

Examples - Advice on units

The advice on units is applied as units are resolved. This means that the advice has an effect during the contexts lifetime, but as soon as the operation is over, the live model will reflect the original meta data translation.

The same advice can be applied again in some other context if needed. Sometimes all that is required is to override some faulty meta data (that the user wants to correct) to make resolution possible. As an example, we want to override a components request for a particular required capability. Once the components has been resolved it will probably not build, the problem is corrected by the user and the unit checked in. The next time, the overriding advice is not needed.

/units[name="org.myorg.projx.broken"]/requiredCapabilities[name="org.myorg.x"]/versionRange = [1.0.0, 3.0.0];

Examples - Creation of complex types

Most of the time values are simply strings, or numbers, or there is a simple translation from string form to the datatype required for a settable attribute. Sometimes this is not possible, and a "new instance operator" is required. This looks like this;

    path-expr = new RequiredCapability {
        name = ...;
        nameSpace = ...;
        ...
    };

Examples - Advising Units

As the advice mechanism is very powerful it is actually possible to create build units directly in the b3 build file. This is useful in some cases where a virtual/configuration unit is required to define the order of dependencies. In many cases, creating an additional bundle or feature that is only used for some special aspect of building/testing can be an overkill - so it is quite convenient to declare it directly in the b3 build file.

Apart from this, the build units are mostly the result of an import, and they may be advised (as described earlier).

Examples - Naming Advice

Typically, a build file contains a single section of unnamed advice, but it is possible to specify reusable named parts. Within a single build file named advice is simply concatenated. The following two examples are equivalent:

advice a {
    something = 10;
}
advice b {
    somethingElse = 20;
}
advice {
    something = 10;
    somethingElse = 20;
}

Naming is however of value when including advice from an external build file.

Examples - Including Advice

It is possible to include advice from other b3 files.

advice http://someplace.org/somebuildfile.b3 { fixOfBrokenX, fixOfBrokenY }

This example shows the inclusion of advice from the build file appointed by the URI. The advice in the appointed file named fixOfBrokenX, and fixOfBrokenY is used as advice. If no list of named advice is used, then all of the advice in the appointed build file is used.

Since a build file may contain unnamed advice, and have more than one unnamed advice statement, it is not possible to refer to a single unnamed advice - they are either all used, or none. Special syntax is needed for this case - for instance @unnamed, used as follows:

advice http://someplace.org/somebuildfile.b3 { @unnamed, fixOfBrokenY }
 

Which specifies used of all unnamed advice statements, and the named fixOfBrokenY.

Alternative: all unnamed advice are simply named "advice", and names does not have to be unique.

Synthetic nodes

It must be possible to address synthetic nodes in the document such as the "before first" and "after last" positions in a list. This is done by suffixing the xpath expression with ":synthetic node name"

e.g.

/units[name=org.myorg.foo]/requiredCapabilities:first = new RequiredCapability { ... }.

Lists (TBD)

It must be possible to reference the list itself. Clear the list. Remove an entry in the list, add an element first or last, and before or after an element. (Have to look more at XPath to see how this is typically done to give examples).

Properties Statement

The build file can define properties, set their values, and declare if they are imutable.

properties-statement
    : 'properties' '{' property-list '}'
    | 'properties' 'using' file-or-uri-reference optional-name-list ';' // see advice-statement for definitions
    ;

property-list 
    :
    | property-spec ';'
    | property-list  property-spec
    ;

property-spec
    : optional-immutable identifier '=' string-expr
    ;

optional-immutable
    : 
    | 'immutable'
    ;

The properties set this way are set in the context's initial property scope. The 'immutable' keyword will flag redefinition of the property as an error.

Examples:

properties {
    useBuild="NBUILD";
    immutable org.myorg.something.special="true";
    immutable myName = "Fred";
}

Property-statements can be named for reuse purposes. The same semantics as for advice applies.

Requests Statement

A requests statement lists the capabilities that will be resolved if the "import" action is performed on the b3 file.

requests-statement
    : 'requests' optional-name '{' requests '}'
    ;

requests
    :
    | request ';'
    | requests request
    ;

request
    ; unit-request optional-options
    ;

unit-request
    : unit-name ',' namespace 
    | unit-name ',' namespace ',' version-range
    ;

optional-options
    :
    | compound-advice
    ;

Examples:

requests {
    org.eclipse.b3.core, osgi.bundle;
    org.someorg.something, eclipse.feature, [0.0.0,3.0.0);
}

A request for a bundle and a feature

The optional-options, is advice on the request options. Here is an example

requests {
    org.eclipse.b3.core, osgi.bundle {
        source=true;
        mutable=true;
   };
}

a request with advice on options

The optional-name makes it possible to specify different import requests for different purposes without having to create multiple b3 files, just to have a small variation. As an example, someone working on extensions that does not need source vs. a committer. When used from a UI, the UI can list the differently named requests/imports. If invoked from a command line, and there are several, an error is issued and the user would need to specify which one to pick.

Action Statement

Action statements expose actions provided by build units. They are important as they exists prior to having resolve the build untis (i.e. before a build unit's meta data has been translated). It is thus possible to expose invokable actions directly in the UI even if only the build file is available.

action-statement
    : 'action' identifier '{' invocation-list '}'
    ;

invocation-list
    : 
    | invocation ';'
    | invocation-list invocation
    ;

invocation
    : identifier
    | identifier '#' identifier
    ;

An action simply specifies a list of build unit parts to execute in sequence. It is possible to a specific part, or the build unit itself (which is a shorthand notation for all of its parts).

Examples:

action buildAll {
    org.myproduct.productSite#site.p2;
    org.myproduct.packaging;
}

This makes it possible to invoke the "buildAll" action on the b3 file without first having to run an import. If the required units are not already available, they will be imported.

Synchronized Statement

The synchronized statement will prevent parallel execution of actions. (Is not likely to be implemented in the first release of b3, but shown for completeness). It is also possible to specify or advice actors, actions, units and build parts that they are synchronized. The general syncronized statement allows syncronization to be specified for disjunct instances and is useful when a composition of units contains actions that would otherwise create conflicts - e.g. producing temporary output to the same directory.

Synchronized statement
synchronized-statement :
    |  'syncronized' '{' sync-expr-list '}'
    ;

sync-expr-list :
    |
    | sync-expr-list sync-expr
    ;

sync-expr :
    | unit-part-ref ',' unit-part-ref
    | sync-expr ',' unit-part-ref
    ;

unit-part-ref :
    | unitname-pattern
    | unitname-pattern '#' partname-pattern
    ;

Examples

synchronize {
    foo#actionA, bar#actionB;
    fee#actionC, fum#actionD;
    fum#actionA, fee#actionB, fee#actionX;
    org.myorg.*, org.yourorg.fum.*;
    org.myorg.foo#binary.*;
} 

The examples shows a specification of 5 groups where parallel execution of the stated units/parts in each groups is prevented (i.e. execution of these are serialized with respect to each other).

TODO: Synchronized-statements should also have the possibility to be named and reference statements in a referenced build file.

Back to the top