View source for Eclipse b3/proposals/Build File Syntax

[[Category:Eclipse b3]]
This information is dated. Please visit the b3 website for the latest information http://www.eclipse.org/modeling/emft/b3/
==Introduction==
This document contains ideas for an xtext based syntax describing build meta data. 

This text includes some explanations of the b3 models as this is needed to understand the proposed syntax.

The idea is that a b3 build file is used as input to b3 commands. Commands in b3  are in practice just two basic commands - either "import" (to populate the workspace), or "invoke" (to get the value (or side effect) of a build unit part  (e.g. <code>MyProduct#productZip</code>). (Although, there may be other commands that can be issued that b3 handle, like querying, debugging, dumping, logging, etc.  - but in principle, there is just "import" and "invoke").

It is possible to do both import and return a result at the same time. The only difference is that the "import" does not require specification of a particular part, and that the typical use of "invoke" is to perform it against what has already been imported to (or preexisted in) the workspace, thus not requiring a resolver specification other than the default (workspace and target platform).

===Context===
The b3 build file can be thought of as a configuration of the context in which commands to b3 execute. The b3 context contains:
* units (the build units in the context)
* resolutions (the resolution of each required capability)
* advice (the set of advice to apply in the context)
* requests (the input request(s), and requests originating from required capabilities)
* resolver (the network of resolvers to use to resolve requests/requirements).

Each request sets up a new context (it may inherit from the default context), and it is (optionally) configured with a build file.

===Requests===
Requests, as you have already seen are the original "import statements" and requests made by "required capabilities" in the resolution being performed (as part of an import or an invocation). The requests can be advised, but are otherwise not terribly interesting :). Note that it is possible ro request something in any namespace, but that the resolution will always resolve this into some build unit providing the capability (analogous to how p2 does this).

===Resolver===
A resolver resolves a request into a resolution. It can handle the entire process of connecting to a repository, reading the information, and translating it to the common build model (i.e. translate it into build units), or it can be a composition of a connector and a meta data translator. Resolvers can be composed into a network of resolvers to facilitate resolution from multiple repositories.

As this is an internal/implementation detail not everyone needs to be aware of we refer to the declaration in the build file as "repositories" - the fact that stating that access is wanted to a repository leads to a resolver for that repository being created is not a primary user concern (it makes sense later when doing advanced things though - the repository itself and the resolver are after all two different things).

===Resolutions===
Resolutions are the result of resolving - they tie the resulting build units to the requirements that caused them to appear in the context. Interesting, advanced things can be done with this as shown in the advise section, but mainly this is a mechanism to tie everything together and enable the invocation of actions that traverse the resulting model.

==Simple Example==
This page describes a lot of syntax. Most of it is used for advanced situations. A typically build file for typical project using eclipse as runtime requires a minimum of specification. So here is a simple example.

<pre>
requests {
    org.myorg.myproj.root, eclipse.feature;
}
repositories {
    platform:/plugin/;
    platform:/resource/;
    p2:http://www.someplace.good/updates-3.5;
    svn:svn+ssh://org.myorg.repo/productx;
}
action buildAll {
    org.eclipse.myproj.site#site.p2;
    org.eclipse.myproj.packaging;
}
advice {
    /requests[name=org.myorg.*]/options {
        source=true;
        mutable=true;
        }
    /resolutions[name=org.eclipse.*]/options {
        location = platform:/plugin/;
        }
}
</pre>
This build file specifies that org.eclipse.myproj.root is a wanted eclipse.feature. The latest version is wanted. The resolution takes place by looking at what is available in the target platform and workspace, in a subversion repository and some p2 repositories. All dependencies are resolved, and everything from myorg is requested as mutable source and stored in the workspace, and everything from eclipse ends up in the target platform.

==Build File==
The build file is a file with the suffix ".b3".

A build file is used to describe top-level invokable actions such as "import", or a particular build result. It is also used to describe embedded advice in units. 

{| border="1" cellborder="0" cellspacing="0"
|-
| ''Build file syntax''
|-
|<pre>
build: import-statements build-statements ;

import-statements
    :
    | import-statements import
    ;

import: 'import' package-name ;

build-statements
    : unit-statement
    | general-statements
    ;

general-statements
    : 
    | general-statements general-statement
    ;

general-statement
    : repositories-statement
    | advice-statement
    | properties-statement
    | request-statement
    | action-statement
    | synchronized-statement
    | include-statement
    ;

</pre>
|}
===Comments===
The build file supports the same type of comments as java, '//' for comments to the end of line, and '/*' and '*/' for block comments.

==Import Statements==
Import statements makes it possible to use short form names in the rest of the build file. Names of resolvers, etc. should be fully qualified to reduce naming collisions. Imports reduces typing, and increases readability. The syntax should be the same as java package imports. 

<pre>
import org.eclipse.b3.resolvers.*; 
</pre>
''Example of import statement''

==Repositories Statement==
A context has a resolver network consisting of resolver nodes. Two special nodes "first" and "best" are used to specify how the search is conducted - resolvers in a "first" list results in the first found resolution, and resolvers in a "best" list results in the resolution with the highest score (compared to the request).

A b3 build file makes it possible to specify the resolution network with a simple syntax.
<pre>
repositories-statement
    : 'repositories' '{' repo-config '}'
    ;
repo-config
    : resolver-class-ref optional-advice ';'
    ;
resolver-class-ref 
    : class-reference   // name of a resolver class
    | uri                          // a uri
    | quoted-uri           // a "uri" - needed when uri contains special characters such as ';'
    ;
optional-advice
    :
    | compound-advice // shown elsewhere
    ;
</pre>

The specified repositories node itself is a "first" type of node.

<pre>
import org.eclipse.b3.resolvers.*; // import names of resolvers to reduce typing

repositories {
    TargetResolver;
    WorkspaceResolver;
    }
</pre>
This specifies that an attempt is made to resolve against the target platform, and if that fails against the Workspace. No options are specified for the resolvers. Resolver class names are used to specify the resolvers to use.

What if I want to resolve first from Target platform, but not for certain tool bundles that, if I have them in the workspace, I would like to get those instead of the installed. This looks like this:

<pre>
repositories {
    WorkspaceResolver {
        namePattern="org.myorg.mytool.*";
        };
    TargetResolver;
    WorkspaceResolver;
    }
</pre>
Here is an example using best:

<pre>
repositories {
    TargetPlatformResolver;
    WorkspaceResolver;
    Best {
        P2Resolver {
            location = "http://www.someplace.good";
            }
        SVNResolver {
            location = "svn+ssh://someplace.else";
            ...;
            };
        }
    }
</pre>
There are several settable properties in the different resolvers. (Compare to Buckminster's Readers in the RMAP). The examples above shows examples of compound-advice statements.

====Short form, using URI schemes====
Since the specification of a resolver typically just consists of a location (a URI), the full syntax can be shortened. The use of a URI instead of a class reference specifies both the resolver class and the location.
The URI may have to be quoted when it contains special characters (such as ';' or '{' and '}'. Using property expansion in these strings will always require quoting. Note that the short form can be combined with additional advice.

<pre>
repositories {
  platform:/plugin/xxx;
  platform:/resource://xxx;
  Best {
    p2:http://www.someplace.good;
    svn:svn+ssh://someplace.else;
    "p2:http://www.someplace.good/updates-${eclipse.release}";
    svn:svn+ssh://someplace.different {
         someProperty="someValue";
         }
  };
}
</pre>
The 'platform' scheme is already recognized by Eclipse giving the host part a very special meaning (plugin basically means TP, resource means workspace, etc.
Mapping of URI scheme to resolver class is done using extensions. 

====Redirection====
Redirection is performed by using a redirection resolver (one is provided by b3). The location of this resolver is interpreted as a file or URI references to another build file. And the resolver specified in that build file is used. This solution gets rid of what would otherwise be a special case. The redirect could be registered as a resolver URI scheme with b3 to make the syntax very compact:
<pre>
redirect:http://someorg.org/somebuildfile.b3;
</pre>

==Advice Statement==
Advice sets values in the model in a fashion similar to how CSS is used with HTML. This proposal uses syntax specified by XPath. An XPath is basically a predicate that results in a set of matched nodes. The specified b3 advice acts on this set of nodes.

<pre>
advice-statement :
    | 'advice' optional-name compound-advice
    ;

optional-name 
    :
    | identifier
    ;

compound-advice :
    | '{' advice-list '}'
    ;

advice-list :
    | 
    | advice
    | advice-list advice
    ;

advice :
    | query '=' advice-expr ';'
    | query compound-advice
    ;

query :
    | XPATHEXPR
    ;

advice-expr
    : numericValue
    | booleanValue
    | 'new' classname optional-advice
    | string-expr
    ;

string-expr 
    : quoted-string
    | identifier '(' arguments ')'  // i.e. string functions - format, toUpper, toLower, split, replace, propertyRef,...
    ;

arguments
    : 
    | string-expr
    | arguments ',' string-expr
    ;

optional-advice :
    | 
    | compound-advice;
    ;
</pre>

The query is relative to the context - which is analogous to the "document" in XML/HTML.
At the root of the context there are:
*units (the build units in the context)
* resolutions (the resolution of each required capability)
* advice (the set of advice to apply in the context)
* requests (the input request(s), and requests originating from required capabilities)
* resolver (the network of resolvers to use to resolve requests/requirements).

It is unclear if it should be allowed to apply advice on all of these (i.e. advice on advice).  The idea is to primarily advice on requests, and resolutions, and in some cases on units.

So here are some examples:
<pre>
/requests[name=org.myorg.projx.*]/options/source = true;
</pre>
''All requests to org.myorg.projx.* should be requests for source''

<pre>
/requests[name=org.myorg.projx.*]/options/mutable = true;
</pre>
''All requests to org.myorg.projx.* should be requests for mutable''

<pre>
/requests[name=org.myorg.projx.*]/options {
    source = true;
    mutable = true;
    }
</pre>
''shorthand for the two pieces of advice above''

<pre>
/requests[name=org.myorg.projx.*, namespace="eclipse.feature"]/options/ {
    source = true;
    mutable = true;
    }
</pre>
''if I only need mutable source for the eclipse.features''

Options include: source, mutable, branchTagPath, timestamp, revision, resolverFilter, filterGroups, overlayPath, includeParts, excludeParts, and prune.

===Examples - Advice on resolutions===
''Place all of my organization's "core" components in the target platform (the rest goes to workspace by default).''
<pre>
/resolutions[name=org.myorg.core.*] /options {
    materializer = org.eclipse.b3.targetPlatformMaterializer;
    location = /where/it/should/be/stored;
    conflictResolution = update;
}
</pre>

''It is possible to make very advanced advice:''
<pre>
/resolutions/[name=org.myorg.*]/subResolutions/resolution[namespace="org.myorg.annotations"]/options {
    materializer = org.eclipse.b3.workspaceMaterialzer;
    location = /where/they/are/stored;
}
</pre>
This would store all build units that provide a capability in the namespace org.myorg.annotations in a particular location on disk and link them to the workspace.

As a shorthand notation, since resolvers can use uri schemes to pick a resolver and a location - the same should be possible when setting a materializer and location. An example is shown in the "Simple Example" at the top of the page.

====Examples - Advice on units====
The advice on units is applied as units are resolved. This means that the advice has an effect during the contexts lifetime, but as soon as the operation is over, the live model will reflect the original meta data translation.

The same advice can be applied again in some other context if needed. Sometimes all that is required is to override some faulty meta data (that the user wants to correct) to make resolution possible. As an example, we want to override a components request for a particular required capability. Once the components has been resolved it will probably not build, the problem is corrected by the user and the unit checked in. The next time, the overriding advice is not needed.

<pre>
/units[name="org.myorg.projx.broken"]/requiredCapabilities[name="org.myorg.x"]/versionRange = [1.0.0, 3.0.0];
</pre>

====Examples - Creation of complex types====
Most of the time values are simply strings, or numbers, or there is a simple translation from string form to the datatype required for a settable attribute. Sometimes this is not possible, and a "new instance operator" is required. This looks like this;
<pre>
    path-expr = new RequiredCapability {
        name = ...;
        nameSpace = ...;
        ...
    };
</pre>

====Examples - Advising Units====
As the advice mechanism is very powerful it is actually possible to create build units directly in the b3 build file. This is useful in some cases where a virtual/configuration unit is required to define the order of dependencies. In many cases, creating an additional bundle or feature that is only used for some special aspect of building/testing can be an overkill - so it is quite convenient to declare it directly in the b3 build file.

Apart from this, the build units are mostly the result of an import, and they may be advised (as described earlier).

====Examples - Naming Advice====
Typically, a build file contains a single section of unnamed advice, but it is possible to specify reusable named parts. Within a single build file named advice is simply concatenated. The following two examples are equivalent:
<pre>
advice a {
    something = 10;
}
advice b {
    somethingElse = 20;
}
</pre>
<pre>
advice {
    something = 10;
    somethingElse = 20;
}
</pre>

Naming is however of value when including advice from an external build file.

Additional rules:
* Unnamed advice is implicitly named "advice"
* Advice must have a unique name (within a file





===Synthetic nodes===
It must be possible to address synthetic nodes in the document such as the "before first" and "after last" positions in a list. This is done by suffixing the xpath expression with ":synthetic node name"

e.g. 
<pre>
/units[name=org.myorg.foo]/requiredCapabilities:first = new RequiredCapability { ... }.
</pre>

====Lists (TBD) ====
It must be possible to reference the list itself. Clear the list. Remove an entry in the list, add an element first or last, and before or after an element. (Have to look more at XPath to see how this is typically done to give examples).

==Properties Statement==
The build file can define properties, set their values, and declare if they are imutable.

<pre>
properties-statement
    : 'properties' '{' property-list '}'
    ;

property-list 
    :
    | property-spec ';'
    | property-list  property-spec
    ;

property-spec
    : optional-immutable identifier '=' string-expr
    ;

optional-immutable
    : 
    | 'immutable'
    ;
</pre>

The properties set this way are set in the context's initial property scope. The 'immutable' keyword will flag redefinition of the property as an error.

Examples:
<pre>
properties {
    useBuild="NBUILD";
    immutable org.myorg.something.special="true";
    immutable myName = "Fred";
}
</pre>

Property-statements can be named for reuse purposes. The same semantics as for advice applies.

==Requests Statement==
A requests statement lists the capabilities that will be resolved if the "import" action is performed on the b3 file.

<pre>
requests-statement
    : 'requests' optional-name '{' requests '}'
    ;

requests
    :
    | request ';'
    | requests request
    ;

request
    ; unit-request optional-options
    ;

unit-request
    : unit-name ',' namespace 
    | unit-name ',' namespace ',' version-range
    ;

optional-options
    :
    | compound-advice
    ;
</pre>

Examples:
<pre>
requests {
    org.eclipse.b3.core, osgi.bundle;
    org.someorg.something, eclipse.feature, [0.0.0,3.0.0);
}
</pre>
''A request for a bundle and a feature''

The optional-options, is advice on the request options. Here is an example
<pre>
requests {
    org.eclipse.b3.core, osgi.bundle {
        source=true;
        mutable=true;
   };
}
</pre>
''a request with advice on options''

The optional-name makes it possible to specify different import requests for different purposes without having to create multiple b3 files, just to have a small variation. As an example, someone working on extensions that does not need source vs. a committer. When used from a UI, the UI can list the differently named requests/imports. If invoked from a command line, and there are several, an error is issued and the user would need to specify which one to pick.

==Action Statement==
Action statements expose actions provided by build units. They are important as they exists prior to having resolve the build untis (i.e. before a build unit's meta data has been translated). It is thus possible to expose invokable actions directly in the UI even if only the build file is available.

<pre>
action-statement
    : 'action' identifier '{' invocation-list '}'
    ;

invocation-list
    : 
    | invocation ';'
    | invocation-list invocation
    ;

invocation
    : identifier
    | identifier '#' identifier
    ;
</pre>

An action simply specifies a list of build unit parts to execute in sequence. It is possible to a specific part, or the build unit itself (which is a shorthand notation for all of its parts).

Examples:
<pre>
action buildAll {
    org.myproduct.productSite#site.p2;
    org.myproduct.packaging;
}
</pre>

This makes it possible to invoke the "buildAll" action on the b3 file without first having to run an import. If the required units are not already available, they will be imported.

==Include statement==
<pre>
include-statement
    | 'include' include-type optional-name '{' include-list '}'
    ;

include-type
    : 'advice'
    | 'properties'
    | 'synchronized'
    ;

include-list
    :
    | inclusion ';'
    | include-list inclusion
    ;

inclusion
    : url optional-names
    ;

optional-names
    : 
    | '{' name-list '}'
    ;

name-list
    : identifier
    | name-list ',' identifier
    ;

</pre>
====Examples - Including Advice====
It is possible to include advice from other b3 files. 
<pre>
include advice { http://someplace.org/somebuildfile.b3 { fixOfBrokenX, fixOfBrokenY } }
</pre>
This example shows the inclusion of advice from the build file appointed by the URI. The advice in the appointed file named fixOfBrokenX, and fixOfBrokenY is used as advice.
If no list of named advice is used, then all of the advice in the appointed build file is used.

Advice that is unnamed is implicitly called "advice", and it can be included like this:
<pre>
include advice { http://someplace.org/somebuildfile.b3 { advice, fixOfBrokenY } }
</pre>
Which specifies the "unnamed" advice statements, and the named fixOfBrokenY.

Multiple includes can be made in one statement, given all of the included advice a new name:
<pre>
include advice c {
    http://someplace.org/somebuildfile.b3 { fixOfBrokenX, fixOfBrokenY }; 
    http://someplace.org/someotherbuildfile.b3 { a, b };
    includes/commonadvice.b3;
}
</pre>

==Unit Statement==
A unit statement is used when the b3 file has the role of describing advice for the unit it is embedded in. When doing this, most other b3 statements are meaningless (resolvers, requests, properties, etc), and this is reflected in the syntax. A unit statement is basically the same as a regular advice statement, only rooted in the unit where the b3 file is as opposed to rooted at the b3 context (as is the case for the build file).

<pre>
unit-statement
    : 'unit' compound-advice
    ;
</pre>

An embedded b3 file should always be called "this.b3".

W.i.P - attempt at writing a unit using the general advice syntax - this gets quite messy... It should really be part of the 3 language.
<pre>
unit {
    name="org.myorg.hopscotch";
    namespace="myorg.special";
    version="1.2.3";
    parts = [
        new ArtifactsPart {
            name="src";
            paths = new PathGroup {
                basePath="src";
                paths = [ "foo.xxx", "bar.xxx", "fee.xxx", "fum.xxx" ];
                };
         },
        new ArtifactsPart {
            name="help";
            paths = new PathGroup {
                basePath="src";
                paths = [ "general.hlp", "other.hlp" ]
                };
        },
        new Action {
               name="zipSource";
               actor = "ZipActor"; // <- TBD... just a rough idea of what is needed here...
               requiredParts = [ new PartRequirements { alias="input"; memberName = "allSource"; } ];
               produces = new ResultingPathGroup {
                     paths = new PathGroup {
                          basePath="bin";
                          paths = [ "output.zip" ];
                     };
                };
        },
        new PartGroup {
            name="allSource"
               requiredParts = [
                    new PartRequirements { memberName = "src"; }
                    new PartRequirement { memberName = "help"; }
                    ];
         }
    ];
// and so on, with required capabilities, provided capabilities, properties etc.
</pre>

==Synchronized Statement==
The synchronized statement will prevent parallel execution of actions. (Is not likely to be implemented in the first release of b3, but shown for completeness).
It is also possible to specify or advice actors, actions, units and build parts that they are synchronized. The general syncronized statement allows syncronization to be specified for disjunct instances and is useful when a composition of units contains actions that would otherwise create conflicts - e.g. producing temporary output to the same directory.

{| border="1" cellborder="0" cellspacing="0"
|-
| ''Synchronized statement''
|-
|<pre>
synchronized-statement :
    :  'synhcronized' optional-name '{' sync-expr-list '}'
    ;

sync-expr-list :
    |
    | sync-expr-list sync-expr
    ;

sync-expr :
    | unit-part-ref ',' unit-part-ref
    | sync-expr ',' unit-part-ref
    ;

unit-part-ref :
    | unitname-pattern
    | unitname-pattern '#' partname-pattern
    ;
</pre>
|}

Examples
<pre>
synchronized {
    foo#actionA, bar#actionB;
    fee#actionC, fum#actionD;
    fum#actionA, fee#actionB, fee#actionX;
    org.myorg.*, org.yourorg.fum.*;
    org.myorg.foo#binary.*;
} 
</pre>
The examples shows a specification of 5 groups where parallel execution of the stated units/parts in each groups is prevented (i.e. execution of these are serialized with respect to each other).