< To: Buckminster Project
Welcome to Buckminster
The purpose of this document is to familiarize you with the key concepts of the Buckminster framework using simple examples. It will provide samples of the available Buckminster editors and relevant Buckminster language artifacts as XML fragments.
If you are new to Buckminster you should read the Buckminster in a nutshell document first.
If you are looking for usage examples or more detailed information you should consult the documentation overview.
How can Buckminster help me ?
Solving a common problem
In its simplest form, Buckminster solves problems like the following:
- Your colleague says "OK, project Overland is ready for you to work on - just check out com.megacorp.overland from CVS and get started".
- You checkout com.megacorp.overland and discover that it has unresolved project dependencies.
- "Oh yeah," he says, "you have to checkout project overwater as well." Of course that doesn't completely solve the problem ...
- "Also you have to get the latest undersea.jar from the central FTP server."
- "But hey, it doesn't build ok!"
- "Oh, I forgot, you have to import this jar here, and rename it ... and let me see what else ..."
- Eventually, after a few rounds of this, you have the whole set of resources and you are ready to go to work.
In its simplest form, Buckminster is a tool for materializing a workspace. In its broadest sense Buckminster puts the world of components at your fingertips by formalising component descriptions and dependencies and materializing them in a context of your choice - one of which is a workspace.
Query the "cloud"
As a component developer, your workspace is the set of components you are working on plus all of their prerequisite components. Under Buckminster, components can be many things, but one of the most important component types is Eclipse projects. In pictures, the workspace (the cloud) is initially empty, then you ask Buckminster to materialize the component (the box), after which the workspace contains the component.
What the simple example above boils down to is what Buckminster calls a component query (or CQUERY). It is the top-level entry point to the Buckminster framework. You are saying "I want "that" component". Buckminster provides everything necessary to ask that question (or express that query) AND materialize "that" component to a location of your choice. Actually, what you are requesting and what Buckminster allows you to express is more like:
- "I want the latest version of "that" component" OR
- "I want version x.y.z or a later versions of "that" component" OR
- "I want version x.y.z or a later versions of "that" component in source form" OR
- ... and so on
Things cannot possibly be that simple ?
This is indeed not a trivial problem. We can continue with the example queries above and extend them with one more important condition. You really are asking "I want the latest version of "that" component AND I don't want to worry about all dependencies".
Using the simple diagram above lets assume that component A has dependencies on B & C which have dependencies themselves, the actual result is something like the example shown in the diagram.
Let's add a bit more context to this simple picture and lets assume your request for "that" component A should result in a project in your workspace which requires B, C & D to build. This is the point where Buckminster really shines.
In order to do so Buckminster will have to understand a lot of things about components. We say "understand" because Buckminster interpretes existing component information to perform its magic. This information comes in many different forms such as plugin.xml or feature.xml files, POM files in Maven, etc. Buckminster understands them all, translates them in a Buckminster component specification (or CSPEC) and is able to learn about new ones.
The component infomation captured in those files (i.e. a plugin.xml or POM) will eventually allow Buckminster to discover that component A needs B, C & D. Buckminster will have to resolve all of those dependencies before it knows what it has to materialize into your workspace. Remember, that you just asked for component A but after the Buckminster resolver has done its work it will be able to tell you that you also need B, C & D.
But how does Buckminster know about all the components in the "cloud" ?
The Buckminster resolver is a bit a like navigation system. You give it a starting point (an empty workspace) and a destination (component A as a project in the workspace without build errors). Like any navigation system Buckminster needs a map to do this job. Buckminster calls this a resource map (or RMAP).
The resource map contains a component index and location descriptions (repository types and addresses). Buckminster looks up the component index which points to a location that Buckminster can visit to find components and their associated component information. Like on any map there are usually alternative routes. The resource map captures this as well. You may prefer the quickest route (get a binary version of D from a remote repository) or the shortest (get a source version of D from your local repository and build a binary).
Buckminster matches the information in the resource map against your request for "that component A with all all dependencies resolved and oh, I prefer to use stuff from a local repository". The component query and resource map together with Buckminster's understanding of components is sufficient to produce a "component shopping list".
Buckminster's "component shopping list"
Buckminster calls this a bill of materials (or BOM). With regard to our picture of Buckminster as a navigation system the BOM would be the computed route. It is a set of instructions and things to get. The Buckminster materializer understands this and performs the final step: it actually physically gets the things you want and those you need.
Component B may require a binary version of D and as part of the materialization process Buckminster may first be required to obtain a source version of B and then build it. All of this is captured in the BOM. The BOM stands alone, that is the entry point to the Buckminster materialization can be the component query or a previously produced bill of materials. In the context of your daily work this means that you can either share a CQUERY and RMAP or a BOM with your team members if they want to get "that" component A as well.
As a final twist Buckminster also recognizes that the content of the same bill of materials should be materialized into different locations depending on your work context, system setup or personal preferences (some of your team members may run on Linux others on Windows). The materializer obtains this information from a materialization specification (or MSPEC) which is associated with a bill of materials. With regard to our picture of a shopping list we can say that two shopping baskets may contain the same items but the items will be put on different shelves at home.
So, is Buckminster just an Eclipse IDE extension ?
Although jumping ahead, a very common question at this point is if it is possible to materialize components with Buckminster without running the Eclipse user interface, for instance for the purpose of building a particular configuration on a remote server. The shortest answer to this question is: Yes, Buckminster can be used both as an integrated part of the Eclipse IDE, and from the command line where Buckminster will materialize a headless Workspace.
What's next ?
The simple example in the previous sections illustrates one usage scenario that Buckminster will help you with. The example covered the main concepts of Buckminster: component query, resource map, bill of materials. In the following sections we explain those concepts in more detail.
Buckminster: the whole story
Once you've grasped that Buckminster materializes components, you will be asking yourself "What are components ?", "Where does Buckminster get the components ?" and "How do my components fit into this ?". We will address those and other questions in more detail in the following sections.
Actually, what do you mean by component ?
We have been talking about components in our introduction assuming that every reader will be able to relate a common notion to this term. Buckminter's view of components is very broad. Basically, in the Buckminster parlance, a component represents a collection of files or sometimes one single file. For a large number of Buckminster users, a component will be an Eclipse project, but components are a larger concept than that - components can be external to Eclipse, components can have attributes, dependencies, perform actions, etc.
Buckminster provides its own language to describes components: it expresses all knowledge about a component in a Component Specification (CSPEC), which in the most general case is stored in an XML file following the Component Specification XML Schema, inside the component.
You will probably ask yourself now how much effort it will be to provide a CSPEC for a component such as an Eclipse project. The answer is "Not much at all" because Buckminster will do the job for you. As mentioned earlier, for many component types the required information is already available through other means (plugin.xml, feature.xml, POM files, etc), and in these cases, Buckminster will translate such information on-the-fly and there is no need to repeat what is already known. The CSPEC is a bit like the Esperanto of component specification languages and Buckminster is the universal translator that understands all of them and can learn about new ones by means of its extension points.
The component specification captures everything that Buckminster needs to know in order to resolve and materilaize components. One of the most important pieces of information in this context is dependencies.
Components Do Not Stand Alone: Dependencies
It is rarely the case that a component, such as an Eclipse project you are working on, is self-contained. Components may depend on other components but a fundamental concept in Buckminster is that it's not until you do something with a component that the prerequisites arise. Depending on what you do, you might need different things. As an example, for a compilation-action of component A you may need interfaces (or header files of some sort) from component B, but you do not need the rest of B, and later, when running the system you may need B in compiled binary loadable library form (but you are not interested in headers or the source of B). Buckminster separates the two concepts - the term dependency denotes the overall dependencies (every dependency), and the term prerequisite is used to denote that there is some action that has a dependency (what is needed for a particular action).
So, as an example; when using Buckminster lingo, you say: "A has a dependency on B", what you really are saying is: "There are actions in A that require B".
Obviously Buckminster finds the closure of the dependency graph when it materializes a workspace. The closure will always be in the form of a Directed Acyclic Graph (DAG). In our pictorial example, component A requires components B and C, and component B requires component D, and component C requires component D, so Buckminster will load all four into the workspace.
As you may have figured out, this is a simplification of what is really going on. Actually, some action was invoked on A, triggering the prerequisites for that action - something is needed from B, and C, and when obtaining those things from B, and C, those actions in turn have pre-requisites on something obtained from D.
A component's dependencies, actions, and the actions' pre-requisites are all expressed in the Component Specification (CSPEC) and serve as a script and/or set of constraints that Buckminster observes when resolving and materializing components.
Why the separation of dependency and prerequisite?
You may wonder why dependencies are declared separately from prerequisites of actions. Surely, the dependencies could be figured out by just following the prerequisites ?
The reason for separating the two is because at the component level many prerequisites will refer to the same dependency and there is a lot to say about a dependency that only applies once - such as which version of a particular component is wanted. Specifying this multiple times, once per prerequisite results in a duplication of dependency information across a component specification and is potentially error-prone as it is easy to make a mistake and include different versions of the same component for different actions.
In summary, a dependency is specified at the component level and can be referred to by many prerequisites. A prerequisite will specify in detail why an action has a particular component dependency and what is actually required from that component to satisfy the prerequisite.
We mentioned earlier that Buckminster really shows off when it comes to resolving a component's dependencies as you ask Buckminster for that component. We will dive into the area of dependency handling a bit more and will look first at how a prerequisite actually specifies what it needs from a component and secondly how Buckminster is able to unambiguously handle "versioned dependencies" that is a dependency on a particular version or version range of a component.
Prerequisites and attributes: I need "this" from "that" component.Remember we said that prerequisites are declared when a component needs something from another component to perform an action. Let's start with an example:
We know that in order to compile component A, we need access to the header files/interfaces provided by component B. To declare this we add a compile action to component A's CSPEC, and add a pre-requisite of component B's "headers" to this action. In component B we need to declare the corresponding "headers" so it represents the set of needed header files. How is this done?
A component can specify a variety of attribute types. Such an attribute can either be data directly, or trigger an action producing the data. Attributes are also referred to as Artifact Groups, which - as you may have guessed - represents a structure of files (one file, a flat list, multiple lists, a tree, a multi rooted tree etc.) An attribute has a name that is unique within a component. Prerequisites simply refer to an attribute in a component by its unique name.
How are attributes specified ?
A component attribute in Buckminster sort of works as a getter method (using Java/OO terminology), and there are three approaches in a Buckminster component specification to declare attributes. They vary with regard to granularity and the means of obtaining the attribute:
- artifact - this is best described as an atomic attribute definition; it specifies none, one, or several paths to either individual files or directories (denoted by a path ending with a slash (/)).
- group - this is a group of artifacts, although any type of attribute (artifact, groups, actions) could be grouped to describe an attribute.
- action - this is an attribute tied to an action that needs to be triggered when this attribute is requested (either by a component or a group or another action); it can produce a path structure on-the-fly returned as the value of the attribute.
So, in our example, component B has an attribute called "headers" that is an artifact element with the path to the directory with headers.
My component/Your component: private and public attributes
When an attribute has a prerequisite on another attribute in the same component, it is said to be local. When the prerequisites needs to reference an attribute in another component it is said to be external. The external reference uses component name, and attribute name.
Components can declare attributes to be private or public. The private attributes can only be used as prerequisistes for attributes in the same component. This makes it possible to create common datasets (artifacts and groups), or actions that are shared internally by the public attributes in the component. You do not have to repeat identical sequences and thus save time, space, and create a more robust, easier maintainable specification.
The most valuable use for private attributes is probably to hide internal complexities from users of the components. The user does not have to understand how certain sets of files are produced. Lets assume for example that the header files are generated from a UML model - the user should not need to know that.
Think big: Attributes that are Components
Sometimes, an entire component is created as the result of some action in another component. Remember that Buckminster distinguishes between the declaration of a component dependency and the prerequisite for a component attribute. That is why Buckminster handles this special case by allowing to express an "obtained from" statement in the dependency declaration.
As an example if component A depends on component X, and component X is obtained by getting attribute X in component B. Then, the declaration in component A states the dependency on "component X obtained from component B's attribute X". The dependency on component B must also be specified.
With this construct in place, it is then possible to have prerequisites on attributes in the component X.
Versioned Dependencies: That version is out, will a later do as well ?
How does Buckminster talk about component versions ?
Software has always been a challenging field because, unlike the continuous (or analog) nature of the real world, software is discrete (or digital). Thus a feature that worked in version 4.5 of the system may no longer work in version 5.0 or even in 4.6 or even in 4.5.1. A CSPEC always declares a dependency with reference to a particular component version. Actually, not just a single version. But lets look at the version information that Buckminster expects and how it is interpreted.
Component versions are referred to by a variety of terms. Although jumping ahead, the interpretation of version information will have a significant impact on how Buckminster will locate a component. When we are talking about component versioning in Buckminster we will usually refer to the following terms. We will discuss them in detail in later sections.
- plain version string - This would be the commonly used way of referring to a version. Something like 3.2.1 or Titanic-1.0 would be typical examples.
- version designator - This is expressing a version range. You may want to say something like version "3.2.1 or later".
- version type - Buckminster knows about a number of version types which have a version string grammar associated (that is it tells you how a version string of a particular type should be structured).
- version selector - Buckminster gets components of a particular type. Depending on their location (i.e. a CVS or SVN repository) the version string will have to be translated in a path (fragment) allowing to access the component.
Normally, the version string would be something simple like "4.5". However, because Buckminster is designed for and by software developers, and because software is not developed in a linear fashion, the version information usually used in a dependency is more complex (and more useful) than the simple plain version string "4.5". Specifically, a version designator denoting a range is used. The characters '[' and ']' are used to denote an inclusive range and the characters '(' and ')' will denote an exclusive range. Here are some examples:
|[1.2.3, 4.5.6)||1.2.3 <= x < 4.5.6|
|[1.2.3, 4.5.6]||1.2.3 <= x <= 4.5.6|
|(1.2.3, 4.5.6)||1.2.3 < x < 4.5.6|
|(1.2.3, 4.5.6]||1.2.3 < x <= 4.5.6|
|1.2.3||1.2.3 <= x|
Examples of version ranges.
Buckminster is not limited to the major.minor.micro number notation used in this example. That notation is just one of several version types. The type interprets the plain version token(s) of a range and assigns a magnitude to the result so that versions can be compared. Buckminster allow news types to be added dynamically via its org.eclipse.buckminster.core.versionTypes extension point.
Mapping Versions to a Revision Control System
The versions used in the dependency specification will often be meaningless to a revision control system. Buckminster will therefore use a Version Converter.
It is the responsibility of the converter to create something that a revision control system can understand and that will Buckminster eventually allow to get to the component. The converter will also be able to convert in the other direction and it plays a very important role when the best match for a version string is to be found in a revision control system. The plain version string is converted into a version selector, a Buckminster data structure describing if the version is a branch, or a tag, if it is based on a time-stamp or a change-number.
A combination of two version converters included with Buckminster and their functionality to use regular expressions makes it easy to set up even a quite complicated bi-directional mapping between a repository and a version selector. As an example, a plain version such as "3.1.0" can be converted into the version selector "main/v3_1_0-xyz" for a tag converter, or the plain version "3.1.0" into the version selector "three_one_zero/LATEST" for a branch converter. It is required that the mapping can be reversed without loss of information. See Version Selector for more information about the syntax of the data, and Version Converter for information about the two supplied converters.
The Simplest Case of Versions
The simplest version numbering (branch specifications) is the linear version numbers: 1.0, 1.1, 1.2, 2.0, etc. Linear version numbers are the base case: a revision control system with just one main branch.
Component B has one main branch, the <default> branch.
When the component dependencies have no branch specifier, Buckminster uses <default>/LATEST which translates to "the latest version of the default branch". The default branch varies by revision control system: CVS uses "HEAD", SVN uses "trunk", ClearCase uses "main", etc.
Component A's dependency on component B has no branch
specifier, thus Buckminster uses B:<default>/LATEST
If the cspec dependency includes a version number, Buckminster uses a version converter to translate the version number into a branch specification. The most common translation is "version" into "<default>/version". For example:
Component A's dependency on component B is version "2.0"
which gets translated to "<default>/2.0". Similarly, the dependency
on component C is "1.1" which gets translated as "<default>/1.1".
Looking at the branching diagram above, we see that "<default>" is
"main", thus Buckminster will load "main/1.1" to satisfy that dependency.
What If There Isn't a Revision Control System?
What if the component is not stored in a revision control system? For example, what if the component is stored in an FTP directory? The answer is that the providers and component readers defined in the rmap handle reading versioned components from the appropriate repository. If the repository does not provide versions, then the component reader treats the repository as if it has just one version: <default>/LATEST. Requesting any other branch specification will fail.
Or, to put the situation more positively, Buckminster treats a "no revision control system" situation as a simple revision control system that provides only one branch and one version.
The Flexible Case of Versions
By using a different version converter, Buckminster can map versions to change numbers, timestamps, or branches instead of tags. We expect the most typical configurations to be based on tags (something stable), or the latest on the main/default branch, or the latest on a particular branch. Combinations can sometimes make sense, but this also depends on the particular repository being used, and how content has been organized, tags set etc.
Checkout Version Converter for more information.
Although not likely to be fully implemented in Buckminster version 1.0, it is of value to understand this simple but yet powerful concept.
A dependency on an Abstract Component means that there is a dependency on a component that is a concrete realization of the abstraction. For example, a component may require that there is a SQL database available in order to run the component. In this case, the component can declare a dependency on the Abstract Component "SQL". When components like "MySQL", "PostgreSQL", and "Oracle" are declared to be concrete realizations of the "SQL" component a component query can fulfill a request (or show available options).
Dynamic CSPECs and extensions
Buckminster is capable of handling components expressed in different "component languages". One such language is the Buckminster Component Specification CSPEC, a language capable of capturing if not in the current version all, a majority of the component languages in use. The CSPEC can be thought of as the Rosetta Stone or Lingua France of Component meta data if you like.
What other component languages are supported?
When this introduction is written there are translators (In Buckminster terms, Component Types) that can read component meta data and translate to CSPEC format on-the-fly for Eclipse PDE, and Maven. A Component Reader is also involved as it is responsible for the protocol used to get to the component data. Buckminster is extendable to handle other component meta data languages.
How does it work?
When a component already has meta data (e.g. Maven or Eclipse PDE), this metadata is read and translated into a CSPEC that resides in the project model (i.e. in memory). It is possible to write this generated CSPEC to an XML file for viewing or other purposes. By generating the CSPEC on the fly, there is no need to maintain the same meta data in more than one place - the natural place; in the original component where the data is required anyway.
ComponentReader and ComponentType cooperate to read component meta data and translates it
into an in-memory CSPEC used by Buckminster. The in-memory CSPEC can also be viewed or exported to a CSPEC file in XML format.
Sometimes, however, the meta data in the component itself is not enough. This could be because the component meta data language does not have the expressive power to handle certain things that are important when materializing and performing component actions (compiling/building/packaging/installing/copying/encrypting or whatever). In these cases, Buckminster provides an extension mechanism to the CSPEC, called CSPECX.
The CSPECX, is stored in a file in the component (with the extension .cspecx), it is in the same format as the CSPEC in general, but here it is also possible to override the generated specification.
Mapping the "cloud": where did you say my components were ?
Where Does Buckminster Get Things?
Once you've grasped that Buckminster materializes components, the next big questions in your mind is probably "what are components?" and "where does Buckminster get the components?". Components are collections of files (a.k.a. resources) and are represented in Eclipse by projects. Components can be retrieved from a variety of different repositories. The most common repositories are a CVS repository or an FTP server, but almost anything (databases, file systems, auto-generator programs, etc) can be used via the org.eclipse.buckminster.core.readerTypes extension point.
Component A being retrieved from a CVS repository
and the required component B from an FTP server.
Of course the story is a bit more complicated than the simple picture because the component retrieval takes into account various version naming schemes, but we'll get into that later. For now, this picture is good enough.
How Does Buckminster Find The Repositories?
So the next question you're asking yourself is how does Buckminster know which component is stored in which repository? Knowing what you know about Eclipse, you know the projects are linked to Team (CVS) repositories (or some other repository like Subversion). However there are actually two concepts: the component and its storage location. Separating these two concepts is like separating interface and implementation in good software design. Buckminster does this separation through a level of indirection known as a resource map. The resource map (rmap) provides location information for families of components. When Buckminster needs to load component A, it looks in the rmap to determine the repository containing A, then goes to that repository and loads A. Then because A requires B and C, Buckminster looks in the rmap for information on B and C, goes to those repositories, loads those components, and so on.
The resource map maps components to storage locations.
Why Is The RMAP Useful?
This separation of component and storage location is useful because it allows Buckminster to retrieve the same component from potentially many different storage locations. The rmap has a number of areas of flexibility. First, the rmap defines storage locations for families of components rather than single components. Thus one can say "find all the Apache components on the Apache ftp server" without having to specify "find Apache Struts on the Apache ftp server and find Apache Cocoon on the Apache ftp server and find ..." ad nausea. Second, the rmap defines a search path of storage locations to search for a component. Thus one can say "first look in my local repository, then look in my team's repository, then look in my corporate group repository, then look in the Eclipse public CVS repository".
One More Useful RMAP Feature
Last, but not least, the rmap allows variability by setting properties. This can be used for many different purposes. One useful thing is for example to have different search paths for different distributed development sites. Thus the rmap for Team Stockholm would list the same component families as the rmap for Team Winnipeg, but with different local repository servers. TCP round-trips from Canada to Sweden are slower than TCP round-trips from Canada to Canada, thus the Winnipeg team maintains a replicated CVS repository for certain components.
Buckminster rmap-properties allows both teams to use the same rmap - the exact same file - but to define a different site, and thus a different set of search paths. Obviously the central servers (probably the last entry in each search path) would point to the same central Swedish servers so that if the components were not cached locally, they would be fetched across the Atlantic.
Components A,B, and C mapped to multiple repositories.
Back to Simplicity
We have described many useful features of the resource map, but to avoid cluttering the picture, let us retreat to the basics: the resource map (rmap) is mapping from components to storage locations.
For details of the rmap, see RMAP.
Just ask for it: I want that component !
A Component Query (CQUERY) is the topmost input to the Buckminster materialization process. It specifies "the thing you want materialized" and provides guidelines for the resolve and materialization stages of the process.
A CQUERY is created using the forms based CQUERY editor, or by editing an XML file. The CQUERY contins things like the name of the top most component, and the wanted version (or range of version), as well as many other parameters that control the materialization - which RMAP to use, if you want to skip certain components, if you want to trust local copies (perhaps materialized previously), if you want to override certain settings, etc.
Once you are done editing the query, you can save it and reuse it. It is also ideal to share with other members of a team (together with the RMAP). They can then easily re-create the same setup as you have created.
Sharing a CQUERY and RMAP
Once you have created a suitable CQUERY and RMAP, you make them available to your team (or the public) by making the files available on a web site (actually, via URLs). Make sure the CQUERY contains the RMAP URL. Tell people to start Eclipse, use the Open File ... dialogue and simply type (copy/paste) the CQUERY URL into the Filename field, and then click Open. That will open the CQUERY editor, from which the configuration can be materialized.
Pulling it all together
Bill of Materials
When Buckinster has finished the resolution of a configuration (i.e. when Buckminster has followed all pre-requisets for an action, finding the correct version of other components via the stated dependencies) a Bill of Materials (BOM) is created. The BOM describes all the components and the version of each component included in the configuration (for full traceability, it also includes the Component Query that was used to produce the BOM). The materialization takes the BOM as input and materializes the required components (i.e. makes them available to the project).
Currently, the BOM can not be exported, it is kept in Buckminster's model. You can expect functionality to be added at some point that both allows you to export the BOM to an XML file, and to materialize from such an externalized BOM.
Variability and 'open/closed' BOMs
A concept that is not likely to be included in Buckminster 1.0 is support for variability. Variability means that the BOM is not fully resolved - there are still things that needs to be specified to make things work. This functionality goes hand in hand with Abstract Components where as an example a BOM may still hold open which database engine to use in runtime. The Buckminster terminology for this is that an "open BOM" is not completely resolved, and a "closed BOM" is.
The BOM is read only
The BOM is always read-only. Even if the BOM would happen to be "open" the resolution of the open dependencies results in a new BOM (that is less open, but not necessarily closed, thus allowing further variability).
As an example, you are perhaps creating a product that can be used with various different "3d party" components in runtime; different databases, different application servers, message queues, etc. Producing pre-built/configured packagings of all combinations is a major shore and often leads to overly bloated software distributions that includes everything (and the kitchen sink).
The idea here is that you instead ship the pieces and let Buckminster resolve the remaining dependencies as part of an "installation"
MOREMORE resource maps, search-paths, providers.