Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "GraphitiBuildProcessDescription"
(→Graphiti Build Process) |
|||
| (4 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
= Graphiti Build Process = | = Graphiti Build Process = | ||
| − | This page describes the Graphiti build setup. Thanks to the Teneo and CDO guys for providing a template for setting up a first version of our build and an excellent description of it (the [http://wiki.eclipse.org/Teneo/Teneo_Build_Setup Teneo build description] was also used as a template for this page). Graphiti uses [http://www.eclipse.org/buckminster/ Buckminster] and the [http://wiki.eclipse.org/Hudson | + | This page describes the Graphiti build setup. Thanks to the Teneo and CDO guys for providing a template for setting up a first version of our build and an excellent description of it (the [http://wiki.eclipse.org/Teneo/Teneo_Build_Setup Teneo build description] was also used as a template for this page). Graphiti uses [http://www.eclipse.org/buckminster/ Buckminster] and the [http://wiki.eclipse.org/graphiti/ Graphiti Hudson instance] for its continuous build. The job [https://hudson.eclipse.org/graphiti/job/graphiti.nightly/ graphiti.nightly] is used to build Graphiti's head version. To understand the content of this page it is necessary to understand the main Buckminster concepts. These are described in detail in the [http://www.eclipse.org/downloads/download.php?file=/tools/buckminster/doc/BuckyBook.pdf Bucky Book]. Some Buckminster concepts can seem fairly abstract initially, the 'Understanding Buckminster' section below may help here. |
== Local Execution<br> == | == Local Execution<br> == | ||
| Line 7: | Line 7: | ||
It should be possible to do a local Graphiti build. [TODO: describe the necessary steps]<br> | It should be possible to do a local Graphiti build. [TODO: describe the necessary steps]<br> | ||
| − | == Understanding Buckminster == | + | == Understanding Buckminster == |
| − | This section may help to make it a | + | This section may help to make it a clearer how Buckminster works. The main functionality (for use within the Graphiti build): |
*Buckminster creates workspaces and target platforms for running the build and test. These workspaces are the same or similar to what you see in your IDE. | *Buckminster creates workspaces and target platforms for running the build and test. These workspaces are the same or similar to what you see in your IDE. | ||
| Line 23: | Line 23: | ||
*'''mspec file''': This file is the starting point. It specifies which cquery file to use and also how the plugins/development projects downloaded as part of resolving the cquery file should be treated, or materialized. Materialized means: where to place the downloaded plugins/development projects, in the workspace or in the target platform. | *'''mspec file''': This file is the starting point. It specifies which cquery file to use and also how the plugins/development projects downloaded as part of resolving the cquery file should be treated, or materialized. Materialized means: where to place the downloaded plugins/development projects, in the workspace or in the target platform. | ||
| − | + | == Main Build Steps == | |
| − | + | The Graphiti build job executes 2 build steps: the first one is a Linux shell script that sets up the environment for the build job, the second one defines the Buckminster steps to process. The latter will be describes in more detail below, here are just a few reamrks for the shell script part: | |
| − | + | *It starts off with deleting the remains from previous builds | |
| + | *Then it creates a properties file containing all the properties defined in the properties file retrieved from the releng project plus defining some more. | ||
| − | + | The next step of the build is the main part. It is a Buckminster script that performs several steps: | |
| − | + | *Set the Java compliance level to Java 6 | |
| + | *Define the folder for the target platform (where BuckminstAllJunitUiTests.launcher syncs all binary usages to) | ||
| + | *Import an API baseline definition. The definition file resides in the releng project and defines different baselines according to the current version. Note that in the 0.7.x codelines no baseline definition file exists, so in case you do a 0.7.x build you will need to comment out this line. | ||
| + | *Do the actual import of the sources and binaries: download the development projects from CVS and use them to create a workspace and download/retrieve the dependencies to the target platform. | ||
| + | *Do the Java build | ||
| + | *Create the JavaDoc (happens via a Buckminster call to execute an Ant script defined in the releng project. | ||
| + | *Creates the site by performing the Buckminster command site.p2.zip. This will first create the site.p2, the - if enabled - sign the site, create teh site.signed and the site.p2.zip file. | ||
| + | *Create an html and xml index for the site (stating which features and plugins in which version is contained). This again happens via a Buckminster call to execute an ANt script resideing in the releng project. | ||
| + | *Start the test runs. There are currently 3 test suites executed defined via launch configs: | ||
| + | **AllJunitTests.launch: Starts the JUnit tests for the plugin org.eclipse.graphiti<br> | ||
| + | **AllJunitUiTests.launch: Starts the JUnit tests for the plugin org.eclipse.graphiti.ui<br> | ||
| + | **AllBotTests.launch: Starts the SWTBot tests for the complete framework<br> | ||
| + | *Copy the build result to the last_success folder. This again happens via a Buckminster call to n Ant script defined in the releng project | ||
| + | *The actual promotion (publishing the site to the Eclipse download area (head or dev download site as stated on the [http://www.eclipse.org/graphiti/download.php Graphiti Downloads page]) is done by a cron job defined for user mwenz that runs once a day in the evening. The reason is that the hudson job does not have write permission for the downloads area. | ||
| − | + | By using a Buckminster script we get the setup of Buckminster for free. We use the Buckminster 3.7 installation on the Eclipse Hudson. The following sections describe the necessary files for the build. | |
| − | + | === Structure of build projects/features === | |
| − | + | The build infrastructure for Graphiti is centered around a releng project containing the all necessary files. A number of other feature projects are used to define which plugins are part of the build, part of the update site and which are part of the tests. The releng and feature projects can be found in the CVS location ''dev.eclipse.org/cvsroot/modeling/org.eclipse.gmp/org.eclipse.gmp.graphiti'' in the folders releng (for the releng project) and features (for the feature projects). | |
| − | + | The ''org.eclipse.graphiti.releng'' project contains the Buckminster files for the build and test steps. The content of this project and the buckminster setup is described in more detail below. This project is the most important one to look at when trying to understand Graphiti's build setup. The feature projects are used to structure the output of the build in the correct way, i.e. which features/plugins are made available via an update site, which features/plugins are involved in testing etc. Here is a summary of these feature projects: | |
| − | + | *''org.eclipse.graphiti.site'' - Defines the update site and the main features. This project contains a buckminster.cspex file to define extra dependencies for stuff added via package imports (this is described in more detail below). The included features are: | |
| + | **org.eclipse.graphiti.feature - The Graphiti runtime feature<br> | ||
| + | **org.eclipse.graphiti.feature.examples - The documentation and the tutorial plugin for Graphiti<br> | ||
| + | *''org.eclipse.graphiti.tests-feature'' - Contains the test plugins and their dependencies. The test plugins contain test launch configurations (see below running test cases).<br> | ||
| − | + | === Main build project: org.eclipse.graphiti.releng<br> === | |
| − | + | After downloading the main build project from cvs you can see that it consists of several files in the root and the xsl subfolder. Here is a summary for the meaning of different files:<br> | |
| − | + | *''build.mspec:'' This file is the starting point for Buckminster. This file specifies two things: firstly, which query file to use, the cquery file is explained below, it defines which root feature project to download as a starting point for the build and secondly, how to materialize: as a default materialize to the target platform (materializer="p2"), only if source is "true" materialize to the workspace. | |
| + | *''build.cquery:'' The cquery file defines which rmap file to use, the rmap file specifies the download locations (update sites) and cvs locations to download dependencies. Buckminster needs a root project to use as a starting point which is defined by the rootRequest tag. The advisorNode tag is used to tell Buckminster to ignore dependencies (defined in feature.xml files) which have a pattern starting with org.eclipse.graphiti and ending with source. This is needed because the SDK feature projects contain these dependencies. However, these *.source plugins/features do not exist in cvs but are created by Buckminster/Eclipse during the build. So they should be ignored when downloading dependencies. See [http://www.eclipse.org/forums/index.php?t=tree&th=162136& www.eclipse.org/forums/index.php] for a thread on the Buckminster newsgroup on this. Besides it can give additional hints for the retrieval of components (e.g. you can define a branch for the source retrieval here).<br> | ||
| + | *''APIbaseLine.target:'' defines a target platform that will be used as the API baseline for the build | ||
| + | *''buckminster.ant'': Defines the Ant tasks used in the build (e.g create the JavaDoc,create the index files, copy build results) | ||
| + | *''buckminster.cspec:'' defines additional dependencies and the actions to trigger for the Ant tasks in buckminster.ant | ||
| + | *''build.properties:'' defines several properties related to how to sign plugins and where the download folder reside (both to download dependencies from and to upload the build result to. This file is used within the central build on the Eclipse Hudson.<br> | ||
| + | *''build.rmap:'' Defines the update sites and CVS locations were to download the different components from (applies for both source and binary components) | ||
| + | *''local.properties:'' same as ''build.properties'' but intended to be used for local builds.<br> | ||
| − | + | === Test Calls<br> === | |
| − | + | The flatXML argument is used to convert the output from the tests to a result which can be read by Hudson. | |
| − | + | === Defining Additional Dependencies: buckminster.cspex === | |
| − | + | Buckminster uses the explicit plugin dependencies in the MANIFEST.MF to download related project and binary dependencies. Buckminster can however not directly/automatically resolve dependencies defined by the import-packages directive. This is logical as Buckminster can not identify the plugins to download, just on the basis of the package name. To solve this, it is possible to define additional dependencies using a so-called buckminster.cspex file. The Graphiti build has a buckminster.cspex file in two feature projects: | |
| − | + | *org.eclipse.graphiti.site | |
| + | *org.eclipse.graphiti.tests-feature | ||
| − | + | The releng project is used in the rootRequest of the cquery file and delegates to the two feature projects. Note, the file containing the additional dependencies must be called buckminster.cspex. This cspex file defines dependencies on several eclipse features. The rmap file define where to find these features (i.e. which update site for example). | |
| − | + | === Testing === | |
| − | <br> | + | Testcases are run through Buckminster using a launch configuration (so Buckminster calls a launch configuration). Graphiti has 3 launch configurations one for running JUnit tests for the core plugin (org.eclipse.graphiti), one for running JUnit tests for the UI plugin (org.eclipse.graphiti.ui) and one for running the SWTBot tests for the complete framework (UI tests). |
| + | |||
| + | The launch configuration is easy to create, after running your test suite in Eclipse, a run configuration should be present in the window Run > Run Configurations. To store the run/launch configuration in a file: in the last tab of the configuration (Common), the save as location is set to a shared file in a project. The file is checked-in into CVS. This shared file is then used as the input to the test run through buckminster, see the call for running the tests above. The output location is specified with the -o parameter. The xml in the output location is used by Hudson to create a test report. | ||
| + | |||
| + | === Defining and creating an update site === | ||
| + | |||
| + | The Graphiti build creates an update site, which site is defined through a feature project: the org.eclipse.graphiti.site project. It has one important file names feature.xml; it defines which features are part of the update site and contains 2 features: | ||
| + | |||
| + | *org.eclipse.graphiti.feature | ||
| + | *org.eclipse.graphiti.feature.examples | ||
| + | |||
| + | To build the update site the build job calls the Buckminster action site.p2.zip which does all necessary stpe for creating the site and archive. Buckminster is called with the org.eclipse.graphiti.site feature (note feature id is used here, not the project name - here this is not relevant since they are the same, but they might differ). As a default (is option controlled) the content of the passed in feature is used for the content of the site, so the feature<br>project (org.eclipse.graphiti.site) itself is not part of the update site. | ||
Revision as of 07:26, 22 November 2013
Contents
Graphiti Build Process
This page describes the Graphiti build setup. Thanks to the Teneo and CDO guys for providing a template for setting up a first version of our build and an excellent description of it (the Teneo build description was also used as a template for this page). Graphiti uses Buckminster and the Graphiti Hudson instance for its continuous build. The job graphiti.nightly is used to build Graphiti's head version. To understand the content of this page it is necessary to understand the main Buckminster concepts. These are described in detail in the Bucky Book. Some Buckminster concepts can seem fairly abstract initially, the 'Understanding Buckminster' section below may help here.
Local Execution
It should be possible to do a local Graphiti build. [TODO: describe the necessary steps]
Understanding Buckminster
This section may help to make it a clearer how Buckminster works. The main functionality (for use within the Graphiti build):
- Buckminster creates workspaces and target platforms for running the build and test. These workspaces are the same or similar to what you see in your IDE.
- Buckminster uses the includes and dependency definitions in the MANIFEST.MF (for a plugin project) and the feature.xml (for a feature project) to download related projects or binaries. Additional dependencies are defined in a buckminster.cspex file. The build and buckminster files only refer to so-called root feature projects. The includes and dependencies of those projects then automatically drive the download of all other relevant projects and binaries into the workspace and target platform.
- Buckminster can search many types of locations for dependencies. The locations and their search order are defined in a configuration file (the graphiti.rmap files in the build and test subfolders). Development projects are downloaded from cvs while binary dependencies are coming from update sites.
- When downloading dependencies, Buckminster determines which dependency to place in the workspace (as a development project) and which dependencies to place in the target platform. The choice where to place what is defined in the graphiti.rmap file using the attribute 'source' with the value 'true', which signals a dependency to be a source (so should be placed in the workspace).
- Buckminster has several commands which can be executed against a workspace/target platform, for example run tests, build a p2.site etc.
A short description on the meaning of some files in the Graphiti build:
- rmap file: Defines where and how to download dependencies and development projects, contains cvs locations, update site url's and specifies an order in which these locations are searched. Also identifies which sources are considered to contain sources (i.e. development projects) and which are used for binary dependencies.
- cquery file: Defines which rmap file to use and defines the root (feature) project, the content (defined via the feature.xml) and the dependencies of the content are downloaded.
- mspec file: This file is the starting point. It specifies which cquery file to use and also how the plugins/development projects downloaded as part of resolving the cquery file should be treated, or materialized. Materialized means: where to place the downloaded plugins/development projects, in the workspace or in the target platform.
Main Build Steps
The Graphiti build job executes 2 build steps: the first one is a Linux shell script that sets up the environment for the build job, the second one defines the Buckminster steps to process. The latter will be describes in more detail below, here are just a few reamrks for the shell script part:
- It starts off with deleting the remains from previous builds
- Then it creates a properties file containing all the properties defined in the properties file retrieved from the releng project plus defining some more.
The next step of the build is the main part. It is a Buckminster script that performs several steps:
- Set the Java compliance level to Java 6
- Define the folder for the target platform (where BuckminstAllJunitUiTests.launcher syncs all binary usages to)
- Import an API baseline definition. The definition file resides in the releng project and defines different baselines according to the current version. Note that in the 0.7.x codelines no baseline definition file exists, so in case you do a 0.7.x build you will need to comment out this line.
- Do the actual import of the sources and binaries: download the development projects from CVS and use them to create a workspace and download/retrieve the dependencies to the target platform.
- Do the Java build
- Create the JavaDoc (happens via a Buckminster call to execute an Ant script defined in the releng project.
- Creates the site by performing the Buckminster command site.p2.zip. This will first create the site.p2, the - if enabled - sign the site, create teh site.signed and the site.p2.zip file.
- Create an html and xml index for the site (stating which features and plugins in which version is contained). This again happens via a Buckminster call to execute an ANt script resideing in the releng project.
- Start the test runs. There are currently 3 test suites executed defined via launch configs:
- AllJunitTests.launch: Starts the JUnit tests for the plugin org.eclipse.graphiti
- AllJunitUiTests.launch: Starts the JUnit tests for the plugin org.eclipse.graphiti.ui
- AllBotTests.launch: Starts the SWTBot tests for the complete framework
- AllJunitTests.launch: Starts the JUnit tests for the plugin org.eclipse.graphiti
- Copy the build result to the last_success folder. This again happens via a Buckminster call to n Ant script defined in the releng project
- The actual promotion (publishing the site to the Eclipse download area (head or dev download site as stated on the Graphiti Downloads page) is done by a cron job defined for user mwenz that runs once a day in the evening. The reason is that the hudson job does not have write permission for the downloads area.
By using a Buckminster script we get the setup of Buckminster for free. We use the Buckminster 3.7 installation on the Eclipse Hudson. The following sections describe the necessary files for the build.
Structure of build projects/features
The build infrastructure for Graphiti is centered around a releng project containing the all necessary files. A number of other feature projects are used to define which plugins are part of the build, part of the update site and which are part of the tests. The releng and feature projects can be found in the CVS location dev.eclipse.org/cvsroot/modeling/org.eclipse.gmp/org.eclipse.gmp.graphiti in the folders releng (for the releng project) and features (for the feature projects).
The org.eclipse.graphiti.releng project contains the Buckminster files for the build and test steps. The content of this project and the buckminster setup is described in more detail below. This project is the most important one to look at when trying to understand Graphiti's build setup. The feature projects are used to structure the output of the build in the correct way, i.e. which features/plugins are made available via an update site, which features/plugins are involved in testing etc. Here is a summary of these feature projects:
- org.eclipse.graphiti.site - Defines the update site and the main features. This project contains a buckminster.cspex file to define extra dependencies for stuff added via package imports (this is described in more detail below). The included features are:
- org.eclipse.graphiti.feature - The Graphiti runtime feature
- org.eclipse.graphiti.feature.examples - The documentation and the tutorial plugin for Graphiti
- org.eclipse.graphiti.feature - The Graphiti runtime feature
- org.eclipse.graphiti.tests-feature - Contains the test plugins and their dependencies. The test plugins contain test launch configurations (see below running test cases).
Main build project: org.eclipse.graphiti.releng
After downloading the main build project from cvs you can see that it consists of several files in the root and the xsl subfolder. Here is a summary for the meaning of different files:
- build.mspec: This file is the starting point for Buckminster. This file specifies two things: firstly, which query file to use, the cquery file is explained below, it defines which root feature project to download as a starting point for the build and secondly, how to materialize: as a default materialize to the target platform (materializer="p2"), only if source is "true" materialize to the workspace.
- build.cquery: The cquery file defines which rmap file to use, the rmap file specifies the download locations (update sites) and cvs locations to download dependencies. Buckminster needs a root project to use as a starting point which is defined by the rootRequest tag. The advisorNode tag is used to tell Buckminster to ignore dependencies (defined in feature.xml files) which have a pattern starting with org.eclipse.graphiti and ending with source. This is needed because the SDK feature projects contain these dependencies. However, these *.source plugins/features do not exist in cvs but are created by Buckminster/Eclipse during the build. So they should be ignored when downloading dependencies. See www.eclipse.org/forums/index.php for a thread on the Buckminster newsgroup on this. Besides it can give additional hints for the retrieval of components (e.g. you can define a branch for the source retrieval here).
- APIbaseLine.target: defines a target platform that will be used as the API baseline for the build
- buckminster.ant: Defines the Ant tasks used in the build (e.g create the JavaDoc,create the index files, copy build results)
- buckminster.cspec: defines additional dependencies and the actions to trigger for the Ant tasks in buckminster.ant
- build.properties: defines several properties related to how to sign plugins and where the download folder reside (both to download dependencies from and to upload the build result to. This file is used within the central build on the Eclipse Hudson.
- build.rmap: Defines the update sites and CVS locations were to download the different components from (applies for both source and binary components)
- local.properties: same as build.properties but intended to be used for local builds.
Test Calls
The flatXML argument is used to convert the output from the tests to a result which can be read by Hudson.
Defining Additional Dependencies: buckminster.cspex
Buckminster uses the explicit plugin dependencies in the MANIFEST.MF to download related project and binary dependencies. Buckminster can however not directly/automatically resolve dependencies defined by the import-packages directive. This is logical as Buckminster can not identify the plugins to download, just on the basis of the package name. To solve this, it is possible to define additional dependencies using a so-called buckminster.cspex file. The Graphiti build has a buckminster.cspex file in two feature projects:
- org.eclipse.graphiti.site
- org.eclipse.graphiti.tests-feature
The releng project is used in the rootRequest of the cquery file and delegates to the two feature projects. Note, the file containing the additional dependencies must be called buckminster.cspex. This cspex file defines dependencies on several eclipse features. The rmap file define where to find these features (i.e. which update site for example).
Testing
Testcases are run through Buckminster using a launch configuration (so Buckminster calls a launch configuration). Graphiti has 3 launch configurations one for running JUnit tests for the core plugin (org.eclipse.graphiti), one for running JUnit tests for the UI plugin (org.eclipse.graphiti.ui) and one for running the SWTBot tests for the complete framework (UI tests).
The launch configuration is easy to create, after running your test suite in Eclipse, a run configuration should be present in the window Run > Run Configurations. To store the run/launch configuration in a file: in the last tab of the configuration (Common), the save as location is set to a shared file in a project. The file is checked-in into CVS. This shared file is then used as the input to the test run through buckminster, see the call for running the tests above. The output location is specified with the -o parameter. The xml in the output location is used by Hudson to create a test report.
Defining and creating an update site
The Graphiti build creates an update site, which site is defined through a feature project: the org.eclipse.graphiti.site project. It has one important file names feature.xml; it defines which features are part of the update site and contains 2 features:
- org.eclipse.graphiti.feature
- org.eclipse.graphiti.feature.examples
To build the update site the build job calls the Buckminster action site.p2.zip which does all necessary stpe for creating the site and archive. Buckminster is called with the org.eclipse.graphiti.site feature (note feature id is used here, not the project name - here this is not relevant since they are the same, but they might differ). As a default (is option controlled) the content of the passed in feature is used for the content of the site, so the feature
project (org.eclipse.graphiti.site) itself is not part of the update site.