Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "VIATRA/DeveloperDocumentation/BuildConfig"

(7 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= The EMF-IncQuery build system  =
+
== Build concepts ==
  
EMF-IncQuery uses Maven/Tycho for continous integration builds.<br>  
+
VIATRA uses Maven/Tycho for continuous integration builds. It is executed in two passes:
 +
 
 +
# In the first pass, a selected subset of projects (the so-called "core" projects) are built. These are Eclipse-independent and include the [https://wiki.eclipse.org/VIATRA/UserDocumentation/Build#viatra-maven-plugin Maven plugin for generating code from VIATRA Query patterns]. This is required to allow projects to reuse the latest compiler version; on the other hand, the projects compiled here are pushed into Maven repositories as well.
 +
# In the second pass, the remaining projects are built, mostly Eclipse-specific. These projects may also use the VIATRA Query Maven compiler to generate pattern matcher code.
 +
 
 +
'''Important''': these two passes have to be executed together, as the language generator mwe workflow of the pattern language (core project) generates code for the Eclipse-dependent UI projects as well; on the other hand, some UI projects depend on the maven compiler to generate matcher code.
 +
 
 +
The Eclipse update site generated for the first pass (''org.eclipse.viatra.update.core'') is not intended for use outside of the build process; it only makes the core plug-ins available for the second pass as Eclipse plug-ins. The second phase gets this site as a Maven property ''core.repository.url''.
 +
 
 +
<source lang="bash">
 +
#First pass
 +
mvn clean install -f releng/org.eclipse.viatra.parent.core/pom.xml
 +
 
 +
#Second pass
 +
mvn clean install -f releng/org.eclipse.viatra.parent.all/pom.xml
 +
</source>
  
 
== Project structure  ==
 
== Project structure  ==
  
*''plugins/*'': The various plug-ins projects EMF-IncQuery project consists of.<br>
+
Each component of the projects are stored as a separate subfolder in git, structured into plugins, tests and features. Test projects are expected not to have external dependencies, e.g. metamodels.
*''features/*'': Installable feature projects<br>
+
 
*''tests/*'': Plug-ins related to testing EMF-IncQuery. Does not include tests with external dependencies (e.g. [https://github.com/ujhelyiz/EMF-IncQuery-Examples/tree/master/school school])<br>
+
*''query'': Querying EMF models (formerly EMF-IncQuery).
*''releng/org.eclipse.incquery.licence.feature'': special licence feature for the update site to avoid storing the EPL licence multiple times. Possibly replaceable in the future with a similar feature from the Juno update site.<br>
+
*''transformation'': Transforming EMF models.
*''releng/org.eclipse.incquery'': The root project of the build, used as parent project for configuration inheritance, and it contains a lists of projects to build.<br>
+
*''cep'': Complex Event Processing framework.
*''releng/org.eclipse.incquery.update'': Update site project - the build creates a p2 repository from this project.  
+
*''dse'': Design Space Exploration framework.
*''releng/org.eclipse.incquery.update.extras'': Update site project - the build creates a p2 repository from this project.<br>
+
*''integration'': Integrations with other technologies.
 +
*''addon'': Addons for implementing useful application functionality with VIATRA queries.
 +
 
 +
The remaining folders describe other elements of the build as follows:
 +
 
 +
*''releng'': Projects related to building and packaging VIATRA:
 +
**''org.eclipse.viatra.parent'': Used as parent project that only includes common configuration for the VIATRA project. This module is safe to reuse outside the Eclipse project
 +
**Container projects for the "core" submodules which can be used outside of Eclipse:
 +
***''org.eclipse.viatra.parent.core'': Root project which contains the core submodules.
 +
***''org.eclipse.viatra.update.core'': Builds a p2 update site from the core submodules.
 +
***''org.eclipse.viatra.target.core'': The target definition necessary for building the core submodules.
 +
**Container projects for all non-core submodules:
 +
***''org.eclipse.viatra.parent.all'': Root Maven project which contains all non-core submodules.
 +
***''org.eclipse.viatra.update.all'': Builds a p2 update site from all submodules.
 +
***''org.eclipse.viatra.target.all'': The target definition necessary for building all submodules.
 +
**Oomph setup:
 +
***''org.eclipse.viatra.setup'': The setup files for setting up the development environments.
 +
***''org.eclipse.viatra.generator'': The generator MWE2 workflows used by the setup.
 +
*''maven'': VIATRA Maven plugins.
 +
*''examples'': Basic examples of various VIATRA features.
 +
*''artwork'': Logos in various formats.
  
 
== Adding a new project to the build  ==
 
== Adding a new project to the build  ==
Line 17: Line 52:
 
There are two important things to configure for adding a new project to the build:  
 
There are two important things to configure for adding a new project to the build:  
  
#In order to technically build the project, a build configuration needs to be created (and registered in the root project).  
+
#In order to technically build the project, a build configuration needs to be created (and registered in the parent project).  
##This consists of a single pom.xml file in the project root.  
+
##This consists of a single pom.xml file in the project root directory.  
 
##And a module declaration in the parent pom.  
 
##And a module declaration in the parent pom.  
 
#Add the created feature to either an existing or a new feature  
 
#Add the created feature to either an existing or a new feature  
Line 25: Line 60:
 
##In both cases, don't forget to add the source plugin to the SDK or another Sources feature.
 
##In both cases, don't forget to add the source plugin to the SDK or another Sources feature.
  
An example commit for executing all these steps (with the creation of a new project) is: https://github.com/ujhelyiz/EMF-IncQuery/commit/4b523260c94ddb2605ce1088... (Note that the addition of a source plugin is not included).<br>
+
An example commit for executing all these steps (with the creation of a new project) is: https://github.com/viatra/EMF-IncQuery/commit/4b523260c94ddb2605ce1088 (Note that the addition of a source plugin is not included).  
  
 
=== Defining the build configuration for a plug-in project  ===
 
=== Defining the build configuration for a plug-in project  ===
  
A project needs to have a pom.xml file in the project root for the build configuration. In most cases, this build configuration is very simple, as all required configurations are inherited either from the project type (Eclipse plug-in project) or the root project. Basically, there are two configuration types used in the project:  
+
A project needs to have a pom.xml file in the project root for the build configuration. In most cases, this build configuration is very simple, as all required configurations are inherited either from the project type (Eclipse plug-in project) or parent project. Basically, there are three configuration types used in the project:  
  
*Projects with xtend files (e.g. the [http://git.eclipse.org/c/incquery/org.eclipse.incquery.git/tree/plugins/org.eclipse.incquery.tooling.core org.eclipse.incquery.tooling.core] project)<br>
+
* Projects with xtend files (e.g. the [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/query/plugins/org.eclipse.viatra.query.tooling.core org.eclipse.viatra.query.tooling.core] project)  
*Projects without xtend files (e.g. the [http://git.eclipse.org/c/incquery/org.eclipse.incquery.git/tree/plugins/org.eclipse.incquery.runtime.base org.eclipse.incquery.runtime.base] project)
+
* Projects without xtend files (e.g. the [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/query/plugins/org.eclipse.viatra.query.runtime.base org.eclipse.viatra.query.runtime.base] project)
 +
* Projects compiling query specifications (e.g. the [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/integration/plugins/org.eclipse.viatra.integration.uml org.eclipse.viatra.integration.uml] project)
  
 
To create such a configuration file for a new project, the recommended solution is to copy an existing corresponding project configuration, and replace the plug-in identifier.  
 
To create such a configuration file for a new project, the recommended solution is to copy an existing corresponding project configuration, and replace the plug-in identifier.  
  
Warning! If a project for any reason uses a different version number then the base IncQuery project (currently 0.7.0.qualifier), it is important to add the version information to the project, replacing '.qualifier' with '-SNAPSHOT' (currently 0.7.0-SNAPSHOT).  
+
'''Warning!''' If a project for any reason uses a different version number than the base VIATRA project, it is important to add the version information to the project, replacing '.qualifier' with '-SNAPSHOT'. [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/query/plugins/org.eclipse.viatra.query.tooling.localsearch.ui org.eclipse.viatra.query.tooling.localsearch.ui].  
  
Additionally, the new project needs to be added to the parent project (org.eclipse.incquery.parent):  
+
Additionally, the new project needs to be added to the corresponding parent project (most commonly org.eclipse.viatra.parent.all): The pom.xml file contains a modules section where a list of submodules are presented - it needs to be extended with a new submodule describing the project location.  
 
+
The pom.xml file contains a module section where a list of submodules are presented - it needs to be extended with a new submodule describing the project location.<br>
+
  
 
=== Adding the plug-in to an existing feature  ===
 
=== Adding the plug-in to an existing feature  ===
  
The existing feature.xml file has to be extended with the new plug-in as a contained plug-in id.<br>
+
The existing feature.xml file has to be extended with the new plug-in as a contained plug-in id.
  
Currently most source plug-ins are installed with the SDK bundle, with the notable exceptions of Graphiti and GMF support plug-ins that can be installed by selecting the source features.<br>
+
Currently most source plug-ins are installed with the SDK bundle, with the notable exceptions of Graphiti and GMF support plug-ins that can be installed by selecting the source features.  
  
 
=== Adding the plug-in to a new feature  ===
 
=== Adding the plug-in to a new feature  ===
  
Creating a new feature project works similarly to creating a new plug-in project, but the feature builder pom.xml files are more simple. The recommended way of creating a new feature is to copy and existing one, and modifying the identifier, descriptions and contained plug-ins (and possibly features). The feature also needs to be registered into the root projects module section.  
+
Creating a new feature project works similarly to creating a new plug-in project, but the feature builder pom.xml files are more simple. The recommended way of creating a new feature is to copy and existing one, and modifying the identifier, descriptions and contained plug-ins (and possibly features). The feature also needs to be registered into the root project's module section.
 +
 
 +
'''Important''': make sure to set up the feature dependencies correctly! This allows including the functionality in Eclipse-based products, but requires a correct identification of the requirement. Also know the difference between feature inclusion and dependency: both are required for your feature to install, but only included features are provided from the VIATRA update site.
 +
 
 +
Additionally, this feature needs to be added to the update site. For this reason, the feature needs to be registered in the category.xml file stored in the update site project (the category.xml file has a form-based editor in PDE).
 +
 
 +
== Special features ==
 +
 
 +
=== Composite repositories ===
 +
Composite repositories are created from integration, milestone and release builds using the p2 Ant tasks. The basic ideas are described at a [http://www.lorenzobettini.it/2015/01/creating-p2-composite-repositories-during-the-build/ blog post], the scripts are available in the EMF-Parsley project, but were customized for the use of VIATRA, as there will be no special folders created for each major version.
 +
 
 +
The [https://hudson.eclipse.org/viatra/job/viatra-master/ Hudson build of VIATRA] allows access to these composites using the ''viatra.download.area'' property.
 +
 
 +
=== Signing binaries ===
 +
Milestone and release builds are signed using the maven-signer-plugin provided by the [[CBI]] project: [[Minerva#Signing]].
 +
 
 +
=== Publishing Maven artifacts ===
 +
Core projects are published to repo.eclipse.org using the [[Services/Nexus]] documentation. It is important that such projects should not depend on Eclipse Platform features, such as extension registry or plug-in activator support. If required, such features can be added optionally; but code should work without them.
  
Additionally, this feature needs to be added to (at least) one the update sites. For this reason, the feature needs to be registered in the category.xml file stored in the corresponding update site project (the category.xml file has a form-based editor in PDE).
+
=== Javadoc generation ===
 +
Javadoc is also generated from the core VIATRA projects using the Tycho Javadoc generator. The generated documentation is configured in the [http://git.eclipse.org/c/viatra/org.eclipse.viatra.git/tree/releng/org.eclipse.viatra.docs org.eclipse.viatra.docs] project.

Revision as of 12:54, 22 November 2019

Build concepts

VIATRA uses Maven/Tycho for continuous integration builds. It is executed in two passes:

  1. In the first pass, a selected subset of projects (the so-called "core" projects) are built. These are Eclipse-independent and include the Maven plugin for generating code from VIATRA Query patterns. This is required to allow projects to reuse the latest compiler version; on the other hand, the projects compiled here are pushed into Maven repositories as well.
  2. In the second pass, the remaining projects are built, mostly Eclipse-specific. These projects may also use the VIATRA Query Maven compiler to generate pattern matcher code.

Important: these two passes have to be executed together, as the language generator mwe workflow of the pattern language (core project) generates code for the Eclipse-dependent UI projects as well; on the other hand, some UI projects depend on the maven compiler to generate matcher code.

The Eclipse update site generated for the first pass (org.eclipse.viatra.update.core) is not intended for use outside of the build process; it only makes the core plug-ins available for the second pass as Eclipse plug-ins. The second phase gets this site as a Maven property core.repository.url.

#First pass
mvn clean install -f releng/org.eclipse.viatra.parent.core/pom.xml
 
#Second pass
mvn clean install -f releng/org.eclipse.viatra.parent.all/pom.xml

Project structure

Each component of the projects are stored as a separate subfolder in git, structured into plugins, tests and features. Test projects are expected not to have external dependencies, e.g. metamodels.

  • query: Querying EMF models (formerly EMF-IncQuery).
  • transformation: Transforming EMF models.
  • cep: Complex Event Processing framework.
  • dse: Design Space Exploration framework.
  • integration: Integrations with other technologies.
  • addon: Addons for implementing useful application functionality with VIATRA queries.

The remaining folders describe other elements of the build as follows:

  • releng: Projects related to building and packaging VIATRA:
    • org.eclipse.viatra.parent: Used as parent project that only includes common configuration for the VIATRA project. This module is safe to reuse outside the Eclipse project
    • Container projects for the "core" submodules which can be used outside of Eclipse:
      • org.eclipse.viatra.parent.core: Root project which contains the core submodules.
      • org.eclipse.viatra.update.core: Builds a p2 update site from the core submodules.
      • org.eclipse.viatra.target.core: The target definition necessary for building the core submodules.
    • Container projects for all non-core submodules:
      • org.eclipse.viatra.parent.all: Root Maven project which contains all non-core submodules.
      • org.eclipse.viatra.update.all: Builds a p2 update site from all submodules.
      • org.eclipse.viatra.target.all: The target definition necessary for building all submodules.
    • Oomph setup:
      • org.eclipse.viatra.setup: The setup files for setting up the development environments.
      • org.eclipse.viatra.generator: The generator MWE2 workflows used by the setup.
  • maven: VIATRA Maven plugins.
  • examples: Basic examples of various VIATRA features.
  • artwork: Logos in various formats.

Adding a new project to the build

There are two important things to configure for adding a new project to the build:

  1. In order to technically build the project, a build configuration needs to be created (and registered in the parent project).
    1. This consists of a single pom.xml file in the project root directory.
    2. And a module declaration in the parent pom.
  2. Add the created feature to either an existing or a new feature
    1. If an existing feature is used, the feature.xml has to be extended with the id of the new project.
    2. If a new feature is created, this feature is required to be registered both in the root project and in the update projects' category.xml file.
    3. In both cases, don't forget to add the source plugin to the SDK or another Sources feature.

An example commit for executing all these steps (with the creation of a new project) is: https://github.com/viatra/EMF-IncQuery/commit/4b523260c94ddb2605ce1088 (Note that the addition of a source plugin is not included).

Defining the build configuration for a plug-in project

A project needs to have a pom.xml file in the project root for the build configuration. In most cases, this build configuration is very simple, as all required configurations are inherited either from the project type (Eclipse plug-in project) or parent project. Basically, there are three configuration types used in the project:

To create such a configuration file for a new project, the recommended solution is to copy an existing corresponding project configuration, and replace the plug-in identifier.

Warning! If a project for any reason uses a different version number than the base VIATRA project, it is important to add the version information to the project, replacing '.qualifier' with '-SNAPSHOT'. org.eclipse.viatra.query.tooling.localsearch.ui.

Additionally, the new project needs to be added to the corresponding parent project (most commonly org.eclipse.viatra.parent.all): The pom.xml file contains a modules section where a list of submodules are presented - it needs to be extended with a new submodule describing the project location.

Adding the plug-in to an existing feature

The existing feature.xml file has to be extended with the new plug-in as a contained plug-in id.

Currently most source plug-ins are installed with the SDK bundle, with the notable exceptions of Graphiti and GMF support plug-ins that can be installed by selecting the source features.

Adding the plug-in to a new feature

Creating a new feature project works similarly to creating a new plug-in project, but the feature builder pom.xml files are more simple. The recommended way of creating a new feature is to copy and existing one, and modifying the identifier, descriptions and contained plug-ins (and possibly features). The feature also needs to be registered into the root project's module section.

Important: make sure to set up the feature dependencies correctly! This allows including the functionality in Eclipse-based products, but requires a correct identification of the requirement. Also know the difference between feature inclusion and dependency: both are required for your feature to install, but only included features are provided from the VIATRA update site.

Additionally, this feature needs to be added to the update site. For this reason, the feature needs to be registered in the category.xml file stored in the update site project (the category.xml file has a form-based editor in PDE).

Special features

Composite repositories

Composite repositories are created from integration, milestone and release builds using the p2 Ant tasks. The basic ideas are described at a blog post, the scripts are available in the EMF-Parsley project, but were customized for the use of VIATRA, as there will be no special folders created for each major version.

The Hudson build of VIATRA allows access to these composites using the viatra.download.area property.

Signing binaries

Milestone and release builds are signed using the maven-signer-plugin provided by the CBI project: Minerva#Signing.

Publishing Maven artifacts

Core projects are published to repo.eclipse.org using the Services/Nexus documentation. It is important that such projects should not depend on Eclipse Platform features, such as extension registry or plug-in activator support. If required, such features can be added optionally; but code should work without them.

Javadoc generation

Javadoc is also generated from the core VIATRA projects using the Tycho Javadoc generator. The generated documentation is configured in the org.eclipse.viatra.docs project.

Copyright © Eclipse Foundation, Inc. All Rights Reserved.