Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "JWT Automatic Build"

(HIPP)
 
(27 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Automatic build of JWT Features and Plugins =
+
= Build of JWT Features and Plugins =
  
== Helpful resources ==
+
This section presents information concerning the JWT 1.3.0 build for eclipse Kepler on Hudson and would be considered as a very useful resource for JWT's future release engineers (starting with Eclipse Luna scheduled for release in 2014). Some of the information/steps that follow might be not necessary for future JWT builds using Maven Tycho since they have already been done but we illustrate them anyway so that we would have a comprehensive and complete documentation of the whole building process. We start by explaining how to prepare the JWT project for the build and then follow by explaining how to configure Hudson to build JWT. Please, note that newly assigned release engineers might have to contact the eclipse webmaster (webmaster@eclipse.org) in case they are missing any permissions that they might require.
  
The following link includes an Eclipse Tycho tutorial that would serve as a very useful resource demonstrating how to build Eclipse plug-ins, features and how to create Eclipse p2 update sites with Maven Tycho.
+
== Helpful resources  ==
* [http://www.vogella.com/articles/EclipseTycho/article.html]
+
  
== Building JWT on Hudson using Maven Tycho ==
+
The following link includes an Eclipse Tycho tutorial that would serve as a very useful resource demonstrating how to build Eclipse plug-ins, features and how to create Eclipse p2 update sites with Maven Tycho.
  
This section presents information concerning the JWT 1.3.0 build for eclipse Kepler on Hudson and would be considered as a very useful resource for JWT's future release engineers (starting with Eclipse Luna scheduled for release in 2014). Some of the information/steps that follow might be not necessary for future JWT builds using Maven Tycho since they have already been done but we illustrate them anyway so that we would have a comprehensive and complete documentation of the whole building process.  
+
*[http://www.vogella.com/articles/EclipseTycho/article.html http://www.vogella.com/articles/EclipseTycho/article.html]<br>
 +
*[http://wiki.eclipse.org/JWT_SVN http://wiki.eclipse.org/JWT_SVN]
 +
*[http://aniszczyk.org/2011/05/18/eclipse-org-signing-support-for-maven-tycho/ http://aniszczyk.org/2011/05/18/eclipse-org-signing-support-for-maven-tycho/]
 +
*[http://wiki.eclipse.org/Kepler http://wiki.eclipse.org/Kepler]
 +
*Project plan:&nbsp;[https://projects.eclipse.org/projects/soa.jwt/releases/1.3.0 https://projects.eclipse.org/projects/soa.jwt/releases/1.3.0]<br>
 +
*Kepler/Simultaneous Release Plan:&nbsp;[http://wiki.eclipse.org/Kepler/Simultaneous_Release_Plan http://wiki.eclipse.org/Kepler/Simultaneous_Release_Plan]
 +
*[http://wiki.eclipse.org/Eclipse_b3/aggregator/manual http://wiki.eclipse.org/Eclipse_b3/aggregator/manual]
  
== Master project for the build  ==
+
== Preparing JWT to be built by Maven Tycho  ==
 +
 
 +
This section provides information concerning preparing JWT to be built using Maven Tycho and preparing the p2 update site contents to be signed by the eclipse signature after the build is done and the jars generated.
 +
 
 +
=== Master project for the build  ===
  
 
The current master project for the JWT build is a project of type '''''General'''''<b>mvn-jwt-builder</b> found at&nbsp;[http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/mvn-jwt-builder/ http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/mvn-jwt-builder/].&nbsp;This project contains the master pom.xml for building the plug-ins. This pom.xml file contains all modules that are related to the build. Modules are the features and plugins to be built in addition to the p2update site of the project.&nbsp;The master pom.xml file also contains general information about the build like the project's repositories in addition to links to any external plugin dependencies the build might need.<br>  
 
The current master project for the JWT build is a project of type '''''General'''''<b>mvn-jwt-builder</b> found at&nbsp;[http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/mvn-jwt-builder/ http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/mvn-jwt-builder/].&nbsp;This project contains the master pom.xml for building the plug-ins. This pom.xml file contains all modules that are related to the build. Modules are the features and plugins to be built in addition to the p2update site of the project.&nbsp;The master pom.xml file also contains general information about the build like the project's repositories in addition to links to any external plugin dependencies the build might need.<br>  
  
== Add new Features/plugins to the build  ==
+
=== Add new Features/plugins to the build  ===
  
 
Future release engineers should add the paths to any new features/plugins in the master pom.xml file. In addition, they should also add any new plugins to one of the features and include any new features needed in the p2update site to the category definition of the p2update site project found at [[Future release engineers should add the paths to any new features/plugins in this file. In addition, they should also add any new plugins to one of the features and include any new features needed in the p2update site to the category definition of the p2update site project found at http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/.|http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/]].&nbsp;Please, note that any new features or plugins have to have their own pom.xml files as well as described in the tutorial recommended above.  
 
Future release engineers should add the paths to any new features/plugins in the master pom.xml file. In addition, they should also add any new plugins to one of the features and include any new features needed in the p2update site to the category definition of the p2update site project found at [[Future release engineers should add the paths to any new features/plugins in this file. In addition, they should also add any new plugins to one of the features and include any new features needed in the p2update site to the category definition of the p2update site project found at http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/.|http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/]].&nbsp;Please, note that any new features or plugins have to have their own pom.xml files as well as described in the tutorial recommended above.  
  
== P2 update Site project  ==
+
=== P2 update Site project  ===
  
The p2 update site project includes jars generated from the build and that are required for installing JWT. The current p2 update site project is a project of type '''''General'''''called&nbsp;'''org.eclipse.jwt.p2updatesite '''and can be found at&nbsp;[http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/ http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/]. This project also has it's own pom.xml file and it includes a profile that signs the JWT jars using the eclipse signature. This is prerequisite for eclipse projects and the details of signing the jars will be explained in the following section.  
+
The p2 update site project includes jars generated from the build and that are required for installing JWT. The current p2 update site project is a project of type '''''General&nbsp;'''''called&nbsp;'''org.eclipse.jwt.p2updatesite '''and can be found at&nbsp;[http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/ http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/]. This project also has it's own pom.xml file and it includes a profile that signs the JWT jars using the eclipse signature. This is prerequisite for eclipse projects and the details of signing the jars will be explained in the following section.<br>
  
<br>
+
=== Signing the JWT jars using the eclipse signature<br>  ===
 
+
== Signing the JWT jars using the eclipse signature<br>  ==
+
  
 
First, the proper maven.eclipse.org repository was added to the master pom.xml.&nbsp;<br>  
 
First, the proper maven.eclipse.org repository was added to the master pom.xml.&nbsp;<br>  
Line 30: Line 37:
 
  &lt;pluginRepositories&gt;
 
  &lt;pluginRepositories&gt;
 
  &lt;pluginRepository&gt;
 
  &lt;pluginRepository&gt;
  &lt;id&gt;maven.eclipse.org&lt;/id&gt;
+
&lt;id&gt;maven.eclipse.org&lt;/id&gt;
    &lt;url&gt;http://maven.eclipse.org/nexus/content/groups/public/&lt;/url&gt;
+
&lt;url&gt;http://maven.eclipse.org/nexus/content/groups/public/&lt;/url&gt;
 
  &lt;/pluginRepository&gt;
 
  &lt;/pluginRepository&gt;
 
  &lt;/pluginRepositories&gt;
 
  &lt;/pluginRepositories&gt;
Line 45: Line 52:
  
 
   &lt;profiles&gt;
 
   &lt;profiles&gt;
    &lt;profile&gt;
+
&lt;profile&gt;
      &lt;id&gt;build-server&lt;/id&gt;
+
&lt;id&gt;build-server&lt;/id&gt;
      &lt;build&gt;
+
&lt;build&gt;
        &lt;plugins&gt;
+
&lt;plugins&gt;
          &lt;plugin&gt;
+
&lt;plugin&gt;
            &lt;groupId&gt;org.eclipse.dash.maven&lt;/groupId&gt;
+
  &lt;groupId&gt;org.eclipse.dash.maven&lt;/groupId&gt;
            &lt;artifactId&gt;eclipse-signing-maven-plugin&lt;/artifactId&gt;
+
  &lt;artifactId&gt;eclipse-signing-maven-plugin&lt;/artifactId&gt;
            &lt;version&gt;1.0.4&lt;/version&gt;
+
  &lt;version&gt;1.0.4&lt;/version&gt;
            &lt;executions&gt;
+
  &lt;executions&gt;
              &lt;!--  
+
    &lt;!--  
              Pack the p2 repository.
+
    Pack the p2 repository.
              --&gt;
+
      --&gt;
              &lt;execution&gt;
+
    &lt;execution&gt;
                &lt;id&gt;pack&lt;/id&gt;
+
      &lt;id&gt;pack&lt;/id&gt;
                &lt;configuration&gt;
+
      &lt;configuration&gt;
                &lt;inputFile&gt;${project.build.directory}/org.eclipse.jwt.p2updatesite-1.3.0.qualifier.zip&lt;/inputFile&gt;
+
        &lt;inputFile&gt;${project.build.directory}/org.eclipse.jwt.p2updatesite-1.3.0.qualifier.zip&lt;/inputFile&gt;
              &lt;/configuration&gt;
+
      &lt;/configuration&gt;
                &lt;phase&gt;package&lt;/phase&gt;
+
      &lt;phase&gt;package&lt;/phase&gt;
                &lt;goals&gt;
+
      &lt;goals&gt;
                  &lt;goal&gt;pack&lt;/goal&gt;
+
        &lt;goal&gt;pack&lt;/goal&gt;
                &lt;/goals&gt;
+
      &lt;/goals&gt;
              &lt;/execution&gt;
+
    &lt;/execution&gt;
              &lt;!--  
+
    &lt;!--  
              Sign the p2 repository
+
    Sign the p2 repository
              --&gt;
+
    --&gt;
              &lt;execution&gt;
+
    &lt;execution&gt;
                &lt;id&gt;sign&lt;/id&gt;
+
      &lt;id&gt;sign&lt;/id&gt;
                &lt;configuration&gt;
+
      &lt;configuration&gt;
                  &lt;inputFile&gt;${project.build.directory}/org.eclipse.jwt.p2updatesite-1.3.0.qualifier.zip&lt;/inputFile&gt;
+
        &lt;inputFile&gt;${project.build.directory}/org.eclipse.jwt.p2updatesite-1.3.0.qualifier.zip&lt;/inputFile&gt;
                  &lt;signerInputDirectory&gt;/home/data/httpd/download-staging.priv/technology/jwt&lt;/signerInputDirectory&gt;
+
        &lt;signerInputDirectory&gt;/home/data/httpd/download-staging.priv/technology/jwt&lt;/signerInputDirectory&gt;
                &lt;/configuration&gt;
+
      &lt;/configuration&gt;
                &lt;phase&gt;package&lt;/phase&gt;
+
      &lt;phase&gt;package&lt;/phase&gt;
                &lt;goals&gt;
+
      &lt;goals&gt;
                  &lt;goal&gt;sign&lt;/goal&gt;
+
        &lt;goal&gt;sign&lt;/goal&gt;
                &lt;/goals&gt;
+
      &lt;/goals&gt;
              &lt;/execution&gt;
+
    &lt;/execution&gt;
              &lt;!--  
+
    &lt;!--  
              Repack the p2 repository
+
    Repack the p2 repository
              --&gt;
+
    --&gt;
              &lt;execution&gt;
+
    &lt;execution&gt;
                &lt;id&gt;repack&lt;/id&gt;
+
      &lt;id&gt;repack&lt;/id&gt;
                &lt;configuration&gt;
+
      &lt;configuration&gt;
                  &lt;inputFile&gt;${project.build.directory}/signed/site_assembly.zip&lt;/inputFile&gt; &lt;!-- this is output from signer mojo --&gt;
+
        &lt;inputFile&gt;${project.build.directory}/signed/site_assembly.zip&lt;/inputFile&gt; &lt;!-- this is output from signer mojo --&gt;
                &lt;/configuration&gt;
+
      &lt;/configuration&gt;
                &lt;phase&gt;package&lt;/phase&gt;
+
      &lt;phase&gt;package&lt;/phase&gt;
                &lt;goals&gt;
+
      &lt;goals&gt;
                  &lt;goal&gt;pack&lt;/goal&gt;
+
        &lt;goal&gt;pack&lt;/goal&gt;
                &lt;/goals&gt;
+
      &lt;/goals&gt;
              &lt;/execution&gt;
+
    &lt;/execution&gt;
              &lt;!--  
+
    &lt;!--  
              Signing and packing alters checksums so fix them
+
    Signing and packing alters checksums so fix them
              --&gt;
+
    --&gt;
              &lt;execution&gt;
+
    &lt;execution&gt;
                &lt;id&gt;fixCheckSums&lt;/id&gt;
+
      &lt;id&gt;fixCheckSums&lt;/id&gt;
                &lt;phase&gt;package&lt;/phase&gt;
+
      &lt;phase&gt;package&lt;/phase&gt;
                &lt;goals&gt;
+
      &lt;goals&gt;
                  &lt;goal&gt;fixCheckSums&lt;/goal&gt;
+
        &lt;goal&gt;fixCheckSums&lt;/goal&gt;
                &lt;/goals&gt;
+
      &lt;/goals&gt;
              &lt;/execution&gt;
+
    &lt;/execution&gt;
            &lt;/executions&gt;
+
  &lt;/executions&gt;
          &lt;/plugin&gt;
+
&lt;/plugin&gt;
          &lt;!--
+
&lt;/plugins&gt;
          This is what I use to deploy a p2 repository someplace to test from before manually making active.           
+
&lt;/build&gt;
          &lt;plugin&gt;
+
&lt;/profile&gt;
              &lt;artifactId&gt;maven-antrun-plugin&lt;/artifactId&gt;
+
&lt;/profiles&gt;
              &lt;executions&gt;
+
                &lt;execution&gt;
+
                  &lt;id&gt;deploy&lt;/id&gt;
+
                  &lt;phase&gt;install&lt;/phase&gt;
+
                  &lt;goals&gt;
+
                    &lt;goal&gt;run&lt;/goal&gt;
+
                  &lt;/goals&gt;
+
                  &lt;configuration&gt;
+
                    &lt;tasks&gt;
+
                      &lt;delete includeemptydirs="false"&gt;
+
                        &lt;fileset
+
                          dir="/home/data/httpd/download.eclipse.org/jwt/integration-update-site"&gt;
+
                          &lt;include name="**" /&gt;
+
                        &lt;/fileset&gt;
+
                      &lt;/delete&gt;
+
                      &lt;copy includeemptydirs="false"
+
                        todir="/home/data/httpd/download.eclipse.org/jwt/integration-update-site"&gt;
+
                        &lt;fileset dir="target/checksumFix"&gt;
+
                          &lt;include name="**" /&gt;
+
                        &lt;/fileset&gt;
+
                      &lt;/copy&gt;
+
                    &lt;/tasks&gt;
+
                  &lt;/configuration&gt;
+
                &lt;/execution&gt;
+
              &lt;/executions&gt;
+
            &lt;/plugin&gt;
+
            --&gt;
+
          &lt;/plugins&gt;
+
      &lt;/build&gt;
+
    &lt;/profile&gt;
+
  &lt;/profiles&gt;
+
 
   
 
   
  
=== Handle automated testing ===
+
== Configuring Hudson to build JWT ==
  
Another main point was to have the ability to run unit test on the just build JWT plugins, to avoid silent regressions. Our steps to do so were:
+
To build JWT on Hudson using Maven, a build step has to be added that uses Maven. The following screenshot presents the details of this build step as used for the Kepler release.  
  
*Copied an example tester (we used GEF and Platform ones as examples)
+
[[Image:Screenshot from 2013-05-07 02-42-20.png|Maven build step on Hudson for the Kepler release.]]
*Same strategy as for the builder: externalize strings from the test.xml to the builder build.properties file, and then adapt values to JWT
+
 
*Added the automated fetching of the test feature in the customTargets.xml  
+
As illustrated by the screenshot, the only build goals needed are "clean install". Moreover, the path to the master pom.xml file had to be specified in the POM file field. Finally, the signing profile added earlier had to be triggered to sign the jars using the eclipse signature. The profile added was called "build-server" and it is enough to simply write its name in the "Profiles" field.
*Added test plugins and features to the map file  
+
<br>
*In the master test plugin, copied a test.xml from another test ''plugin'' (not a test.xml from a tester)
+
 
*In the tester's test.xml, adapt the creation of the runtime we'll use for testing
+
Now everything is ready and the final step would be to click "Build now" from the main page of the build on Hudson.
**The test runtime is created dynamically as a copy of the build base
+
**And we add the JWT plugins and JWT tests plugins in it (expanding zips into the test runtime directory)
+
*Update the createUpdateSite.xml so that it copies test results into the update site
+
  
=== P2  ===
 
  
We encountered a problem with p2 file generation: Indeed, setting the generate.p2.metadata to true made metadate generated also for the test feature, whereas we do not expect to deliver them.  
+
== HIPP builds & Simultaneous Release ==
 +
An Eclipse.org [https://hudson.eclipse.org/jwt/ Hudson instance dedicated to building JWT] ([https://hudson.eclipse.org/ https://wiki.eclipse.org/Hudson#HIPP HIPP] - Hudson Instance Per Project) is available as [https://bugs.eclipse.org/bugs/show_bug.cgi?id=419955 requested on Bugzilla]. It features :
 +
* a "soa-jwt-integration" build, which builds JWT and makes it available in its own integration update site
 +
* and a "soa-jwt-luna" build, which directly promotes JWT built artifacts to the Eclipse Luna repository. Have a look at the Simultaneous Release [https://hudson.eclipse.org/hudson/job/simrel.luna.runaggregator Aggregator job] to check that it's been done successfully, for instance look for "jwt" in its console output ([https://hudson.eclipse.org/hudson/job/simrel.luna.runaggregator/546/consoleFull sample])
  
Then the solution was to set generate.p2.metadata to false, and then to run the p2 generator on the generated update site. The p2.generate ant task does not seem to support the ''site'' attribute, thus we simply used an Ant ''exec'' task calling the eclipse p2 generator application as we would have done with command-line.  
+
NB. Builds in the [common Hudson instance https://hudson.eclipse.org/hudson/] are obsolete.
  
=== What can still be done  ===
+
= <span style="line-height: 1.5em;">What can still be done</span> =
  
 
*Try to use a build/continuous integration server  
 
*Try to use a build/continuous integration server  
 
*Try to have nightly build  
 
*Try to have nightly build  
*Merge builder and tester: Move the test.xml from tester to builder, test and replace references to tester by builder.
+
*Merge builder and tester: Move the test.xml from tester to builder, test and replace references to tester by builder.  
 +
*Enable automatic deployment of JWT to the update site through after the build is run
 +
*Run JUnit tests on Hudson
 +
*Take care of the stable build on Hudson&nbsp;
  
== What we have ==
+
<br>
  
=== JWT build minimal Eclipse base ===
 
  
In order for the build and automated testing to be working as fast as possible, it is necessary to maintain a build base (in ''/shared/technology/jwt/base'' on build.eclipse.org), that contains only the projects that are necessary to build and test JWT, and to ensure that these project are built for for the build server you are using.
+
= OBSOLETE :  =
  
==== Creating the base with from repositories (recommended) ====
+
= What we have  =
  
P2 is the recommended way to install feature since it checks dependencies and avoids to forget some bundles that are necessary at runtime for testing.
+
=== JWT build minimal Eclipse base  ===
  
* Download a platform that is OK with you build host architecture => ''~/downloads/eclipse/downloads/[buildId]/eclipse-SDK-[version]-linux-gtk-ppc.tar.gz'' from [http://download.eclipse.org/eclipse/downloads/ http://download.eclipse.org/eclipse/downloads/]
+
In order for the build and automated testing to be working as fast as possible, it is necessary to maintain a build base (in ''/shared/technology/jwt/base'' on build.eclipse.org), that contains only the projects that are necessary to build and test JWT, and to ensure that these project are built for for the build server you are using.  
* Transfer to the /base directory on build.eclipse.org (e.g. with SCP) and expand
+
* Go into the new eclipse folder, then type
+
**  ''export LAUNCHER="java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.equinox.p2.director"''
+
** ''$LAUNCHER -metadataRepository http://download.eclipse.org/releases/galileo -artifactRepository http://download.eclipse.org/releases/galileo -installIU org.eclipse.pde.feature.group -installIU org.eclipse.jdt.feature.group -installIU org.eclipse.emf.feature.group -installIU org.eclipse.gef -installIU org.eclipse.m2m.atl.feature.group'' (dependency list to be updated following JWT deps)
+
  
In case you want to install a feature from a repository that is not p2-ready, you can use
+
==== Creating the base with from repositories (recommended)  ====
* ''java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.update.core.standaloneUpdate -command install -featureId [FEATURE_ID] -from [SITE_URL] -version [VERSION]''
+
  
==== Creating the base using drops (not fair) ====
+
P2 is the recommended way to install feature since it checks dependencies and avoids to forget some bundles that are necessary at runtime for testing.
  
To create a build base, you'll need to expand the following project in the directory you want to use ase the eclipse base:
+
*Download a platform that is OK with you build host architecture =&gt; ''~/downloads/eclipse/downloads/[buildId]/eclipse-SDK-[version]-linux-gtk-ppc.tar.gz'' from [http://download.eclipse.org/eclipse/downloads/ http://download.eclipse.org/eclipse/downloads/]  
* Eclipse platform linux/ppc/gtk => ''~/downloads/eclipse/downloads/[buildId]/eclipse-platform-[version]-linux-gtk-ppc.tar.gz''
+
*Transfer to the /base directory on build.eclipse.org (e.g. with SCP) and expand
* Eclipse PDE => ''~/downloads/eclipse/downloads/[buildId]/eclipse-PDE-[version].zip''
+
*Go into the new eclipse folder, then type
* Eclipse JDT => ''~/downloads/eclipse/downloads/[buildId]/eclipse-JDT-[version].zip''
+
**''export LAUNCHER="java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.equinox.p2.director"''  
* EMF Runtime => ''~/downloads/modeling/emf/emf/downloads/drops/[version]/[buildId]/emf-runtime-[buildId].zip''
+
**''$LAUNCHER -metadataRepository http://download.eclipse.org/releases/galileo -artifactRepository http://download.eclipse.org/releases/galileo -installIU org.eclipse.pde.feature.group -installIU org.eclipse.jdt.feature.group -installIU org.eclipse.emf.feature.group -installIU org.eclipse.gef -installIU org.eclipse.m2m.atl.feature.group'' (dependency list to be updated following JWT deps)
* GEF Runtime => ''~/downloads/tools/gef/downloads/drops/[version]/[buildId]/GEF-runtime-[buildId].zip''
+
* ATL Runtime (for transformations) => ''~/downloads/modeling/m2m/atl/downloads/drops/[version]/[buildId]/m2m-atl-runtime-[version].zip''
+
  
Choosing the version of each project:
+
In case you want to install a feature from a repository that is not p2-ready, you can use
* Currently, JWT build on Eclipse server uses the latest version that contains a Maintenance or Release build (ie buildId starting with R or M)
+
* However, we could discuss about trying to build usng stable or milestone builds, to ensure the compatibility with the latest versions of projects. But this would require much more work until we succeed to automate the creation of the base with the latest drops...
+
  
==== Check an existing base ====
+
*''java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.update.core.standaloneUpdate -command install -featureId [FEATURE_ID] -from [SITE_URL] -version [VERSION]''
  
There are 2 ways to check that an eclipse installation matches your machine platform:
+
==== Creating the base using drops (not fair) ====
* Try to run the eclipse launcher program (''./eclipse''). If eclipse starts, it is OK
+
* From the ''eclipse'' directory, try command ''ls plugins/ | grep org.eclipse.swt'', then you should get a list of plugins, and one of them specifies a os/ws/arch 3-uplet in its name. It must be match your server platform.
+
  
If you don't have the same eclipse target and server platforms, then you might be able to build, but testing will be impossible for you.
+
To create a build base, you'll need to expand the following project in the directory you want to use ase the eclipse base:
  
== How to use it ==
+
*Eclipse platform linux/ppc/gtk =&gt; ''~/downloads/eclipse/downloads/[buildId]/eclipse-platform-[version]-linux-gtk-ppc.tar.gz''
 +
*Eclipse PDE =&gt; ''~/downloads/eclipse/downloads/[buildId]/eclipse-PDE-[version].zip''
 +
*Eclipse JDT =&gt; ''~/downloads/eclipse/downloads/[buildId]/eclipse-JDT-[version].zip''
 +
*EMF Runtime =&gt; ''~/downloads/modeling/emf/emf/downloads/drops/[version]/[buildId]/emf-runtime-[buildId].zip''
 +
*GEF Runtime =&gt; ''~/downloads/tools/gef/downloads/drops/[version]/[buildId]/GEF-runtime-[buildId].zip''
 +
*ATL Runtime (for transformations) =&gt; ''~/downloads/modeling/m2m/atl/downloads/drops/[version]/[buildId]/m2m-atl-runtime-[version].zip''
  
=== Update sites organisation ===
+
Choosing the version of each project:
  
First, as it is the case in most of "one feature" projects (such as GEF), JWT now has 3 update sites:
+
*Currently, JWT build on Eclipse server uses the latest version that contains a Maintenance or Release build (ie buildId starting with R or M)  
* http://download.eclipse.org/technology/jwt/update-site is the update-site that contains the '''current release of JWT'''
+
*However, we could discuss about trying to build usng stable or milestone builds, to ensure the compatibility with the latest versions of projects. But this would require much more work until we succeed to automate the creation of the base with the latest drops...
* http://download.eclipse.org/technology/jwt/integration-update-site is the update-site the could contain any '''development or nightly build of JWT'''. Contents of this update-site should only be used by JWT contributors for testing.
+
* http://download.eclipse.org/technology/jwt/stable-update-site is the update-site for stable development build, and also for '''Galileo milestones'''
+
  
=== Building on build.eclipse.org ===
+
==== Check an existing base  ====
  
==== Requirements ====
+
There are 2 ways to check that an eclipse installation matches your machine platform:
  
Since testing requires the ability to start the eclipse UI, an X server must be started and the DISPLAY environment variable must be correctly set on the build server. See [[EMF Build Server Setup#Verify X Server|EMF advices about X configuration]] for more details.
+
*Try to run the eclipse launcher program (''./eclipse''). If eclipse starts, it is OK
 +
*From the ''eclipse'' directory, try command ''ls plugins/ | grep org.eclipse.swt'', then you should get a list of plugins, and one of them specifies a os/ws/arch 3-uplet in its name. It must be match your server platform.
  
On build.eclipse.org, you can use the Xvfb server started by root on localhost:1.0 (DISPLAY is automatically set to this value when running build.sh)
+
If you don't have the same eclipse target and server platforms, then you might be able to build, but testing will be impossible for you.
 
+
If you there is a problem with X server, you'll have the bahavior described above:
+
 
+
* In the build trace, the Java Result: 13 tells you that eclipse could not start
+
<source lang="text">
+
java-test:
+
    [echo] Running org.eclipse.jwt.we.tests.AllTests. Result file: /shared/technology/jwt/tester/results//org.eclipse.jwt.we.tests.AllTests.xml.
+
    [java] Java Result: 13
+
</source>
+
* In the result file, you see a desperately empty test result
+
<source lang="xml">
+
<?xml version="1.0" encoding="UTF-8" ?>
+
<testsuites />
+
</source>
+
* In the eclipse log (tester/results/org.eclipse.jwt.we.tests.AllTests.txt), you can see a message telling you that a problems occurs with display:
+
<source lang="text" >
+
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
+
        at org.eclipse.swt.SWT.error(SWT.java:3803)
+
</source>
+
 
+
==== Start a build ====
+
 
+
Anyone who has access to build.eclipse.org can now start a build of cvs HEAD by browsing to JWT home folder (''/shared/technology/jwt''), going into the ''builder'' folder and typing ''./build.sh [buildType]'' where build type is I (default), S or R, respectively for integration, stable or release build.
+
 
+
This script will start the whole build process and update the update-site for the specified build type.
+
 
+
Follow your build on [https://build.eclipse.org/hudson/view/JWT/ Hudson].
+
 
+
=== Start a build on your local machine ===
+
 
+
If you want to build locally JWT, simply checkout the org.eclipse.jwt/releng CVS module and write a script that starts the build with your local configuration (please '''do not modify build.sh or build.properties''', that are build.eclipse.org files). See build.mistria.openwide.sh for example.
+
  
=== Continuous integration ===
+
= Update sites organisation  =
  
'''TODO''' Note that the PDE build process supports unit testing. I've read about several Eclipse projects that used a continuous integration tool (such as cruise, continuum or hudson) to monitor their builds and test results. I'll try to choose if one of these solutions is fine for JWT.
+
First, as it is the case in most of "one feature" projects (such as GEF), JWT now has 3 update sites:
  
= Releasing Policy =
+
*http://download.eclipse.org/technology/jwt/update-site is the update-site that contains the '''current release of JWT'''
 +
*http://download.eclipse.org/technology/jwt/integration-update-site is the update-site the could contain any '''development or nightly build of JWT'''. Contents of this update-site should only be used by JWT contributors for testing.
 +
*http://download.eclipse.org/technology/jwt/stable-update-site is the update-site for stable development build, and also for '''Galileo milestones'''<br>
  
When releasing JWT, here are the steps to follow:
+
= Releasing Policy  =
# Tag the feature and all plugins included into the release with tag R{releaseVersion} (for example R0.5.0)
+
# Build on build.eclipse.org the feature to release (''./build.sh R''). The update site is automatically updated.
+
# Increment the feature version number (See [[Version Numbering]])
+
# Increment all plugins versions that will be modified in next release)
+
# Create a buzz about this release!
+
  
= Adding new plugins to the JWT release =
+
When releasing JWT, here are the steps to follow:
  
This section describes the steps to follow to add a new plugin to JWT release and get the build working with this add-ons. We'll use an example to facilitate the explanations.
+
#Tag the feature and all plugins included into the release with tag R{releaseVersion} (for example R0.5.0)
 +
#Build on build.eclipse.org the feature to release (''./build.sh R''). The update site is automatically updated.
 +
#Increment the feature version number (See [[Version Numbering]])
 +
#Increment all plugins versions that will be modified in next release)
 +
#Create a buzz about this release!
  
Assume that jwt-we (plugin) now depends on a new plugin (jwt.we.conf.model) that was not part of the automated build before. In such case, some things needs to be updated to build the jwt-we automatically, else you'll get a "missing bundle" error telling you that the build cannot find the jwt.we.conf.model.
+
<div><br></div>
* Ensure it conforms to Eclipse.org legal and technical requirements
+
** Plugin version must endswith .qualifier (see [[Version Numbering]])
+
** Message strings must be externalized and messaging must be based on NLS (See [[Message Bundle Conversion Tool]] and related docs)
+
** More generally, it must conform with Eclipse requirements (See list on [[Galileo Simultaneous Release]])
+
** Ensure it conforms to legal Eclipse requirements
+
*** Provider must be Eclipse.org
+
*** Plugin must include license and about files
+
*** License headers are necessary on all source files
+
*** More generally, see [http://www.eclipse.org/legal/]
+
* Add those plugins to the top-level feature (currently releng/jwt-feature) with version 0.0.0 (automatically chosen at build time)
+
* Add those plugins CVS path to the builder map file (currently releng/builder/maps/jwt.map)
+
* If your new plugin depends on other plugins (eg EMFT or M2M projects) that are not included in the build base, add its dependencies to the build base
+
* Add the test plugins as dependencies to the top-level test plugin (currently jwt-tests-plugin)
+
** Add them as dependencies to the MANIFEST.MF
+
** Add them into the AllTests testSuite of the plugin
+
** Add them to the top-level test feature (so that they will be fetched from CVS at build time, currently jwt-tests-feature, edit the feature.xml)
+
** Add those plugins CVS path to the builder map file (currently releng/maps/jwt.map)
+

Latest revision as of 10:02, 3 June 2014

Build of JWT Features and Plugins

This section presents information concerning the JWT 1.3.0 build for eclipse Kepler on Hudson and would be considered as a very useful resource for JWT's future release engineers (starting with Eclipse Luna scheduled for release in 2014). Some of the information/steps that follow might be not necessary for future JWT builds using Maven Tycho since they have already been done but we illustrate them anyway so that we would have a comprehensive and complete documentation of the whole building process. We start by explaining how to prepare the JWT project for the build and then follow by explaining how to configure Hudson to build JWT. Please, note that newly assigned release engineers might have to contact the eclipse webmaster (webmaster@eclipse.org) in case they are missing any permissions that they might require.

Helpful resources

The following link includes an Eclipse Tycho tutorial that would serve as a very useful resource demonstrating how to build Eclipse plug-ins, features and how to create Eclipse p2 update sites with Maven Tycho.

Preparing JWT to be built by Maven Tycho

This section provides information concerning preparing JWT to be built using Maven Tycho and preparing the p2 update site contents to be signed by the eclipse signature after the build is done and the jars generated.

Master project for the build

The current master project for the JWT build is a project of type Generalmvn-jwt-builder found at http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/mvn-jwt-builder/. This project contains the master pom.xml for building the plug-ins. This pom.xml file contains all modules that are related to the build. Modules are the features and plugins to be built in addition to the p2update site of the project. The master pom.xml file also contains general information about the build like the project's repositories in addition to links to any external plugin dependencies the build might need.

Add new Features/plugins to the build

Future release engineers should add the paths to any new features/plugins in the master pom.xml file. In addition, they should also add any new plugins to one of the features and include any new features needed in the p2update site to the category definition of the p2update site project found at [[Future release engineers should add the paths to any new features/plugins in this file. In addition, they should also add any new plugins to one of the features and include any new features needed in the p2update site to the category definition of the p2update site project found at http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/.%7Chttp://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/]]. Please, note that any new features or plugins have to have their own pom.xml files as well as described in the tutorial recommended above.

P2 update Site project

The p2 update site project includes jars generated from the build and that are required for installing JWT. The current p2 update site project is a project of type General called org.eclipse.jwt.p2updatesite and can be found at http://dev.eclipse.org/svnroot/soa/org.eclipse.jwt/trunk/releng/org.eclipse.jwt.p2updatesite/. This project also has it's own pom.xml file and it includes a profile that signs the JWT jars using the eclipse signature. This is prerequisite for eclipse projects and the details of signing the jars will be explained in the following section.

Signing the JWT jars using the eclipse signature

First, the proper maven.eclipse.org repository was added to the master pom.xml. 

<pluginRepositories>
<pluginRepository>
<id>maven.eclipse.org</id>
<url>http://maven.eclipse.org/nexus/content/groups/public/</url>
</pluginRepository>
</pluginRepositories>

After that, a profile was added to do the signing. To sign the jars, the following has to be done:

  1. Pack the p2 repository
  2. Sign the p2 repository
  3. Repack the p2 repository
  4. Fix checksums because signing and packing alters them.

The following profile performs the mentioned steps.

  <profiles>
<profile>
<id>build-server</id>
<build>
<plugins>
<plugin>
  <groupId>org.eclipse.dash.maven</groupId>
  <artifactId>eclipse-signing-maven-plugin</artifactId>
  <version>1.0.4</version>
  <executions>
    <!-- 
    Pack the p2 repository.
     -->
    <execution>
      <id>pack</id>
      <configuration>
       <inputFile>${project.build.directory}/org.eclipse.jwt.p2updatesite-1.3.0.qualifier.zip</inputFile>
     </configuration>
      <phase>package</phase>
      <goals>
        <goal>pack</goal>
      </goals>
    </execution>
    <!-- 
    Sign the p2 repository
    -->
    <execution>
      <id>sign</id>
      <configuration>
        <inputFile>${project.build.directory}/org.eclipse.jwt.p2updatesite-1.3.0.qualifier.zip</inputFile>
        <signerInputDirectory>/home/data/httpd/download-staging.priv/technology/jwt</signerInputDirectory>
      </configuration>
      <phase>package</phase>
      <goals>
        <goal>sign</goal>
      </goals>
    </execution>
    <!-- 
    Repack the p2 repository
    -->
    <execution>
      <id>repack</id>
      <configuration>
        <inputFile>${project.build.directory}/signed/site_assembly.zip</inputFile> <!-- this is output from signer mojo -->
      </configuration>
      <phase>package</phase>
      <goals>
        <goal>pack</goal>
      </goals>
    </execution>
    <!-- 
    Signing and packing alters checksums so fix them
    -->
    <execution>
      <id>fixCheckSums</id>
      <phase>package</phase>
      <goals>
        <goal>fixCheckSums</goal>
      </goals>
    </execution>
  </executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>

Configuring Hudson to build JWT

To build JWT on Hudson using Maven, a build step has to be added that uses Maven. The following screenshot presents the details of this build step as used for the Kepler release.

Maven build step on Hudson for the Kepler release.

As illustrated by the screenshot, the only build goals needed are "clean install". Moreover, the path to the master pom.xml file had to be specified in the POM file field. Finally, the signing profile added earlier had to be triggered to sign the jars using the eclipse signature. The profile added was called "build-server" and it is enough to simply write its name in the "Profiles" field.

Now everything is ready and the final step would be to click "Build now" from the main page of the build on Hudson.


HIPP builds & Simultaneous Release

An Eclipse.org Hudson instance dedicated to building JWT (https://wiki.eclipse.org/Hudson#HIPP HIPP - Hudson Instance Per Project) is available as requested on Bugzilla. It features :

  • a "soa-jwt-integration" build, which builds JWT and makes it available in its own integration update site
  • and a "soa-jwt-luna" build, which directly promotes JWT built artifacts to the Eclipse Luna repository. Have a look at the Simultaneous Release Aggregator job to check that it's been done successfully, for instance look for "jwt" in its console output (sample)

NB. Builds in the [common Hudson instance https://hudson.eclipse.org/hudson/] are obsolete.

What can still be done

  • Try to use a build/continuous integration server
  • Try to have nightly build
  • Merge builder and tester: Move the test.xml from tester to builder, test and replace references to tester by builder.
  • Enable automatic deployment of JWT to the update site through after the build is run
  • Run JUnit tests on Hudson
  • Take care of the stable build on Hudson 



OBSOLETE :

What we have

JWT build minimal Eclipse base

In order for the build and automated testing to be working as fast as possible, it is necessary to maintain a build base (in /shared/technology/jwt/base on build.eclipse.org), that contains only the projects that are necessary to build and test JWT, and to ensure that these project are built for for the build server you are using.

Creating the base with from repositories (recommended)

P2 is the recommended way to install feature since it checks dependencies and avoids to forget some bundles that are necessary at runtime for testing.

  • Download a platform that is OK with you build host architecture => ~/downloads/eclipse/downloads/[buildId]/eclipse-SDK-[version]-linux-gtk-ppc.tar.gz from http://download.eclipse.org/eclipse/downloads/
  • Transfer to the /base directory on build.eclipse.org (e.g. with SCP) and expand
  • Go into the new eclipse folder, then type
    • export LAUNCHER="java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.equinox.p2.director"
    • $LAUNCHER -metadataRepository http://download.eclipse.org/releases/galileo -artifactRepository http://download.eclipse.org/releases/galileo -installIU org.eclipse.pde.feature.group -installIU org.eclipse.jdt.feature.group -installIU org.eclipse.emf.feature.group -installIU org.eclipse.gef -installIU org.eclipse.m2m.atl.feature.group (dependency list to be updated following JWT deps)

In case you want to install a feature from a repository that is not p2-ready, you can use

  • java -jar plugins/org.eclipse.equinox.launcher_*.jar -application org.eclipse.update.core.standaloneUpdate -command install -featureId [FEATURE_ID] -from [SITE_URL] -version [VERSION]

Creating the base using drops (not fair)

To create a build base, you'll need to expand the following project in the directory you want to use ase the eclipse base:

  • Eclipse platform linux/ppc/gtk => ~/downloads/eclipse/downloads/[buildId]/eclipse-platform-[version]-linux-gtk-ppc.tar.gz
  • Eclipse PDE => ~/downloads/eclipse/downloads/[buildId]/eclipse-PDE-[version].zip
  • Eclipse JDT => ~/downloads/eclipse/downloads/[buildId]/eclipse-JDT-[version].zip
  • EMF Runtime => ~/downloads/modeling/emf/emf/downloads/drops/[version]/[buildId]/emf-runtime-[buildId].zip
  • GEF Runtime => ~/downloads/tools/gef/downloads/drops/[version]/[buildId]/GEF-runtime-[buildId].zip
  • ATL Runtime (for transformations) => ~/downloads/modeling/m2m/atl/downloads/drops/[version]/[buildId]/m2m-atl-runtime-[version].zip

Choosing the version of each project:

  • Currently, JWT build on Eclipse server uses the latest version that contains a Maintenance or Release build (ie buildId starting with R or M)
  • However, we could discuss about trying to build usng stable or milestone builds, to ensure the compatibility with the latest versions of projects. But this would require much more work until we succeed to automate the creation of the base with the latest drops...

Check an existing base

There are 2 ways to check that an eclipse installation matches your machine platform:

  • Try to run the eclipse launcher program (./eclipse). If eclipse starts, it is OK
  • From the eclipse directory, try command ls plugins/ | grep org.eclipse.swt, then you should get a list of plugins, and one of them specifies a os/ws/arch 3-uplet in its name. It must be match your server platform.

If you don't have the same eclipse target and server platforms, then you might be able to build, but testing will be impossible for you.

Update sites organisation

First, as it is the case in most of "one feature" projects (such as GEF), JWT now has 3 update sites:

Releasing Policy

When releasing JWT, here are the steps to follow:

  1. Tag the feature and all plugins included into the release with tag R{releaseVersion} (for example R0.5.0)
  2. Build on build.eclipse.org the feature to release (./build.sh R). The update site is automatically updated.
  3. Increment the feature version number (See Version Numbering)
  4. Increment all plugins versions that will be modified in next release)
  5. Create a buzz about this release!

Copyright © Eclipse Foundation, Inc. All Rights Reserved.