Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Modeling Project Releng/Building/Signing And Packing"

(UPDATE: Signed Sources: provide new example as Query is broken)
 
(20 intermediate revisions by 2 users not shown)
Line 7: Line 7:
 
2. You must be able to ssh from your build server to build.eclipse.org without being prompted for a host key or password. The first time you try to connect from your build server, accept the host key and you should never be prompted again. If you can ssh from your build server to dev.eclipse.org or download1.eclipse.org, you should be able to connect to build.eclipse.org using the same credentials and ssh key.
 
2. You must be able to ssh from your build server to build.eclipse.org without being prompted for a host key or password. The first time you try to connect from your build server, accept the host key and you should never be prompted again. If you can ssh from your build server to dev.eclipse.org or download1.eclipse.org, you should be able to connect to build.eclipse.org using the same credentials and ssh key.
  
3. While on build.eclipse.org, you must be able to create files in your staging folder, eg., /home/data/httpd/download-staging.priv/modeling/emft.
+
3. While on build.eclipse.org, you must be able to create files in your staging folder, eg., '''/home/data/httpd/download-staging.priv/modeling/emft'''.
  
4. While on build.eclipse.org, you must be able to run /usr/bin/jarsigner.
+
4. While on build.eclipse.org, you must be able to run '''/usr/bin/sign'''.
  
If you cannot do any of the above things, [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=community&component=Servers open a bug] and ask for access. cc: your PMC for approval and optionally, [[User:Nickb]].
+
If you cannot do any of the above things, [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=community&component=Servers open a bug] and ask for access: "''Please add user foo to group signers to permit jar signing for project bar''". cc: your PMC for approval and optionally, [[User:Nickb]].
  
 
== Process ==
 
== Process ==
  
1. Create an "all" or "master" feature, which will build SDK, examples, and any other features you might build. This obsoletes the need for an SDK builder, runtime builder, doc builder, and examples builder. You can still use your custom doc plugin builder; this just cleans up some redundancies in your .releng project.
+
=== All/Master Feature ===
 +
 
 +
Create an "all" or "master" feature, which will build SDK, examples, and any other features you might build. This obsoletes the need for an SDK builder, runtime builder, doc builder, and examples builder. You can still use your custom doc plugin builder; this just cleans up some redundancies in your .releng project.
 +
 
 +
If your SDK feature already contains all your smaller features (including runtime, source, doc, and examples, but not tests) then you don't need a new "all" feature.
  
 
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet/plugins/org.eclipse.jet.all-feature?root=Modeling_Project JET All-In-One ("Master") Feature]
 
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet/plugins/org.eclipse.jet.all-feature?root=Modeling_Project JET All-In-One ("Master") Feature]
 
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/features/org.eclipse.gef.all-feature?root=Tools_Project GEF All-In-One ("Master") Feature]
 
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/features/org.eclipse.gef.all-feature?root=Tools_Project GEF All-In-One ("Master") Feature]
 +
* Example 3: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.uml2/plugins/org.eclipse.uml2.sdk-feature?root=Modeling_Project UML2 SDK Feature] (already includes Examples)
 +
* Example 4: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools/features/org.eclipse.emf.ecoretools.all-feature/?root=Modeling_Project Ecore Tools All-In-One ("Master") Feature]
  
2. Add this new feature to your map file.
+
=== Map File ===
 +
 
 +
Add this new feature to your map file.
  
 
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/maps/jet.map?root=Modeling_Project JET map]
 
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/maps/jet.map?root=Modeling_Project JET map]
 
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/maps/gef.map?root=Tools_Project GEF map]
 
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/maps/gef.map?root=Tools_Project GEF map]
 +
* Example 3: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.uml2.releng/maps/uml2.map?root=Modeling_Project UML2 map] (no change)
 +
* Example 4: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/maps/?root=Modeling_Project Ecore Tools map]
  
3. Add a new builder/all/ folder in your .releng project.
+
=== All/Master Builder ===
 +
 
 +
Add a new builder/all/ folder in your .releng project. Your old builder/sdk, builder/doc, builder/examples folders are no longer required and can be deleted. As in step 1, if you're reusing your SDK feature, this step is not required. Just update your customTargets.xml to create the Master Zip.
  
 
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/builder/all/?root=Modeling_Project JET's /builder/all/]
 
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/builder/all/?root=Modeling_Project JET's /builder/all/]
 
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/builder/all/?root=Tools_Project GEF's /builder/all/]
 
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/builder/all/?root=Tools_Project GEF's /builder/all/]
 +
* Example 3: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.uml2.releng/builder/sdk/customTargets.xml?root=Modeling_Project&r1=1.5&r2=1.6 UML2's /builder/sdk/customTargets.xml]
 +
* Example 4: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/builder/all/?root=Modeling_Project Ecore Tools' /builder/all/]
  
4. Change your .releng/buildAll.xml to use new [http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/buildAllHelper.xml?root=Modeling_Project&view=markup signing/packing code]. You'll notice that many targets have been removed to buildAllHelper.xml to simplify this file, and some things have been reordered. The most important change is that you must now define your own packaging; however, by only doing a single PDE build for the whole "All" feature (and a second one for your tests), your build will run faster. This is important because jar signing can take a few minutes or as much as an hour, depending on the queue ahead of your build.
+
=== buildAll.xml ===
  
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/buildAll.xml?root=Modeling_Project&view=markup JET's buildAll.xml] [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/buildAll.xml?root=Modeling_Project&r1=1.12&r2=1.25 changes]
+
Change your .releng/buildAll.xml to use new [http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/buildAllHelper.xml?root=Modeling_Project&view=markup signing/packing code]. You'll notice that many targets have been removed to buildAllHelper.xml to simplify this file, and some things have been reordered. The most important change is that you must now define your own packaging; however, by only doing a single PDE build for the whole "All" feature (and a second one for your tests), your build will run faster. This is important because jar signing can take a few minutes or as much as an hour, depending on the queue ahead of your build.
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/buildAll.xml?root=Tools_Project&view=markup GEF's buildAll.xml] [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/buildAll.xml?root=Tools_Project&r1=1.8&r2=1.43 changes]
+
  
5. Add pack200="true" to all your update site's site*.xml files:
+
:<b style="color:red">NOTE!</b> This step may not be the most efficient way to collect all your build artifacts into one zip. Another approach is to continue to build multiple targets (SDK, runtime, examples, doc, tests) instead of just the single "all" build, then combine the results of those builds into a single zip. After signing, you would update your individual zips from the Master (signed) zip.  
  
{{codeblock|<nowiki><site pack200="true" ...></nowiki>}}
+
* Example 1: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/buildAll.xml?root=Modeling_Project&view=markup JET's buildAll.xml] [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/buildAll.xml?root=Modeling_Project&r1=1.12&r2=1.26 changes]
 +
* Example 2: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/buildAll.xml?root=Tools_Project&view=markup GEF's buildAll.xml] [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/buildAll.xml?root=Tools_Project&r1=1.8&r2=1.44 changes]
 +
* Example 3: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.uml2.releng/buildAll.xml?root=Modeling_Project&view=markup UML2's buildAll.xml] [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.uml2.releng/buildAll.xml?root=Modeling_Project&r1=1.27&r2=1.33 changes]
 +
* Example 4: [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/buildAll.xml?root=Modeling_Project&view=markup Ecore Tools' buildAll.xml] [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.emf/org.eclipse.emf.ecoretools.releng/buildAll.xml?root=Modeling_Project&r1=1.5&r2=1.16 changes]
  
6. Use new update site builder (coming soon) -- see {{bug|173651}}.
+
:<b style="color:red">NOTE!</b> Make sure that '''all''' your files are properly appearing the the newly-constructed zips. As discovered in {{bug_comment|173651|10}}, you need to be sure that your source plugins' MANIFEST.MF files are properly included.
 +
 
 +
{{codeblock|<nowiki><include name="**/org.eclipse.gef*" />
 +
<include name="**/org.eclipse.gef*/**" />
 +
<include name="**/org.eclipse.gef*/**/**" /></nowiki>}}
 +
 
 +
:<b style="color:red">NOTE!</b> Make sure that you don't accidentally add the same file to a zip (eg., rootfiles like eclipse/epl-v10.html) more than once. To prevent this, add the following attribute to your <zip> tasks ({{bugstrike|221590}}):
 +
 
 +
{{codeblock|<nowiki><zip ... duplicate="preserve"></nowiki>}}
 +
 
 +
Should you want to disable signing, comment out the "sign" variable declaration:
 +
 
 +
{{codeblock|<nowiki><!-- <property name="sign" value="true" /> DISABLED FOR NOW --></nowiki>}}
 +
 
 +
Or, instead of a <code><property></code>, use a <code><condition></code> so that you can enable signing only for certain build types (eg., to make N builds run faster):
 +
 
 +
{{codeblock|<nowiki><condition property="sign">
 +
  <or>
 +
    <!-- want to enable/disable signing for a given build type? add/remove types here -->
 +
    <equals arg1="${buildType}" arg2="I"/>
 +
    <equals arg1="${buildType}" arg2="M"/>
 +
    <equals arg1="${buildType}" arg2="S"/>
 +
    <equals arg1="${buildType}" arg2="R"/>
 +
  </or>
 +
</condition></nowiki>}}
 +
 
 +
=== promoteToEclipse.*.properties ===
 +
 
 +
Use the Master zip as input to the [http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/scripts/buildUpdateSite.sh?root=Modeling_Project&view=markup buildUpdateSite.sh] script, by updating your [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.m2t/org.eclipse.jet.releng/promoteToEclipse.jet.properties?root=Modeling_Project&view=markup promoteToEclipse.*.properties] [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/org.eclipse.gef.releng/promoteToEclipse.gef.properties?root=Tools_Project&view=markup file].
 +
 
 +
{{codeblock|<nowiki># zip to use for UM jar generation
 +
filePrefixesToUnzipArray=( "$projectName-$subprojectName-Master-" );</nowiki>}}
 +
:- or -
 +
{{codeblock|<nowiki># zip to use for UM jar generation
 +
filePrefixesToUnzipArray=( "$projectName-$subprojectName-Master-incubation-" );</nowiki>}}
 +
 
 +
=== UPDATE: Signed Sources ===
 +
 
 +
As discussed in {{bug|235734}}, the above process does not sign *everything* in the zips. So, to correct this, we need to reverse the flow so that we create update jars (features and plugins, all jarred) FIRST and zip contents (folders + jars) SECOND.
 +
 
 +
For the case of UML2, here are the two steps:
 +
 
 +
: 1. [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.uml2.releng/builder/sdk/build.properties?root=Modeling_Project&r1=1.1&r2=1.2 *.releng/builder/all/build.properties]
 +
 
 +
: 2. [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.mdt/org.eclipse.uml2.releng/buildAll.xml?root=Modeling_Project&r1=1.34&r2=1.38 *.releng/buildAll.xml]
 +
 
 +
=== And finally... ===
 +
 
 +
Promote as before, using build/promo.php or promoteToEclipse.sh. Or, to test creating an update site without publishing a build, you can use buildUpdateSite.sh without the -promote flag to build an update site locally on the build server in /var/www/html/.
 +
 
 +
== Testing / Install Verification ==
 +
 
 +
When you install your signed features from the update site, you should see something like this:
 +
 
 +
[[Image:Gef-3.4-signed-features.png]]
  
 
== Bugs ==
 
== Bugs ==
Line 54: Line 124:
 
* [[Pack200]]
 
* [[Pack200]]
 
* [[JarProcessor Options]]
 
* [[JarProcessor Options]]
 +
 +
 +
[[Category:Modeling]] [[Category:Releng]]

Latest revision as of 16:39, 9 June 2008

This document is a work in progress.

Requirements

1. You must have permission to ssh to build.eclipse.org. This is where the signing will occur.

2. You must be able to ssh from your build server to build.eclipse.org without being prompted for a host key or password. The first time you try to connect from your build server, accept the host key and you should never be prompted again. If you can ssh from your build server to dev.eclipse.org or download1.eclipse.org, you should be able to connect to build.eclipse.org using the same credentials and ssh key.

3. While on build.eclipse.org, you must be able to create files in your staging folder, eg., /home/data/httpd/download-staging.priv/modeling/emft.

4. While on build.eclipse.org, you must be able to run /usr/bin/sign.

If you cannot do any of the above things, open a bug and ask for access: "Please add user foo to group signers to permit jar signing for project bar". cc: your PMC for approval and optionally, User:Nickb.

Process

All/Master Feature

Create an "all" or "master" feature, which will build SDK, examples, and any other features you might build. This obsoletes the need for an SDK builder, runtime builder, doc builder, and examples builder. You can still use your custom doc plugin builder; this just cleans up some redundancies in your .releng project.

If your SDK feature already contains all your smaller features (including runtime, source, doc, and examples, but not tests) then you don't need a new "all" feature.

Map File

Add this new feature to your map file.

All/Master Builder

Add a new builder/all/ folder in your .releng project. Your old builder/sdk, builder/doc, builder/examples folders are no longer required and can be deleted. As in step 1, if you're reusing your SDK feature, this step is not required. Just update your customTargets.xml to create the Master Zip.

buildAll.xml

Change your .releng/buildAll.xml to use new signing/packing code. You'll notice that many targets have been removed to buildAllHelper.xml to simplify this file, and some things have been reordered. The most important change is that you must now define your own packaging; however, by only doing a single PDE build for the whole "All" feature (and a second one for your tests), your build will run faster. This is important because jar signing can take a few minutes or as much as an hour, depending on the queue ahead of your build.

NOTE! This step may not be the most efficient way to collect all your build artifacts into one zip. Another approach is to continue to build multiple targets (SDK, runtime, examples, doc, tests) instead of just the single "all" build, then combine the results of those builds into a single zip. After signing, you would update your individual zips from the Master (signed) zip.
NOTE! Make sure that all your files are properly appearing the the newly-constructed zips. As discovered in bug 173651 comment 10, you need to be sure that your source plugins' MANIFEST.MF files are properly included.
<include name="**/org.eclipse.gef*" />
<include name="**/org.eclipse.gef*/**" />
<include name="**/org.eclipse.gef*/**/**" />
NOTE! Make sure that you don't accidentally add the same file to a zip (eg., rootfiles like eclipse/epl-v10.html) more than once. To prevent this, add the following attribute to your <zip> tasks (bug 221590):
<zip ... duplicate="preserve">

Should you want to disable signing, comment out the "sign" variable declaration:

<!-- <property name="sign" value="true" /> DISABLED FOR NOW -->

Or, instead of a <property>, use a <condition> so that you can enable signing only for certain build types (eg., to make N builds run faster):

<condition property="sign">
  <or>
    <!-- want to enable/disable signing for a given build type? add/remove types here -->
    <equals arg1="${buildType}" arg2="I"/>
    <equals arg1="${buildType}" arg2="M"/>
    <equals arg1="${buildType}" arg2="S"/>
    <equals arg1="${buildType}" arg2="R"/>
  </or>
</condition>

promoteToEclipse.*.properties

Use the Master zip as input to the buildUpdateSite.sh script, by updating your promoteToEclipse.*.properties file.

# zip to use for UM jar generation
filePrefixesToUnzipArray=( "$projectName-$subprojectName-Master-" );
- or -
# zip to use for UM jar generation
filePrefixesToUnzipArray=( "$projectName-$subprojectName-Master-incubation-" );

UPDATE: Signed Sources

As discussed in bug 235734, the above process does not sign *everything* in the zips. So, to correct this, we need to reverse the flow so that we create update jars (features and plugins, all jarred) FIRST and zip contents (folders + jars) SECOND.

For the case of UML2, here are the two steps:

1. *.releng/builder/all/build.properties
2. *.releng/buildAll.xml

And finally...

Promote as before, using build/promo.php or promoteToEclipse.sh. Or, to test creating an update site without publishing a build, you can use buildUpdateSite.sh without the -promote flag to build an update site locally on the build server in /var/www/html/.

Testing / Install Verification

When you install your signed features from the update site, you should see something like this:

Gef-3.4-signed-features.png

Bugs

  • bug 173651: Support Pack200 in Modeling builds / update sites
  • bug 217929: M2T Signing & Packing
  • bug 217945: Problems with file permissions in signed/packed jars
  • bug 211613: JDK 6's Pack200 is not backward compatible with JDK 5 if packed jars contain no classfiles

References

Back to the top