Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Tycho/Reproducible Version Qualifiers"

m
m (Feature version qualifiers)
 
(20 intermediate revisions by 10 users not shown)
Line 4: Line 4:
 
* Generate the same version qualifier when building from the same git commit.
 
* Generate the same version qualifier when building from the same git commit.
 
* Guarantee consistent artifact id, version and contents.
 
* Guarantee consistent artifact id, version and contents.
 +
* Available since Tycho 0.16
  
 
=== Configuration (just copy&paste!) ===
 
=== Configuration (just copy&paste!) ===
Line 28: Line 29:
 
             <!--
 
             <!--
 
               Do not consider changes to pom.xml file when calculating most recent git commit.  
 
               Do not consider changes to pom.xml file when calculating most recent git commit.  
 +
              Avoid failure from uncommitted temporary polyglot files.
 
             -->
 
             -->
 
             <jgit.ignore>
 
             <jgit.ignore>
 
               pom.xml
 
               pom.xml
 +
              .polyglot.build.properties
 
             </jgit.ignore>
 
             </jgit.ignore>
 
           </configuration>
 
           </configuration>
Line 51: Line 54:
 
           </configuration>
 
           </configuration>
 
         </plugin>
 
         </plugin>
 +
      </plugins>
 
     </pluginManagement>
 
     </pluginManagement>
  
Line 56: Line 60:
  
 
Version qualifier is calculated based on timestamp of the most recent commit that touches any file under project base directory.
 
Version qualifier is calculated based on timestamp of the most recent commit that touches any file under project base directory.
 +
As of Tycho 0.19.0, by default if any uncommitted changes are detected, the build will fail (see <tt>jgit.dirtyWorkingTree</tt> option below).
 
This produces stable version qualifier when building projects from HEAD of a branch and when building projects
 
This produces stable version qualifier when building projects from HEAD of a branch and when building projects
 
from a tagged or any commit in general. To put it differently, version qualifier reflects build contents, not build timestamp.  
 
from a tagged or any commit in general. To put it differently, version qualifier reflects build contents, not build timestamp.  
Line 63: Line 68:
 
Generated artifacts are then compared to artifacts available from configured baseline repositories.
 
Generated artifacts are then compared to artifacts available from configured baseline repositories.
 
If baseline repositories contain artifacts with the same id and version and equal contents, the generated artifacts are replaced with baseline version.
 
If baseline repositories contain artifacts with the same id and version and equal contents, the generated artifacts are replaced with baseline version.
If baseline repositories contain artifacts with the same id and version but different contents, depending on <stringBaseline>
+
If baseline repositories contain artifacts with the same id and version but different contents, depending on <tt><baselineMode></tt>
 
configuration parameter the build will either fail or generated artifacts will be replaced with baseline version.
 
configuration parameter the build will either fail or generated artifacts will be replaced with baseline version.
 
If the base repository does not contain artifact with the same id and version, the generated artifacts are used.
 
If the base repository does not contain artifact with the same id and version, the generated artifacts are used.
Line 73: Line 78:
 
=== Rebuilding from tag or, in fact, any commit ===
 
=== Rebuilding from tag or, in fact, any commit ===
  
Use -Dtycho.baseline.replace=none command line parameter to make sure Tycho does not replace build artifacts with
+
Use <tt>-Dtycho.baseline.replace=none</tt> command-line parameter to make sure Tycho does not replace build artifacts with
 
their corresponding baseline version. Tycho will still validate new build artifacts match corresponding artifacts from
 
their corresponding baseline version. Tycho will still validate new build artifacts match corresponding artifacts from
 
original build and depending on tycho.baseline configuration will either warn or fail the build if there are any  
 
original build and depending on tycho.baseline configuration will either warn or fail the build if there are any  
 
discrepancies.
 
discrepancies.
  
 +
 +
=== Dirty working tree behaviour ===
 +
 +
As of Tycho 0.19.0, by default if any uncommitted changes are detected, the build will fail.
 +
Configuration option <tt>jgit.dirtyWorkingTree</tt> of <tt>tycho-packaging-plugin</tt> allows to control behaviour in case of uncommitted changes detected for a module:
 +
 +
* <tt>error</tt> (default): fail the build if git status is not clean. This is useful for production builds to make sure the commit timestamp build qualifier used is really reproducible
 +
* <tt>warning</tt>: warn about any uncommitted changes and use the current build timestamp as build qualifier for the "dirty" module. This is useful for local builds with uncommitted changes.
 +
* <tt>ignore</tt>: do not check whether git status is clean and use the timestamp of the last git commit as build qualifier regardless of uncommitted changes done on top. This can be used in case jgit should report a false positive dirty working tree or in case you want to restore pre-0.19.0 behaviour
  
 
=== Feature version qualifiers ===
 
=== Feature version qualifiers ===
  
For feature projects, version qualifiers are generated from the last of the timestamp of the most recent common that touches any file under feature project basedir and timestamps of directly included bundles and features. Timestamps of included bundles and features and derived from their version qualifiers and any qualifier that cannot be interpreted as timestamp is ignored.
+
For feature projects, version qualifiers are generated from the last of the timestamp of the most recent commit that touches any file under feature project basedir and timestamps of directly included bundles and features. Timestamps of included bundles and features are derived from their version qualifiers. Besides the configured build qualifier timestamp format, the following timestamp formats occuring anywhere inside the qualifier are recognized since Tycho 0.25.0:
 +
 
 +
* <code>yyyyMMddHHmm</code>
 +
* <code>yyyyMMdd-HHmm</code>
 +
* <code>yyyyMMdd</code>
 +
 
 +
Any qualifier that cannot be interpreted as timestamp is ignored.
  
 
For the majority of feature projects that generates stable version qualifier that changes when feature contents changes, however manual "bump-version" commit in feature project will be necessary if included bundle/feature changes but its version qualifier cannot be interpreted and therefor ignored.
 
For the majority of feature projects that generates stable version qualifier that changes when feature contents changes, however manual "bump-version" commit in feature project will be necessary if included bundle/feature changes but its version qualifier cannot be interpreted and therefor ignored.
 +
 +
'''External includes in features'''
 +
 +
Please note that if a feature directly includes bundles or features not part of the same code base but consumed via a target platform, the stability of a reproducible feature version qualifier also depends on the stability of the target platform in addition to the stability of the code base. If a rebuild occurs with a change in target platform, a different feature version qualifier might be generated.
  
 
=== Other notes and hints. ===
 
=== Other notes and hints. ===
  
Do not use -SNAPSHOT dependencies resolved from Maven repositories. This is a good idea in general. For Tycho projects
+
* Do not use -SNAPSHOT dependencies resolved from Maven repositories. Using RELEASE versions of all external dependencies for Tycho projects
 
that use reproducible version qualifier this will allow calling any build a fully reproducible release build.
 
that use reproducible version qualifier this will allow calling any build a fully reproducible release build.
 +
* Maven <tt>-X</tt> command line option will print list of all files that differ between baseline and build.
 +
* <tt>-Dtycho.debug.artifactcomparator</tt> can be used to log disassembly of class files that are different between baseline and build. Disassembled class files are written under project target/ directory. The files have "-baseline" and "-build" suffixes and can be compared using 'diff' command line utility of 'Compare with each other' in Eclipse IDE.
 +
* You can optionally use the [https://www.eclipse.org/tycho/sitedocs/tycho-extras/tycho-p2-extras-plugin/compare-version-with-baselines-mojo.html compare-version-with-baselines mojo] on the verify step to add some checks and guarantee that version seems consistent with baselines (either strictly higher or same fully-qualified versions and same content). The baseline set to this mojo is expected to be a ''release'' of your p2 repository.
 +
* If there is more than one file you want to "ignore", it must be written in a line-delimited ".gitignore" style, with no leading whitespace, as in the following example. But, this may change in a future version, see {{bug|436325}}.
 +
 +
<pre><nowiki>
 +
<jgit.ignore>pom.xml
 +
buildnotes_*.html</jgit.ignore>
 +
</nowiki></pre>
  
  
 
[[Category:Tycho]]
 
[[Category:Tycho]]

Latest revision as of 07:36, 15 March 2022

"Executive" summary

  • Only produce new artifact version when there are changes.
  • Generate the same version qualifier when building from the same git commit.
  • Guarantee consistent artifact id, version and contents.
  • Available since Tycho 0.16

Configuration (just copy&paste!)

   <pluginManagement>
     <plugins>
       <plugin>
         <groupId>org.eclipse.tycho</groupId>
         <artifactId>tycho-packaging-plugin</artifactId>
         <version>${tycho.version}</version>
         <dependencies>
           <dependency>
             <groupId>org.eclipse.tycho.extras</groupId>
             <artifactId>tycho-buildtimestamp-jgit</artifactId>
             <version>${tycho-extras.version}</version>
           </dependency>
         </dependencies>
         <configuration>
           <timestampProvider>jgit</timestampProvider>
           <jgit.ignore>
             pom.xml
             .polyglot.build.properties
           </jgit.ignore>
         </configuration>
       </plugin>
       <plugin>
         <groupId>org.eclipse.tycho</groupId>
         <artifactId>tycho-p2-plugin</artifactId>
         <version>${tycho.version}</version>
         <configuration>
           <baselineRepositories>
             <repository>
               <url>[some url]</url>
             </repository>
           </baselineRepositories>
         </configuration>
       </plugin>
     </plugins>
   </pluginManagement>

What does this actually do?

Version qualifier is calculated based on timestamp of the most recent commit that touches any file under project base directory. As of Tycho 0.19.0, by default if any uncommitted changes are detected, the build will fail (see jgit.dirtyWorkingTree option below). This produces stable version qualifier when building projects from HEAD of a branch and when building projects from a tagged or any commit in general. To put it differently, version qualifier reflects build contents, not build timestamp. Although as of 2012-06-30 this is only implemented for git, it should be possible to provide implementation for other modern version control systems.

Generated artifacts are then compared to artifacts available from configured baseline repositories. If baseline repositories contain artifacts with the same id and version and equal contents, the generated artifacts are replaced with baseline version. If baseline repositories contain artifacts with the same id and version but different contents, depending on <baselineMode> configuration parameter the build will either fail or generated artifacts will be replaced with baseline version. If the base repository does not contain artifact with the same id and version, the generated artifacts are used. In all three cases the same artifact id/version are guaranteed to represent the same artifact contents.

The end result is that build output contains new versions of the artifacts that did change and baseline version of the artifacts that did not.

Rebuilding from tag or, in fact, any commit

Use -Dtycho.baseline.replace=none command-line parameter to make sure Tycho does not replace build artifacts with their corresponding baseline version. Tycho will still validate new build artifacts match corresponding artifacts from original build and depending on tycho.baseline configuration will either warn or fail the build if there are any discrepancies.


Dirty working tree behaviour

As of Tycho 0.19.0, by default if any uncommitted changes are detected, the build will fail. Configuration option jgit.dirtyWorkingTree of tycho-packaging-plugin allows to control behaviour in case of uncommitted changes detected for a module:

  • error (default): fail the build if git status is not clean. This is useful for production builds to make sure the commit timestamp build qualifier used is really reproducible
  • warning: warn about any uncommitted changes and use the current build timestamp as build qualifier for the "dirty" module. This is useful for local builds with uncommitted changes.
  • ignore: do not check whether git status is clean and use the timestamp of the last git commit as build qualifier regardless of uncommitted changes done on top. This can be used in case jgit should report a false positive dirty working tree or in case you want to restore pre-0.19.0 behaviour

Feature version qualifiers

For feature projects, version qualifiers are generated from the last of the timestamp of the most recent commit that touches any file under feature project basedir and timestamps of directly included bundles and features. Timestamps of included bundles and features are derived from their version qualifiers. Besides the configured build qualifier timestamp format, the following timestamp formats occuring anywhere inside the qualifier are recognized since Tycho 0.25.0:

  • yyyyMMddHHmm
  • yyyyMMdd-HHmm
  • yyyyMMdd

Any qualifier that cannot be interpreted as timestamp is ignored.

For the majority of feature projects that generates stable version qualifier that changes when feature contents changes, however manual "bump-version" commit in feature project will be necessary if included bundle/feature changes but its version qualifier cannot be interpreted and therefor ignored.

External includes in features

Please note that if a feature directly includes bundles or features not part of the same code base but consumed via a target platform, the stability of a reproducible feature version qualifier also depends on the stability of the target platform in addition to the stability of the code base. If a rebuild occurs with a change in target platform, a different feature version qualifier might be generated.

Other notes and hints.

  • Do not use -SNAPSHOT dependencies resolved from Maven repositories. Using RELEASE versions of all external dependencies for Tycho projects

that use reproducible version qualifier this will allow calling any build a fully reproducible release build.

  • Maven -X command line option will print list of all files that differ between baseline and build.
  • -Dtycho.debug.artifactcomparator can be used to log disassembly of class files that are different between baseline and build. Disassembled class files are written under project target/ directory. The files have "-baseline" and "-build" suffixes and can be compared using 'diff' command line utility of 'Compare with each other' in Eclipse IDE.
  • You can optionally use the compare-version-with-baselines mojo on the verify step to add some checks and guarantee that version seems consistent with baselines (either strictly higher or same fully-qualified versions and same content). The baseline set to this mojo is expected to be a release of your p2 repository.
  • If there is more than one file you want to "ignore", it must be written in a line-delimited ".gitignore" style, with no leading whitespace, as in the following example. But, this may change in a future version, see bug 436325.
 <jgit.ignore>pom.xml
 buildnotes_*.html</jgit.ignore>

Back to the top