Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Platform-releng/Juno Git Migration"

(Current state)
(Process)
(39 intermediate revisions by 7 users not shown)
Line 3: Line 3:
 
= Current state =
 
= Current state =
  
* We have been doing test builds fetching the org.eclipse.equinox.p2.* bundles from the git mirror automatically created by the foundation
+
Eclipse project migration to Git is complete. For all the details see the plan item {{bug|345479}}.
* This mirror is missing some tag/branch information so isn't reliable for test builds
+
 
* Denis set up area where we can run test migrations to Git
+
* Some common workflows are recorded in [[Platform-releng/Git Workflows]]
* Kim has run a test migration which she will then run a test build on
+
* Still need to figure out how to exclude binaries from the migration to Git step
+
* Then we can do a full test migration of all repositories and run a complete build
+
  
 
= Issues blocking migration =
 
= Issues blocking migration =
Line 18: Line 15:
 
* Equinox launcher and security fragments
 
* Equinox launcher and security fragments
 
* Misc platform binaries such as file system, resources
 
* Misc platform binaries such as file system, resources
 +
* Archived examples and samples
 
These binaries bloat the repository which will make cloning repositories painful for developers. We decided on the following action:
 
These binaries bloat the repository which will make cloning repositories painful for developers. We decided on the following action:
 
* Binaries are instead published to a p2 repository (the ''native binaries repository'')
 
* Binaries are instead published to a p2 repository (the ''native binaries repository'')
 
* Build consumes binaries from p2 repositories and source from Git
 
* Build consumes binaries from p2 repositories and source from Git
 
* These binary p2 repositories are only used as build input and not intended for wider consumption
 
* These binary p2 repositories are only used as build input and not intended for wider consumption
* Retention policy likely to be a single release for these binary repositories. Can always regenerate old binaries from source in the future so no need to retain them forever
+
* Retention policy likely to be a single release for these binary repositories. Can always regenerate old binaries from source in the future so no need to retain them forever (NOTE: Unfortunately, the statement "Can always regenerate old binaries from source in the future so no need to retain them forever" is almost certainly false. Suggest that we keep all milestone and release binaries indefinitely)
 +
* For developers at dev time, the source projects can have an Ant builder on them which will automatically fetch the corresponding binaries for the source project
  
 
Bootstrap process for the base builder becomes:
 
Bootstrap process for the base builder becomes:
 
* wget a stable platform runtime binary (milestone)
 
* wget a stable platform runtime binary (milestone)
 
* invoke command line p2 director to get the base builder from milestone repository
 
* invoke command line p2 director to get the base builder from milestone repository
* We will need a base builder product (bug ref??)
+
* We will need a base builder product ([https://bugs.eclipse.org/324682 bug 324682])
  
 
SWT build process:
 
SWT build process:
Line 39: Line 38:
 
== Git Ant task ==
 
== Git Ant task ==
 
We need a Git Ant task that invokes JGit for fetching SWT source as part of SWT hudson build. See {{bug|321237}}. This is apparently done (need to confirm it is available and works for us on all platforms).
 
We need a Git Ant task that invokes JGit for fetching SWT source as part of SWT hudson build. See {{bug|321237}}. This is apparently done (need to confirm it is available and works for us on all platforms).
 +
 +
== Story for maintenance streams ==
 +
 +
We need to maintain a CVS repository for performing builds on older branches (3.6.2+, 3.5.2+, etc). Until we have a solution for this, we will need to keep our CVS repository writeable after the Git migration. We can delete all content from HEAD in CVS to avoid confusion.
 +
 +
One option we are considering for maintenance is having a read-only CVS repository that mirrors our Git repository. We could then push changes to Git for maintenance streams, and our old builders can pick the change up from the CVS mirror/clone. We need to work with the Foundation staff to see if this is feasible ( [https://bugs.eclipse.org/bugs/show_bug.cgi?id=348976 bug 34876]).
 +
 +
Current thinking is that 3.7.x is built from Git. This means we need to proof that we can build 3.7.1 against Git before we do the migration ( [https://bugs.eclipse.org/bugs/show_bug.cgi?id=349460 bug 349460]).
 +
 +
== Migration script ==
 +
 +
We need to gather various bits of information to plug into the CVS->Git migration tool:
 +
* Need author information for all committers past and present [https://bugs.eclipse.org/bugs/show_bug.cgi?id=349350 bug 349350]
 +
* Figure out the CVS module -> Git repository mapping for all CVS modules
 +
* Figure out what binaries to exclude from migration, and how to exclude them
 +
* Determine what legacy CVS modules we can omit from migration altogether
 +
* [[Platform-releng/Juno_Git_Migration_Recipe | Steps to test the migration]].
  
 
= Resolved issues =
 
= Resolved issues =
  
== Git repository granularity ==
+
== Git repository granularity ==
 +
 
 +
We decided on a single Git repository per ACL. One common Eclipse TLP repository for doc and map files. See [https://bugs.eclipse.org/bugs/show_bug.cgi?id=345471 bug 345471] for the discussion.
 +
 
 +
The same holds for Equinox, although Tom considering folding security component into bundles due to lack of active committers. The repositories for Equinox will be:
  
We decided on a single Git repository per ACL. One common Eclipse TLP repository for doc and map files.
+
*Framework
 +
*Bundles
 +
*p2
 +
*Security (maybe)
 +
*Incubator
  
The same holds for Equinox, although Tom considering folding security component into bundles due to lack of active committers. The repositories for Equinox will be:
+
<br> The&nbsp; [http://wiki.eclipse.org/Platform-releng/Git_Migration_Granularity Git_Migration_Granularity] page documents the proposed bundles per repo.&nbsp; (Nested bundles are assumed to be included by indicating the top level directory).
* Framework
+
* Bundles
+
* p2
+
* Security (maybe)
+
* Incubator
+
  
 
== Tagging process ==
 
== Tagging process ==
Line 57: Line 76:
 
* If you don't want to contribute to build, put it in another branch
 
* If you don't want to contribute to build, put it in another branch
 
* Eliminate the manual tagging step taken by all teams today
 
* Eliminate the manual tagging step taken by all teams today
 +
* builds should still be map file based
 +
* We use tagging for two purposes: 1. reproducible builds and 2. bundle version numbers.  We need to ensure that if a bundle's contents doesn't change, that the version number doesn't change. Paul W. had some thoughts on how this will be done by actually looking at the commit history.
  
 
= Issues for after migration =
 
= Issues for after migration =
 
== 3.x/4.x migration ==
 
== 3.x/4.x migration ==
 
* Renaming of e4 bundles
 
* Renaming of e4 bundles
* Stitching in R4 branch of org.eclipse.ui.workbench that is in separate location
+
* Stitching in R4 branch of org.eclipse.ui.workbench that is in separate location {{bug|349177}}
  
 
== Including eGit in Eclipse SDK ==
 
== Including eGit in Eclipse SDK ==
Line 67: Line 88:
 
Traditionally we have included everything in the SDK needed to build the SDK. This implies including EGit/JGit in the Eclipse SDK. However this introduces a cyclic build dependency which always causes problems and introduces end-game risk. We need to reconsider this policy of including everything we need. Using p2 having multiple "root" features installed and updating them on an ongoing basis is no longer painful. The advantage of putting everything into a single build is perhaps outweighed by the pain of build-time dependency entanglement.
 
Traditionally we have included everything in the SDK needed to build the SDK. This implies including EGit/JGit in the Eclipse SDK. However this introduces a cyclic build dependency which always causes problems and introduces end-game risk. We need to reconsider this policy of including everything we need. Using p2 having multiple "root" features installed and updating them on an ongoing basis is no longer painful. The advantage of putting everything into a single build is perhaps outweighed by the pain of build-time dependency entanglement.
  
Recommendation: Remove all VCM support from the Eclipse SDK and users install what they need separately.
+
Recommendation: Remove all VCM support (including CVS) from the Eclipse SDK and users install what they need separately.
  
Need to discuss this at the PMC call and architecture meeting to find consensus
+
'''edit''' After excluding CVS from the product, the SDK build could still install it and the EGit feature using the director.
 +
 
 +
This was discussed at the June 15, 2011 [[Eclipse/PMC|PMC Meeting]].
  
 
== Git fetch factory ==
 
== Git fetch factory ==
  
 
The Git fetch factory still uses command line git. This is not yet ported to JGit but that's not blocking us because our builds all run on a single platform (Linux). Unclear who will do this and when it will happen. In the long term this will be beneficial to remove a specific platform dependency from our build process.
 
The Git fetch factory still uses command line git. This is not yet ported to JGit but that's not blocking us because our builds all run on a single platform (Linux). Unclear who will do this and when it will happen. In the long term this will be beneficial to remove a specific platform dependency from our build process.
 +
 +
It was ant tasks that were holding this up.  {{bug|349056}}
  
 
== Source headers ==
 
== Source headers ==
Line 79: Line 104:
 
Eclipse-Source-Header support for Git still missing. This is needed for Juno but not for Git migration. Consider PDE plan item to support this (possibly in collaboration with EGit team)
 
Eclipse-Source-Header support for Git still missing. This is needed for Juno but not for Git migration. Consider PDE plan item to support this (possibly in collaboration with EGit team)
  
= Timing =
+
See {{bug|327381}}
 +
 
 +
== Copyright Tool ==
 +
The copyright tool may need to be updated for integration with git.
 +
 
 +
See {{bug|345669}}
 +
 
 +
== Git Ignore file ==
 +
 
 +
All of the repos should probably commit a '''.gitignore''' file at the base of their repositories.  I would imagine that it needs to be done either on each branch that the component thinks is important going forward, or perhaps added as the root commit ''after'' the CVS to Git migration has been verified.
 +
 
 +
An example '''.gitignore''' file:
 +
 
 +
 
 +
bin/
 +
*~
 +
*.rej
 +
*.bak
 +
core
 +
*.so
 +
*.patch
 +
javacore.*
 +
heapdump.*
 +
core.*
 +
Snap.*
 +
target/
 +
 
  
June 13-24:
 
* Run test migrations and test builds
 
* SWT investigating migration of SWT builder
 
June 27-July 1:
 
* Migrate Equinox repositories to Git
 
July 11-15
 
* Migrate Eclipse TLP repositories
 
  
Remove org.eclipse.team.cvs feature from SDK?
+
= Process =
  
CVS mirror of Git for maintenance streams
+
The process that a component can expect during the git migration is described in this document
  
Leave CVS writable until we figure out 3.4.2+ story
+
[[Platform-releng/Juno_Git_Migration_Process]]
  
Need author information for all committers past and present for migration tool
+
[[Category:Eclipse_Platform_Releng| ]]

Revision as of 16:57, 15 March 2012

The Eclipse top-level project is investigating migration to Git during the summer of 2011. Development towards both Juno and Indigo SR1 would continue in Git after migration

Current state

Eclipse project migration to Git is complete. For all the details see the plan item bug 345479.

Issues blocking migration

Binaries in repositories

We currently check binaries into CVS in several places:

  • Base builder
  • SWT fragments
  • Equinox launcher and security fragments
  • Misc platform binaries such as file system, resources
  • Archived examples and samples

These binaries bloat the repository which will make cloning repositories painful for developers. We decided on the following action:

  • Binaries are instead published to a p2 repository (the native binaries repository)
  • Build consumes binaries from p2 repositories and source from Git
  • These binary p2 repositories are only used as build input and not intended for wider consumption
  • Retention policy likely to be a single release for these binary repositories. Can always regenerate old binaries from source in the future so no need to retain them forever (NOTE: Unfortunately, the statement "Can always regenerate old binaries from source in the future so no need to retain them forever" is almost certainly false. Suggest that we keep all milestone and release binaries indefinitely)
  • For developers at dev time, the source projects can have an Ant builder on them which will automatically fetch the corresponding binaries for the source project

Bootstrap process for the base builder becomes:

  • wget a stable platform runtime binary (milestone)
  • invoke command line p2 director to get the base builder from milestone repository
  • We will need a base builder product (bug 324682)

SWT build process:

  • SWT has a hudson build for generating their native binaries
  • Builder will fetch native source from Git using Git Ant task
  • Builder will push resulting fragment to the Eclipse project native binaries repository

Equinox launcher natives:

  • Similar to SWT above

Git Ant task

We need a Git Ant task that invokes JGit for fetching SWT source as part of SWT hudson build. See bug 321237. This is apparently done (need to confirm it is available and works for us on all platforms).

Story for maintenance streams

We need to maintain a CVS repository for performing builds on older branches (3.6.2+, 3.5.2+, etc). Until we have a solution for this, we will need to keep our CVS repository writeable after the Git migration. We can delete all content from HEAD in CVS to avoid confusion.

One option we are considering for maintenance is having a read-only CVS repository that mirrors our Git repository. We could then push changes to Git for maintenance streams, and our old builders can pick the change up from the CVS mirror/clone. We need to work with the Foundation staff to see if this is feasible ( bug 34876).

Current thinking is that 3.7.x is built from Git. This means we need to proof that we can build 3.7.1 against Git before we do the migration ( bug 349460).

Migration script

We need to gather various bits of information to plug into the CVS->Git migration tool:

  • Need author information for all committers past and present bug 349350
  • Figure out the CVS module -> Git repository mapping for all CVS modules
  • Figure out what binaries to exclude from migration, and how to exclude them
  • Determine what legacy CVS modules we can omit from migration altogether
  • Steps to test the migration.

Resolved issues

Git repository granularity

We decided on a single Git repository per ACL. One common Eclipse TLP repository for doc and map files. See bug 345471 for the discussion.

The same holds for Equinox, although Tom considering folding security component into bundles due to lack of active committers. The repositories for Equinox will be:

  • Framework
  • Bundles
  • p2
  • Security (maybe)
  • Incubator


The  Git_Migration_Granularity page documents the proposed bundles per repo.  (Nested bundles are assumed to be included by indicating the top level directory).

Tagging process

  • Build tags everything that is in master
  • If you don't want to contribute to build, put it in another branch
  • Eliminate the manual tagging step taken by all teams today
  • builds should still be map file based
  • We use tagging for two purposes: 1. reproducible builds and 2. bundle version numbers. We need to ensure that if a bundle's contents doesn't change, that the version number doesn't change. Paul W. had some thoughts on how this will be done by actually looking at the commit history.

Issues for after migration

3.x/4.x migration

  • Renaming of e4 bundles
  • Stitching in R4 branch of org.eclipse.ui.workbench that is in separate location bug 349177

Including eGit in Eclipse SDK

Traditionally we have included everything in the SDK needed to build the SDK. This implies including EGit/JGit in the Eclipse SDK. However this introduces a cyclic build dependency which always causes problems and introduces end-game risk. We need to reconsider this policy of including everything we need. Using p2 having multiple "root" features installed and updating them on an ongoing basis is no longer painful. The advantage of putting everything into a single build is perhaps outweighed by the pain of build-time dependency entanglement.

Recommendation: Remove all VCM support (including CVS) from the Eclipse SDK and users install what they need separately.

edit After excluding CVS from the product, the SDK build could still install it and the EGit feature using the director.

This was discussed at the June 15, 2011 PMC Meeting.

Git fetch factory

The Git fetch factory still uses command line git. This is not yet ported to JGit but that's not blocking us because our builds all run on a single platform (Linux). Unclear who will do this and when it will happen. In the long term this will be beneficial to remove a specific platform dependency from our build process.

It was ant tasks that were holding this up. bug 349056

Source headers

Eclipse-Source-Header support for Git still missing. This is needed for Juno but not for Git migration. Consider PDE plan item to support this (possibly in collaboration with EGit team)

See bug 327381

Copyright Tool

The copyright tool may need to be updated for integration with git.

See bug 345669

Git Ignore file

All of the repos should probably commit a .gitignore file at the base of their repositories. I would imagine that it needs to be done either on each branch that the component thinks is important going forward, or perhaps added as the root commit after the CVS to Git migration has been verified.

An example .gitignore file:


bin/
*~
*.rej
*.bak
core
*.so
*.patch
javacore.*
heapdump.*
core.*
Snap.*
target/


Process

The process that a component can expect during the git migration is described in this document

Platform-releng/Juno_Git_Migration_Process

Back to the top