Jump to: navigation, search

Difference between revisions of "Platform-releng/Juno Git Migration Recipe"

m
(Running the tool)
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
This is a modified version of the steps described by the CDT team in bug [https://bugs.eclipse.org/bugs/show_bug.cgi?id=316208 bug 316208]  
 
This is a modified version of the steps described by the CDT team in bug [https://bugs.eclipse.org/bugs/show_bug.cgi?id=316208 bug 316208]  
  
Recipe for test conversion for p2  
+
=Overview of the steps for platform conversion:=
 +
 
 +
This is taken from the Platform UI/Eclipse4 migration script, [http://git.eclipse.org/c/e4/org.eclipse.migration.git/tree/scripts/migration_script.sh migration_script.sh]
 +
 
 +
=== Precondition the CVS repo ===
 +
 
 +
If you have maintenance releases where your release was tagged '''R3_6''' but only projects that were changed were branched '''R3_6_maintenance''' the cvs2git conversion tool will create a maintenance branch with only the projects that were changed in it.  This makes the maintenance branch useless.  But if you branch every project in your CVS repo before the conversion, you will get correct branches in git.
 +
 
 +
For each maintenance branch you care about you have to fix the branch + the tags, ex: R3_6, R3_6_1, R3_6_2, and R3_6_maintenance.
 +
 
 +
 
 +
=== Create a temp repo ===
 +
 
 +
cvs2git changes the cvs repo as it goes (it reads and modifies the ''',v''' files), so you should do your conversion on a temp repo.  For a component like Platform UI, that allowed us to shape our repository from many top level CVS modules to its current shape: repo/bundles/[plugin_projects], repo/tests/[plugin_projects], etc.  Just copy the projects to where you would like them to show up.
 +
 
 +
=== Running the tool ===
 +
 
 +
The options file for running the tool needs 2 things.
 +
#the root of the temporary CVS repo you created
 +
#the list of authors
 +
 
 +
There is a quick way to generate a list of authors from a bash shell for your CVS projects.  The Eclipse Foundation requires that we use "Name" <committer.id> or "committer.id" <committer.id> as the authors for the conversion, unless that committer has specifically allowed his committer email to be used.
 +
 
 +
=== Fixing Delete-only tags ===
 +
 
 +
For most tags used in build cvs2git generates "delete-only" commits, because we only tag projects that changed it feels it has to remove all of the other ones.  It makes our history look like a little fir tree.
 +
 
 +
There are scripts that can fix them (in Platform UI, it fixed ~1700 out of 2200 tags).
 +
 
 +
=== Fixing a Delete-only commit at the base of a branch ===
 +
 
 +
If you have a small branch that you didn't pre-condition the base of the branch will have a delete-only commit.  To remove it in git, you have to rebase the delete-only commit onto its parent.  rebase doesn't come with a move tag option, so you will have to save the tags from before and after, and move them manually to the new commits created by the rebase.  I only had one small branch effecting one project that this mattered for.
 +
 
 +
=== adding a .gitignore ===
 +
 
 +
You only need one at the base of your repo.  Ideally, it should look like it was there from the beginning.  There might be a way to insert a .gitignore,v file into CVS before the conversion to get it there, but after the conversion you can use git filter-branch to make it look like .gitignore has been there from the beginning.  This should be the last step before your publish your repo, as it re-writes every commit currently in git.
 +
 
 +
=Recipe for test conversion for p2 =
  
Steps to reproduce:
 
  
 
==Copy the existing repository to a temp location==
 
==Copy the existing repository to a temp location==
Line 16: Line 52:
 
Note: This didn't apply to us.  
 
Note: This didn't apply to us.  
  
Move old content into old and exclude old content from migration. Need to create exclude list, preliminary one is here [[Platform-releng/Git Migration Granularity|here]]
+
==Move old content into old and exclude old content from migration==
 +
 
 +
Need to create exclude list [[Platform-releng/Git Migration Granularity|preliminary list is here]]
  
 
==Run cvs2git to do the conversion==
 
==Run cvs2git to do the conversion==
Line 28: Line 66:
 
<pre>
 
<pre>
 
   svn co --username=guest --password="" http://cvs2svn.tigris.org/svn/cvs2svn/trunk cvs2svn-trunk  
 
   svn co --username=guest --password="" http://cvs2svn.tigris.org/svn/cvs2svn/trunk cvs2svn-trunk  
 +
 +
</pre>
  
 
Export from cvs
 
Export from cvs
.../cvs2svn/cvs2git --options=cvs2git.options
 
  
 +
<pre>
 +
cvs2svn/cvs2git --options=cvs2git.options
 
</pre>
 
</pre>
 +
 
     (To do: Add options file link)
 
     (To do: Add options file link)
  
Line 45: Line 87:
 
==Move tags into place==
 
==Move tags into place==
  
   python ..../cvs2svn/contrib/git-move-refs.py  
+
   python ~/cvs2svn-trunk/contrib/git-move-refs.py
  
 
==Prune + Repack the repository==
 
==Prune + Repack the repository==
Line 58: Line 100:
  
 
   mkdir /tmp/compare-kmoir/
 
   mkdir /tmp/compare-kmoir/
.../verify-cvs2svn.py --git .../cvs/tools/org.eclipse.cdt/ .../path_to_git_repo/ --tmp=/tmp/compare-kmoir/ --diff
+
  python /home/data/users/kmoir/cvs2svn-trunk/contrib/verify-cvs2svn.py --git /home/data/users/kmoir/cvs/p2/ /gitroot/p2/org.eclipse.equinox.p2/org.eclipse.equinox.p2.git/ --tmp=/tmp/compare-kmoir/ --diff
 +
 
 +
==Update map files==
 +
 
 +
<pre>
 +
replace CVS= with GIT=
 +
tag should remain
 +
replace cvsRoot=:pserver:anonymous@dev.eclipse.org:/cvsroot/rt/ with file:///gitroot/equinox/rt.equinox.p2.git
 +
remove org.eclipse.equinox/p2/ in path=
 +
update path
 +
</pre>
 +
 
 +
==Run test build==
 +
 
 +
Run test build on Hudson - job name is [https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-equinox-git-test-N/ eclipse-equinox-git-test-N]

Latest revision as of 09:48, 21 July 2011

This is a modified version of the steps described by the CDT team in bug bug 316208

Overview of the steps for platform conversion:

This is taken from the Platform UI/Eclipse4 migration script, migration_script.sh

Precondition the CVS repo

If you have maintenance releases where your release was tagged R3_6 but only projects that were changed were branched R3_6_maintenance the cvs2git conversion tool will create a maintenance branch with only the projects that were changed in it. This makes the maintenance branch useless. But if you branch every project in your CVS repo before the conversion, you will get correct branches in git.

For each maintenance branch you care about you have to fix the branch + the tags, ex: R3_6, R3_6_1, R3_6_2, and R3_6_maintenance.


Create a temp repo

cvs2git changes the cvs repo as it goes (it reads and modifies the ,v files), so you should do your conversion on a temp repo. For a component like Platform UI, that allowed us to shape our repository from many top level CVS modules to its current shape: repo/bundles/[plugin_projects], repo/tests/[plugin_projects], etc. Just copy the projects to where you would like them to show up.

Running the tool

The options file for running the tool needs 2 things.

  1. the root of the temporary CVS repo you created
  2. the list of authors

There is a quick way to generate a list of authors from a bash shell for your CVS projects. The Eclipse Foundation requires that we use "Name" <committer.id> or "committer.id" <committer.id> as the authors for the conversion, unless that committer has specifically allowed his committer email to be used.

Fixing Delete-only tags

For most tags used in build cvs2git generates "delete-only" commits, because we only tag projects that changed it feels it has to remove all of the other ones. It makes our history look like a little fir tree.

There are scripts that can fix them (in Platform UI, it fixed ~1700 out of 2200 tags).

Fixing a Delete-only commit at the base of a branch

If you have a small branch that you didn't pre-condition the base of the branch will have a delete-only commit. To remove it in git, you have to rebase the delete-only commit onto its parent. rebase doesn't come with a move tag option, so you will have to save the tags from before and after, and move them manually to the new commits created by the rebase. I only had one small branch effecting one project that this mattered for.

adding a .gitignore

You only need one at the base of your repo. Ideally, it should look like it was there from the beginning. There might be a way to insert a .gitignore,v file into CVS before the conversion to get it there, but after the conversion you can use git filter-branch to make it look like .gitignore has been there from the beginning. This should be the last step before your publish your repo, as it re-writes every commit currently in git.

Recipe for test conversion for p2

Copy the existing repository to a temp location

mkdir ~/cvs; cd ~/cvs

cp /home/data/cvs/rt/org.eclipse.equinox.p2 .

Remove broken symlinks in the repo from all to the components we moved above

  find cvs/ -type l|xargs -n 1 rm

Note: This didn't apply to us.

Move old content into old and exclude old content from migration

Need to create exclude list preliminary list is here

Run cvs2git to do the conversion

  http://cvs2svn.tigris.org/cvs2git.html
 Get latest cvs2git:
  svn co --username=guest --password="" http://cvs2svn.tigris.org/svn/cvs2svn/trunk cvs2svn-trunk 

Export from cvs

 cvs2svn/cvs2git --options=cvs2git.options
   (To do: Add options file link)

Import the cvs2git output into git

  cd /gitroot/p2
  mkdir -p org.eclipse.equinox.p2/org.eclipse.equinox.p2.git
  cd org.eclipse.equinox.p2/org.eclipse.equinox.p2.git
  git init
  cat ~/cvs2svn-tmp/git-blob.dat ~/cvs2svn-tmp/git-dump.dat | git fast-import

Move tags into place

 python ~/cvs2svn-trunk/contrib/git-move-refs.py

Prune + Repack the repository

git prune
git repack -a -d --depth=250 --window=250
git gc --aggressive
git repack -a -d --depth=250 --window=250

Verify repo

 mkdir /tmp/compare-kmoir/
 python /home/data/users/kmoir/cvs2svn-trunk/contrib/verify-cvs2svn.py --git /home/data/users/kmoir/cvs/p2/ /gitroot/p2/org.eclipse.equinox.p2/org.eclipse.equinox.p2.git/ --tmp=/tmp/compare-kmoir/ --diff

Update map files

replace CVS= with GIT=
tag should remain
replace cvsRoot=:pserver:anonymous@dev.eclipse.org:/cvsroot/rt/ with file:///gitroot/equinox/rt.equinox.p2.git
remove org.eclipse.equinox/p2/ in path=
update path

Run test build

Run test build on Hudson - job name is eclipse-equinox-git-test-N