Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "STEM/Git Migration"

(Checking out STEM into Eclipse using EGit)
(Migration Timeline)
 
(26 intermediate revisions by the same user not shown)
Line 9: Line 9:
 
* [http://git-scm Official Web site]
 
* [http://git-scm Official Web site]
 
* [[Git|Eclipse Foundation's Git server information]]
 
* [[Git|Eclipse Foundation's Git server information]]
 +
* [[Git#Resources | Eclipse's Recommended Git Resources]]
 
* [[Git/Migrating_to_Git|Eclipse's Guide for Projects Migrating to Git from SVN/CVS]]
 
* [[Git/Migrating_to_Git|Eclipse's Guide for Projects Migrating to Git from SVN/CVS]]
 
* [http://eclipse.org/egit/ EGit Team Provider for Eclipse IDE]
 
* [http://eclipse.org/egit/ EGit Team Provider for Eclipse IDE]
Line 23: Line 24:
 
* April 2:  Public announcement of migration with STEM 2.0 milestone 1
 
* April 2:  Public announcement of migration with STEM 2.0 milestone 1
 
* April:  test runs of repository split and import, prepare infrastructure (build scripts, CI server, project sets, etc) and update documentation
 
* April:  test runs of repository split and import, prepare infrastructure (build scripts, CI server, project sets, etc) and update documentation
* (Tentative) April 30:  Freeze SVN, execute import, roll out new Git repositories
+
* April 30:  Freeze SVN, execute import, roll out new Git repositories
* (Tenative) May 1:  Declare migration complete
+
* May 1:  Declare migration complete
  
 
=== Steps for Developers ===
 
=== Steps for Developers ===
Line 33: Line 34:
 
** Developers will have to manually reconcile changes that aren't ready for commit after re-checking out from Git
 
** Developers will have to manually reconcile changes that aren't ready for commit after re-checking out from Git
 
* After the SVN freeze date and migration is complete, create a new workspace or remove the SVN-linked STEM projects from your existing workspace then clone and checkout STEM from the new Git repositories
 
* After the SVN freeze date and migration is complete, create a new workspace or remove the SVN-linked STEM projects from your existing workspace then clone and checkout STEM from the new Git repositories
 +
 +
=== Additional Steps for Committers ===
 +
Committers should read the Eclipse Foundation's information on [[Git#Committing_and_pushing|committing to Git]] and [[Development_Resources/Handling_Git_Contributions|handling Git Contributions]].  These links contain very important details.
 +
 +
==== Make sure Git's user.email matches the email Eclipse has on file for your committer account ====
 +
This step is critical.  Git registers a name and e-mail address with every commit ref.  '''Eclipse requires that all commit ref emails match the email address that Eclipse has on record for your committer account'''.  Eclipse's Git servers will reject all commit refs pushed to it from unknown email addresses.
 +
 +
Configuring a Git repository's name and email can be done a number of ways.  The easiest is to set it up globally for your whole system:
 +
<pre>
 +
git config --global user.email my_committer_email@address.com
 +
git config --global user.name "John Doe"
 +
</pre>
 +
 +
You can also set it on a per-repository basis (remove --global) as well as configuring it using EGit. In Eclipse, <code>Window > Preferences > Team > Git > Configuration</code>.
 +
 +
==== Set Up SSH Public Key Authentication ====
 +
With this migration, committers should consider migrating to SSH private key authentication to make Git clone/pull/push easier. '''Eclipse recommends using this form of authentication over password authentication.'''  See this [[IT_Infrastructure_Doc#Upload_my_public_key|document from Eclipse IT]] for more details.
 +
 +
Additionally, on some systems, you can configure SSH to automatically use a specific username for a host or set of hosts.  This can be useful if your local system's username does not match your Eclipse committer username.
 +
 +
For Linux and OS X machines, try these steps:
 +
* Open a command line / terminal
 +
* <code>cd ~/.ssh</code>
 +
* Create or edit <code>config</code>
 +
* Enter the following lines:
 +
<pre>Host *.eclipse.org
 +
  User YOUR_COMMITTER_USERNAME
 +
</pre>
 +
* Save config file
 +
* You can now SSH directly to Eclipse without having to enter a username
 +
*: e.g you can do <code>git clone ssh://git.eclipse.org/gitroot/stem/org.eclipse.stem.git</code>
 +
See the <code>ssh_config</code> man page for more details.
  
 
=== Repository Layout ===
 
=== Repository Layout ===
Due to the nature of distributed versioning system like Git, an entire repository must be "cloned" onto a developer's machine before it can be worked with.  While Git's compression is impressive, it still requires significant network transfer time and local storage for large repositories - even if you only need to work with a small subset.  In STEM's case, we store a large amount of denominator in our SVN repository, on the order of 2 GB.
+
The nature of distributed versioning system like Git means an entire repository must be "cloned" onto a developer's machine before it can be worked with.  While Git's compression is impressive, it still requires significant network transfer time and local storage for large repositories - even if you only need to work with a small subset.  In STEM's case, we store a large amount of denominator in our SVN repository, on the order of 2 GB.
  
STEM has will split our code base into multiple repositories that can be fetched independently as needed.  The proposed repositories are:
+
Due to the large repository requirements, STEM has decided to split the code base into multiple repositories that can be fetched independently as needed.  The proposed repositories are:
  
* org.eclipse.stem.git - Core repository, including majority of source code and epidemiological modeling components
+
* {{Git|stem|org.eclipse.stem.git}}
* org.eclipse.stem.data.git - Common denominator data, including administrative borders and human population data
+
: Core repository, including majority of source code and epidemiological modeling components
* org.eclipse.stem.data.earthscience.git - Less common Earth Science denominator data
+
* {{Git|stem|org.eclipse.stem.data.git}}
 +
: Common denominator data, including administrative borders and human population data
 +
* {{Git|stem|org.eclipse.stem.data.earthscience.git}}
 +
: Less common Earth Science denominator data
 +
* {{Git|www.eclipse.org|stem.git}}
 +
: STEM Web site hosted at eclipse.org/stem
  
 
The idea is you can work independently with STEM without needing the additional repositories and they can be added as needed (although most developers will benefit from having the org.eclipse.stem.data repository cloned).
 
The idea is you can work independently with STEM without needing the additional repositories and they can be added as needed (although most developers will benefit from having the org.eclipse.stem.data repository cloned).
Line 54: Line 92:
 
# On the ''' Import Team Project Set ''' page, enter the correct URL...
 
# On the ''' Import Team Project Set ''' page, enter the correct URL...
 
#* ''For non-committers:''
 
#* ''For non-committers:''
#*: <code><nowiki>http://www.eclipse.org/stem/psf-git.php</nowiki></code>
+
#*: <code><nowiki>http://www.eclipse.org/stem/psf.php</nowiki></code>
 
#* ''For committers:''
 
#* ''For committers:''
#*: <code><nowiki>http://www.eclipse.org/stem/psf-git.php?r=ssh&u=YOUR_COMMITTER_USERNAME</nowiki></code>
+
#*: <code><nowiki>http://www.eclipse.org/stem/psf.php?r=ssh&u=YOUR_COMMITTER_USERNAME</nowiki></code>
 
#*: '''Note: Replace YOUR_COMMITTER_USERNAME with your committer username, not e-mail address, in the above link'''
 
#*: '''Note: Replace YOUR_COMMITTER_USERNAME with your committer username, not e-mail address, in the above link'''
 
# Click '''Finish'''
 
# Click '''Finish'''
Line 62: Line 100:
 
=== Repository URLs ===
 
=== Repository URLs ===
 
The STEM repositories will be available at the following URLs:
 
The STEM repositories will be available at the following URLs:
 +
 +
==== Committer Access ====
 +
Committers should use Git over SSH to access the repositories. Other proxy methods are available, see the [[Git#Connecting.2C_cloning_a_repo|Eclipse Git guide]] for more details
 +
 +
* ssh://committer_username@git.eclipse.org/gitroot/stem/org.eclipse.stem.git
 +
* ssh://committer_username@git.eclipse.org/gitroot/stem/org.eclipse.stem.data.git
 +
* ssh://committer_username@git.eclipse.org/gitroot/stem/org.eclipse.stem.earthscience.git
 +
* ssh://committer_username@git.eclipse.org/gitroot/www.eclipse.org/stem.git (STEM web site)
  
 
==== Read-only Access ====
 
==== Read-only Access ====
Line 67: Line 113:
 
* git://git.eclipse.org/gitroot/stem/org.eclipse.stem.data.git
 
* git://git.eclipse.org/gitroot/stem/org.eclipse.stem.data.git
 
* git://git.eclipse.org/gitroot/stem/org.eclipse.stem.earthscience.git
 
* git://git.eclipse.org/gitroot/stem/org.eclipse.stem.earthscience.git
 +
* git://git.eclipse.org/gitroot/www.eclipse.org/stem.git (STEM web site)
  
 
Alternately, the repositories can be accessed using HTTP.  See the [[Git|Eclipse Git]] guide for more details
 
Alternately, the repositories can be accessed using HTTP.  See the [[Git|Eclipse Git]] guide for more details
 
==== Committer Access ====
 
Committers should use Git over SSH to access the repositories. Other proxy methods are available, see the [[Git#Connecting.2C_cloning_a_repo|Eclipse Git guide]] for more details
 
 
* ssh://git.eclipse.org/gitroot/stem/org.eclipse.stem.git
 
* ssh://git.eclipse.org/gitroot/stem/org.eclipse.stem.data.git
 
* ssh://git.eclipse.org/gitroot/stem/org.eclipse.stem.earthscience.git
 
  
 
==== Web access ====
 
==== Web access ====

Latest revision as of 12:09, 6 May 2013

Background

In April 2013, the STEM Project initiated its move from Subversion (SVN) to Git. This process is expected to take about a month and impacts all users of the STEM source code.

The Eclipse Foundation has encouraged projects to migrate from CVS and SVN, including shutting down CVS access late last year.


About Git

There are countless resources for learning about Git. Some useful ones:

Details

Our migration is being tracked in Bugzilla under Bug 404584

Migration Timeline

  • March 21: Initial discussion on committer call re/ moving to Git (complete)
  • March 28: Further discussion on committer call re/ moving to Git. Committer motion to develop migration plan and initiate process passes. Most committers have experience with Git.
  • March 29: Request Git migration from Eclipse Webmaster
  • April 1: Create and publish test repositories for testing, begin migrating infrastructure
  • April 2: Public announcement of migration with STEM 2.0 milestone 1
  • April: test runs of repository split and import, prepare infrastructure (build scripts, CI server, project sets, etc) and update documentation
  • April 30: Freeze SVN, execute import, roll out new Git repositories
  • May 1: Declare migration complete

Steps for Developers

  • Install EGit into your Eclipse IDE or use your favorite Git client (You can use the Install New Software feature in Eclipse to install from the Juno repository)
  • Before the official freeze date (TBD, tentatively April 30), test that you can connect to the test repositories (see below) with your needed level of access.
  • Committers should try making a commit and push to the test repositories to again verify your needed access
  • Before the official freeze date, make all outstanding commits to the SVN repository
    • Developers will have to manually reconcile changes that aren't ready for commit after re-checking out from Git
  • After the SVN freeze date and migration is complete, create a new workspace or remove the SVN-linked STEM projects from your existing workspace then clone and checkout STEM from the new Git repositories

Additional Steps for Committers

Committers should read the Eclipse Foundation's information on committing to Git and handling Git Contributions. These links contain very important details.

Make sure Git's user.email matches the email Eclipse has on file for your committer account

This step is critical. Git registers a name and e-mail address with every commit ref. Eclipse requires that all commit ref emails match the email address that Eclipse has on record for your committer account. Eclipse's Git servers will reject all commit refs pushed to it from unknown email addresses.

Configuring a Git repository's name and email can be done a number of ways. The easiest is to set it up globally for your whole system:

git config --global user.email my_committer_email@address.com
git config --global user.name "John Doe"

You can also set it on a per-repository basis (remove --global) as well as configuring it using EGit. In Eclipse, Window > Preferences > Team > Git > Configuration.

Set Up SSH Public Key Authentication

With this migration, committers should consider migrating to SSH private key authentication to make Git clone/pull/push easier. Eclipse recommends using this form of authentication over password authentication. See this document from Eclipse IT for more details.

Additionally, on some systems, you can configure SSH to automatically use a specific username for a host or set of hosts. This can be useful if your local system's username does not match your Eclipse committer username.

For Linux and OS X machines, try these steps:

  • Open a command line / terminal
  • cd ~/.ssh
  • Create or edit config
  • Enter the following lines:
Host *.eclipse.org
  User YOUR_COMMITTER_USERNAME

See the ssh_config man page for more details.

Repository Layout

The nature of distributed versioning system like Git means an entire repository must be "cloned" onto a developer's machine before it can be worked with. While Git's compression is impressive, it still requires significant network transfer time and local storage for large repositories - even if you only need to work with a small subset. In STEM's case, we store a large amount of denominator in our SVN repository, on the order of 2 GB.

Due to the large repository requirements, STEM has decided to split the code base into multiple repositories that can be fetched independently as needed. The proposed repositories are:

Core repository, including majority of source code and epidemiological modeling components
Common denominator data, including administrative borders and human population data
Less common Earth Science denominator data
STEM Web site hosted at eclipse.org/stem

The idea is you can work independently with STEM without needing the additional repositories and they can be added as needed (although most developers will benefit from having the org.eclipse.stem.data repository cloned).

In July 2011, STEM did a logical refactor that changed the layout of our SVN repository from a flat to hierarchical structure reflecting components. We're happy with this layout and will not be changing the overall structure of our repository.

Checking out STEM into Eclipse using EGit

To quickly checkout the required STEM source code, the project set generator has been updated to generate EGit-compatible Team Project set.

  1. In Eclipse, open the Git Repository Exploring perspective
  2. Launch the Project Set Import Wizard File > Import > Team > Team Project Set
  3. On the Import Team Project Set page, enter the correct URL...
    • For non-committers:
      http://www.eclipse.org/stem/psf.php
    • For committers:
      http://www.eclipse.org/stem/psf.php?r=ssh&u=YOUR_COMMITTER_USERNAME
      Note: Replace YOUR_COMMITTER_USERNAME with your committer username, not e-mail address, in the above link
  4. Click Finish

Repository URLs

The STEM repositories will be available at the following URLs:

Committer Access

Committers should use Git over SSH to access the repositories. Other proxy methods are available, see the Eclipse Git guide for more details

Read-only Access

Alternately, the repositories can be accessed using HTTP. See the Eclipse Git guide for more details

Web access

Eclipse publishes a Gitweb view of the repositories. For the STEM, the URLs are:

Copyright © Eclipse Foundation, Inc. All Rights Reserved.