Git conversion with svn-all-fast-export
You need: - svn-all-fast-export (from the Debian/Ubuntu package of the same name; upstream homepage is http://gitorious.org/svn2git, not related to a Ruby tool of the same name) - a copy of the Subversion repository Good thing I have one set up, using svnsync: # http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposad... svnadmin create /stuff/zope-mirror svnadmin setuuid /stuff/zope-mirror 62d5b8a3-27da-0310-9561-8e5933582275 svnsync init file:///stuff/zope-mirror svn://svn.zope.org/repos/main/ svnsync sync file:///stuff/zope-mirror # repeat last command periodically It needs about 3.3 gigs of disk space. - a copy of authors.txt that maps svn usernames to real names and emails (ask Tres or Jim) - an empty repository on Github (ask Tres or Jim to create one at https://github.com/zopefoundation) Speaking of this, could somebody create a zope.dottedname repo? Thanks. The conversion process goes like this: - write a rules.txt like this one I used for zope.dottedname: create repository zope.dottedname end repository # feel free to create multiple repositories in one go # order of matches matters in this file # trailing slashes in match rules are very important match / # skip revisions 129030, 129031 and 129032, since they're empty # in my svn mirror (no files, no committer, no date) for some reason # (bug in svnsync?), and they make svn-all-fast-export segfault min revision 129030 max revision 129032 end match match /(zope\.dottedname)/trunk/ repository \1 branch master end match match /(zope\.dottedname)/branches/([^/]+)/ repository \1 branch \2 end match match /(zope\.dottedname)/tags/([^/]+)/ repository \1 branch refs/tags/\2 end match match / # ignore all other projects end match - run svn-all-fast-export --identity-map=authors.txt --rules=rules.txt --stats \ /path/to/your/zope-svn-mirror - wait a bit The first time I ran it it took ~18 wall clock minutes (~4 CPU minutes) and ended in a segfault. The second run took 12 wall clock minutes (hot disk cache, I suppose) and also ended in a segfault. Then I discovered that if I don't remove the git repository, svn-all-fast-export will resume the process (a few thousand revisions before it crashed, or maybe that just happened to be the last successfully converted commit before the crash), which is considerably faster than starting from scratch. - inspect ./zope.dottedname for sanity I recommend tig as a very nice console-mode interactive git history viewer. For an example of things to inspect, e.g., there was a deleted 3.4.1 tag from http://zope3.pov.lt/trac/changeset/80495, which shouldn't have been deleted, according to http://zope3.pov.lt/trac/changeset/80499, so I've re-created the tag from refs/backups/r80495/tags/3.4.1 that was left by the conversion tool. - upload to github Advantages of svn-all-fast-export - it's fast (<5 CPU minutes for 129128 svn revisions; I wish I had an SSD on the server that hosts my svn mirror) - it can simultaneously convert multiple packages (add more 'create repository/end repository' statements to rules.txt, and extend the match regexps to catch the packages you're interested in) - it's very flexible and can handle gnarly repository history, if you write rules for it - it was written for and used by the KDE project to convert and explode their humongous svn repository into a multitude of git projects, so it's been stress-tested rather well Disadvantages of svn-all-fast-export - it requires a local copy of the entire subversion repository - it doesn't produce error messages if something's wrong, instead it segfaults Incidentally, I'd like to hear reports about conversion speed from people (e.g. Jim ;-) who have used Jim's conversion script. Is 12 minutes fast or slow? zope.dottedname is not a very large package; it had merely 34 revisions in trunk's history. I would also like to offer my services as a git conversion specialist, since I have access to an svn mirror. Ask me for a package by name, and I'll send you a git repository in a few days. It'll be up to you to inspect the fidelity of the conversion, though; I don't have the spare cycles for that. Regards, Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/04/2013 02:59 PM, Marius Gedminas wrote:
Speaking of this, could somebody create a zope.dottedname repo? Thanks.
Done. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEQIcEACgkQ+gerLs4ltQ5d9wCgq9HEYwd7BiL1KuuzQCV+Ks2n dgUAni4Eod+a4m5l2IhDDFoUJDRM+bC7 =KzJ+ -----END PGP SIGNATURE-----
Some minor updates: On Mon, Feb 04, 2013 at 09:59:37PM +0200, Marius Gedminas wrote:
The conversion process goes like this:
- write a rules.txt like this one I used for zope.dottedname:
create repository zope.dottedname end repository
# feel free to create multiple repositories in one go
# order of matches matters in this file # trailing slashes in match rules are very important
match / # skip revisions 129030, 129031 and 129032, since they're empty # in my svn mirror (no files, no committer, no date) for some reason # (bug in svnsync?), and they make svn-all-fast-export segfault min revision 129030 max revision 129032 end match
This bit doesn't work (I still get the segfault). What works is re-running the conversion script so it picks up the pieces. The three missing commits are test commits in /2github, so can be safely ignored. Besides, I doubt your Zope mirror will have the same defect. A more interesting thing to add to the rules is this bit: match /Zope3/trunk/src/zope/dottedname/ repository zope.dottedname branch monolithic-zope3 end match This way you can 'git blame' and see where the code actually originated. Just be sure not to use "branch master" here, or you'll get a big mess. For packages created before the namegeddon you might need more rules. http://zope3.pov.lt/trac/log/zope.dottedname is useful for finding the old location of code (click on r75116 aka "Moving code to satellite" to see).
match /(zope\.dottedname)/trunk/ repository \1 branch master end match
match /(zope\.dottedname)/branches/([^/]+)/ repository \1 branch \2 end match
match /(zope\.dottedname)/tags/([^/]+)/ repository \1 branch refs/tags/\2 end match
match / # ignore all other projects end match
- run svn-all-fast-export --identity-map=authors.txt --rules=rules.txt --stats \ /path/to/your/zope-svn-mirror
- wait a bit
The first time I ran it it took ~18 wall clock minutes
On my laptop (faster Core i5 CPU, SSD) it takes less than 4 wall clock minutes.
- inspect ./zope.dottedname for sanity
I recommend tig as a very nice console-mode interactive git history viewer.
'gitk --all' also works very well.
For an example of things to inspect, e.g., there was a deleted 3.4.1 tag from http://zope3.pov.lt/trac/changeset/80495, which shouldn't have been deleted, according to http://zope3.pov.lt/trac/changeset/80499, so I've re-created the tag from refs/backups/r80495/tags/3.4.1 that was left by the conversion tool.
This bit was 'git tag 3.4.1 refs/backups/r80495/tags/3.4.1'
- upload to github
This bit is git remote add origin git@github.com:zopefoundation/zope.dottedname.git git push -u origin --all git push origin --tags One more step at the end: - Remove old code from subversion, once you're sure the conversion is fine: svn rm * echo 'See https://github.com/zopefoundation/zope.dottedname' > MOVED_TO_GITHUB svn add MOVED_TO_GITHUB svn ci -m "Moved to github" Next I was going to add Travis CI integration, but, apparently, I need to be an admin of the ZopeFoundation organization to set up the needed hooks. Hint, hint. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
On Tue, Feb 05, 2013 at 05:36:38PM +0200, Marius Gedminas wrote:
One more step at the end:
- Remove old code from subversion, once you're sure the conversion is fine:
svn rm * echo 'See https://github.com/zopefoundation/zope.dottedname' > MOVED_TO_GITHUB svn add MOVED_TO_GITHUB svn ci -m "Moved to github"
More steps: - update any buildouts that used to check code out from svn, e.g. wineggbuilder:: svn co svn+ssh://svn.zope.org/repos/main/zope.wineggbuilder/trunk cd zope.wineggbuilder vim project-list.cfg replace zope.dottedname,svn://svn.zope.org/repos/main/ with zope.dottedname,git://github.com/zopefoundation/zope.dottedname.git svn ci -m "zope.dottedname moved to github" - update zopetoolkit too:: svn co svn+ssh://svn.zope.org/repos/main/zopetoolkit/trunk cd zopetoolkit vim ztk-sources.cfg replace zope.dottedname = svn ${buildout:svn-zope-org}/zope.dottedname/trunk with zope.dottedname = git ${buildout:github}/zope.dottedname svn ci -m "zope.dottedname moved to github" This seems to be it. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
Since I keep learning new and interesting things about git conversion, and wiki.zope.org is read-only (or I'm incapable of figuring out how to log in there), I've moved my notes to https://gist.github.com/mgedmin/4944724 Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
On Mon, Feb 04, 2013 at 09:59:37PM +0200, Marius Gedminas wrote:
You need:
- svn-all-fast-export (from the Debian/Ubuntu package of the same name; upstream homepage is http://gitorious.org/svn2git, not related to a Ruby tool of the same name)
- a copy of the Subversion repository
Good thing I have one set up, using svnsync:
# http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposad... svnadmin create /stuff/zope-mirror svnadmin setuuid /stuff/zope-mirror 62d5b8a3-27da-0310-9561-8e5933582275 svnsync init file:///stuff/zope-mirror svn://svn.zope.org/repos/main/ svnsync sync file:///stuff/zope-mirror # repeat last command periodically
It needs about 3.3 gigs of disk space.
- a copy of authors.txt that maps svn usernames to real names and emails (ask Tres or Jim)
- an empty repository on Github (ask Tres or Jim to create one at https://github.com/zopefoundation)
The conversion process goes like this:
- write a rules.txt like this one I used for zope.dottedname:
create repository zope.dottedname end repository
# feel free to create multiple repositories in one go
# order of matches matters in this file # trailing slashes in match rules are very important
match /(zope\.dottedname)/trunk/ repository \1 branch master end match
match /(zope\.dottedname)/branches/([^/]+)/ repository \1 branch \2 end match
match /(zope\.dottedname)/tags/([^/]+)/ repository \1 branch refs/tags/\2 end match
match / # ignore all other projects end match
- run svn-all-fast-export --identity-map=authors.txt --rules=rules.txt --stats \ /path/to/your/zope-svn-mirror
You can also pass --svn-branches for a slightly more accurate conversion (branch merge commits do not go away, even when the diff is empty), if I understand it correctly. And if you pass --add-metadata-notes, you'll get to see svn path and revno attached to a note on each commit. These are shown by git log. These notes are easy to lose (git push --all/--tags doesn't push them; git clone doesn't fetch them). Read more about them at http://git-scm.com/2010/08/25/notes.html The notes are shown on Github like this: https://github.com/zopefoundation/zope.traversing/commit/c10f103#gitnotes
- inspect ./zope.dottedname for sanity
I recommend tig as a very nice console-mode interactive git history viewer.
Sometimes the conversion tool produces strands of unrelated history. tig --all interleaves them which makes this hard to notice. gitk --all shows them separately. You can connect the strands by creating a grafts file, and you can make the connections permanent (I don't think the grafts file survives a git push) by running git-filter-branch.
For an example of things to inspect, e.g., there was a deleted 3.4.1 tag from http://zope3.pov.lt/trac/changeset/80495, which shouldn't have been deleted, according to http://zope3.pov.lt/trac/changeset/80499, so I've re-created the tag from refs/backups/r80495/tags/3.4.1 that was left by the conversion tool.
And 'git shortlog -s' is a good way to check if your authors.txt was complete and correct. Note: if you find you need to re-run the conversion, make sure to rm -rf the git repository _and_ the log-zope.whateverpackage file that svn-all-fast-export uses to remember where it last crashed.
- upload to github
This'll be git remote add origin git@github.com:zopefoundation/zope.dottedname.git git push -u origin --mirror if you want to push everything (branches, tags, notes). Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
participants (2)
-
Marius Gedminas -
Tres Seaver