[Python-Dev] SVN <-> HG workflow to split Python Library by Module

[anatoly techtonik]

I planned to publish this proposal when it is finally ready and tested
with an assumption that Subversion repository will be online and
up-to-date after Mercurial migration. But recent threads showed that
currently there is no tested mechanism to sync Subversion repository
back with Mercurial, so it will probably quickly outdate, and the
proposal won't have a chance to be evaluated. So now is better than
never.

So, this is a way to split modules from monolithic Subversion
repository into several Mercurial mirrors - one mirror for each module
(or whatever directory structure you like). This will allow to
concentrate your work on only one module at a time ("distutils",
"CGIHTTPServer" etc.) without caring much about anything else.
Exceptionally useful for occasional external "contributors" like me,
and folks on Windows, who don't possess Visual Studio to compile
Python and are forced to use whatever version they have installed to
create and test patches.

Here is a picture if you feel bored -
https://docs.google.com/drawings/edit?id=1c9FDQ27BnaIew_1T7Tr-rFg1OCdPVS9w3TQREOkzyjk&hl=en
An example of the split distutils module -
http://bitbucket.org/techtonik/distutils

The split is not perfect, but the process can be polished - it is the
first version I managed to get only this morning. More important is
that HG repository is incrementally synchronized. The split is not
perfect, because in particular I see that documentation dir is not
sucked in. But it is a working proof on concept you can test yourself
using the code from:
http://bitbucket.org/techtonik/python-split
You will also need patched version of `hgsvn` from
http://bitbucket.org/techtonik/hgsvn

How does it work
-------------------------
The module is described as a series of paths inside typical Subversion checkout.
On the first run `refresh.py` script from `python-split` creates
shallow SVN checkout with only required files using
distutils.module.def module definition
Second run of `refresh.py` imports shallow checkout into Mercurial
And the third run imports the rest of the history pulling only
changesets relevant to given paths.

Workflow
-------------
Diagram showed patches that are pulled from local clones of split
repositories to master Mercurial mirror, but it won't work this way,
because hashes of revisions in direct mirror wont't match hashes in
split repositories - that's why some hash lookup/sync procedure is
needed to correctly process incoming patches. This workflow works with
hash sync only when changes are pushed back to central Subversion
repository from local clones (possibly through another intermediate
normalizing repository). Changes pushed this way are streamlined and
could be downloaded into stable branch of other mirrors as a single
line of development. I borrowed streamlining concept from Go
contribution guide as it really helps to review chaotic Mercurial
commits. http://golang.org/doc/contribute.html#Code_review

Maintaining centralized Subversion repository will require additional
properties to be set, but this is doable. I don't how to make module
split with Mercurial alone.
http://mercurial.selenic.com/wiki/ShallowClone is still a draft (and
complicated one) and Mercurial 1.6 that released today doesn't contain
anything revolutionary to propose an alternative.

I am exhausted.
-- 
anatoly t.