On Jul 28, 2005, at 4:20 PM, Guido van Rossum wrote:

Managing users is especially important -- if a user is compromised (as has happened in the past for python.org users) the whole repository is compromised. Now this could happen to SF users too, but I'm not sure that we know all the tricks in the book to prevent attacks; SF has been doing this for years and that's an aspect of SF that I trust (I think I've heard that they have even modified their SSH server to be stricter).

If you use the fsfs storage mechanism for subversion, it is somewhat simpler to verify that the repository is not compromised. Each commit is represented as a separate file, and thus old commits are never modified. Only new files are appended to the directory. If you have a filesystem that allows "append-only" permissions on a directory, you can enforce this directly. Additionally, it is possible in your backup script to verify that only new files were added and nothing else changed. Then at least you know how much you need to examine instead of having to treat the entire repository as possibly contaminated. James

On Thu, 2005-07-28 at 16:20, Guido van Rossum wrote:

I hope we're correctly estimating the effort required to manage the server and the users.

Yah, me too! ;) We are building some experience with this though, having moved many of the system files, and all of the web pages, to svn. So far, the management overhead has been almost nil (um, None :). We'll have a bit of ongoing work to add users, but the infrastructure team is also building up some community knowledge about how to do that.

Managing users is especially important -- if a user is compromised (as has happened in the past for python.org users) the whole repository is compromised. Now this could happen to SF users too, but I'm not sure that we know all the tricks in the book to prevent attacks; SF has been doing this for years and that's an aspect of SF that I trust (I think I've heard that they have even modified their SSH server to be stricter).

James has a very interesting idea for mitigating this. Presumably <heh>, we'll have backups of everything. I'll feel better when we have coverage from maybe 6 admins spanning as many timezones as possible. -Barry

Guido van Rossum wrote:

I hope we're correctly estimating the effort required to manage the server and the users. Managing users is especially important -- if a user is compromised (as has happened in the past for python.org users) the whole repository is compromised. Now this could happen to SF users too, but I'm not sure that we know all the tricks in the book to prevent attacks; SF has been doing this for years and that's an aspect of SF that I trust (I think I've heard that they have even modified their SSH server to be stricter).

There are three issues that I see: - management of access control. This is actually something that we do on SF as well; if the pydotorg admins get overloaded, I'm sure the current projects/python admins would be willing to help there. - assignment of passwords. This I don't like about the current pydotorg setup - there should be a way to chose your own password; perhaps without involving an administrator. I could imagine a web form for password change, and administrator interaction in case of a lost password. - compromised passwords. The only tricky question then is: was the repository altered? Fortunately, for Subversion, there should be an easy way to tell: in fsfs, files never change (only new files are added). So we could generate md5sums of all files in the repository, and download these to an offsite place. If the md5sum of an immutable file changes, we were compromised (there are, of course, a few files that do change regularly). Of course, we also need regular backups of the entire data so we can restore them if they got compromised. Regards, Martin

Barry Warsaw wrote:

I disagree. By reserving password generation to the pydotorg admins, we can better insure the passwords are more robust against dictionary attacks. See my previous message. I actually /don't/ want individuals to be able to set their own passwords. In practice, you only have to know your password once, because svn caches the authentication (yes, that opens up opportunities for compromise, but that's how svn works).

See Michael's (I think) message: that is a much greater risk than the one of a brute-force attack. In our environment, a determined student could easily read out my home directory, and get at my pydotorg password (if I would allow svn to cache it). They would have to break all kinds of rules in doing so; yet, it would be technically possible - so I just can't turn on this svn setting, and have to type the password every time. This is surely inconvenient, as I cannot even remember the password. Regards, Martin

Barry Warsaw wrote:

I know that SF has promised svn access to projects for a long time now, but I haven't heard anything from them in a long time. It's listed under their "Strategic Projects" but the last update to that news item was back in April. Question: do we wait for SF to make the service available (after getting more up-to-date status and a realistic timetable), or do we go straight to svn.python.org?

My proposal is to go straight to svn.python.org, for two reasons: - It might well be 2006 before they update the status and provide a realistic timetable. It's not that I lost faith into SF, but they seem to be continually overworked (as anybody), so anything that is not *really* essential to the operation of the service (such as support for subversion) has to wait "until we have time" -- which means it has to wait forever. Remember the plan to provide shell access to the project admins on the CVS server? - They currently have a separate anonymous access for CVS which is behind the real CVS, for load sharing reasons. They also had severe performance issues in the past. It might be that we also get performance problems, but we only need to support a single project (or two if you count pydotorg) on Subversion, not thousands of them. So our load evolution is much more predictable.

...

1. Assign passwords for all current committers for use on svn.python.org. User names on SF and svn.python.org should be identical, unless some committer requests a different user name.

We've been going with firstname.lastname (with some exceptions -- hi Aahz! :) for the current svn access. Is it better to stay with that convention or to maintain consistency with SF's CVS committer names? Maybe the latter for revision history consistency.

It's also a convenience issue, and has social aspects. For example, firstname.lastname does not work quite well for me, either (martin.v.loewis or martin.von.loewis would work better; likewise guido.van.rossum?), other users prefer not to use their "true" real name (e.g. tim_one vs. tim.peters). Another issue is password assignment - currently, users cannot choose their passwords on svn.python.org, right?

We can probably play games with the various CVS hooks to disable all checkins during this time. We might also want to disable the u/i access to CVS at the same time.

That should be tested in advance, of course.

...

4. Convert the CVS repository into two subversion repositories, one for distutils and one for Python.

Do we also want to split off nondist and encodings?

Well, encodings is empty right now, so that might be a good idea. Technically, I would just move the files in the CVS repository before conversion. As for nondist: how precisely would you do that? Make it a separate repository? If so, where? I could envision a "/projects" repository, that has the current nondist.

Note that we might want to provide different access permission to different parts of the repository (but I think we can do that even if we don't split those off into separate repos).

I don't know how cvs2svn supports it, but one option would be to revert the trunk/ structure in /projects, so you get http://www.python.org/projects/peps/trunk http://www.python.org/projects/peps/tags http://www.python.org/projects/peps/branches http://www.python.org/projects/sandbox/trunk http://www.python.org/projects/sandbox/tags http://www.python.org/projects/sandbox/branches Then you could give different access to peps and to sandbox. Perhaps there isn't even a need for tags/branches on PEPs? Or we put everything in nondist into /projects, and then put tags + branches as siblings (so only people with write access to tags could create them)? Regards, Martin

On 7/29/05, Fred L. Drake, Jr. wrote:

On Thursday 28 July 2005 20:07, Fernando Perez wrote:

...

or something similar. It's an extra few chars, and it would give a convenient way to branch off pieces of the main code into their own subprojects in the future if needed.

More interestingly, keeping it in a single repository makes it easier to merge projects, or parts of projects, together, without losing the history. This would be useful when developing packages that may be considered for the standard library, but which also need to continue separate releases to support older versions of Python. We've found it very handy to keep multiple projects in a single repository for zope.org.

I thought I would chime in on this with some more experience. We use SVN (migated from CVS about 2 years ago) for everything, and we keep it all in one repository (even though there's several "products") for exactly this reason. The major "downside" that I, and some others, find is changeset explosion. When you start getting into 5 digit changeset numbers it can become a bit unwieldy to remember to type them all, but otherwise it works well. We also moved from BerkeleyDB-based to fsfs recently, and it seems to have fixed some intermittent problems we had had in the past. Another thing to look at would be Trac, which we are in the process of moving to from the horrendous nightmare of Bugzilla. It's integration with SVN as well as Wiki is quite amazing. Chris -- | Christopher Petrilli | petrilli@gmail.com

Barry Warsaw wrote:

Now's the time I pipe in to remind everyone that Atlassian has offered free (as in beer) versions of Jira and Confluence for the Python project (actually any open source project). If you haven't used these, they're definitely worth a look. Jira is the issue tracker, Confluence the wiki. They're extremely professional, usable, well-integrated, and you can integrate them with Subversion. We've used them at work for maybe a year now and I've been very happy with them. Jira is definitely one of the better issue trackers, free or not free, that I've used.

www.atlassian.com

I've also used Confluence at work, and been very impressed. A very nice feature which I haven't seen anywhere else is that the Wiki pages allow comments as well as direct editing - it allows discussion *about* the page to occur in a normal answer/response fashion, possibly leading to eventual update of the page text itself.

They're not Python and they're not open source, so perhaps it's legitimate to dismiss them because of that. But they are also definitely cool. At the Atlassian folks are very cool too, and fans of FOSS.

I'd hope we wouldn't dismiss them out of hand - one of the things that appeals to me about Python is the philosophy that open-source and protected-source groups can both benefit from working together. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

"Martin v. Löwis" wrote:

Fred L. Drake, Jr. wrote:

...

More interestingly, keeping it in a single repository makes it easier to merge projects, or parts of projects, together, without losing the history. This would be useful when developing packages that may be considered for the standard library, but which also need to continue separate releases to support older versions of Python. We've found it very handy to keep multiple projects in a single repository for zope.org.

So do you use project/trunk or trunk/project? If the former, I would need to get instructions on how to do the conversion from CVS.

For ipython, which recently went through cvs2svn, I found that moving over to a project/trunk structure was a few minutes worth of work. Since svn has moving commands, it was just a matter of making the extra project/ directory and moving things one level down the hierarchy. Even if cvs2svn doesn't quite create things the way you want them in the long run, svn is flexible enough that a few manual tweaks should be quite easy to perform. Cheers, f

Fernando Perez wrote:

For ipython, which recently went through cvs2svn, I found that moving over to a project/trunk structure was a few minutes worth of work. Since svn has moving commands, it was just a matter of making the extra project/ directory and moving things one level down the hierarchy. Even if cvs2svn doesn't quite create things the way you want them in the long run, svn is flexible enough that a few manual tweaks should be quite easy to perform.

Doesn't this give issues as *every* file the starts out renamed? e.g. what if you want "revision 100 of project/trunk/foo", but, at revision 100, it really was trunk/project/foo? Regards, Martin

On Fri, 2005-07-29 at 17:21, "Martin v. Löwis" wrote:

Doesn't this give issues as *every* file the starts out renamed? e.g. what if you want "revision 100 of project/trunk/foo", but, at revision 100, it really was trunk/project/foo?

I think it only matters if you use urls. I working directories, I think everything gets tracked correctly. I could be wrong about that though. -Barry

On Thu, 2005-07-28 at 20:15, Leif Hedstrom wrote:

I'm definitely positive to a migration to Subversion, but I'd be really concerned about using plain text authentication mechanisms.

We won't use plain text, but we may (or, we currently do) use basic auth over ssl. The security then is in the passwords, so we have to make sure they're generated securely.

We should have enough man powers to come up with some secure solution here :).

Volunteers (i.e. those with actual free time to give on implementing any solution) are definitely welcome. -Barry

Barry Warsaw wrote:

We won't use plain text, but we may (or, we currently do) use basic auth over ssl. The security then is in the passwords, so we have to make sure they're generated securely.

That (sort of) *is* plain text passwords. Somebody who took over svn.python.org can get the password. In public-key or digest authentication, this won't be possible. Regards, Martin

Barry Warsaw wrote:

Public/private keys would be better, and if anybody knows how to set up a Subversion server to use these without having to create accounts for everyone, I think we (the pythong.org admins) would love your help.

I'll play around with it some this weekend, see what's possible, and was isn't. I'm thinking we ought to be able to use SSH's configuration to only allow one specific command, e.g. something like this in the authorized_keys: no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/usr/bin/svnserve" -- Leif

At 06:39 PM 7/29/2005 -0400, Barry Warsaw wrote:

But that would still require us to create accounts for every committer, right?

No. Just one account. You can have more than one key listed in authorized_keys, using svnserve --tunnel-user and sshd will select the right command based on the public key the client authenticates with.

At 05:54 PM 7/29/2005 -0400, Barry Warsaw wrote:

Public/private keys would be better, and if anybody knows how to set up a Subversion server to use these without having to create accounts for everyone, I think we (the pythong.org admins) would love your help.

From the svnserve man page: -t, --tunnel Causes svnserve to run in tunnel mode, which is just like the inetd mode of operation (serve one connection over stdin/stdout) except that the connection is considered to be pre-authenticated with the username of the current uid. This flag is selected by the client when running over a tunnel agent. --tunnel-user=username When combined with --tunnel, overrides the pre-authenticated username with the supplied username. This is useful in combina- tion with the ssh authorized_key file's "command" directive to allow a single system account to be used by multiple committers, each having a distinct ssh identity. So, it looks like you'd just need to set up public keys for each user, and list them in authorized_keys. Presumably doing something like this: command="/usr/bin/svnserve --root=/svnroot -t --tunnel-user=username",no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa [key info here] would therefore do the trick. I've used a similar arrangement for my own CVS repository, but haven't tried it for SVN yet.

On Thursday 28 July 2005 07:21 pm, Tim Peters wrote:

[Martin v. Löwis]

...

The conversion should be done using cvs2svn utility, available e.g. in the cvs2svn Debian package. The command for converting the Python repository is

<snip>

...

Sample results of this conversion are available at

http://www.dcl.hpi.uni-potsdam.de/python/ http://www.dcl.hpi.uni-potsdam.de/distutils/

I'm sending this to Jim Fulton because he did the conversion of Zope Corp's code base to SVN. Unfortunately, Jim will soon be out of touch for several weeks. Jim, do you have time to summarize the high bits of the problems you hit? IIRC, you didn't find any conversion script at the time capable of doing the whole job as-is. If that's wrong, it would be good to know that too.

The conversion script isn't perfect and does fail on complex CVS arrangements or where there is extensive history to migate. However it appears above that Martin has already tried the script out, with success. BTW, re SSH access on python.org, using Apache's SSL support re https would provide as good of security without the risk of giving out shell accounts. SSL would encrypt the link and require a password or permit cert auth instead, same as SSH. Cert admin needn't be hard if only a single server cert is used, with client passwords, instead of client certs. -Jeff

On Thu, 2005-07-28 at 22:14, Tim Peters wrote:

Ah, before I forget, "single repository" has worked very well for Zope (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... projects):

http://svn.zope.org/

Long URLs don't really get in the way in practice (rarely a need to type one after initial checkout; even "svn switch" is usually just a tail-end cmdline edit starting from a copy+paste of "svn info" output).

It depends. In my use of svn, I do a lot of cross-branch merging and repo-side tagging. Those are done with urls and in those cases, long urls can suck. But we may not do a ton of that with the Python project, and besides it might not be important enough to split the directories. -Barry

On Thu, 2005-07-28 at 22:59, Tim Peters wrote:

Yup, me too -- between the two of us, we don't have enough fingers to count how many trunks, branches, and tags of ZODB and Zope I have to fiddle with.

I have a small inkling of your pain.

They're all still copy, paste, tail-edit for me, and-- indeed! --having them all in the same repository is what makes just-the-tail editing possible.

Good point!

Ya, in Python we make a branch about once per release, + once per 5 years when Jeremy underestimates how long it will take to finish a crusade <wink>.

So are you saying that moving to svn will let us do more long lived branches? Yay! -B

On Sun, 2005-07-31 at 23:54, Stephen J. Turnbull wrote:

...

...

...

...

...

"BAW" == Barry Warsaw writes:

BAW> So are you saying that moving to svn will let us do more long BAW> lived branches? Yay!

Yes, but you still have to be disciplined about it. svn is not much better than cvs about detecting and ignoring spurious conflicts due to code that gets merged from branch A to branch B, then back to branch A. Unrestricted cherry-picking is still out.

Yeah. IMHO the sadest thing about SVN is it doesn't do branch/merge properly. All the other cool stuff like renames etc is kinda undone by that. For a definition of properly, see; http://prcs.sourceforge.net/merge.html This is why I don't bother migrating any existing CVS projects to SVN; the benefits don't yet outweigh the pain of migrating. For new projects sure, SVN is a better choice than CVS. -- Donovan Baarda

At Mon, 01 Aug 2005 10:52:03 -0700, Donovan Baarda wrote:

On Sun, 2005-07-31 at 23:54, Stephen J. Turnbull wrote:

...

...

...

...

...

> "BAW" == Barry Warsaw writes:

BAW> So are you saying that moving to svn will let us do more long BAW> lived branches? Yay!

Yes, but you still have to be disciplined about it. svn is not much better than cvs about detecting and ignoring spurious conflicts due to code that gets merged from branch A to branch B, then back to branch A. Unrestricted cherry-picking is still out.

Yeah. IMHO the sadest thing about SVN is it doesn't do branch/merge properly. All the other cool stuff like renames etc is kinda undone by that. For a definition of properly, see;

http://prcs.sourceforge.net/merge.html

This is why I don't bother migrating any existing CVS projects to SVN; the benefits don't yet outweigh the pain of migrating. For new projects sure, SVN is a better choice than CVS.

Since Python is Open Source are you looking at Per Force which you can use for free and seems to be a happy medium between something like CVS and something horrific like Clear Case? Later, George

[Martin von Löwis]

The PEP is only about Subversion. I think anything but Subversion is ruled out because:

- there is no offer to host that anywhere (for subversion, there is already svn.python.org)

- there is no support for converting a CVS repository (for subversion, there is cvs2svn)

I quickly discussed Subversion with a few friends. While some say Subversion is the most reasonable avenue nowadays, others them told me they found something more appealing than Subversion: http://www.venge.net/monotone/ The hosting paradigm is fairly different, and for a few weeks now, they have a CVS repository converter. In my very naive eyes, the centralised aspects of Python development are be better represented with Subversion. It is notable also that Subversion if more Python-friendly than Monotone, with its Lua-based scripting. I did not deepen why, but at first glance, Monotone does not seduce me. On the other hand, the two guys saying good about Monotone are well informed (and also well known), so I would not dismiss their opinion so lightly. So, it might be worth at least a quick look? :-) -- François Pinard http://pinard.progiciels-bpi.ca

[Raymond Hettinger]

...

http://www.venge.net/monotone/

The current release is 0.21 which suggests that it is not ready for primetime.

It suggests it, yes, and to me as well. On the other hand, there is a common prejudice that something requires many releases, or frequent releases, to be qualified as good. While it might be true on average, this is not necessarily true: some packages need not so many steps for becoming very usable, mature or stable. (Note that I'm not asserting anything about Monotone, here.) We should merely keep an open mind. -- François Pinard http://pinard.progiciels-bpi.ca

[Donovan Baarda]

It is true that some well designed/developed software becomes reliable very quicky. However, it still takes heavy use over time to prove that.

There is wisdom in your say! :-) -- François Pinard http://pinard.progiciels-bpi.ca

On 8/2/05, "Martin v. Löwis" wrote:

George V. Neville-Neil wrote:

...

Since Python is Open Source are you looking at Per Force which you can use for free and seems to be a happy medium between something like CVS and something horrific like Clear Case?

No. The PEP is only about Subversion. Why should we be looking at Per Force? Only because Python is Open Source?

Perforce is a commercial product, but it can be had for free for verified Open Source projects, which Python shouldn't have any problem with. There are other problems, like you have to renew the agreement every year, but it might be worth considering, given the fact that it's an excellent system.

I think anything but Subversion is ruled out because: - there is no offer to host that anywhere (for subversion, there is already svn.python.org)

We could host a Perforce repository just as easily, I would think.

- there is no support for converting a CVS repository (for subversion, there is cvs2svn)

I'd put $20 on the fact that cvs2svn will *not* work out of the box for converting the python repository. Just call it a hunch. In any case, the Perforce-supplied cvs2p4 should work at least as well. -- Nick

...

...

...

...

"aahz" == aahz writes:

aahz> I'd rather not rely on licensing of a closed-source system; aahz> one of the points made during the talk was that the Linux aahz> project had to scramble when they lost their Bitkeeper aahz> license Python is unlikely to throw away its license in the same way, I should think. For additional security, you could try to negotiate a perpetual license on a particular version, or a license that required substantial notice (say, six months) for termination. I would imagine you could get them; the only reason for the vendor not to give them would be spite. The problem with both of those options is the one that Martin already pointed out: negotiation takes effort. There are several good open source alternatives, one of which (svn) is well-established and gets excellent reviews for those goals it sets itself, which happen to be solving the problems (as opposed to missing features) of CVS. Why spend effort on negotiating licenses and preparing for potential vendor relationship problems, unless there's acknowledged need for features svn doesn't provide? -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

On 8/4/05, "Martin v. Löwis" wrote:

Nicholas Bastin wrote:

...

Perforce is a commercial product, but it can be had for free for verified Open Source projects, which Python shouldn't have any problem with. There are other problems, like you have to renew the agreement every year, but it might be worth considering, given the fact that it's an excellent system.

So we should consider it because it is an excellent system... I don't know what that means, in precise, day-to-day usage terms (i.e. what precisely would it do for us that, say, Subversion can't do).

It's a mature product. I would hope that that would count for something. I've had enough corrupted subversion repositories that I'm not crazy about the thought of using it in a production system. I know I'm not the only person with this experience. Sure, you can keep backups, and not really lose any work, but we're moving over because we have uptime and availability problems, so lets not just create them again.

...

...

I think anything but Subversion is ruled out because: - there is no offer to host that anywhere (for subversion, there is already svn.python.org)

We could host a Perforce repository just as easily, I would think.

Interesting offer. I'll add this to the PEP - who is "we" in this context?

Uh, the Python community. Which is currently hosting a subversion repository, so it doesn't seem like a stretch to imagine that p4.python.org could exist just as easily.

...

...

- there is no support for converting a CVS repository (for subversion, there is cvs2svn)

I'd put $20 on the fact that cvs2svn will *not* work out of the box for converting the python repository. Just call it a hunch.

You could have read the PEP before losing that money :-) It did work out of the box.

Pardon me if I don't feel that I'd like to see a system in production for a few weeks before we declare victory. The problems with this kind of conversion can be very subtle, and very painful. I'm not saying we shouldn't do this, I'm just saying that we should take an appropriate measure of how much greener the grass really is on the other side, and how much work we're willing to put in to make it that way. -- Nick

On Sun, Aug 07, 2005 at 11:51:49PM -0400, Barry Warsaw wrote:

Has anyone experienced svn corruptions with the fsfs backend? I haven't, across quite a few repositories.

I haven't. But I must admit that the repositories I'm working with aren't big. The bigest is at svn.colorstudy.com (I am working on SQLObject) and since the time Ian has switched from dbfs to fsfs I don't remember any problems with the repo. Speaking of merge. SVN relived much pain that CVS had gave me. With CVS I had a lot of conflicts - if the code to be merged is already there (had been merged from another branch) one got conflict. If the code contains CVS keywords (__version__ = "$Id$") cvs merge always produced conflicts. SVN fixed both problems so now I see only real conflicts. SVN just ignores the code to be merged if it has ben already merged; and SVN convert keywords internally to its default form ($Id$ instead of $Id: python.c 42 phd $) before merging. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

On Sun, Aug 07, 2005, Barry Warsaw wrote:

We'd also have to teach the current crop of developers how to use the client tools effectively. I think it's fairly simple to teach a CVS user how to use Subversion, but have no idea if translating CVS experience to Perforce is as straightforward.

The impression I got from Alex Martelli is that it's not particularly straightforward. (Google apparently uses Perforce.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ The way to build large Python applications is to componentize and loosely-couple the hell out of everything.

Nicholas Bastin wrote:

It's a mature product. I would hope that that would count for something.

Sure. But so is subversion.

I've had enough corrupted subversion repositories that I'm not crazy about the thought of using it in a production system.

I had the last corrupted repository with subversion 0.23. It has matured since then. Even then, invoking db_recover would restore the operation, without losing data (i.e. I did not need to go to backup).

...

Interesting offer. I'll add this to the PEP - who is "we" in this context?

Uh, the Python community. Which is currently hosting a subversion repository, so it doesn't seem like a stretch to imagine that p4.python.org could exist just as easily.

Ah. But these people have no expertise with Perforce, and there is no Debian Perforce package, so it *is* a stretch assuming that they could also host a perforce directory. So I should then remove your offer to host a perforce installation, as you never made such an offer, right?

Pardon me if I don't feel that I'd like to see a system in production for a few weeks before we declare victory. The problems with this kind of conversion can be very subtle, and very painful. I'm not saying we shouldn't do this, I'm just saying that we should take an appropriate measure of how much greener the grass really is on the other side, and how much work we're willing to put in to make it that way.

Yes. That's what this PEP is for. So I guess you are -1 on the PEP. Regards, Martin

On Mon, 2005-08-15 at 12:27 -0400, Nicholas Bastin wrote:

On 8/8/05, "Martin v. Löwis" wrote:

...

Nicholas Bastin wrote:

...

It's a mature product. I would hope that that would count for something.

Sure. But so is subversion.

I will then assume that you and I have different ideas of what 'mature' means.

Bigger projects than Python use it and consider it mature for real use (All the Apache projects, all of KDE, GNOME is planning on switching soon, etc). I've never seen a corrupted FSFS repo, only corrupted BDB repos, and I will happily grant that using BDB ended up being a big mistake for Subversion. Not one that could have easily been foreseen at the time, but such is life. But this is why FSFS is the default for 1.2+ I've never seen you post about a corrupted repository to svn-users or svn-dev or file a bug, so i can't say why you see corrupted repositories if they are FSFS ones. --Dan

Nicholas Bastin wrote:

It's a mature product. I would hope that that would count for something. I've had enough corrupted subversion repositories that I'm not crazy about the thought of using it in a production system. I know I'm not the only person with this experience.

compared to Perforce, SVN is extremely fragile. I've used both for years, and I've never had Perforce repository break down on me. our SVN repositories are relatively stable these days, but the clients are still buggy as hell (mostly along the "I don't feel like doing this today, despite the fact that it worked yesterday, and I don't feel like telling you what's wrong either" lines. having to nuke workspaces from time to time gets boring, quickly.) in contrast, Perforce just runs and runs and runs. the clients always do what you tell them. and server maintenance is trivial; just make sure that the server starts when the host computer boots, and if you have enough disk, just leave it running. if you're tight on disk space, trim away some log files now and then. that's it. but despite this, if all you need is a better CVS, I'd say SVN is good enough for today's python-dev. I'd still think that a more distributed, mail-driven system (built on top of Mercurial, Bazaar-NG, or some such (*)) would speed up both development and patch processing, and also make it a lot easier for "casual contributors" and "drive-by developers" to help develop Python, but that's another story. </F> *) being able to ship a fully working Python-powered SCM with the Python source code would be an extra coolness bonus, of course.

Guido van Rossum wrote:

I'm intrigued by Linus Torvald's preference for extremely distributed source control, but I have no experience and it seems a bit, um, experimental.

"git", which is Linus' home-grown replacement for BitKeeper, quickly attracted a development community and has grown into a reasonably full-featured distributed RCS. It is apparently already stable enough for serious use. If I was trying to pick an RCS for a large, distributed project, I would at least investigate it as a possibility. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ -----------------------------------------------------------------------

...

Since Python is Open Source are you looking at Per Force which you can use for free and seems to be a happy medium between something like CVS and something horrific like Clear Case?

No. The PEP is only about Subversion. Why should we be looking at Per Force? Only because Python is Open Source?

Perforce offers free licensing to open source projects.

I think anything but Subversion is ruled out because: - there is no offer to host that anywhere (for subversion, there is already svn.python.org) - there is no support for converting a CVS repository (for subversion, there is cvs2svn)

There *is* support for converting a CVS repository to Perforce [1]. Perforce is very good, stable, solid, reliable, good tools, etc. etc. but I'd tend to support SVN over Perforce for Python development. Perforce usage is quite different than CVS (would be a painful re-learning for old CVS-hands) and SVN tends to better support highly distributed development: most operations don't need to talk to the server, with Perforce (aka p4), almost *all* operations talk to the server. This can be somewhat mitigated with "p4proxy" (a tool that Perforce also provides) but people would be happier with SVN, I'd bet. [1] It is a project called VCP. Some details here (I didn't look too hard): http://www.cpan.org/modules/by-module/LWP/AUTRIJUS/VCP-autrijus-snapshot-0.9... Trent -- Trent Mick TrentM@ActiveState.com

Trent Mick wrote:

...

No. The PEP is only about Subversion. Why should we be looking at Per Force? Only because Python is Open Source?

Perforce offers free licensing to open source projects.

Ok, so I now got "it's mature", and "it would be without charges". Given that it is now running against Subversion, I would be still interested in advantages that it offers compared to svn. The biggest disadvantage, to me, is that few people know how to use it (myself included). I don't trust tools I've never used. So for me, as the author of this PEP, usage of the revsion control software is non-negotiable (selection of hoster, to a limited degree, is). If you want to see Perforce used for the Python development, you should write a counter-PEP, so we could let the BDFL decide. [This is a theoretical "you" here, since you then explain that you would still prefer to use subversion] Regards, Martin

[Trent Mick]

... There are other little things, like not being able to trim the check-in filelist when editing the check-in message. For example, say you have 10 files checked out scattered around the Python source tree and you want to check 9 of those in.

This seems dubious, since you're not checking in the state you actually have locally, and you were careful to run the full Python test suite with your full local state ;-)

Currently with svn you have to manually specify those 9 to be sure to not include the remaining one. With p4 you just say to check-in the whole tree and then remove that one from the list give you in your editor with entering the check-in message. Not that big of a deal.

As a purely theoretical exercise <wink>, the last time I faced that under SVN, I opened the single file I didn't want to check-in in my editor, did "svn revert" on it from the cmdline, checked in the whole tree, and then hit the editor's "save" button. This doesn't scale well to skipping 25 of 50, but it's effective enough for 1 or 2.

On Mon, 2005-08-08 at 19:29, Tim Peters wrote:

...

Currently with svn you have to manually specify those 9 to be sure to not include the remaining one. With p4 you just say to check-in the whole tree and then remove that one from the list give you in your editor with entering the check-in message. Not that big of a deal.

As a purely theoretical exercise <wink>, the last time I faced that under SVN, I opened the single file I didn't want to check-in in my editor, did "svn revert" on it from the cmdline, checked in the whole tree, and then hit the editor's "save" button. This doesn't scale well to skipping 25 of 50, but it's effective enough for 1 or 2.

Or one could use a decent client, like say psvn under XEmacs <wink> which presents you a list of all modified files and lets you select which ones you want to commit. The one thing I dislike about svn (in my day-to-day use of it) is that it can take a VERY long time to do updates at the roots of very large trees. I once tried to check out the root of our dev tree, which contains all branches and tags. Of course the initial checkout took forever. But an update at the root made this approach unusable. svn would sit there, seemingly idle for 30-45 minutes and then take another 30-45 minutes updating the changes, which typically consisted of maybe 50 files out of thousands. And this on a gig LAN with fast h/w all around (and for Tim's sake I won't even complain about how some operating systems appear to perform much worse than others :). The smaller you can keep your working copies, the better. -Barry

Who made me the Perforce-bitch? Here I am screaming "Subversion! Subversion!" and y'all think I just using that as cover for a p4 lover affair. :) [Donovan Baarda wrote]

On Mon, 2005-08-08 at 15:49, Trent Mick wrote:

...

One feature I like in Perforce (which Subversion doesn't have) is the ability to have pending changesets. A changeset is, as with subversion, something you check-in atomically. Pending changesets in Perforce allow you to (1) group related files in a source tree where you might be working on multiple things at once to ensure and (2) to build a change description as you go. In a large source tree this can be useful for separating chunks of work.

This seems like a poor workaround for crappy branch/merge support.

More like a pretty nice independent self-organizing feature that was necessitated as a workaround for a crappy solution (clientspecs) for huge data trees.

I'm new to perforce, but the pending changesets seem dodgey to me... you are accumulating changes gradually without recording any history during the process... ie, no checkins until the end.

You want to do checkins of code in a consisten state. Some large changes take a couple of days to write. During which one may have to do a couple minor things in unrelated sections of a project. Having some mechanism to capture some thoughts and be able to say "these are the relevant source files for this work" is handy. Creating a branch for something that takes a couple of days is overkill. Perforce branching is pretty good in my experience. For very long projects one can easily create a branch.

Even worse, perforce seems to treat clients like "unversioned branches", allowing you to review and test pending changesets in other clients.

I'm not sure what you are talking about here. Yes, client information is stored on the server, but the *changes* (i.e. the diffs) on the client aren't so you must be talking about some other tool. Actually, if there *were* such a feature that would be quite handy. I'd love to be able to easily transfer my diffs developed on my Windows box to my Linux or Mac OS X box to quickly test changes there before checking in.

This supposedly allows people to review/test each others changes before they are committed. The problem is, since these changes are not committed, there is no firm history of what what was reviewed/tested vs what gets committed... ie they could be different.

The alternative being either that you have separate branches for everything (can be a pain) or just check-in for review (possibly breaking the build or functionality for other developers until the review is done). Actually the Perl guys working on PureMessage downstairs have two branches going in Perforce: one for checking into right away and then a cleaner tree to which only reviewed check-ins from the first are integrated. I'm not saying I am awash in pending changelists here. Nor that they should be used for what is better handled with branching. It is a tool (and a minor one).

Trying to develop and test a mixture of different changes in one source tree is asking for trouble... they can interact.

...within reason. Trent -- Trent Mick TrentM@ActiveState.com

...

...

...

...

"Donovan" == Donovan Baarda writes:

Donovan> It all comes down to how painless branch/merge is. Many Donovan> esoteric "features" of version control systems feel like Donovan> they are there to workaround the absence of proper Donovan> branch/merge histories. It's not that simple. I've followed both the Arch and the darcs lists---they handle a lot more branch/merge scenarios than Subversion does, but you still can't get away with zero discipline. On the other hand, for the purpose of the main repository for a well-factored project with disciplined workflow like Python, it's not obvious to me that the middle-complexity scenarios are that important. Furthermore, taking good advantage of the more complex branch/merge scenarios will require a change to Python workflow (for example, push- to-tracker will no longer be a convenient way to submit patches for most developers); that will be costly. More important, since none of the core Python people have spoken up strongly in favor of an advanced system, I would guess there's little experience to support planning a new workflow. (Cf. the Linux case, where Linus opted to roll his own.) I know many people in the Emacs communities who are successfully using CVS for the main repositories and various advanced systems (prcs, darcs, arch at least) for local branching and small group project communication. It seems fairly easy to automate that (much easier than extracting changeset information from CVS!) I think that as developers find they have need for such capabilities, the practice will grow in Python too, and then there may be a case to be built for moving the main repository to such a system. -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Donovan Baarda writes:

This is why I don't bother migrating any existing CVS projects to SVN; the benefits don't yet outweigh the pain of migrating.

I think they do. I was on dialup for a while, and would have _loved_ Python to be using SVN then -- and given how long diffs can take even over my broadband connection... Cheers, mwh PS: Wot, noone's suggested git yet? :) -- C++ is a siren song. It *looks* like a HLL in which you ought to be able to write an application, but it really isn't. -- Alain Picard, comp.lang.lisp

Martin v. Löwis wrote:

Donovan Baarda wrote:

...

Yeah. IMHO the sadest thing about SVN is it doesn't do branch/merge properly. All the other cool stuff like renames etc is kinda undone by that. For a definition of properly, see;

http://prcs.sourceforge.net/merge.html

Can you please elaborate? I read the page, and it seems to me that subversion's merge command works exactly the way described on the page.

maybe it's changed since I last looked at it, but last time I looked SVN didn't track merge histories. From the svnbook; "Unfortunately, Subversion is not such a system. Like CVS, Subversion 1.0 does not yet record any information about merge operations. When you commit local modifications, the repository has no idea whether those changes came from running svn merge, or from just hand-editing the files." What this means is SVN has no way of automatically identifying the common version. An svn merge requires you to manually identify and specify the "last common point" where the branch was created or last merged. PRCS automatically finds the common version from the branch/merge history, and even remembers the merge/replace/nothing/delete decision you make for each file as the default to use for future merges. You can see this in the command line differences. For subversion; # create and checkout branch my-calc-branch $ svn copy http://svn.example.com/repos/calc/trunk \ http://svn.example.com/repos/calc/branches/my-calc-branch \ -m "Creating a private branch of /calc/trunk." $ svn checkout http://svn.example.com/repos/calc/branches/my-calc-branch # merge and commit changes from trunk $ svn merge -r 341:HEAD http://svn.example.com/repos/calc/trunk $ svn commit -m "Merged trunc changes to my-calc-branch." # merge and commit more changes from trunk $ svn merge -r 345:HEAD http://svn.example.com/repos/calc/trunk $ svn commit -m "Merged trunc changes to my-calc-branch." Note that 341 and 345 are "magic" version numbers which correspond to the trunc version at the time of branch and first merge respectively. It is up to the user to figure out these versions using either meticulous use of tags or svn logs. In PRCS; # create and checkout branch my-calc-branch $ prcs checkout calc -r 0 $ prcs checkin -r my-calc-branch -m "Creating my-calc-branch" # merge and commit changes from trunk $ prcs merge -r 0 $ prcs checkin -m " merged changes from trunk" # merge and commit more changes from trunk $ prcs merge -r 0 $ prcs checkin -m " merged changes from trunk" Note that "-R 0" means "HEAD of trunk branch", and "-r my-calc-branch" means "HEAD of my-calc-branch". There is no need to figure out what versions of those branches to use as the "changes from" point, because PRCS figures it out for you. Not only that, but if you chose to ignore changes in certain files during the first merge, the second merge will remember that as the default action for the second merge. -- Donovan Baarda

Martin v. Löwis wrote:

Donovan Baarda wrote:

...

What this means is SVN has no way of automatically identifying the common version.

If this is too painful, you can probably use subversion to store the relevant information. For example, you could define a custom property on the directory

A script named "svnmerge" that does just that is included in the contrib directory of the Subversion tar. We (ZC) have just started using it to track two-way merge operations, but I don't have much experience with it personally yet. -- Benji York

On Thu, Jul 28, 2005, Barry Warsaw wrote:

On Thu, 2005-07-28 at 22:14, Tim Peters wrote:

...

Ah, before I forget, "single repository" has worked very well for Zope (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... projects):

http://svn.zope.org/

Long URLs don't really get in the way in practice (rarely a need to type one after initial checkout; even "svn switch" is usually just a tail-end cmdline edit starting from a copy+paste of "svn info" output).

It depends. In my use of svn, I do a lot of cross-branch merging and repo-side tagging. Those are done with urls and in those cases, long urls can suck. But we may not do a ton of that with the Python project, and besides it might not be important enough to split the directories.

Why can't you write a Python script to generate the URLs? <0.3 wink> -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ The way to build large Python applications is to componentize and loosely-couple the hell out of everything.

Martin v. Löwis wrote:

Tim Peters wrote:

...

Ah, before I forget, "single repository" has worked very well for Zope (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... projects):

http://svn.zope.org/

Long URLs don't really get in the way in practice (rarely a need to type one after initial checkout; even "svn switch" is usually just a tail-end cmdline edit starting from a copy+paste of "svn info" output).

That would indeed give conversion problems:

Nah

cvs2svn automatically generates tags/trunk/branches in the root, with the original CVS modules below; there is a single space for tags and branches (as is in the original CVS repository).

I would see two possible conversion procedures: 1. convert the modules one-by-one, adding to the same repository. I.e. python would get all the small revision numbers, then peps, then sandbox, and so on. Cross-module tags would not be supported (but likely don't exist in the first place), and the revision number would not increase in historical order. 2. convert the entire CVS, then rearrange things through svn move. This would be tedious to do (atleast for tags/branches), and it would cause all files to be renamed right from the scratch (some svn clients fail to display history past moves).

You reminded me of another reason why I used a custom conversion script. :) I did convert projects individually. I told cvs2svn to just create dump files. I then used svnload to load the dump files myself so that I could make each project a top-level directory with it's own trunk, branches and tags. I'd be happy to share my scrips, although there's nothing all that complicated about them.

So for all who would prefer to see a single repository, could somebody please 1. state how precisely the repository should be organized (with exact URLs on svn.python.org - eg. which of the non-dist directories becomes toplevel, which ones get their own tags/branches/trunk, etc).

I'm not close enough to the Python repo to offer much of an opinion other than: - IMO, a single repository is good - The repository should be orgabized by "projects", where separate projects reflect more or less independent development efforts with their own teams and schedules. (Maybe Python doesn't have many of these.)

2. state how the conversion should be performed

Once you decide what the projects are, I suggest converting each project separately. I'd be happy to share my scrips and experenice, although, as Tim noted I'll be off-line for two or three weeks starting in the next few days. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Martin v. Löwis wrote:

Jim Fulton wrote:

...

I did convert projects individually. I told cvs2svn to just create dump files. I then used svnload to load the dump files myself so that I could make each project a top-level directory with it's own trunk, branches and tags.

I'd be happy to share my scrips, although there's nothing all that complicated about them.

If that's how it worked, I'm sure I can cook my own. Just for confirmation: the svn revision numbers don't increase chronologically across 'modules', right: i.e. the first revision of the module that was converted second has a higher revision than the last revision of the first module, even though historically, the order should have been reverse.

Not that this worries me much, but I'd like to confirm there is no other way.

Right. The revision numbers reflect the order in which they are added to the svn repo. I'm pretty sure the revision meta data, in particular the date, was retained. This is an advantage advantage of using the low-level dump/load mechanism. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jeff Rush wrote:

BTW, re SSH access on python.org, using Apache's SSL support re https would provide as good of security without the risk of giving out shell accounts. SSL would encrypt the link and require a password or permit cert auth instead, same as SSH. Cert admin needn't be hard if only a single server cert is used, with client passwords, instead of client certs.

That is the currently-proposed setup. However, with the current subversion clients, you will have to save your password to disk, or type it in every time. This is the real security disk: if somebody attacks the client machine, they get access to the python source repository. Regards, Martin

Fernando Perez wrote:

I know Joe was in contact with the SVN devs to work on this, so perhaps he's using a patched version of cvs2svn, I simply don't know. But I mention it in case it proves useful to the python.org conversion.

Thanks for the pointer. It turns out that I could resolve all my conversion problems myself (following Jim Fulton's suggestion of creating dump files). I found that somebody created a patch to support different structures in cvs2svn directly, but these patches have not been integrated yet. Regards, Martin

On Friday 29 July 2005 09:17, Jim Fulton wrote:

1. We were making extensive use of symbolic links to share packages among various Zope projects. This requires special care and was the main reason I wrote my own scrips.

I don't expect that this would be an issue for Python.

I know of three symlinks in the Python repository, all from the distutils project. There's one for the package and one for each of the documents.

2. I initially tried to conver our entire repository, including all branches. This would have taken days. I finally decided to just convert our trunk, which took several hours. The main time sink was in the load step of the conversion process.

This might be a possibility for Python as well, though we have a much less complex branching structure, so the conversion may not be so difficult. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Martin v. Löwis wrote:

Jim Fulton wrote:

...

I don't expect that this would be an issue for Python.

Right.

...

2. I initially tried to conver our entire repository, including all branches. This would have taken days. I finally decided to just convert our trunk, which took several hours. The main time sink was in the load step of the conversion process.

I suspect that much of the time hit was due to the Berkely DB back end. I think I've heard that the file-system-based back end is much faster in general.

Dunno. For the Python CVS (which translates into some 40,000 revisions), on the machine which I was originally using (1GHz Pentium), conversion took 7h. On my current workstation, it takes a little over an hour.

Was this with the file-system back end? What is your current system? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Fri, 2005-07-29 at 17:32, "Martin v. Löwis" wrote:

...

Was this with the file-system back end? What is your current system?

Yes, and it is a 3 GHz dual processor (although I don't think it uses two processors) with 1GB RAM (I believe RAM size matters a lot for this process; the older machine has 512MB).

That matches my experience w/ the fsfs backend. If we do the conversion on one of the new python.org boxes, I expect it to go pretty quickly. -Barry

Michael Hudson wrote:

Would it work/how much risk would it be to create accounts with shell /bin/false?

It seems that the pydotorg admins are worried about such a prospect. I believe this alone either won't work or won't be good enough (not sure which one): If you have /bin/false as login shell, and still manage to invoke /usr/bin/svnserve remotely, you can likely also invoke /usr/bin/cat /etc/passwd remotely (or download and build the root exploit via ssh). So you would have restrict the set of valid programs to *only* svnserve. This is possible, but difficult to manage (AFAIK).

(still faintly bothered by ~/.subversion/auth/svn.simple/*...)

Indeed. I personally would prefer SSL client certificates. This is still tricky (where do you get the passphrase for the private key from (*)), but slightly better. Regards, Martin (*) In case you wonder, I'm personally using the following techniques here: - on windows, remove the passphrase from the private key/.p12 file, and encrypt the file through NTFS encryption. Then you don't need to enter a passphrase, but still nobody can steal the private key from your disk (as long as you are not logged in; the same holds for ssh-agent) - on Linux, my issue is that .subversion is on NFS, so any root user in our net can connect to the file. Therefore, I copy the .p12 file to /tmp/private_dir, and remove the passphrase there. No other machine can read the file (as /tmp is not exported), and the file goes away after machine shutdown latest (as tmp is cleaned on reboot). - again on windows, I'm working on teaching subversion the Microsoft certificate store.

M.-A. Lemburg wrote:

...

and on python.org instead of sf.net. To facilitate discussion, I have drafted a PEP describing the rationale for doing so, and the technical procedure to be performed.

Not sure about the move to svn.python.org. This would bind additional developer resources for doing administration work.

True. OTOH, there already is a subversion on svn.python.org, and administrative work is likely only about creating new accounts. I guess the current SF project admins would help out if help is needed.

If SF is such a show-stopper

It is at the moment, atleast: there currently is not svn support on sf.net.

would it be reasonable to look for managed alternatives, such as eg. CollabNet (who funded Subversion development) ?

Perhaps. Somebody would need to research the precise migration procedure. I once tried to move the Python CVS to Sunsite (or its successors), and gave up after half a year of getting nowhere; I'm personally not keen on repeating such an experience. However, if somebody stepped forward to do the actual work (perhaps based on the draft PEP), I would happily hand this project over (it would be good if that volunteer would set a deadline for negotiations with the new host). Regards, Martin

Barry Warsaw wrote:

On Fri, 2005-07-29 at 16:59, "Martin v. Löwis" wrote:

...

Perhaps. Somebody would need to research the precise migration procedure. I once tried to move the Python CVS to Sunsite (or its successors), and gave up after half a year of getting nowhere; I'm personally not keen on repeating such an experience.

I'm with you. SF is a devil we know. If we don't stick with them (and it looks like that's not an option if we want to switch to svn soon), then I say we move it to svn.python.org, where we already have a server set up and running. If we need more volunteers, well the community has always stepped up before and I'm sure it will when we come asking again.

Actually, it's a good idea to /always/ be soliciting for help. People get burned out and busy and I'd love to have a bullpen of volunteers that gets constantly refreshed.

I guess my point in suggesting a company to do the hosting was to overcome any issues with people getting burned or tired of administration. The PSF does have a reasonable budget, so why not use it to maintain the infrastructure needed for Python development and let a company do the administration of the needed servers and the importing of the CSV and tracker items into their systems ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 30 2005)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Martin v. Löwis wrote:

M.-A. Lemburg wrote:

...

The PSF does have a reasonable budget, so why not use it to maintain the infrastructure needed for Python development and let a company do the administration of the needed servers and the importing of the CSV and tracker items into their systems ?

In principle, this might be a good idea. In practice, it falls short of details: which company, what precisely are their procedures, etc. It's not always the case that giving money to somebody really gives you back the value you expect.

True, but if we never ask, we'll never know :-) My question was: Would asking a professional hosting company be a reasonable approach ?

From the answers, I take it that there's not much trust in these offers, so I guess there's not much desire to PSF money into this.

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 02 2005)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Martin v. Löwis wrote:

M.-A. Lemburg wrote:

...

True, but if we never ask, we'll never know :-)

My question was: Would asking a professional hosting company be a reasonable approach ?

It would be an option, yes, of course. It's not an approach that *I* would be willing to implement, though.

Fair enough.

...

...

From the answers, I take it that there's not much trust in these offers, so I guess there's not much desire to PSF money into this.

I haven't received any offers to make a qualified statement. I only know that I would oppose an approach to ask somebody but our volunteers to do it for free, and I also know that I don't want to spend my time researching commercial alternatives (although I wouldn't mind if you spent your time).

I don't quite understand what you meant here: are you opposing spending PSF money on a hosting company if and only if volunteers who take on the job don't get paid ? I've done a bit of research on the subject and so far only found CollabNet and VA offering commercial services in this area. VA hosts SourceForge so that's a non-option, I guess :-) I know that Greg Stein worked for CollabNet, so thought it might be a good idea to ask him about the idea to move things to CollabNet. Of course, before taking this route, I wanted to get a feeling for the general attitude towards a commercial approach, which is why I tossed in the idea. Other non-commercial alternatives are Berlios and Savannah, but I'm not sure whether they'd offer Subversion support. BTW, have you considered using Trac as issue tracker on svn.python.org ? They have a very good subversion integration, it's easy to use, comes with a wiki and looks great. Oh, and it's written in Python :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 03 2005)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Martin v. Löwis wrote:

M.-A. Lemburg wrote:

...

...

I haven't received any offers to make a qualified statement. I only know that I would oppose an approach to ask somebody but our volunteers to do it for free, and I also know that I don't want to spend my time researching commercial alternatives (although I wouldn't mind if you spent your time).

I don't quite understand what you meant here: are you opposing spending PSF money on a hosting company if and only if volunteers who take on the job don't get paid ?

No. I'm opposed to approaching somebody to do it for free, except the somebody are the pydotorg volunteers (IOW, I won't take gifts from anybody else in this matter).

Ok.

...

I've done a bit of research on the subject and so far only found CollabNet and VA offering commercial services in this area. VA hosts SourceForge so that's a non-option, I guess :-)

It's not that I dislike VA - I personally think they are doing a great job with SourceForge, and I like SourceForge a lot. There are just some issues with it (like that they offer no Subversion).

The question would be: what precisely is the commercial offering from VA: does it provide subversion? how is the user management done? etc.

I guess this was a misunderstanding on my part: VA doesn't offer their commercial solution in an ASP-like way. Their product, called SourceForge Enterprise, is a J2EE application which we'd have to install and run. They do mention Subversion as being supported by the Enterprise edition.

...

I know that Greg Stein worked for CollabNet, so thought it might be a good idea to ask him about the idea to move things to CollabNet. Of course, before taking this route, I wanted to get a feeling for the general attitude towards a commercial approach, which is why I tossed in the idea.

Ok - I expect that the project might be *done* before we even have a single commercial offer, with a precise service description, and a precise price tag. That makes commercial offers so difficult: that it is so time expensive to use them, that you might spend less time doing it yourself.

For (more or less) simple things like setting up SVN, I'd agree, but for hosting a complete development system, I have my doubts - things start to get rather complicated and integration of various different tools tends to be very time consuming. Sysadmin tasks like doing backups, emergency recovery, etc. also get more complicated once you have to deal with many different ways of data storage deployed by such tools, e.g. many of them require use of special tools to do hot backups.

...

Other non-commercial alternatives are Berlios and Savannah, but I'm not sure whether they'd offer Subversion support.

For me, they fall into the "I won't take gifts" category.

Ok, I'll drop the idea.

...

BTW, have you considered using Trac as issue tracker on svn.python.org ?

You mean, me personally? I quite like the Subversion tracker, and don't want to trade it for anything else. I know Guido wants to use Roundup (which is also written in Python), and obviously so does Richard Jones.

The main questions are the same as with this PEP: how to do the migration from SF (without losing data), and how to do the ongoing maintenance. It's just that finding answers to these questions is so much harder, therefore, this PEP is *only* about CVS.

Ok. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 04 2005)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Martin v. Löwis wrote:

M.-A. Lemburg wrote:

...

I guess this was a misunderstanding on my part: VA doesn't offer their commercial solution in an ASP-like way. Their product, called SourceForge Enterprise, is a J2EE application which we'd have to install and run. They do mention Subversion as being supported by the Enterprise edition.

Ah, ok. I don't think I want to operate such a software (and, strictly speaking, this is out of the scope of the PEP). I had the "pleasure" once of having to maintain a SourceForge installation (before SourceForge became closed source), and it was a nightmare to operate.

With J2EE I doubt that things got any easier to maintain... (assuming that you had to run the version of the software which is used on SF.net).

...

For (more or less) simple things like setting up SVN, I'd agree, but for hosting a complete development system, I have my doubts - things start to get rather complicated and integration of various different tools tends to be very time consuming.

I guess Python's development process is very simple then. We use mailing lists, CVS, newsgroups, web servers, and bug trackers, but these don't have to integrate. Many of these services are already on pydotorg, and I propose to add an additional one (revision control).

...

Sysadmin tasks like doing backups, emergency recovery, etc. also get more complicated once you have to deal with many different ways of data storage deployed by such tools, e.g. many of them require use of special tools to do hot backups.

We are doing quite well here. XS4ALL kindly does disk backup for us, and, in the specific case of Subversion's fsfs, this is all that is needed. For Postgres, we backup to disk, which then gets picked up by the disk backup.

Sounds like you have everything under control, which is good :-) BTW, in one of your replies I read that you had a problem with how cvs2svn handles trunk, branches and tags. In reality, this is no problem at all, since Subversion is very good at handling moves within the repository: you can easily change the repository layout after the import to whatevery layout you see fit - without losing any of the version history. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 04 2005)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2005-07-18: Released mxODBC.Zope.DA for Zope 2.8 ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Phillip J. Eby wrote:

At 08:28 PM 8/4/2005 +0200, M.-A. Lemburg wrote:

...

BTW, in one of your replies I read that you had a problem with how cvs2svn handles trunk, branches and tags. In reality, this is no problem at all, since Subversion is very good at handling moves within the repository: you can easily change the repository layout after the import to whatevery layout you see fit - without losing any of the version history.

Yeah, in my use of SVN I find that this is more theoretical than actual for certain use cases. You can see the history of a file including the history of any file it was copied from. However, if you want to try to look at the whole layout, you can't easily get to the old locations. This can be a royal pain, whereas at least in CVS you can use viewcvs to show you the "attic". Subversion doesn't have an attic, which makes looking at structural history very difficult.

Hmm, I usually create a tag before doing such changes in our Subversion repo. This makes it very easy to look at layouts before a restructuring. And because Subversion doesn't really care whether you do a tag, branch, or some other form of diverting versions into different namespaces (it's all just copying data), you can easily create a directory called "attic" for just this purpose and copy your structural change tags in there :-)

That having been said, I generally like Subversion, I just know that when I moved my projects to it I felt it was worth taking extra care to convert them in a way that didn't require me to reorganize the repository immediately thereafter, because I didn't want a sudden discontinuity, beyond which history would be difficult to follow.

Therefore, I'm saying that taking some care with the conversion process to get things the way we like them would be a good idea.

Still very true indeed. The fact that cvs2svn is written in Python should make this even easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 04 2005)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2005-07-18: Released mxODBC.Zope.DA for Zope 2.8 ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

M.-A. Lemburg wrote:

BTW, in one of your replies I read that you had a problem with how cvs2svn handles trunk, branches and tags. In reality, this is no problem at all, since Subversion is very good at handling moves within the repository: you can easily change the repository layout after the import to whatevery layout you see fit - without losing any of the version history.

Yes, however, I recall that some clients have problems with displaying history across renames (in particular, I believe viewcvs has this problem); also, it becomes difficult to refer to an old version by path name, since the old versions had all different path names. Jim Fulton has suggested a different approach: cvs2svn can create a dump file, and svnadmin load accepts a parent directory. Then, no renames are necessary. Regards, Martin

On Wednesday 03 August 2005 15:01, M.-A. Lemburg wrote:

Other non-commercial alternatives are Berlios and Savannah, but I'm not sure whether they'd offer Subversion support.

Berlios does offer Subversion; the docutils project is using the Berlios Subversion and SourceForge for everything else. I don't know whether Savannah is offering Subversion right now, but the last time I looked at it, it appeared nearly un-maintained. But that may just be the understated nature of that community. :-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>