On Mon, Feb 23, 2009 at 11:17 AM, Robert Cimrman <cimrman3@ntc.zcu.cz> wrote:

Hi Stéfan,

Stéfan van der Walt wrote:

...

Implementation:

Enthought generously hosts SciPy, and I hope they will continue doing so. New software will need to be installed on the server, but we have many hands willing to tackle that task: David Cournapeau and myself included. Before deploying to scipy.org, *we will configure a *different* server as a proof of concept.*

1) *Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous. *

Going git would make my life as an occasional numpy/scipy contributor really a lot easier, so big +1!

cheers, r.

I'm pretty happy with svn; it is relatively simple and has good integration and GUI tools on Windows. From all I read, git would be a big barrier for casual users (of git). From all the descriptions I've read, git is powerful for "command line junkies" who remember a large number of commands and options but not for occasional users of it. But I never installed git, because some time ago when I compared bazar, mercurial, git still didn't have much support on windows. One problem is that the bazar mirror of scipy on launchpad still fails to import, but otherwise working with mirrors for the different version control system would creating patches easier for users of scipy, for example before my committ access to scipy, I used bzrsvn to maintain my local branches. My main problem with trac tickets are missing tests, not the actual applying of the patch or bugfixes. I think low test coverage and weak testing "culture" is more of a problem than the revision control system. From what I have seen in the scipy code, it is true that, if it doesn't have a test, it is broken with high probability. Josef

On Feb 23, 2009, at 11:44 AM, David Cournapeau wrote:

50 % of the time I create a branch for numpy, I screw up because I need like 10 commands, which fail half of the time for stupid errors or time out.

I have no opinion on a switch of SciPy to git or anything else, and I'm generally interested in the prospects for distributed version control, but I really have to ask, what 10 commands could you possibly need to execute to create a branch in svn?

On Feb 23, 2009, at 1:10 PM, David Cournapeau wrote:

svn cp trunk -> branch svnmerge switch branch svnmerge init trunk svn ci -F svnmerge-commit.txt svn switch trunk svnmerge init branch svn ci -F svnmerge-commit.txt

Ahah. The answer to that is, don't use svnmerge. I tried it after you told me about it on this list and it's a disaster, at least from our perspective. We have a protocol for merges <http://matforge.org/fipy/browser/trunk/documentation/ADMINISTRATA.txt

, based directly on the guidance of the The SVN Book, and it works very well. I was prepared to concede to you that *merging* changes takes way too many steps, but you said "create a branch", which piqued my curiosity.

While svnmerge appeared to dramatically simplify all of the tagging and commenting that we presently have to do, in practice I found that it made a complete hash of things. I have no doubt that it could be used safely, but I don't believe that it actually saves any effort over doing it manually. As a case in point, seven steps for what should only be one.

Ok, that's 7 :)

J'accuse! 8^)

On Feb 23, 2009, at 1:19 PM, David Cournapeau wrote:

Let's say "using" branches is a PITA in svn.

I don't quite concede that, but I'll grant that it's harder than it ought to be. Some of my tolerance probably stems from the fact that svn beats the snot out of cvs in this regard. That doesn't mean that svn shouldn't be better (and, as I understand, the latest revs are better, but we're not using them yet).

On Tue, Feb 24, 2009 at 7:28 PM, Ralf Ahlbrink <Ralf_Ahlbrink@web.de> wrote:

David Cournapeau wrote:

...

On Tue, Feb 24, 2009 at 2:24 AM, Jonathan Guyer <guyer@nist.gov> wrote:

...

On Feb 23, 2009, at 11:44 AM, David Cournapeau wrote:

...

50 % of the time I create a branch for numpy, I screw up because I need like 10 commands, which fail half of the time for stupid errors or time out.

I have no opinion on a switch of SciPy to git or anything else, and I'm generally interested in the prospects for distributed version control, but I really have to ask, what 10 commands could you possibly need to execute to create a branch in svn?

svn cp trunk -> branch svnmerge switch branch svnmerge init trunk svn ci -F svnmerge-commit.txt svn switch trunk svnmerge init branch svn ci -F svnmerge-commit.txt

Ok, that's 7 :)

Hi David,

your statement here about subversion branching/merging is somewhat misleading, because you presume subversion <= 1.4. The current version (1.5) supports 'merge-tracking', i.e. the svnmerge functionality is transparently incorporated into svn. See e.g. http://blog.red-bean.com/sussman/?p=92 or http://svnbook.red-bean.com/en/1.5/index.html (especially chapter 4). Migration to 1.5 repositories by svnadmin dump and load actions works well.

AFAIK, svn 1.5 only solve some of the problems, but it it still very slow, which is one of the main issue. It also fails in case of renames, etc... I have seen reports of people sticking to svnmerge with 1.5. Also, building subversion is a royal PITA, I had to do it once on a CENTOS system, it took me a while - not all distributions have svn 1.5. Frankly, if we change, better change to a better system. svn is just an inferior tool in almost every possible way. Changing svn to 1.5 brings most of the pain that would bring DVCS, and for no clear improvement. I think at the moment, we would be better to stick to 1.4 for now, bring some official git mirrors, and work on other issues, David

On Mon, Feb 23, 2009 at 1:00 PM, <josef.pktd@gmail.com> wrote:

On Mon, Feb 23, 2009 at 11:44 AM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

...

I agree that tools cannot solve the problem of lack of test. But to give you a scenario: I had a couple of hours to spend on triaging bugs on numpy for 1.3 release last WE. I have done almost nothing: I can't easily get tickets with attached patches, I can't control the bug tracker from the command line (I can't say: give me all the tickets since 1.2 with attached patches, give me all tickets on numpy.core since 1.2, etc...). I find the whole workflow extremely frustrating personally.

This seems to me to be a problem with the (older) trac ticketing. Queries for attachment, I found, work ok, but I didn't find a query for tickets with recent comments. Going through the stats review tickets to see which have any relevant information is really slow, and I still never went through all of them to see if someone added a comment or not. Has this improved with the new version of trac, scipy trac is 0.10.2 and current trac is 0.11.3?

But I don't see how changing the revision control system helps with this.

I installed msysgit following the links provided and the git gui looks ok, although the file browser is missing the basic file information (revision numbers, dates, added to repository or not). I don't like the bash shell because I don't know the keys for basic things (or I have to look them up), but that's only a smaller problem.

Bash on windows is ugly anyways ;)

The question is whether the revision control should be easier for developer or easier for users to create patches, and that might not be the same system.

Also, at least with bazaar and bzrsvn and, I guess, git-svn, I still don't see what the (major) disadvantage for branching is of using a mirror of the central svn repository in a decentralized version control.

I think git-svn takes care of most of the branching problem on a local basis. Where it fall down, IMHO, is in testing a branch on the builtbots and sharing a branch among two or three people.

(However, I usually just work with only short lived branches to try out fixes and new code, or do selective merging manually. Personally, my second incentive for keeping essentially only one main branch (svn) in my own code is, that I am not very well organized with branches, and having 10 to 15 versions/branches of a package on my hard drive ends up just wasting space and time. Also, I like the svn integration in eclipse.)

I tend to agree that changing the VCS shouldn't be the first priority. It sounds like tracking the tickets and svn changes relevant to particular tagged releases might be where the effort should go. Chuck

On Tue, Feb 24, 2009 at 6:11 AM, Robert Kern <robert.kern@gmail.com> wrote:

On Mon, Feb 23, 2009 at 15:06, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

I think git-svn takes care of most of the branching problem on a local basis. Where it fall down, IMHO, is in testing a branch on the builtbots and sharing a branch among two or three people.

Why is that? Push your git branch to github.

This can cause trouble when dcommitting back to svn. I don't remember the exact scenario, but I managed to break some things (like committing twice to svn) because I was not careful. It is easy to forget getting svn references in addition to git ones, maybe this is linked - I have not tried really hard to understand the problem to be honest. cheers, David

-- Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev

On Mon, Feb 23, 2009 at 2:11 PM, Robert Kern <robert.kern@gmail.com> wrote:

On Mon, Feb 23, 2009 at 15:06, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

I think git-svn takes care of most of the branching problem on a local basis. Where it fall down, IMHO, is in testing a branch on the builtbots and sharing a branch among two or three people.

Why is that? Push your git branch to github.

Does github work with our buildbots? Chuck

2009/2/23 Robert Kern <robert.kern@gmail.com>:

...

Does github work with our buildbots?

Probably not right now. But that's exactly the same problem if the main repo were a git one, too.

The latest version of Buildbot has support for git. S.

On Mon, Feb 23, 2009 at 9:46 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

On Mon, Feb 23, 2009 at 2:27 PM, Robert Kern <robert.kern@gmail.com> wrote:

...

On Mon, Feb 23, 2009 at 15:19, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

On Mon, Feb 23, 2009 at 2:11 PM, Robert Kern <robert.kern@gmail.com> wrote:

...

On Mon, Feb 23, 2009 at 15:06, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

I think git-svn takes care of most of the branching problem on a local basis. Where it fall down, IMHO, is in testing a branch on the builtbots and sharing a branch among two or three people.

Why is that? Push your git branch to github.

Does github work with our buildbots?

Probably not right now. But that's exactly the same problem if the main repo were a git one, too.

True enough, I'm not arguing for changing the main repository, just pointing out what I miss when making branches locally with git-svn.

Watching this discussion with interest... Although git seems to have more momentum here, I though I would point out bzr-svn, which I think allows distributed collaboration around a central svn-hosted branch without the problems people have mentioned with git-svn. By storing the bzr metadata in svn properties, a full bazaar branch can be hosted in svn, so people can branch off it, merge with each other, merge back etc. In my experience it's worked very well for giving the distributed benefits of branching from a centralised svn repo. Cheers Robin

On Mon, Feb 23, 2009 at 10:59 PM, Andrew Straw <strawman@astraw.com> wrote:

...

...

True enough, I'm not arguing for changing the main repository, just pointing out what I miss when making branches locally with git-svn.

Watching this discussion with interest... Although git seems to have more momentum here, I though I would point out bzr-svn, which I think allows distributed collaboration around a central svn-hosted branch without the problems people have mentioned with git-svn. By storing the bzr metadata in svn properties, a full bazaar branch can be hosted in svn, so people can branch off it, merge with each other, merge back etc.

In my experience it's worked very well for giving the distributed benefits of branching from a centralised svn repo.

Robin, can you point us to a public svn repo where non-trivial branching is happening with bzr? I had lots of trouble trying to get anything working in my attempts.

Hi, I'm not sure what qualifies as non-trivial branching (DVCS people say all branching should be trivial!). I have only used it on my private repo myself, but I haven't had any problems. Asking on IRC, a good example is GNOME. Here are the developer branches created using bzr-svn http://bzr-playground.gnome.org/ - although it looks like they aren't pushing directly back to svn (but going through patches). This is the best I could find documentating a workflow for using bzr with svn: http://www.serverzen.net/starting-with-bazaar-bzr-svn But I think to avoid problems with merging the trick is to have a bzr checkout from svn as your sort of trunk branch, which you can then branch with bzr to create feature branches. Upstream branches can be pulled to the trunk branch, then merged to your feature branches, and when you want to push stuff back you merge it into your trunk checkout (which also commits it to svn). There are options to either push each individual commit as an svn commit, or just have a single merge commit in svn (the metadata for the bzr commits are there so they can be seen by other bzr users). Because all the metadata is in svn, you should be able to merge with anyone else who has their branch based on a svn checkout... Again not 100% on all this, but I believe that's how it works. When I looked the DVCS I settled on this since I wanted to work with my existing svn repo and it seemed to be the best dvcs subversion interface. Cheers Robin

On Tue, Feb 24, 2009 at 12:28 AM, Andrew Straw <strawman@astraw.com> wrote:

Robin wrote:

...

...

Robin, can you point us to a public svn repo where non-trivial branching is happening with bzr? I had lots of trouble trying to get anything working in my attempts. I'm not sure what qualifies as non-trivial branching (DVCS people say all branching should be trivial!). I have only used it on my private repo myself, but I haven't had any problems. Asking on IRC, a good example is GNOME. Here are the developer branches created using bzr-svn http://bzr-playground.gnome.org/ - although it looks like they aren't

On Mon, Feb 23, 2009 at 10:59 PM, Andrew Straw <strawman@astraw.com> wrote: pushing directly back to svn (but going through patches).

This is the best I could find documentating a workflow for using bzr with svn: http://www.serverzen.net/starting-with-bazaar-bzr-svn

Thanks for the links.

...

But I think to avoid problems with merging the trick is to have a bzr checkout from svn as your sort of trunk branch, which you can then branch with bzr to create feature branches. Upstream branches can be pulled to the trunk branch, then merged to your feature branches, and when you want to push stuff back you merge it into your trunk checkout (which also commits it to svn). There are options to either push each individual commit as an svn commit, or just have a single merge commit in svn (the metadata for the bzr commits are there so they can be seen by other bzr users). Because all the metadata is in svn, you should be able to merge with anyone else who has their branch based on a svn checkout...

My problem when I tried this out was that the svn metadata wasn't the same across different bzr repos cloned from the same svn repo -- thus no ability to actually share the bzr branches between bzr repos. It sounds like you have only tried this from a single bzr repo? Plus, I didn't like polluting the svn repo with the bzr metadata, particularly given this no-ability-to-create-the-same-bzr-clones issue. Git solves the first issue (different git clones of the same svn repo produce the same git repo) and thereby somewhat eliminates the need to store git metadata in the svn repo. Which is why I like git-svn more than bzr-svn. But the lack of one-to-one bidirectional mapping between DVCS branches and svn branches it was prevents any of these schemes from working on any DVCS, as far as I can see.

I guess this is getting slightly off-topic - also I'm not trying to advocate anything over anything else (and don't have the experience to do so!) but for the record I'm pretty sure what you say here about bzr-svn is wrong. Seperate bzr clones of the same svn repository *are* the same, and one can merge between them. This works without bzr metadata in the svn repo (simple example below). Bzr metadata is only needed to provide the one-to-one mapping you mention, between bzr branches and svn branches (for example, one can checkout from subversion using bzr-svn, branch using bazaar, modify, and push back to a different location in svn and it will be a svn branch). Bzr metadata is not visible if you are running svn >= 1.5, and for other versions of svn whatever you are using can be configured to ignore it (it is just svn properties as far as I understand). I know there are lots of arguments for/against the different DVCS's - but I think the one thing bazaar is clearly ahead on is this svn integration. # two separate checkouts jm-g26b101:tmp robince$ bzr co http://svn.scipy.org/svn/scikits/trunk/mlabwrap mlabwrap-trunk jm-g26b101:tmp robince$ bzr co http://svn.scipy.org/svn/scikits/trunk/mlabwrap mlabwrap-trunk2 # branch each checkout jm-g26b101:tmp robince$ bzr branch mlabwrap-trunk mlabwrap-branch1 Branched 93 revision(s). jm-g26b101:tmp robince$ bzr branch mlabwrap-trunk2 mlabwrap-branch2 Branched 93 revision(s). # add a file to each jm-g26b101:tmp robince$ cd mlabwrap-branch1 jm-g26b101:mlabwrap-branch1 robince$ echo "a new file" > new.txt jm-g26b101:mlabwrap-branch1 robince$ bzr add new.txt adding new.txt add completed jm-g26b101:mlabwrap-branch1 robince$ bzr ci -m "add a new file" Committing to: /Users/robince/tmp/mlabwrap-branch1/ added new.txt Committed revision 94. jm-g26b101:mlabwrap-branch1 robince$ cd ../mlabwrap-branch2/ jm-g26b101:mlabwrap-branch2 robince$ echo "another new file" > new.txt jm-g26b101:mlabwrap-branch2 robince$ bzr add new.txt adding new.txt add completed jm-g26b101:mlabwrap-branch2 robince$ bzr ci -m "a new file" Committing to: /Users/robince/tmp/mlabwrap-branch2/ added new.txt Committed revision 94. # merge works fine jm-g26b101:mlabwrap-branch2 robince$ bzr merge ../mlabwrap-branch1/ +N new.txt R new.txt => new.txt.moved Conflict adding file new.txt. Moved existing file to new.txt.moved. 1 conflicts encountered. Cheers Robin

On Mon, Feb 23, 2009 at 15:24, Stéfan van der Walt <stefan@sun.ac.za> wrote:

Hi Robert

2009/2/23 Robert Kern <robert.kern@gmail.com>:

...

On Mon, Feb 23, 2009 at 15:06, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

I think git-svn takes care of most of the branching problem on a local basis. Where it fall down, IMHO, is in testing a branch on the builtbots and sharing a branch among two or three people.

Why is that? Push your git branch to github.

What do you say to the merging difficulties that SVN causes? Git-svn doesn't address that, unfortunately.

Scenario? There are several merging difficulties of SVN, and I do believe that git-svn addresses at least some of them. Precisely what are you thinking of? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

2009/2/23 Robert Kern <robert.kern@gmail.com>:

On Mon, Feb 23, 2009 at 15:24, Stéfan van der Walt <stefan@sun.ac.za> wrote:

...

Hi Robert

2009/2/23 Robert Kern <robert.kern@gmail.com>:

...

On Mon, Feb 23, 2009 at 15:06, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

I think git-svn takes care of most of the branching problem on a local basis. Where it fall down, IMHO, is in testing a branch on the builtbots and sharing a branch among two or three people.

Why is that? Push your git branch to github.

What do you say to the merging difficulties that SVN causes? Git-svn doesn't address that, unfortunately.

Scenario? There are several merging difficulties of SVN, and I do believe that git-svn addresses at least some of them. Precisely what are you thinking of?

I was thinking of this, from http://www.kernel.org/pub/software/scm/git/docs/git-svn.html """ Running git-merge or git-pull is NOT recommended on a branch you plan to dcommit from. Subversion does not represent merges in any reasonable or useful fashion; so users using Subversion cannot see any merges you've made. Furthermore, if you merge or pull from a git branch that is a mirror of an SVN branch, dcommit may commit to the wrong branch. """ Cheers Stéfan

2009/2/24 Pauli Virtanen <pav@iki.fi>:

I'm wondering if we would benefit from an official, pull-only, automatically updating, Git mirror of the SVN repository:

I'm still wondering what the advantages are of staying with SVN. I haven't heard any compelling arguments so far, whereas we've heard numerous accounts from developers regarding the positive aspects of DVC systems. Cheers Stéfan

Hello, On Mon, Feb 23, 2009 at 2:54 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

I think the first thing to do is take a look at what sage is doing and see if we can't refurbish our current system to make it more useable.

On Mon, Feb 23, 2009 at 1:03 PM, Stéfan van der Walt <stefan@sun.ac.za> wrote:

Proposed workflow:

1. Cook up a patch 2. Attach the patch (or a URL to the patchset/branch) to the issue tracker with a REVIEW tag 3. Ping the mailing list or IRC to request a review (rinse and repeat)

Workflow for dev:

1. Request a list of patches ready for review: review - Has tests [check] - Has docs [check] - Does what it is supposed to do [check] 2. Add a POSITIVE_REVIEW or NEGATIVE_REVIEW tag as appropriate 3. Request a list of patches ready to be merged (code can be merged if seen by two pairs of eyes: reviewer + committer, reviewer + reviewer, etc. In the end it must have "positive_reviews - negative_reviews >= 2"). Review the patch (this adds one pair of eyes) and merge if appropriate.

This is roughly what we do for Sage, and it's fairly effective at getting code merged in. Also, _every_ piece of code that goes in does so through via Trac ticket. It's easy to see when bugs have been fixed. --Mike

On Tue, Feb 24, 2009 at 7:54 AM, Charles R Harris <charlesr.harris@gmail.com> wrote:

I think the first thing to do is take a look at what sage is doing and see if we can't refurbish our current system to make it more useable.

sage uses mercurial, David

Hi Josef 2009/2/23 <josef.pktd@gmail.com>

I'm pretty happy with svn; it is relatively simple and has good integration and GUI tools on Windows. From all I read, git would be a big barrier for casual users (of git). From all the descriptions I've read, git is powerful for "command line junkies" who remember a large number of commands and options but not for occasional users of it. But I never installed git, because some time ago when I compared bazar, mercurial, git still didn't have much support on windows.

Git used to be hard to use, even casually, but that changed. Most of the basic commands are very similar to what they are in SVN. Some are even simpler, for example, it is much easier to merge under Git. I think many people fall into the trap of trying to do Revision Control Acrobatics before having mastered the basics (and I count myself in that group), but the link given (http://git.or.cz/course/svn.html) shows that simple things remain simple. Apart from the technical benefits, distributed revision control has a profound impact upon the social structuring of a project. Flat is better than nested, even when it comes to code development :)

My main problem with trac tickets are missing tests, not the actual applying of the patch or bugfixes.

I would like for contributers to take responsibility for their own patches. If we have a clearly designated set of criteria for inclusion (i.e. tested, documented, peer reviewed), it becomes easier to get code *into* SciPy, providing the scaffolding needed to develop and mature code.

I think low test coverage and weak testing "culture" is more of a problem than the revision control system. From what I have seen in the scipy code, it is true that, if it doesn't have a test, it is broken with high probability.

I agree, but I think that the two subjects go hand-in-hand. In essence, by making each user part of the development team, we give them the mandate to develop or solicit unit tests for their own patches. Regards Stéfan

Charles R Harris wrote:

On Mon, Feb 23, 2009 at 9:04 AM, Stéfan van der Walt <stefan@sun.ac.za <mailto:stefan@sun.ac.za>> wrote:

*[If you only have 30 seconds to read this email, read the **bold text only]*

*Dear* SciPy *developer*s

The past while has seen a rocky ride with the SciPy servers, but yesterday Peter Wang announced that he is attending to the situation. This, then, seems like the perfect time to *stand back and take a look at our infrastructure*, and whether we should continue with the current setup.

To put this conversation into context, we have to face the facts: SciPy has a large user community relative to the number of developers. A big library of code, used by many scientists, is supported by a small handful of people all over the world. *We cannot afford* *a high barrier to contribution*, and we have to lower the effort it takes for a developer to merge contributed code.

*I'd like to propose two changes* to the status quo:

1. *Change to a distributed revision control system*, encouraging more open collaboration. 2. *Determine guidelines for code acceptance*, in terms of unit tests, documentation and peer review.

Allow me to motivate these changes, and then suggest practical approaches for their implementation:

Subversion allows only a selected group of developers to change the SciPy source code. This does not encourage a culture of meritocracy, but worse, has practical implications, in that users cannot merge their own patches. I won't discuss the advantages of distributed revision control here, but note that it shifts responsibility from the current core developers to contributers; *that benefits us all!*

This ties in with my second point: code review. The current developers have access to SVN because they are experienced programmers with knowledge of SciPy's scientific domains of application. We are unable to employ this scarce resource fully, because it simply takes too long to merge a patch from Trac, review it, *bring it up to scratch*, and commit it. *We have to put a system in place which allows contributers to take responsibility for their own patches, and for core developers to guide and advise during this process.* As it is, we have many patches waiting on Trac for up to a year or more without any feedback; that is not acceptable.

My view on testing is simple: *untested code is probably broken code* (and I can show examples from the past year's commit logs to corroborate this statement). *As for documentation, we cannot afford to be without it. * Implementation:

Enthought generously hosts SciPy, and I hope they will continue doing so. New software will need to be installed on the server, but we have many hands willing to tackle that task: David Cournapeau and myself included. Before deploying to scipy.org <http://scipy.org>, *we will configure a *different* server as a proof of concept.*

1) *Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous.*

I really like Git, but... the last time I looked windows support wasn't up to snuff. Does anyone have more recent feedback on the windows situation?

It is not ideal: it is based on a bash shell. But it does not require cygwin anymore - you can grab an exe, and get it installed on your machine for e.g. cloning and submitting a patch to the bug tracker. If you want GUI, it won't work (but no DVCS has a decent GUI: TortoiseBZR and TortoiseHG are really far behind what I would expect from a reasonable GUI on windows). I think git will never be on par compared to other tools, because git is fundamentally engrained into the unix mentality (set of tools who communicate together through text). But after having used bzr for > 2 years, I am entirely convinced that git is far ahead bzr or even hg (I don't know much hg - I looked at it at some point because it had the best svn support, but I have not followed it recently - I still closely follow bzr development). One thing about git is that the speed factor is too much emphasized IMHO - even if git was as slow as bzr, I would prefer git today. My main worries about git usage for numpy/scipy are related to the bug tracker; tracking branches is more of a problem than I initially thought. Github has some very nice concepts, but still none of the git hosting projects can for example display the history graph, which is very helpful for newcomers I think (the tools exists locally, though). cheers, David

On Mon, Feb 23, 2009 at 4:29 PM, Charles R Harris

MSysGit works pretty well these days, and is apparently being merged with official git.

http://code.google.com/p/msysgit/

And there is of course Cygwin, which includes git. S.M.

Hi Chuck 2009/2/23 Charles R Harris <charlesr.harris@gmail.com>:

...

1) Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous.

I really like Git, but... the last time I looked windows support wasn't up to snuff. Does anyone have more recent feedback on the windows situation?

I've read that msysgit (http://code.google.com/p/msysgit/) works well. From http://garrys-brain.blogspot.com/2008/04/git-for-windows-msysgit.html : """ I would say Git For Windows is very close to being "ready" and providing you are not in need of the more difficult corner cases it is ready for production use. The guys working on it have done a great job. """ I can't vouch for this information, so it would be great to hear from someone who tried it themselves. Cheers Stéfan

Robert Cimrman wrote:

Stéfan van der Walt wrote:

...

Hi Chuck

2009/2/23 Charles R Harris <charlesr.harris@gmail.com>:

...

...

1) Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous. I really like Git, but... the last time I looked windows support wasn't up to snuff. Does anyone have more recent feedback on the windows situation? I've read that msysgit (http://code.google.com/p/msysgit/) works well. From http://garrys-brain.blogspot.com/2008/04/git-for-windows-msysgit.html :

""" I would say Git For Windows is very close to being "ready" and providing you are not in need of the more difficult corner cases it is ready for production use. The guys working on it have done a great job. """

I can't vouch for this information, so it would be great to hear from someone who tried it themselves.

I have tried git on windows XP a few days ago, and it was smooth - the installation using the usual wizard, launching git gui (no command line needed!), looking at an existing project...

And did not done anything fancy, though.

A little correction: And -> I I have tried: http://msysgit.googlecode.com/files/Git-1.6.1-preview20081227.exe r.

Matthieu Brucher wrote:

...

2) Ticketing back-end: David is exploring RedMine [2], and I'd like to take a look at InDefero [3], but we'll do a careful analysis of trac-git (like FedoraHosted) too.

Why not staying with Trac ?

- no multiple project support (mostly a problem for scikits) - the ticket workflow is very awkward and too simplistic - no command line interface (may be solved with the xmlrpc plugin, though, I have no experience with it) - query system very primitive - etc... cheers, David

[I'm +1 on git, although my level of developer-ness should be weighted appropriately. (However, it's likely to be at least marginally higher if the switch to git is made.)] I just came across bugs everywhere, which looks like an interesting in-repo bug tracker with potential for web GUIs: http://bugseverywhere.org/be/show/HomePage As a big plus for this crowd, it's written in Python. -Andrew Stéfan van der Walt wrote:

2009/2/23 Matthieu Brucher <matthieu.brucher@gmail.com>:

...

...

2) Ticketing back-end: David is exploring RedMine [2], and I'd like to take a look at InDefero [3], but we'll do a careful analysis of trac-git (like FedoraHosted) too. Why not staying with Trac ?

I'm still investigating Trac. It is installed on our students' code hosting service, and I would like to see whether trac-git is mature and well integrated. Do you have any experience with it?

Cheers Stéfan _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev

Hi Matthew 2009/2/23 Matthew Brett <matthew.brett@gmail.com>:

2) Yes. Please. That would really help. How do we think we should best get there? If we don't switch to DVCS immediately?

It won't require a revolution (hopefully), just consensus amongst developers. I view anyone with SVN access as a gatekeeper. Gatekeepers should agree to a code of conduct, by which they are held. 1. No code enters SciPy unless it had two pairs of eyes on it: reviewer and committer, reviewer and reviewer, reviewer and release manager, etc. All tickets ready for merging are marked in Trac for convenience. 2. No code enters SciPy unless it is fully documented. 3. No code enters SciPy unless it is fully tested (this holds for both bug-fixes and enhancements) It sounds tough, but it guarantees fewer bugs and a higher quality code base. I think it would be *less* intimidating for new contributers if they knew that everybody's code got reviewed, not just theirs, and that we'd like to work with them to improve their code to the point where it can be included. Cheers Stéfan

2009/2/23 Matthew Brett <matthew.brett@gmail.com>:

A) Do we agree in general to a more disciplined tests / review / accept cycle. B) What specifically are the problems that y'all are having, and what options are there for solving them.

Current workflow: 1. Cook up a patch 2. Apply the patch or, if you are not a dev, upload to trac So, currently, unreviewed, untested code ends up in SciPy, or languishes on Trac for a long time. Proposed workflow: 1. Cook up a patch 2. Attach the patch (or a URL to the patchset/branch) to the issue tracker with a REVIEW tag 3. Ping the mailing list or IRC to request a review (rinse and repeat) Workflow for dev: 1. Request a list of patches ready for review: review - Has tests [check] - Has docs [check] - Does what it is supposed to do [check] 2. Add a POSITIVE_REVIEW or NEGATIVE_REVIEW tag as appropriate 3. Request a list of patches ready to be merged (code can be merged if seen by two pairs of eyes: reviewer + committer, reviewer + reviewer, etc. In the end it must have "positive_reviews - negative_reviews >= 2"). Review the patch (this adds one pair of eyes) and merge if appropriate. That's the rough idea. Comments welcome. Cheers Stéfan

On Tue, Feb 24, 2009 at 6:15 AM, <josef.pktd@gmail.com> wrote:

On Mon, Feb 23, 2009 at 4:03 PM, Stéfan van der Walt <stefan@sun.ac.za> wrote:

...

2009/2/23 Matthew Brett <matthew.brett@gmail.com>:

...

A) Do we agree in general to a more disciplined tests / review / accept cycle. B) What specifically are the problems that y'all are having, and what options are there for solving them.

Current workflow:

1. Cook up a patch 2. Apply the patch or, if you are not a dev, upload to trac

So, currently, unreviewed, untested code ends up in SciPy, or languishes on Trac for a long time.

Proposed workflow:

1. Cook up a patch 2. Attach the patch (or a URL to the patchset/branch) to the issue tracker with a REVIEW tag 3. Ping the mailing list or IRC to request a review (rinse and repeat)

Workflow for dev:

1. Request a list of patches ready for review: review - Has tests [check] - Has docs [check] - Does what it is supposed to do [check] 2. Add a POSITIVE_REVIEW or NEGATIVE_REVIEW tag as appropriate 3. Request a list of patches ready to be merged (code can be merged if seen by two pairs of eyes: reviewer + committer, reviewer + reviewer, etc. In the end it must have "positive_reviews - negative_reviews >= 2"). Review the patch (this adds one pair of eyes) and merge if appropriate.

That's the rough idea. Comments welcome.

Cheers Stéfan

I agree it is a good idea, theoretically, but

Maybe I'm slightly pessimistic, but almost the only comment or review for my bugfixes in scipy.stats that I got, were from Per Brodtkorb, and my tickets and patches were sitting for half a year in trac.. If I have to wait for a review, then ...

For patches: not being able to even retrieve them is one problem. Let say right now I feel guilty about your email, and look into trac: I can't easily retrieve all your patches which are > 6 months old without getting into a SQL query :) Or if I look at them, and think I have nothing to say, I can't mark them as "read" so I won't bother reading them next time I look at the bugs. I don't know if this scenario makes the problems I have with trac ATM clearer ? David

Hi Josef 2009/2/23 <josef.pktd@gmail.com>:

I agree it is a good idea, theoretically, but

Maybe I'm slightly pessimistic, but almost the only comment or review for my bugfixes in scipy.stats that I got, were from Per Brodtkorb, and my tickets and patches were sitting for half a year in trac.. If I have to wait for a review, then ...

I hope that, if we have a decent workflow in place, this kind of thing won't happen any longer. I can't go onto trac and search for "tickets with patches that needs review". Whenever I have time, I try to close bugs and review patches, but more often than not it takes a very long time just to access all the required info from Trac. I remember spending a full day reviewing your changes to distributions. Unfortunately, technology thwarted us there as well. You had one huge patch, that you then very carefullly split into parts, but I never managed to merge your changes with my tree. Then you got SVN access and, well, the rest is SVN history.

(and I'm still waiting for 2 fixes to numpy.random)

See, I didn't know that :) I would review it for you right now, but I waited 60 seconds and then got 500 Internal Server Error. Regards Stéfan

Matthew Brett wrote:

Hi Stefan,

...

1. No code enters SciPy unless it had two pairs of eyes on it: reviewer and committer, reviewer and reviewer, reviewer and release manager, etc. All tickets ready for merging are marked in Trac for convenience. 2. No code enters SciPy unless it is fully documented. 3. No code enters SciPy unless it is fully tested (this holds for both bug-fixes and enhancements)

Right.

So, the real problem here is that the people doing the actual work have severe problems with the current workflow.

It seems to me the issue

A) Do we agree in general to a more disciplined tests / review / accept cycle.

I'm a bit concerned about getting too top-heavy here. I think the biggest problem has been time and adding too formal of a process will just increase the time it takes to get code into SciPy. I'm fine with emphasizing documentation and tests as we discuss things and we should encourage each other, but I'm not comfortable with hard-line statements like the ones being made above. Yes, such things are helpful, but they are also expensive and I worry more about what we lose in contributions. The quality of what we create should emerge as all interested parties critically look at the code that is available in SciPy. Not everyone can do that on the same schedule. I'm opposed to trying to force that to happen. I very much favor cultivating a culture that wants someone to fix the problems in their code. Once we have a git-svn integration working, then I can support a simple policy like 1) "this list of people can push from git to svn" 2) "code to submit must either be O.K.'d by one other or have a certain time limit expired with no response" But, my favorite workflow is a bit more chaotic, than that. People create their own DVCS versions of SciPy using their best judgment and publish revisions they consider to be working code. Branches that are given the thumbs up by 2 people (or 1 on the steering committee) get pushed to the main branch. This review happens regularly, on IRC channels at regularly scheduled times. Good conversation... -Travis

Hi Matthew 2009/2/24 Matthew Brett <matthew.brett@gmail.com>:

I think Stefan's position is that, as more people start using and contributing to Scipy, it's become near impossible to maintain in a release-worthy way (Stefan - is that right)?    That if we want to keep going without collapsing we need a more formal process.

Exactly. If we keep introducing new bugs ourselves, there's not enough time in the world to bring SciPy up to standard.

I guess the alternative position is that not having a discipline of code review, testing and documentation will make it more likely we'll have contributors, and the code will get better that way.

That's an interesting position, and one I don't understand entirely. I think that a clear, structured guideline for contributions would make SciPy *easier* to collaborate on. It is much easier to please someone if you know what they want! I would have committed many patches in the past if I had had a guarantee that they were working as advertised. That guarantee is provided by tests. With the nose framework in place, writing tests is so very easy: def test_myfoo(): assert 1 == 1 So I hope that everyone would agree that proper testing and documentation improves life, not only for the user community, but also for the contributor. Regards Stéfan

Stéfan van der Walt wrote:

Hi Matthew

2009/2/24 Matthew Brett <matthew.brett@gmail.com>:

...

I think Stefan's position is that, as more people start using and contributing to Scipy, it's become near impossible to maintain in a release-worthy way (Stefan - is that right)? That if we want to keep going without collapsing we need a more formal process.

Exactly. If we keep introducing new bugs ourselves, there's not enough time in the world to bring SciPy up to standard.

When have you done enough "testing" or "documentation"? We need bright, dedicated people working on SciPy and using good judgment. Sometimes that means emphasizing tests. Sometimes that means emphasizing documentation. Sometimes it means thinking hard about the algorithm you are implementing and carefully coding it. Unit-testing is a tool, but it requires more than that to create code that others can use and rely on --- a skill I don't think we can quantify sufficiently to formalize a process that by itself "produces more contributions"

With the nose framework in place, writing tests is so very easy:

def test_myfoo(): assert 1 == 1

So I hope that everyone would agree that proper testing and documentation improves life, not only for the user community, but also for the contributor.

Yes, unit-testing is essential when you need to re-factor --- but it comes with a cost. Code that is unit-tested requires those tests to also be re-factored when the code gets refactored, so there is such a thing as "unit-testing an API too early". Thus, there is a life-cycle question for unit-test coverage. Early-on fewer core unit-tests are appropriate. Later, when the API is stabilized, more unit-tests are appropriate. I can't give hard numbers for what is "right" or when the transition is made. -Travis

I'll add a couple comments regarding this whole discussion (including the tools that will be used for scipy/numpy software development). 1) I'm not sure that it is a good idea to change everything at once (e.g., svn->git, trac->roundup, etc), particularly if these changes can be done incrementally. It's easy for those that hunger for these changes to think of why doing sooner is better. I suspect there are many other that may feel otherwise that may not be as vocal. And some of the impacts may not be obvious. So if it is at all possible, even if it means some extra work, try just doing one thing at a time and evaluating its impact first before making other changes. Arguably the same point can be made about process changes. 2) While I understand the desire to increase the quality of commits to scipy by putting in a more formal process, like making sure code is reviewed, tests are present, and documentation is provided, I too, like Travis, worry that this may inhibit many useful contributions. Rather than act as a barrier, why not just have some sort of "seal of approval" for things that have gone through that process. As a user, I'd rather have the choice of using an unreviewed, poorly tested, or poorly documented module than have someone else decide that I can't make that choice myself. Who knows, I might find it useful enough to improve. Yes, it can be put in another area (e.g., scikits), but it should be just as easy to get at and see that it is available. If one thing should be most required it would be tests to ensure the main functionality works on all supported platforms so that building releases isn't held up by problems discovered on yet untested platforms. I don't think the reviewing or documentation issues generally affect how much work is involved in making releases though. Perry

On Wed, Feb 25, 2009 at 15:22, Matthew Turk <matthewturk@gmail.com> wrote:

Hi there,

I've only just subscribed to this list (after following on GMANE-RSS) because I wanted to contribute to the discussion.  I'd like to echo Perry's first point below, and add on some specific concerns I have about the entire workflow discussion.

An aspect I worry is being overlooked is that in some communities change comes rather slowly.  I have helped out a number of people with user-space deployment of Python packages, and the biggest impediment is -- as with many things -- installation.  I worry that if the release schedule of SciPy doesn't speed up substantially, accessing source control will be the primary means of getting the code. Installing git and mercurial (and maybe Bazaar, but I've had the most trouble with that) into some user-space area is not difficult, but it adds on another layer of overhead.  Until all of the supercomputing centers provide DVCS, users (and developers!) targeting deployment there will have yet another barrier to entry for using SciPy.  (And as a result, they may fall back on old habits: IDL, for instance.)  To that end, I'd like to strongly and plaintively request that some kind of mirror in SVN, or even archived nightly tarballs, be kept of the primary tree of development.

Pretty much all of the DVCS web frontends allow users to get tarballs of any revision they like. For example, the "bz2", "zip" and "gz" links at the top of my Mercurial repo for line_profiler: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ Click the "files" link for any of the previous revisions to go back in the history, and click on the "bz2", etc., links to get past revisions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

I've written a lot of response to various comments and I'd like to summarize and extend a few of my comments: 1) We absolutely need to improve the quality of SciPy, and that does mean more tests, documentation, and reviews --- and most importantly faster releases. Right now, a release happens when someone steps up to be a release manager and commits to making it happen. I don't know how to promise that on a regular cycle with only volunteer effort. I would love to have the resources to fund SciPy release management. 2) I think we are doing a decent job of commits having tests and documentation. We should continue to remind each other of the need for quality code in SciPy (and continue to clean up code that is there). 3) There are pieces of SciPy that need work (interpolate stands out most in my mind right now). I have changes to the interpolate code that I have not yet committed because I was waiting for the release of 0.7 but I really want to commit. Who is interested in reviewing this? I'm happy to work with additional eyes, but my current workflow is "commit code I think is working along with some tests and docstrings", and then let review/improve happen on the trunk. I don't really like having lots of branches checked out of a code-base in order to manage a different workflow. I'm open to being educated about approaches that work better. 4) Bug-fix commits are a different thing than feature-enhancement commits. We should have different expectations of them. 5) We do have scikits for more experimental additions to live so that SciPy should become more of a stable, documentation-rich library. But, the problem there is distribution. EPD and Enstaller (our BSD-licensed version of setuptools) is one answer to that distribution problem. There are others. 6) I very much appreciate all the work people do on SciPy. I think our biggest lack more than anything else is the "full-time" person that can respond to the user community and keep the momentum moving. -Travis

Charles R Harris wrote:

Interpolate stands out in my mind also, along with signal processing. Mostly because last time I looked -- a long time ago -- they were pretty messy and I haven't seen much work done on them since. I think in this case it would be helpful if you summarized your intended changes and interfaces and gave a short explanation of your motivation. By the time the code actually shows up in SVN might be a little late.

An intern did some work here last summer (and I did some work two summers ago). Last summer's work never got integrated. The goal is to unify and expand the interface to interp1d and interp2d and interpnd --- to allow for future improvement while improving the API as well as a few of the algorithms. Basically, interpolate is one thing we teach quite a bit and it's a little embarrassing every time we teach it.

I think a similar approach would have helped with curve_fit, not least since I have found Gauss-Newton with numerical derivatives to out-perform Levenburg-Marquardt by ~50x in some problems and give better answers. Then there is the question of when to stop the iterations. Also, I couldn't see if the function and data could be array valued, which can be handy in some cases. For instance, I recently fit a case where the parameters moved a large set of points around on a unit sphere in order to minimize the distance to data points on the sphere. In this case the output of f was most conveniently represented as an array of vectors. Mind, I don't mind the function itself so much as giving it a name that implies more generality than I think it has.

I'm very happy to change the name and/or the algorithm to improve the generality / speed. Anything you have would be welcome. Perhaps other people have a different perspective, but my perspective of the trunk is that it is not a release and so changes to new functionality in the trunk can be made with no concern of "backward-compatibility" until a release is made. Given the review tools I've seen. My perspective on the best way to review is to look at changes to the trunk and just make the changes that you see as needed, or if it's not clear to you how to make the changes, you can put comments in the code about what you would like to see done and then perhaps somebody else can figure out how to make that change. In that process, we should all play nicely with each other and respect each other's opinions. If an occasional point can't be resolved between the interested parties because of basically differences of opinion, then we can: 1) vote on the list and if that doesn't clearly resolve the question, 2) the steering committee votes and makes a decision. That's my perspective on how changes are made. -Travis

Pauli Virtanen wrote:

Wed, 25 Feb 2009 17:18:37 -0600, Travis E. Oliphant wrote: [clip]

...

3) There are pieces of SciPy that need work (interpolate stands out most in my mind right now). I have changes to the interpolate code that I have not yet committed because I was waiting for the release of 0.7 but I really want to commit. Who is interested in reviewing this? I'm happy to work with additional eyes, but my current workflow is "commit code I think is working along with some tests and docstrings", and then let review/improve happen on the trunk.

The codereview.appspot.com tool is very fast to use, eg. via the

http://codereview.appspot.com/static/upload.py

tool. So I'd suggest to just uploading the patches there even before commit; it can't do any harm.

The harm is the effort to do it. Interacting with a web-page is slower than svn commit. This extra step in the process does make a difference when you are time-crunched.

The problem with reviewing code after commit in trunk is that it takes more effort to correct or ask about dubious points.

I disagree with this statement. Why does it take more effort than reviewing code on the trunk? You can do an svn diff to get the code changes, and do the review exactly as you could with any other tool. One way to see this is the difference between asking for permission or asking for forgiveness. Both have their place in social activities, but we shouldn't institutionalize one over the other.

...

I don't really like having lots of branches checked out of a code-base in order to manage a different workflow. I'm open to being educated about approaches that work better.

I've found git-svn quite good for maintaining topic branches. It can switch easily between them using the same working tree, so that compiles are fast, and editor just needs M-x revert-buffer.

Thanks for the tip. At some point I may be able to invest some time in learning about git-svn. How do you switch between branches using git-svn. With svn it's svn switch http://some-name-I-always-have-to-look-up-and-takes-time. -Travis

On Thu, Feb 26, 2009 at 09:58, Travis E. Oliphant <oliphant@enthought.com> wrote:

Pauli Virtanen wrote:

...

Wed, 25 Feb 2009 17:18:37 -0600, Travis E. Oliphant wrote: [clip]

...

3) There are pieces of SciPy that need work (interpolate stands out most in my mind right now).    I have changes to the interpolate code that I have not yet committed because I was waiting for the release of 0.7 but I really want to commit.  Who is interested in reviewing this?  I'm happy to work with additional eyes, but my current workflow is "commit code I think is working along with some tests and docstrings", and then let review/improve happen on the trunk.

The codereview.appspot.com tool is very fast to use, eg. via the

      http://codereview.appspot.com/static/upload.py

tool. So I'd suggest to just uploading the patches there even before commit; it can't do any harm.

The harm is the effort to do it.    Interacting with a web-page is slower than svn commit.

That's why there are CLI tools to submit the review.

  This extra step in the process does make a difference when you are time-crunched.

We're usually not.

...

The problem with reviewing code after commit in trunk is that it takes more effort to correct or ask about dubious points.

I disagree with this statement.   Why does it take more effort than reviewing code on the trunk?    You can do an svn diff to get the code changes, and do the review exactly as you could with any other tool.

Because looking at a web page is easier, I've found. The communicating that happens afterwards is also easier. Please, *try* it for a month. I believe that you are speaking from ignorance. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Thu, Feb 26, 2009 at 10:36:03AM -0700, Charles R Harris wrote:

I think a brief howto somewhere would help. It isn't easy to keep up with new tools and learn new habits without some help.

Especially when you are very busy. Its a common problem: learning new tools makes you more productive, so you do more when you have very little time. But if you have very little free time, you don't have time to learn new tools... Ga�l

Thu, 26 Feb 2009 09:44:43 -0800, Ondrej Certik wrote: [clip]

I am sure Stefan will write a howto, if not, I will, e.g. I'll try to fix something in scipy (hint: broyden2), go through the whole procedure (e.g. git, review, ...) and document the way, so that you can then just follow my howto.

I now notice that I still haven't found time to make progress on the scipy.optimize.nonlin rewrite... I moved my git branches around a bit, you can find my current work here: http://github.com/pv/scipy-work/tree/ticket-791-optimize-nonlin-rewrite It'd be great if we managed to finish this, as it's been on hold now for some time. -- Pauli Virtanen

Charles R Harris wrote:

...

Because looking at a web page is easier, I've found. The communicating that happens afterwards is also easier.

Please, *try* it for a month. I believe that you are speaking from ignorance.

I think a brief howto somewhere would help. It isn't easy to keep up with new tools and learn new habits without some help.

+1 to some way of having the reviews easily accessible. I just looked at a review that Charles did and already learned some things about conventions in scipy, just based on the line-by-line commenting he did on a review. Jason

Hi Travis 2009/2/27 Travis E. Oliphant <oliphant@enthought.com>:

My biggest concern is having a bunch of code sitting in a queue and not reviewed, nor committed --- or the review processes become too onerous and code not making it through because of what I would consider to be "ticky-tacky technicalities."

That's a very valid concern. David and I are experimenting with different issue trackers and plugins for trac, to see how best to generate a "review pool". I.e., what I'd like to see is that, if you only have 5 minutes to work on SciPy in the evening, you can a) Go to trac and click on "tickets for review" b) Review a couple of tickets or a) Go to trac and click on "reviewed tickets" b) Apply those patches or a) Go to trac and click on "unresolved issues" b) Fix the bug c) Upload the patch for review Technically, (c) is a bit challenging. I note your concern that it would become difficult to check in, so what I would like is to have a script such as scipy-submit -t 212 -m "Do not deallocate memory after object disposal." which then uploads the patch to the codereview site, and adds a link to ticket 212 with the commit message and review URL. All of this can be done via the web, but I'd prefer to have a CLI available. Do you have any suggestions or further concerns? Thanks Stéfan

Prabhu Ramachandran wrote:

On 02/27/09 11:07, Stéfan van der Walt wrote:

...

scipy-submit -t 212 -m "Do not deallocate memory after object disposal."

which then uploads the patch to the codereview site, and adds a link to ticket 212 with the commit message and review URL. All of this can be done via the web, but I'd prefer to have a CLI available.

Do you have any suggestions or further concerns?

I've not contributed anything in years to scipy but I have a practical problem that might be worth addressing eventually (others might be in a similar position) -- my entire network is firewalled and I can only access the web behind an authenticated http proxy.

I have similar issues, and I agree those are valid concerns. Those can be very painful to handle. In my case, there is no DNS server, the names are resolved by the proxy; my workstation can only resolve the proxy name. This breaks most applications out there. ssh is not easy, because ssh cannot resolve names - for git, I managed to get things worked out for github using corkscrew. This is the kind of things which I managed to do once and hope never have to do again, so I can't tell you exactly how to do it: http://en.wikipedia.org/wiki/Corkscrew_(program) My .ssh/config looks like this for github Host gitproxy User git HostName ssh.github.com Port 443 ProxyCommand /usr/bin/corkscrew www 3128 %h %p IdentityFile /home/david/.ssh/id_rsa.pub Where www is the name of my proxy and 3128 the port. FWIW, svn has similar problems. I could never commit anything from a former internship location because of some proxy limitations - it is one of the reasons which pushed me into git for scipy development, actually. If you can't access either ssh or proxy, my experience is that you are more or less screwed with any tool out there - but with DVCS, you can at least put your changes aside and commit them later from an easier connection. cheers, David

On Sat, Feb 28, 2009 at 4:53 AM, Prabhu Ramachandran <prabhu@aero.iitb.ac.in> wrote:

Host github.com         User git         HostName github.com # also tried ssh.github.com         Port 443         ProxyCommand /usr/bin/corkscrew my_proxy.iitb.ac.in  80 %h %p /home/prabhu/.ssh/auth         IdentityFile /home/prabhu/.ssh/id_dsa.pub

If you cannot go through port 443, that may explain it. One way to check the connection is to ssh directly to github.com (with user git). It will fail (it is not support to work), but will tell you something like: PTY allocation request failed on channel 0 Hi cournape! You've successfully authenticated, but GitHub does not provide shell access. Connection to github.com closed.

svn has worked well for me in this regard.  I have always been able to checkin and checkout stuff with svn.

Yes, it may work for svn and not for git: the related network requirements are not the same. In some cases, http is the only method - on the draconian environment, I used http + push at the end of the day at home. David

On Sat, Feb 28, 2009 at 5:46 AM, Prabhu Ramachandran <prabhu@aero.iitb.ac.in> wrote:

On 02/28/09 01:37, David Cournapeau wrote:

...

On Sat, Feb 28, 2009 at 4:53 AM, Prabhu Ramachandran <prabhu@aero.iitb.ac.in> wrote:

...

Host github.com         User git         HostName github.com # also tried ssh.github.com         Port 443         ProxyCommand /usr/bin/corkscrew my_proxy.iitb.ac.in  80 %h %p /home/prabhu/.ssh/auth         IdentityFile /home/prabhu/.ssh/id_dsa.pub

If you cannot go through port 443, that may explain it. One way to check the connection is to ssh directly to github.com (with user git). It will fail (it is not support to work), but will tell you something like:

PTY allocation request failed on channel 0 Hi cournape! You've successfully authenticated, but GitHub does not provide shell access.          Connection to github.com closed.

Mine tells me this:

$ ssh -p 443 github.com The authenticity of host '[ssh.github.com]:443 (<no hostip for proxy command>)' can't be established. RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[ssh.github.com]:443' (RSA) to the list of known hosts. Permission denied (publickey).

So it looks like it does work but does not authenticate.  I hope I don t have to setup a login with github.

If you want to connect through ssh, I am afraid you don't have a choice. But of course, the no hassle solution is to just clone from http - or even simpler, to get the autogenerated tarball (independently of DVCS or nor, I think that's something we should support anyway). cheers, David

On 02/27/09 14:48, Pauli Virtanen wrote:

Fri, 27 Feb 2009 11:22:55 +0530, Prabhu Ramachandran wrote: [clip]

...

I've not contributed anything in years to scipy but I have a practical problem that might be worth addressing eventually (others might be in a similar position) -- my entire network is firewalled and I can only access the web behind an authenticated http proxy. The firewall does allow ssh connections out though but that seems useless to access a git repository hosted on github say.

Git can clone over HTTP, just change git:// to http:// and it seems to work. I can also clone through a proxy with

export http_proxy=http://username:password@proxy:port/ git clone http://whatever

Thanks, this works.

Pushing over HTTP is another question... It's probably not possible to push to Github over HTTPS, but maybe there are places that you can push to with only HTTP authentication.

OK, thanks. prabhu

On 02/27/09 23:58, Pauli Virtanen wrote:

Fri, 27 Feb 2009 11:22:55 +0530, Prabhu Ramachandran wrote: [clip]

...

I've not contributed anything in years to scipy but I have a practical problem that might be worth addressing eventually (others might be in a similar position) -- my entire network is firewalled and I can only access the web behind an authenticated http proxy. The firewall does allow ssh connections out though but that seems useless to access a git ^^^^^^^^^^^^^^^^^^^^^^^^^

SSH is the default transport protocol for Git.

Hmm, this doesn't seem to work. My guess is that it uses a different port which clearly won't work unless I force the admins here to open up the git port. $ git clone --origin svn git://github.com/pv/scipy-svn.git scipy.git Initialized empty Git repository in /home/prabhu/src/git/scipy.git/.git/ github.com[0: 65.74.177.129]: errno=Connection timed out fatal: unable to connect a socket (Connection timed out) fetch-pack from 'git://github.com/pv/scipy-svn.git' failed. Note, that I can certainly ssh just fine to the outside world. cheers, prabhu

Stéfan van der Walt wrote:

Hi Travis

2009/2/27 Travis E. Oliphant <oliphant@enthought.com>:

...

My biggest concern is having a bunch of code sitting in a queue and not reviewed, nor committed --- or the review processes become too onerous and code not making it through because of what I would consider to be "ticky-tacky technicalities."

That's a very valid concern. David and I are experimenting with different issue trackers and plugins for trac, to see how best to generate a "review pool". I.e., what I'd like to see is that, if you only have 5 minutes to work on SciPy in the evening, you can

a) Go to trac and click on "tickets for review" b) Review a couple of tickets

or

a) Go to trac and click on "reviewed tickets" b) Apply those patches

or

a) Go to trac and click on "unresolved issues" b) Fix the bug c) Upload the patch for review

Technically, (c) is a bit challenging. I note your concern that it would become difficult to check in, so what I would like is to have a script such as

scipy-submit -t 212 -m "Do not deallocate memory after object disposal."

Something like that would be nice! Thanks for the continued effort at improving workflow. -Travis

Pauli Virtanen wrote:

Thu, 26 Feb 2009 09:58:03 -0600, Travis E. Oliphant wrote: [clip]

Anyway, difficult to tell how a mandatory review policy would affect Scipy's development. I'd be reluctant to jump headfirst to requiring it, without experimenting, even if Sage and Sympy have had good experience about it. Nevertheless, I'm going to try to change my own workflow and see what happens.

I'm willing to do this as well. I love seeing the enthusiasm behind getting a description of how to contribute to SciPy up on the wiki and the effort put in to testing DVCS. That alone is worth encouraging if I can by switching my own workflow. -Travis

On Thu, Feb 26, 2009 at 7:11 PM, David Cournapeau <cournape@gmail.com> wrote:

On Fri, Feb 27, 2009 at 8:21 AM, Pauli Virtanen <pav@iki.fi> wrote:

...

Thu, 26 Feb 2009 09:58:03 -0600, Travis E. Oliphant wrote: [clip]

...

The harm is the effort to do it.    Interacting with a web-page is slower than svn commit.   This extra step in the process does make a difference when you are time-crunched.

...

The problem with reviewing code after commit in trunk is that it takes more effort to correct or ask about dubious points.

I disagree with this statement.   Why does it take more effort than reviewing code on the trunk?    You can do an svn diff to get the code changes, and do the review exactly as you could with any other tool.

I partly agree with this: if you want to immediately fix something "wrong" in the suggested change yourself, a web-based code review tool gets in the way. (But I haven't yet tried how well the command-line tool would work...)

But if you want to suggest some changes to the author of the commit, or ask some specifics, the code review tool works fairly well as a communication tool. Less hassle than commenting on a mailing list, and more organized.

I note that Github offers a similar "remarks-in-commits" feature. Could be worth a try to check how this works in practice.

...

One way to see this is the difference between asking for permission or asking for forgiveness. Both have their place in social activities, but we shouldn't institutionalize one over the other.

Another social aspect is that asking for permission has much more positive connotations than asking for forgiveness.

Anyway, difficult to tell how a mandatory review policy would affect Scipy's development. I'd be reluctant to jump headfirst to requiring it, without experimenting, even if Sage and Sympy have had good experience about it. Nevertheless, I'm going to try to change my own workflow and see what happens.

[clip]

...

...

I've found git-svn quite good for maintaining topic branches. It can switch easily between them using the same working tree, so that compiles are fast, and editor just needs M-x revert-buffer.

Thanks for the tip. At some point I may be able to invest some time in learning about git-svn. How do you switch between branches using git-svn.  With svn it's svn switch http://some-name-I-always-have-to-look-up-and-takes-time.

Check what branches you have

       git branch

Check what branches other people have

       git branch -r

Switch working tree to a different branch

       git checkout BRANCHNAME

And it's fast. I think here Git beats SVN, Bzr and Mercurial in ease of use. (Mercurial does have some similar features, but IMO they are a bit less mature.)

For me, that's one of the big killer feature of git. Switching branches is really cheap, both in time and in terms of command lines. When you add the ability to compare branches between them, git just blows away bzr.

For example, when I want to get an idea of the development between two branches, I can do:

git diff branch1..branch2 # --stat option is useful too to get a global view git log branch1..branch2

In scipy, both of those takes less than a second, even for thousand of commits of difference. For release, of to make sure I merge what I think I am merging, this is very helpful. You just never do it with svn, because it is so slow (takes minutes) and the syntax to compare branches is awful. This alone is one of the reason why I vastly prefer git to bzr, too.

Exactly, and also git cherry-pick, to pickup some particular patches from the other branch (e.g. some fixes etc.). Those are things that really rock and once you get used to it, you never want to come back. Ondrej

Stéfan van der Walt wrote:

Hi Travis

...

3) There are pieces of SciPy that need work (interpolate stands out most in my mind right now). I have changes to the interpolate code that I have not yet committed because I was waiting for the release of 0.7 but I really want to commit. Who is interested in reviewing this?

I'd be glad to. Pauli's suggestion of codereview.appspot.com sounds good, since we don't have any better infrastructure in place.

As I've mentioned. I'm all for others doing this if it helps them feel more comfortable contributing to SciPy. I'm not interested in using this tool because of the increased effort. I have always been and remain very interested in reviews / feedback / comments / fixes to check-ins that I make to the trunk. If the check-in email is not sufficient for large changes, I am willing to send an email to interested parties about changes that have been made pointing to the svn diff in the Trac. If someone else would like to take those changes and have a discussion with some other tool, that is also fine. I'm very impatient because my windows of time to work on something are small and if I have to "wait-for-review" before something gets checked-in, I suspect I will get impatient because it increases the mental-time I have to spend on getting something fixed / improved in SciPy. I'm very aware of many of the improvements that need to be made, but don't have a lot of time to spend on them.

...

4) Bug-fix commits are a different thing than feature-enhancement commits. We should have different expectations of them.

I agree, to an extent. I think it is an ideal opportunity to add a test (since, clearly, the current test suite didn't catch the problem, and since you had to study the broken code in order to fix it); but in such a case it's more important to have the bug fixed. Unfortunately, without a test you won't be absolutely certain that it's fixed everywhere, but the process at least converges in the right direction.

I agree, but would mention that a unit test only lets you test against that particular feature/bug that was called out in the test. Without code-coverage you don't have any guarantees or even a guarantee that the unit-test is written well-enough to catch the more subtle and harder to replicate bugs. So, yes, unit-tests are good, but they are not a panacea to the goal of quality code.

...

6) I very much appreciate all the work people do on SciPy. I think our biggest lack more than anything else is the "full-time" person that can respond to the user community and keep the momentum moving.

Absolutely. I've often wondered how hard it would be to obtain such funding, but to date I haven't made any proposals.

Right now it looks to me that we have a steady-stream of students, academics, and the dedicated Robert Kern. I was hopeful that I would be able to make SciPy-growth work while I was in academia but it didn't work out for me. Right now, I'm excited to continue to help Enthought in its support of SciPy. I'm hopeful that will lead to me having more time to spend on SciPy myself, but it's possible that it won't work out that way. It's great to see others that have stepped up and are continuing to step up to move SciPy forward. Best regards, -Travis

Thu, 26 Feb 2009 14:17:07 +0200, Stéfan van der Walt wrote: [clip]

I don't know if you've visited the portal to SciKits: http://scikits.appspot.com. If we can make any changes to facilitate packaging, let me know.

I think a couple of things should be done on the portal to make it feel more finished: 1. Write a blurb for all of the scikits. (I can also do this, if you tell me how...) The portal page currently looks quite empty and somewhat discouraging: http://scikits.appspot.com/scikits Or should this happen automatically via PyPi? 2. A link to the portal should be added in a visible place @ scipy.org, when it's ready. 3. PyPi links & instructions for packages that are not in PyPi should be hidden. 4. openopt seems to live nowadays at openopt.org -- Pauli Virtanen

On Fri, Feb 27, 2009 at 4:45 AM, Pauli Virtanen <pav@iki.fi> wrote:

Thu, 26 Feb 2009 14:17:07 +0200, Stéfan van der Walt wrote: [clip]

...

I don't know if you've visited the portal to SciKits: http://scikits.appspot.com.  If we can make any changes to facilitate packaging, let me know.

I think a couple of things should be done on the portal to make it feel more finished:

1. Write a blurb for all of the scikits. (I can also do this, if you tell   me how...) The portal page currently looks quite empty and somewhat   discouraging:

  http://scikits.appspot.com/scikits

  Or should this happen automatically via PyPi?

Yes, it should be automatic. It corresponds to the short description of the package. I will fix it for my own packages, at least. There seems to be a problem with some packages which should not be there, though (scikits.em, for example - they are parts of scikits.learn - maybe a bug somewhere, since they don't exist in pypi). David

On 26-Feb-09, at 2:45 PM, Pauli Virtanen wrote:

3. PyPi links & instructions for packages that are not in PyPi should be hidden.

It sticks out to me that there are a lot of broken PyPI links throughout the portal site. Also, I'm not sure what can be done about it, but the whole site is quite slow for me. David

Sat, 28 Feb 2009 11:14:08 +0200, Stéfan van der Walt wrote:

2009/2/27 David Warde-Farley <dwf@cs.toronto.edu>:

...

...

3. PyPi links & instructions for packages that are not in PyPi should   be hidden.

It sticks out to me that there are a lot of broken PyPI links throughout the portal site.

Those are the SciKits that haven't been registered with PyPi. We can easily remove them from the list, but I thought it's better to have them there to start off with.

I think it's good to have the scikits there, but if a Scikit isn't in PyPi, it's best not to show broken PyPi links or easy_install instructions. -- Pauli Virtanen

jason-sage@creativetrax.com wrote:

Perry Greenfield wrote:

...

2) While I understand the desire to increase the quality of commits to scipy by putting in a more formal process, like making sure code is reviewed, tests are present, and documentation is provided, I too, like Travis, worry that this may inhibit many useful contributions. Rather than act as a barrier, why not just have some sort of "seal of approval" for things that have gone through that process.

Lots of projects have -stable and -dev branches. The -stable branch for scipy could involve the "seal of approval" with review, doctests, etc. The -dev branch could be the unreviewed code. This lets Travis commit to something and get his patches out there, but also clearly defines a line in the sand between reviewed and unreviewed code. I realize that scipy already has something of -dev and -stable branches, based on releases. Maybe this idea boils down to: only reviewed code is allowed in an official release, but there is a -dev branch with all code available as well. As code is reviewed, it is moved into the -stable branch and released in the next release.

This may be a good solution for us in the short term, prior to choosing a DVCS.

In reality, using a DVCS, each developer's copy of the repository then becomes a private -dev branch that can be pulled from. Developers get to commit and publish unreviewed changes, and someone (the release manager) can pull in to -stable the changes that are reviewed. The release manager could also pull all changes from developer repositories into an official -dev branch if you wanted to have a central clearing house for what everyone is working on.

This sounds like a good workflow that solves the concerns I have while still allowing a stable branch to emerge with well-documented / tested / reviewed code. It seems like we could do this today --- I like it. -Travis

On Thursday 26 February 2009 10:40:24 jason-sage@creativetrax.com wrote:

...

2) While I understand the desire to increase the quality of commits to   scipy by putting in a more formal process, like making sure code is   reviewed, tests are present, and documentation is provided, I too,   like Travis, worry that this may inhibit many useful contributions.   Rather than act as a barrier, why not just have some sort of "seal of   approval" for things that have gone through that process.

Lots of projects have -stable and -dev branches.

Please, NO! This does not scale. This is the process used in Boost and makes the release process a nightmare. Boost svn commit access is not easily gifted; virtually every library in Boost goes through extensive review and its authors are generally way-above-average programmers. In spite of all the preceding, getting a Boost release out is a superhuman effort: - tracking commits between branches is a full-time job - the -dev branch becomes "the wild west" very easily (slippery slope) - if no feature freeze, bug fixes get done even slower What you want, in my humble opinion, is what the KDE people call "always summer in the trunk". There really is only one scalable way to manage this: a DVCS. Instead of using trunk as a playground, use the "trunk" in your clone ("master" in git). So long as your clone is published somewhere (anywhere on the web, for instance), any authorized committer into the main repository can pull it. There is *exactly* one more step compared to using svn trunk, viz., sending a mail out to the mailing list indicating the change. But then, you were going to do that anyway, weren't you? The best example I can think of is Xorg. See http://cgit.freedesktop.org for a list of everyone's local repositories. (Of course, scipy may choose bzr over git, but the point still stands.) Let's say "airlied" just finished implementing the feature "drm" for the submodule "radeon r6xx". *His workflow is exactly the same as svn*: git clone ... = svn co / svn up # make changes git commit -a -m "drm rework finished" = svn commit # send mail to the mailing list Any interested party in the mailing list reviews the code and pulls it into the main repository. Note that "airlied" does no more work with the DVCS than with svn; the workflow for "airlied" has not changed other than the command substitution above. The above is actually a true story. Of course, now note that instead of "airlied" (a sanctified committer), it could be you and anyone interested (core developer or not) can simply pull from you and test it. This lowers the barriers to newbies, in my limited experience. Regards, Ravi

Hi,

Yes, I claim this. Doing bug triaging in a web interface is already not a pleasant experience, but with trac, it just becomes very frustrating. When preparing for releases (massive bug triaging), I waste hours doing things which could be at least partially automated.

Can I just check we've got a clear idea who is doing the most work of general scipy maintenance and release? I have the feeling that this group is David, Stefan, Jarrod at least? Then there are those of us who maintain or wrote packages. As the matlab io maintainer (more or less) I would not claim to have a good overview on general scipy release problems. Is there a successful template project workflow we can take up formally - matplotlib, ipython, python? Best, Matthew

On Mon, Feb 23, 2009 at 2:47 PM, David Cournapeau <cournape@gmail.com> wrote:

On Tue, Feb 24, 2009 at 4:03 AM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

...

Yes, I claim this. Doing bug triaging in a web interface is already not a pleasant experience, but with trac, it just becomes very frustrating. When preparing for releases (massive bug triaging), I waste hours doing things which could be at least partially automated.

Can I just check we've got a clear idea who is doing the most work of general scipy maintenance and release?

I have the feeling that this group is David, Stefan, Jarrod at least?

Pauli has been involved too - for last scipy, Nathan has been heavily involved thanks to his great sparse contributions.

...

Then there are those of us who maintain or wrote packages. As the matlab io maintainer (more or less) I would not claim to have a good overview on general scipy release problems.

Is there a successful template project workflow we can take up formally - matplotlib, ipython, python?

I don't know about those projects workflow, but I can tell you about the workflow I would enjoy: - launch the bug tracker interface - get all the new bugs since last time I did some bug triage - select N bugs to assign to contributor joe - look at all the bugs with patchs without review etc....

Doing this in trac takes a lot of time (or is even not possible, unless you type your own SQL queries). With something integrated to vi, a bit like an email interface, it would be much more pleasant.

Imagine that instead of svn, you would have to use a web interface to commit, diff, log every revision. I think people would quickly become crazy,

cheers,

David

I was looking at the changes in 0.11. in trac and I found a trac commandshell (I just found it, I didn't try it out, the google code source is very recent) http://code.google.com/p/tracshell/ http://trac-hacks.org/wiki/TracShellScript Also the trac instance of edgwall.org has ticket query by last modified date. So, in comparing ticket systems the current trac release and not 0.10.2 should be compared to the alternatives Josef

On Tue, Feb 24, 2009 at 4:10 AM, Nathan Bell <wnbell@gmail.com> wrote:

On Mon, Feb 23, 2009 at 1:34 PM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

...

...

And I'd argue for: - someone who we can spam when scipy.X fails - a setup.py that didn't lead to questions about Fortran ABI incompatibilities - a setup.py (or equivalent) with bdist_foo for every foo we care about - a ~6 month cycle and nightly builds (with binary installers) - a website where the scipy.X maintainer can see errors for their module on a dozen different platforms

I would argue those issues are not orthogonal to the quality of the tools we are using. The time I waste on trac and svn is time I don't spend on those issues, and this time easily go up to hours now.

The collective time wasted by Fortran ABI problems *alone* is 10x more than that wasted by the problems you seek to remedy.

I think that's a bit unfair. I built ubuntu packages for scipy/numpy, I built a win32 binary which solved another common ABI problem, I have added unit tests to detect fortran ABI problems, I have set up a build bot on the build service from open source for automatic rpm builds with correct fortran ABI. I spent hours on a broken platform I don't even use because I think it is important for windows to be a first class citizen for numpy/scipy. But I think you don't realize all the work that some of us need to do to make decent releases, properly tested on a vast range of platforms, without a lot of resources. Just looking at new bugs and assigning them takes a lot of time, and with an email interface and/or command line interface, it would be a matter of one minute or two. I personally don't care too much about using git instead of svn - because git-svn gets me a lot of advantages already, without changing anyone workflow. But concerning the bug tracker, something has to be done: I don't care what, as long as it enables a command-line and offline handling. cheers, David

On Mon, Feb 23, 2009 at 02:10:24PM -0500, Nathan Bell wrote:

The collective time wasted by Fortran ABI problems *alone* is 10x more than that wasted by the problems you seek to remedy. I've been using development versions of scipy for 2 years now and even I get burned by which fortran/BLAS/LAPACK I need to install.

While I mostly agree with you that changing VCS will probably not bring huge improvements to the development process, that fact is that resolving the problem you are talking about (fortran ABI compatibility) is very hard, and probably much harder than changing VCS. So I claim your remarks is unfair. Gaël

On Tue, Feb 24, 2009 at 5:06 AM, Robert Kern <robert.kern@gmail.com> wrote:

On Mon, Feb 23, 2009 at 10:04, Stéfan van der Walt <stefan@sun.ac.za> wrote:

...

1) Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous.

While I really like DVCS in general, I don't think there is much benefit to switching. The various DVCS-SVN bridges account for most of the benefits, I think.

I agree on this - having a "blessed" mirror so that anyone into DVCS could have a "reference" would be enough for me, at least as long as there is no good solution for bug tracking (the one advantage of switching to a DVCS is easier merging/branching, but I am worried about the workflow if the bug tracker cannot track branches).

...

2) Ticketing back-end: David is exploring RedMine [2], and I'd like to take a look at InDefero [3], but we'll do a careful analysis of trac-git (like FedoraHosted) too.

You may also want to consider using Roundup for just bug tracking and forgoing "integrated" solutions like the above entirely. We don't use the Trac wiki for anything we couldn't do on the Moin site.

Ah, I did not know about roundup, it looks really nice: at least from the feature set, it has everything I miss from trac. THe reason why I thought about redmine is that it is easy to migrate, and has support for the things I care the most. I otherwise do not care about the solution as long as it is scriptable and ideally can be used offline. Would hosting roundup be an option ? cheers, David

Peter Wang wrote:

On Feb 23, 2009, at 2:16 PM, David Cournapeau wrote:

...

Would hosting roundup be an option ?

Certainly, especially if moving to roundup is done as part of "The Big Transition" off of the old hardware,

As it happens, Roundup itself is right now in (somewhat) active development again, so this would be a good time to evaluation it and let us know if anything not available through extensions is amiss. ('us' as in 'the Roundup developers') Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...

On Mon, Feb 23, 2009 at 14:16, David Cournapeau <cournape@gmail.com> wrote:

On Tue, Feb 24, 2009 at 5:06 AM, Robert Kern <robert.kern@gmail.com> wrote:

...

On Mon, Feb 23, 2009 at 10:04, Stéfan van der Walt <stefan@sun.ac.za> wrote:

...

1) Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous.

While I really like DVCS in general, I don't think there is much benefit to switching. The various DVCS-SVN bridges account for most of the benefits, I think.

I agree on this - having a "blessed" mirror so that anyone into DVCS could have a "reference" would be enough for me, at least as long as there is no good solution for bug tracking (the one advantage of switching to a DVCS is easier merging/branching, but I am worried about the workflow if the bug tracker cannot track branches).

The user pastes in the URL of his branch. Bug trackers don't really "track branches"; they track the status of issues. Some bug trackers, like that in Trac and Redmine, have some special mark up to make it easy to refer to a particular revision in their repo browser. Is that what you are talking about? Remember that with a DVCS, it is intrinsically difficult to accomplish this; anyone can host their branches anywhere. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Mon, Feb 23, 2009 at 14:41, Pauli Virtanen <pav@iki.fi> wrote:

Mon, 23 Feb 2009 14:32:54 -0600, Robert Kern wrote: [clip]

...

...

I agree on this - having a "blessed" mirror so that anyone into DVCS could have a "reference" would be enough for me, at least as long as there is no good solution for bug tracking (the one advantage of switching to a DVCS is easier merging/branching, but I am worried about the workflow if the bug tracker cannot track branches).

The user pastes in the URL of his branch. Bug trackers don't really "track branches"; they track the status of issues. Some bug trackers, like that in Trac and Redmine, have some special mark up to make it easy to refer to a particular revision in their repo browser. Is that what you are talking about? Remember that with a DVCS, it is intrinsically difficult to accomplish this; anyone can host their branches anywhere.

I'd think the markup/URLs shouldn't be a problem, since, as Robert says, one can just paste URLs.

I would be a bit more worried about support for 'Milestones' and 'Versions'. AFAIK Trac (at least the 0.10.3 we have) is a bit primitive here, as a bug can be only assigned to a single version/branch. So it's not easy to query for bugs that have been fixed in trunk, but haven't been backported eg. to 0.7.x branch.

It's worth noting that Roundup isn't a bug tracker so much as it is a toolkit for constructing bug trackers (this is not always a good thing). The database schema is ridiculously flexible. The only required table is "user". You can make whatever fields you like according to the semantics that are appropriate for the workflow you want to use. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Mon, Feb 23, 2009 at 14:45, David Cournapeau <cournape@gmail.com> wrote:

On Tue, Feb 24, 2009 at 5:32 AM, Robert Kern <robert.kern@gmail.com> wrote:

...

On Mon, Feb 23, 2009 at 14:16, David Cournapeau <cournape@gmail.com> wrote:

...

On Tue, Feb 24, 2009 at 5:06 AM, Robert Kern <robert.kern@gmail.com> wrote:

...

On Mon, Feb 23, 2009 at 10:04, Stéfan van der Walt <stefan@sun.ac.za> wrote:

...

1) Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous.

While I really like DVCS in general, I don't think there is much benefit to switching. The various DVCS-SVN bridges account for most of the benefits, I think.

I agree on this - having a "blessed" mirror so that anyone into DVCS could have a "reference" would be enough for me, at least as long as there is no good solution for bug tracking (the one advantage of switching to a DVCS is easier merging/branching, but I am worried about the workflow if the bug tracker cannot track branches).

The user pastes in the URL of his branch. Bug trackers don't really "track branches"; they track the status of issues. Some bug trackers, like that in Trac and Redmine, have some special mark up to make it easy to refer to a particular revision in their repo browser. Is that what you are talking about?

Yes: specially when reviewing for release, I like the ability to go from a bug to the revision and vice-versa.

With Roundup, for example, we can make a field for people to paste in the relevant branch URL.

But my concern is a bit more general: I think there are still a lof of issues on how to integrate DVCS and bug tracking, and nobody really solved it yet. Some people have suggested putting the bugs themselves in the repository, launchpad is trying a lot of things, but is clearly struggling to get what's important and what's not, etc... DVCS have some very interesting UI issues, and I haven't seen a clear solution yet.

I think we should recognize that we will almost certainly not solve the general problem. We should not let this failure impede us from solving the more immediate problems. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Hi Robert 2009/2/23 Robert Kern <robert.kern@gmail.com>:

...

1) Distributed revision control system: David Cournapeau and myself have been test driving Git [1] on SciPy and NumPy for a while. It is fast, well supported, has great branch support, and is simple to use for the average contributor, while allowing powerful patch-carving for the more adventurous.

While I really like DVCS in general, I don't think there is much benefit to switching. The various DVCS-SVN bridges account for most of the benefits, I think.

The thing I miss most is merge support. They are working on that for 1.6, but all the DVC systems have it now. If we don't switch, we should consider an official DVCS-based mirror, so that patches can be supplied as links to Github or elsewhere.

You may also want to consider using Roundup for just bug tracking and forgoing "integrated" solutions like the above entirely. We don't use the Trac wiki for anything we couldn't do on the Moin site.

Thanks, I completely forgot about Roundup. I remember it from yonks ago when it won the Software Carpentry competition! Cheers Stéfan

On Mon, Feb 23, 2009 at 20:42, Pauli Virtanen <pav@iki.fi> wrote:

Mon, 23 Feb 2009 19:40:27 -0600, Robert Kern wrote: [clip]

...

Pauli, you seem familiar with setting up a git-to-svn bridge. Can you do this?

Sure. I'll need a box on which to deploy the update script, though. Would one (which?) of the virtual hosts of conference.scipy.org do?

Probably. That machine is where all of the services will be moving to. Peter might be able to say which one.

I'd guess what's needed of the web server would be only to enable CGI for a single script. When poked, it would then fetch new stuff from SVN and either

- Push to github or some such service - Push to a HTTP location on the machine, served statically - Push to a HTTP location on the machine, served by gitweb (cgi)

The first option is probably the easiest, if account/password issues can be sorted out. The second option is probably enough for practical purposes.

1) probably has the advantage of having smaller overhead for people publishing their own branches on github.

...

What do you need done one the actual SVN server to support this?

A post-commit hook poking a CGI script somewhere should be enough:

/usr/bin/curl -d "revision=$REV&repository=$REPOS" \ http://host/path/to/script.cgi &

If all this runs on the same machine as the SVN server, going over the network can be skipped, too.

Anyway, I'll think about the details tomorrow.

Great. Thank you! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

2009/2/24 Pauli Virtanen <pav@iki.fi>:

The first option is probably the easiest, if account/password issues can be sorted out. The second option is probably enough for practical purposes.

Authentication to github is done with SSH keys, so it should be easy to automate. Cheers Stéfan

Hi, Only two small things to add to this (interesting) discussion on git stuff: 1. tortoisegit is in active development (http://code.google.com/p/tortoisegit/). Haven't tried it, but looks good already. 2. I've found that, for a new user, having a git cheat sheet (e.g. http://zrusin.blogspot.com/2007/09/git-cheat-sheet.html, the one I prefer) gives about the same power, as using svn. For basic users, the features of svn are pretty much just a subset of those of git, and such a cheat sheet greatly helps the learning curve. Just a few thoughts. (I use git for all my stuff, and love it). --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak@gmail.com ++++++++++++++++++++++++++++++++++++++++++

On Mon, Feb 23, 2009 at 6:40 PM, Robert Kern <robert.kern@gmail.com> wrote:

Here is my take at the current time:

The experimentalist in me cries out that we should make just one major infrastructure change at a time. For a variety of reasons, from IT support issues to just plain hating {{{}}}, I think replacing the bug tracker should be the change to make right now.

I think this is a sensible approach. Getting the up-time and bug tracking issues straightened will be a big help.

That said, we can try a blessed DVCS-SVN bridge and see how it works out. It doesn't solve all problems, but it should enable a better workflow for casual contributors, the raison d'être for this discussion. It also doesn't commit us to anything while we are separately seeing how the tracker changes work out.

Git on windows has improved a lot in the last year. Another year or two might make it a good solution for everyone. Chuck

On Tue, Feb 24, 2009 at 1:21 PM, Bill Baxter <wbaxter@gmail.com> wrote:

...

Git on windows has improved a lot in the last year. Another year or two might make it a good solution for everyone.

I gave msys-Git a try last week and it was not smooth sailing for me. Seemed to maybe be having conflicts with the Unix tool ports I usually use.

IIRC, there is an option to check/uncheck to avoid this exact issue in the installer. Concerning the bzr vs git thing, it is hard to really know the problems without having used both of them extensively; a lot are just "look how cool git is", look how cool bzr is. I wrote a couple of months ago a comparison which totally omits the speed aspect if that interests you: http://cournape.wordpress.com/2008/10/30/going-away-from-bzr-toward-git/ David

On Tue, Feb 24, 2009 at 10:40 AM, Robert Kern <robert.kern@gmail.com> wrote:

David and Stéfan, can you work on proposing a new tracker configuration? I ask that you take a glance at Roundup, but I'll leave that up to your schedules.

That sounds like a plan. One thing which I think would be helpful is a svn + trac dump, so that we have something to try things on, thank you very much, David