On Wed, May 26, 2010 at 4:12 PM, Travis Oliphant wrote:

I think we are ready for such a move.    Someone should think about the implications, though (with Trac integration, check-in mailings, etc.) and make sure we get something we all like.   Somebody probably has thought through all of these things already, though.

Cool. At this point, I am just testing the water. If enough people seem to be OK with the idea in general, I can spend some time looking into the details more closely. Before we make an actual decision, it would be worth turning this into an actual NEP and then asking people to review it. Jarrod

2010/5/26 Stéfan van der Walt :

On 26 May 2010 16:12, Travis Oliphant wrote:

...

I changed the subject line for this thread, since I didn't want to hijack another thread.  Anyway, I am not proposing that we actually decide whether to move to git and github now, but I am just curious how people would feel.  We had a conversation about this a few years ago and it was quite contentious at the time.  Since then, I believe a number of us have started using git and github for most of our work. And there are a number of developers using git-svn to develop numpy now.  So I was curious to get a feeling for what people would think about it, if we moved to git.  (I don't want to rehash the arguments for the move.)

I think we are ready for such a move.    Someone should think about the implications, though (with Trac integration, check-in mailings, etc.) and make sure we get something we all like.   Somebody probably has thought through all of these things already, though.

Awesome, if there's enough interest I'll help Jarrod out on the NEP. I've been looking at GitHub's Trac integration, and it seems that we should be able to have the same level of integration with the bugtracker as we currently do.  Their plugin is available here:

http://github.com/davglass/github-trac/

The SVN-checkout functionality should take care of the build bot.  As a bonus, we no longer have to administrate user accounts.  Converting the SVN repo to Git should pose no problem.

Regards Stéfan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

You are all probably aware of this, but I just wanted it said. I do understand the advantage of being able to pull from someone's Python 3 branch (like scipy) as well as some of the more experimental side like the proposed refactoring. All that I ask is that there is one official place to do 'git clone' or 'git pull' from a single official branch. I do not think that it is good to tell users to pull from different branches especially if these branches have conflicts. It also provides a common foundation to troubleshoot problems (of course you don't see it because you don't have that branch...). Yet I do understand that any release candidate can be pulled from any tree (as happens with the Linux kernel) and that this should be more of guide than a fixed rule. Bruce

On Wed, May 26, 2010 at 6:37 PM, Bruce Southey wrote:

All that I ask is that there is one official place to do 'git clone' or 'git pull' from a single official branch. I do not think that it is good to tell users to pull from different branches especially if these branches have conflicts. It also provides a common foundation to troubleshoot problems (of course you don't see it because you don't have that branch...). Yet I do understand that any release candidate can be pulled from any tree (as happens with the Linux kernel) and that this should be more of guide than a fixed rule.

That seems to be a very reasonable request and one that I agree with. If we do move to git we will have an official master branch, which will be a single official branch. Thanks for the feedback. Jarrod

Hi,

It seems to me that git's flexibility in how people collaborate means we can do a certain amount of figuring out after the switch.

This is very well said and true to our recent experience with nipy and ipython: http://github.com/ipython/ipython http://github.com/nipy/nipy

My experience with a small project has been that anyone who wants to make major changes just clones the repository on github and makes the changes; then we email the main author to ask him to pull particular branches into the main repo. It works well enough.

That's the model we've gone for in nipy and ipython too. We wrote it up in a workflow doc project. Here are the example docs giving the git workflow for ipython: https://cirl.berkeley.edu/mb312/gitwash/ and in particular: https://cirl.berkeley.edu/mb312/gitwash/development_workflow.html Cheers, Matthew

On Wed, May 26, 2010 at 9:49 PM, Jarrod Millman wrote:

On Wed, May 26, 2010 at 8:08 PM, Matthew Brett wrote:

...

That's the model we've gone for in nipy and ipython too. We wrote it up in a workflow doc project. Here are the example docs giving the git workflow for ipython:

https://cirl.berkeley.edu/mb312/gitwash/

and in particular:

https://cirl.berkeley.edu/mb312/gitwash/development_workflow.html

I would highly recommend using this workflow. Ideally, we should use the same git workflow for all the scipy-related projects. That way developers can switch between projects without having to switch workflows. The model that Matthew and Fernando developed for nipy and ipython seem like a very reasonable place to start. __

I wouldn't. Who is going to be the gate keeper and pull the stuff? No vacations for him/her, on 24 hour call, yes? They might as well run a dairy. And do we really want all pull requests cross-posted to the list? Linus works full time as gatekeeper for Linux and gets paid for the effort. I think a central repository model would work better for us. Chuck

On 27 May 2010 01:22, Charles R Harris wrote:

On Wed, May 26, 2010 at 9:49 PM, Jarrod Millman wrote:

...

On Wed, May 26, 2010 at 8:08 PM, Matthew Brett wrote:

...

That's the model we've gone for in nipy and ipython too.  We wrote it up in a workflow doc project.  Here are the example docs giving the git workflow for ipython:

https://cirl.berkeley.edu/mb312/gitwash/

and in particular:

https://cirl.berkeley.edu/mb312/gitwash/development_workflow.html

I would highly recommend using this workflow.  Ideally, we should use the same git workflow for all the scipy-related projects.  That way developers can switch between projects without having to switch workflows.  The model that Matthew and Fernando developed for nipy and ipython seem like a very reasonable place to start. __

I wouldn't. Who is going to be the gate keeper and pull the stuff? No vacations for him/her, on 24 hour call, yes? They might as well run a dairy. And do we really want all pull requests cross-posted to the list? Linus works full time as gatekeeper for Linux and gets paid for the effort. I think a central repository model would work better for us.

I don't think this is as big a problem as it sounds. If the gatekeeper takes a week-long vacation, so what? People keep working on their changes independently and they can get merged when the gatekeeper gets around to it. If they want to accelerate the ultimate merging they can pull the central repository into their own and resolve all conflicts, so that the pull into the central repository goes smoothly. If the gatekeeper's away and the users want to swap patches, well, they just pull from each other's public git repositories. Anne

Chuck

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi,

Linux has Linus, ipython has Fernando, nipy has... well, I'm sure it is somebody. Numpy and Scipy no longer have a central figure and I like it that way. There is no reason that DVCS has to inevitably lead to a central authority.

I think I was trying to say that the way it looks as if it will be - before you try it - is very different from the way it actually is when you get there. Anne put the idea very well - but I still think it is very hard to understand, without trying it, just how liberating the workflow is from anxieties about central authorities and so on. You can just get on with what you want to do, talk with or merge from whoever you want, and the whole development process becomes much more fluid and productive. And I know that sounds chaotic but - it just works. Really really well. See you, Matthew

On Wed, May 26, 2010 at 11:06 PM, Anne Archibald

wrote:

...

Hi,

...

Linux has Linus, ipython has Fernando, nipy has... well, I'm sure it is somebody. Numpy and Scipy no longer have a central figure and I like it

On 27 May 2010 01:55, Matthew Brett wrote: that

...

...

way. There is no reason that DVCS has to inevitably lead to a central authority.

I think I was trying to say that the way it looks as if it will be - before you try it - is very different from the way it actually is when you get there. Anne put the idea very well - but I still think it is very hard to understand, without trying it, just how liberating the workflow is from anxieties about central authorities and so on. You can just get on with what you want to do, talk with or merge from whoever you want, and the whole development process becomes much more fluid and productive. And I know that sounds chaotic but - it just works. Really really well.

One way to think of it is that there is no "main line" of development. The only time the central repository needs to pull from the others is when a release is being prepared. As it stands we do have a single release manager, though it's not necessarily the same for each version. So if we wanted, they could just go and pull and merge the repositories of everyone who's made a useful change, then release the results. Of course, this will be vastly easier if all those other people have already merged each other's results (into different branches if appropriate). But just like now, it's the release manager's decision which changes end up in the next version.

No, at this point we don't have a release manager, we haven't since 1.2. We have people who do the builds and put them up on sourceforge, but they aren't release managers, they don't decide what is in the release or organise the effort. We haven't had a central figure since Travis got a real job ;) And now David has a real job too. I'm just pointing out that that projects like Linux and IPython have central figures because the originators are still active in the development. Let me put it this way, right now, who would you choose to pull the changes and release the official version? Chuck

On Wed, May 26, 2010 at 11:28 PM, David wrote:

On 05/27/2010 02:16 PM, Charles R Harris wrote:

...

On Wed, May 26, 2010 at 11:06 PM, Anne Archibald mailto:aarchiba@physics.mcgill.ca> wrote:

On 27 May 2010 01:55, Matthew Brett mailto:matthew.brett@gmail.com> wrote: > Hi, > >> Linux has Linus, ipython has Fernando, nipy has... well, I'm sure it is >> somebody. Numpy and Scipy no longer have a central figure and I like it that >> way. There is no reason that DVCS has to inevitably lead to a central >> authority. > > I think I was trying to say that the way it looks as if it will be

-

...

> before you try it - is very different from the way it actually is when > you get there. Anne put the idea very well - but I still think it is > very hard to understand, without trying it, just how liberating

the

...

> workflow is from anxieties about central authorities and so on. You > can just get on with what you want to do, talk with or merge from > whoever you want, and the whole development process becomes much

more

...

> fluid and productive. And I know that sounds chaotic but - it

just

...

> works. Really really well.

One way to think of it is that there is no "main line" of

development.

...

The only time the central repository needs to pull from the others is when a release is being prepared. As it stands we do have a single release manager, though it's not necessarily the same for each version. So if we wanted, they could just go and pull and merge the repositories of everyone who's made a useful change, then release the results. Of course, this will be vastly easier if all those other people have already merged each other's results (into different branches if appropriate). But just like now, it's the release manager's decision which changes end up in the next version.

No, at this point we don't have a release manager, we haven't since 1.2. We have people who do the builds and put them up on sourceforge, but they aren't release managers, they don't decide what is in the release or organise the effort. We haven't had a central figure since Travis got a real job ;) And now David has a real job too. I'm just pointing out that that projects like Linux and IPython have central figures because the originators are still active in the development. Let me put it this way, right now, who would you choose to pull the changes and release the official version?

Ralf is the release manager, and for deciding what goes into the release, we do just as we do now. For small changes which do not warrant discussion, they would be handled through pull requests in github at first, but we can improve after that (for example having an automatic gatekeeper which only pulls something that would at least compile and pass the test on a linux machine).

So you are saying that Ralf has to manage all the pull requests? Have you asked Ralf about that? An automatic gatekeeper is pretty much a central repository, as I was suggesting. Chuck

On Thu, May 27, 2010 at 1:34 PM, Charles R Harris

wrote:

On Wed, May 26, 2010 at 11:28 PM, David wrote:

...

On 05/27/2010 02:16 PM, Charles R Harris wrote:

...

On Wed, May 26, 2010 at 11:06 PM, Anne Archibald mailto:aarchiba@physics.mcgill.ca> wrote:

On 27 May 2010 01:55, Matthew Brett mailto:matthew.brett@gmail.com> wrote: > Hi, > >> Linux has Linus, ipython has Fernando, nipy has... well, I'm sure it is >> somebody. Numpy and Scipy no longer have a central figure and I like it that >> way. There is no reason that DVCS has to inevitably lead to a central >> authority. > > I think I was trying to say that the way it looks as if it will

be -

...

> before you try it - is very different from the way it actually is when > you get there. Anne put the idea very well - but I still think it is > very hard to understand, without trying it, just how liberating

the

...

> workflow is from anxieties about central authorities and so on. You > can just get on with what you want to do, talk with or merge from > whoever you want, and the whole development process becomes much

more

...

> fluid and productive. And I know that sounds chaotic but - it

just

...

> works. Really really well.

One way to think of it is that there is no "main line" of

development.

...

The only time the central repository needs to pull from the others

is

...

when a release is being prepared. As it stands we do have a single release manager, though it's not necessarily the same for each version. So if we wanted, they could just go and pull and merge the repositories of everyone who's made a useful change, then release

the

...

results. Of course, this will be vastly easier if all those other people have already merged each other's results (into different branches if appropriate). But just like now, it's the release manager's decision which changes end up in the next version.

No, at this point we don't have a release manager, we haven't since 1.2. We have people who do the builds and put them up on sourceforge, but they aren't release managers, they don't decide what is in the release or organise the effort. We haven't had a central figure since Travis got a real job ;) And now David has a real job too. I'm just pointing out that that projects like Linux and IPython have central figures because the originators are still active in the development. Let me put it this way, right now, who would you choose to pull the changes and release the official version?

Ralf is the release manager, and for deciding what goes into the release, we do just as we do now. For small changes which do not warrant discussion, they would be handled through pull requests in github at first, but we can improve after that (for example having an automatic gatekeeper which only pulls something that would at least compile and pass the test on a linux machine).

So you are saying that Ralf has to manage all the pull requests?

I'd hope not. For the record, I really like the development model Matthew proposed. About deciding what goes into a release, I'm sure that David meant small stuff like "this can't go in, it's too late in the release cycle" or "this code needs tests if you want it to be in this release". Cheers, Ralf

2010/5/27 Friedrich Romstedt :

I just want to say that I used Git on Windows without any problem using a minGW built Git, i.e. msysgit:

Hm, I read the other thread too late to recognise this to be discussed already - Sorry And hey, even Windows has Tab completion of path names, even in the, agreed, terrible, non-PowerShell console.

On Wed, May 26, 2010 at 11:34 PM, Matthew Brett wrote:

Hi,

...

No, at this point we don't have a release manager, we haven't since 1.2. We have people who do the builds and put them up on sourceforge, but they aren't release managers, they don't decide what is in the release or organise the effort. We haven't had a central figure since Travis got a real job ;) And now David has a real job too. I'm just pointing out that that projects like Linux and IPython have central figures because the originators are still active in the development. Let me put it this way, right now, who would you choose to pull the changes and release the official version?

OK - for nipy - we have - I think - 5 people who can commit into the main repository. Any one of those 5 people can review someone's work, and commit into the main repository. My guess is - with numpy - there would be some number of people with the same permissions - I imagine you among them. But the rule is -

No-one commits into the main repo without someone reviewing and agreeing the work

Any trusted person can review. But the point is:

No development in the main repo. Merges only.

Why?

Let's flip your question the other way round.

You are saying - I want to continue (as for SVN) to develop in the main repo.

No, I am saying we need at least five people who can commit to the main repo. That is the central repository model.

But the main repo is where everyone merges from. That means that

a) It makes it much harder for anyone to review your changes because they are mixed up in a lot of other changes and

b) You force everyone following numpy to adopt your changes

In practice - that means that you make it harder for others by making them follow your line of development when they may not want to - until it's ready.

Review is fine, and it would be nice if more people were reviewing code. At the moment I think it is just Pauli, Stefan, and myself. I guess you'd agree that code review is essential to good code quality

- both for improving code - and for teaching. It encourages new developers because they know their work will be checked. It helps developers learn the coding guidelines and to share good practice. It helps the developers have a broad knowledge of the code base.

With SVN / central repo development - that's really hard - because all the development lines get mixed up as people work in different places.

But a repo that five folks can commit to *is* a central repository, by definition. DVCS and central repository are orthogonal concepts. With git / DVCS - it suddenly becomes absolutely natural.

I think that's why people like Joel Spolsy say stuff like 'This is possibly the biggest advance in software development technology in the ten years I’ve been writing articles here.' : http://www.joelonsoftware.com/items/2010/03/17.html

Please - try it - see - I am absolutely sure you'll love it after a very short time...

Chuck

On Wed, May 26, 2010 at 11:55 PM, Matthew Brett wrote:

Hi,

...

No, I am saying we need at least five people who can commit to the main repo. That is the central repository model.

Excellent - yes - that's reasonable. Then if you also agree to this:

...

No development in the main repo. Merges only.

then we're all in full agreement.

...

Review is fine, and it would be nice if more people were reviewing code. At the moment I think it is just Pauli, Stefan, and myself.

Right - and that is partly because it so much harder to do review with the model that we have at the moment, and partly because we don't yet have the tradition in numpy of review. I think - honestly - if we're going to be able to encourage and train new developers - we'll have to get on that as soon as we can...

See you,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi,

How does that differ from what we do now? Review? I develop in my own branches as is.

Right - so - then do you always ask for a review from someone before merging into trunk? If so, then git is just a much more fluid, reliable and faster tool to do what you are doing now.

True, but what happens when there is no review? I might point out that there are currently tickets with patches for review going back two years and reviewing a patch isn't *that* much harder than visiting github. Using git makes merging changes much easier, but it doesn't solve the review problem.

Well - that's true and not true. The joy of git branches and the ease of merging is that you quickly get into the habit of making feature branches for each piece of work. This makes it extremely easy for someone else to review the changes that you have made. So, it greatly lowers the work needed for someone to review your code, and therefore makes it more likely. Having said that - it will of course happen that you ask for review and no-one responds. That's not a very big problem, because git merges are so easy that you can - as Anne said earlier - just keep on developing without worrying that your changes will go out of date. But if there's a long wait - or it's urgent - then what I do is just email with 'If I don't hear anything I'll merge these changes in a few days'. See you, Matthew

Hi,

...

Having said that - it will of course happen that you ask for review and no-one responds.  That's not a very big problem, because git merges are so easy that you can - as Anne said earlier - just keep on developing without worrying that your changes will go out of date. But if there's a long wait - or it's urgent - then what I do is just email with 'If I don't hear anything I'll merge these changes in a few days'.

Exactly. I had a private bet with myself that that would be the case. See, it isn't so much different after all. The tools change, but the problems and solutions remain much the same. Given that there are only three people doing reviews, and really only two really looking at the c code, I expect that a lot of stuff will be merged without much in the way of review.

Well - I do honestly think that a decentralized git workflow is the best tool to improve that.

Now if git leads to more developers that might change. Here's hoping.

I hope so too. I accidentally ran across this a few days ago: http://www.erlang.org/ - "This [Erlang/OTP R13B04] is the first release after the introduction of the official Git repository at Github and it is amazing to notice that the number of contributions from the community has increased significantly. As many as 32 contributors have provided 1 or more patches each until now, resulting in 51 integrated patches from the open source community in this service release." Here's hoping... See you, Matthew

2010/5/27 Stéfan van der Walt

On 26 May 2010 23:27, Charles R Harris wrote:

...

Exactly. I had a private bet with myself that that would be the case. See, it isn't so much different after all. The tools change, but the problems and solutions remain much the same.

In this case, I believe the tool may be part of the solution. With limited manpower at our disposal, having a somewhat painful process certainly doesn't help.

It should help. A commitment to doing reviews is probably more important here than submitting for review. It's less fun than development and takes a certain commitment. Of course, there are probably some perverts out there who find it enjoyable. I hope we find some.

- Working with patches is unreliable (check out all the patches in Trac that don't apply cleanly and how much effort it will be to fix them). Distributed revision control provides a much better structure within which to manage patches.

Two year old patches are always going to be a problem. The real fix here is not to let things languish.

- Merging in SVN is horrible and will never encourage branches. Without branches, trunk becomes turbulent easily.

True. Although there would need to be more activity to get to true turbulence.

- We currently don't have any code review in place. This isn't SVN's fault, but tools such as GitHub's compare view (http://github.com/blog/612-introducing-github-compare-view) look really promising

Maybe most importantly, distributed revision control places any possible contributor on equal footing with those with commit access; this is one important step in making contributors feel valued.

Well, not quite. They can't commit to the main repository. I think the main thing is to be responsive: fast review, quick commit. And quick to offer commit rights to anyone who sends in more that a couple of decent patches. Maybe we should take a vow to review one patch a week. Chuck

On 27 May 2010 04:43, Charles R Harris wrote:

...

Maybe most importantly, distributed revision control places any possible contributor on equal footing with those with commit access; this is one important step in making contributors feel valued.

Well, not quite. They can't commit to the main repository. I think the main thing is to be responsive: fast review, quick commit. And quick to offer commit rights to anyone who sends in more that a couple of decent patches. Maybe we should take a vow to review one patch a week.

Okay. Suppose we wanted to replicate the current permissions arrangement as closely as possible with git. It seems to me it would look something like this: * Set up a git repository somewhere on scipy.org. * Give everyone who currently has permission to commit to SVN permission to write to this repository. * git submissions would become possible: a user would make some changes but instead of posting a patch would link to a particular git state. The changes could be reviewed and incorporated like a patch, but with easier merging and better history. If the changes became out of date the user could easily merge from the central repository and resolve the conflict themselves. * Patch submissions would be reviewed as now and committed to git by one of the people who do this now. Alternatively they could be integrated to the mainline by someone without write access and published as a git change, to be incorporated (easily) as above by someone with write access. * if review and inclusion were slow it would nevertheless be easy for users to pull from each other and build on each other's changes without making the eventual merge a nightmare. So, no major change to who controls what. The nipy/ipython model takes this a step further, reasoning that git makes branching and merging so easy there's no need for such a large group of people with write access to the central repository, but if that doesn't work for numpy/scipy we don't need to do it. And we can change in either direction at any time with no major changes to infrastructure or workflow. To get back to the original point of the thread: nobody has yet objected to git, and all we have are some debates about the ultimate workflow that don't make much difference to whether or how git should be adopted. Is this a fair description? Anne

On Thu, May 27, 2010 at 2:28 AM, Anne Archibald wrote:

* Set up a git repository somewhere on scipy.org.

It's a minor point, but setting up and maintaining our own git repository will require extra work without gaining anything useful. Github has a number of very useful features and is gaining new functionality all the time. It also greatly simplifies account management, which is a royal pain with our current system. Obviously this is a separate issue from whether we move to git or not, but I just wanted to address it quickly. I've registered the following github accounts just to reserve them for now: http://github.com/numpy http://github.com/scipy http://github.com/scikits Jarrod