Hi Glyph, That is great news. I already helped with Braid and would be interested in contributing some work in this area. Cheers, Jonathan On Jun 3, 2013, at 16:59, Glyph wrote:

Hi Twisted developers,

This weekend I had a discussion with many Twisted developers, both local to and visiting San Francisco. The topic came up of how to get more long-term contributors to participate more regularly in the project - particularly, doing code reviews, but also, developing and contributing to complex fixes and features that new contributors might not be able to tackle.

One suggestion that almost everybody made immediately was: we should use Github for code reviews.

In the past, I've heard this suggestion given mainly as a way to contribute more code, which does not appeal to me, since we are already swamped reviewing all the code that is currently being contributed.

This time, however, it's been pitched as a way to get people to do more reviews, which I'm keenly interested in. Why would people do more reviews on Github? In a nutshell, it's a lot less work. Here are some reasons why:

• Instead of having to run 'force-builds' on the command line, or load a buildbot status page, Github has a way for a build system to report build success automatically, so you can see immediately within a pull request if the changes that it proposes are "good to merge". You can see this at work with Travis here: https://github.com/twisted/klein/pull/11. Originally I thought that this was a Travis-CI feature, but have since learned that this is apparently easy - trivial even - to hook up to Buildbot, since it's a simple HTTP API to invoke when a build completes, and there is even some existing buildbot infrastructure (deployed by Django, among others) to automate it. • Instead of having to describe each patch location so that you can comment on it in a single message, if you want to put a comment on a particular part of a diff in a Github pull request, you can just click on it and start typing. • In addition to the diff, it's reasonably easy to see the code in context on the web, which is faster than getting it into one's local development environment. • If a review is successful, instead of having to have a local development environment, a committer can just hit the "approve" button and it's landed immediately. • Instead of having to read through all history ever to see what's still relevant, a pull request will hide comments that address outdated diffs, allowing the change author to easily see what remains.

These advantages are not comprehensive, but they're the more significant ones I remember from this discussion.

A prerequisite for using Github for code reviews would be using Git rather than Subversion. Luckily there's not much work to do in this area, thanks to Tom's excellent work on the Git import and automatic Github mirror. As a bonus, by using Git instead of Subversion, we can start properly recording merge metadata.

In this discussion, Alex Gaynor pointed out that Django has a hybrid workflow where they still use Trac for bug tracking, and Github for code review. We would therefore not need to come up with a way to migrate all of our tickets to Github issues (which seems, oddly, to be fairly unpopular even among those who like github a lot).

What would need to happen in order for this to take place?

• We'd need some consensus (hence this message). • We'd need to update the release process http://twistedmatrix.com/trac/wiki/ReleaseProcess and our development documentation http://twistedmatrix.com/trac/wiki/BasicGuideToContributingCode to refer to the relevant Git commands rather than Subversion commands. • We'd need a redirect from http://twistedmatrix.com/trac/browser/ and http://twistedmatrix.com/trac/changeset/ that would point at https://github.com/twisted/twisted and https://github.com/twisted/twisted/commits/ respectively. • We'd need a Github web hook that could poke Buildbot to kick off commits. • We'd need Buildbot integration to update Github pull requests with build results when builds complete. • We'd need someone to install git rather than bzr on all the buildbots, and update the configuration of the builders to get the code from a git rather than Subversion URL. • Someone will need to convert every open ticket in review to a pull request.

I do anticipate some objections.

One objection is that each of the above tasks is going to take some work.

I am fairly confident that some of the people who have educated me here will step forward to volunteer to do it. Please reply to this message if you'd like to volunteer, saying what you'd like to volunteer to do. If not, then I guess that objection stands :-).

Another is that this might not be worth that investment of effort. This is why it was nice to have Alex contributing to the discussion: Django did basically this very change (right down to the "Trac for tickets / Github for pull requests" distinction), at a much higher scale than we have, and as he described it the change was *well* worth it.

Another objection is that Github is proprietary software, and an externally-maintained service that we'd be depending upon.

One solution to the "proprietary software" thing is the availability of the MIT-licensed http://gitlab.org. It's a largely feature-complete clone of Github; if, for some reason, we need to migrate away from Github in a hurry, it will be relatively painless to set up Gitlab instead, and the fact that Git is a DVCS means every contributor will serve as a backup. The main reason I would not suggest just deploying it is that it creates another sticky infrastructure-management problem, and while Braid is great, I'd prefer to avoid creating more work in that area. Github also has APIs for literally all of their features, so we can create a backup script.

(Also worth noting: Gitlab is an open-source competitor to Github, but they still trust Github enough to https://github.com/gitlabhq/ host their own development there.)

Finally, my own minor concern: Github has no notion of a "code review" as a unit of work. A pull request is just "open" until it is "closed". Closing pull requests to request changes would be jarring to the cultural norms associated with Github's UI. All the github users I've spoken with, even those who follow processes which are effectively identical to Twisted's, have assured me that this is not really an issue. A code review is "accepted" when you merge it; it's "rejected" if the pull request is still open but has some comments on it. This will make porting over http://twistedmatrix.com/highscores/ a bit challenging, but I think it would be worth letting that break for the time being.

-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Am 03.06.2013 22:59, schrieb Glyph:

Another objection is that Github is proprietary software, and an externally-maintained service that we'd be depending upon.

One solution to the "proprietary software" thing is the availability of the MIT-licensed http://gitlab.org. It's a largely feature-complete clone of Github; if, for some reason, we need to migrate away from Github in a hurry, it will be relatively painless to set up Gitlab instead, and the fact that Git is a DVCS means every contributor will serve as a backup. The main reason I would not suggest just deploying it is that it creates another sticky infrastructure-management problem, and while Braid is great, I'd prefer to avoid creating /more/ work in that area. Github also has APIs for literally all of their features, so we can create a backup script.

(Also worth noting: Gitlab is an open-source competitor to Github, but they still trust Github enough to https://github.com/gitlabhq/ host their own development there.)

It may be worth noting that the reason for gitlab being hosted on github is probably due to the fact that up until very recently, gitlab had no way for a non-registered user to access gitlab at all. Even though they have introduced a "public area" for projects this does only include anonymously cloning of a repository, you still cannot browse code or look at issues, pull/merge request or wikis without a user account. Although gitlab is great for internal projects, this lack of a proper support for public features makes it imo not that suitable for open projects. Cheers, Chris

I sent most of the below off-list to Glyph earlier, as my comments were a bit half-assed and I'm not really (or not at all) a Twisted contributor. Glyph suggested I mail them to the list anyway, and to try adding some more concrete reasons for being +1 on the suggested change. --- [ Original mail ] I'm +1 bigtime on moving towards git/github. I really dislike git, but it gets the job done, and github is awesome. Github changed the way we work, it removed a ton of friction and overhead in reviewing (you're right, the click to comment on a diff is really convenient). I don't have too many complaints about the ticket system, but sure it could be better. The really impressive thing about Github is how incredibly quickly they move. It gets better and better and better all the time. We migrated away from svn + trac to bzr + launchpad and finally to git + github. The latter blows the former away massively, IMO. I'm not trying to give formal quantitative feedback, just my subjective opinion. I've also felt for ages that there's too much overhead in trying to contribute to Twisted. I'm a lazy/selfish kind of member of the Twisted community, and when I'm faced with the thought of having to set up all that old-fashioned (from my POV) machinery like svn and combinator and what-not to think about really contributing or running tests or whatever, some part of me just thinks "no, no, no, I'm not going back to that world". (see above re lazy, if that wasn't clear). OK, sorry for a bit of a rambling subjective mail. I'm sure you've heard all this before. The funny thing about these changes is that before you make them you always think things are pretty much fine. After you make them you wonder how you ever lived with the old system. That may not be the case here for you or for you & the Twisted community, but it certainly was for me with the move to github. --- Glyph> Maybe you could be a bit more specific as to the steps that Github eliminates, and _concrete_ ways in which it is more efficient. I'd say there are probably dozens of ways in which Gthub improved things for us. Many of them are minor, of course, but they all add up and the usability difference between Github and Trac or Launchpad is extreme. Ease of use and the anticipated amount of effort something is going to take are very important aspects of tools, in my opinion. If you know some action - like starting a review or branching or merging or commenting etc. - is going to take a certain amount of effort, that correlates (unless you're being paid or doing this out of some extreme reason, like being a core dev on a project) with how likely you are to do it. (E.g., making branches - it was possible in sccs, in rcs, in cvs, in svn, but.... oh, the pain! When making branches *and merging them* became so extraordinarily simple and pain free, it changed the way people worked.) As I said above, I'm a lazy/selfish Twisted onlooker. Many has been the time when I'd have been happy to chip in on something (e.g., last week someone was asking for some reviewing help) but the thought of getting the required machinery in place, and using it, even if it's just svn (ugh, ugh, ugh) stops me. That's a feeling I've had about Twisted many times over the last years. It's the feeling I have when someone asks me to go back to helping them on some old perl code - you just don't wanna go there :-) It's ugly and not fun. Sorry if these are wimpy reasons, but to me usability friction (real or perceived) is very important. A very nice thing about Github is that you can have conversations about the diff right on the page, inline, where the diff is shown. You don't need to download the branch, to use the command-line (and i LOVE the command line BTW), or to do anything like that. Github is smart enough to hide the discussion once a subsequent commit comes in that addresses the line being discussed (sure, that could go wrong but in practice it doesn't seem to). One click to merge branches is great - in fact we made a rule that merging is always done via Github. Adding of milestones and labels to issues is smooth, as is viewing various simple subsets of issues. The project activity graphs in Github are very interesting. GFM (Github flavored markdown) is really easy to use and the results are attractive. Gists are great. It's nice being able to have a wiki that's also just a repo. I very much like the "pull requests are a place where discussion takes place" approach - as opposed to holding that pull requests are just where branches go when they're "finished" and ready to be merged. BTW, it's very interesting to read about how the folks at Github use Github themselves [1, 2]. The workflow we developed is very similar, only slightly more complex. If people want more examples, I can try to go more systematically through my usage of Github & say more. As mentioned above, Github moves (improves) incredibly quickly. They put pretty much every other project I know to shame (including all of mine) in terms of how fast they get stuff done and improve the product and how attractively it's done. In comparison, Trac and Launchpad felt static, butt-ugly, and like they virtually never improved. Github gets better underneath you all the time. E.g., you happen to be sitting on a page looking at a pull request, and it updates dynamically without you needing to reload. The pull request is merged maybe, or the author pushes into the branch again, or conflicts with master get resolved by someone and the page updates. Little things, but they work damned well and make using the site a pleasure. I can't resist adding: a friend in IRC just said re this thread: "2003 called and asked for their shitty tools back." Terry [1] http://scottchacon.com/2011/08/31/github-flow.html [2] https://github.com/blog/1124-how-we-use-pull-requests-to-build-github On Mon, Jun 3, 2013 at 9:59 PM, Glyph wrote:

Hi Twisted developers,

This weekend I had a discussion with many Twisted developers, both local to and visiting San Francisco. The topic came up of how to get more long-term contributors to participate more regularly in the project - particularly, doing code reviews, but also, developing and contributing to complex fixes and features that new contributors might not be able to tackle.

One suggestion that almost everybody made immediately was: we should use Github for code reviews.

In the past, I've heard this suggestion given mainly as a way to *contribute more code*, which does not appeal to me, since we are already swamped reviewing all the code that is currently being contributed.

This time, however, it's been pitched as a way to get people to *do more reviews*, which I'm keenly interested in. Why would people do more reviews on Github? In a nutshell, it's a lot less work. Here are some reasons why:

- Instead of having to run 'force-builds' on the command line, or load a buildbot status page, Github has a way for a build system to report build success automatically, so you can see immediately within a pull request if the changes that it proposes are "good to merge". You can see this at work with Travis here: https://github.com/twisted/klein/pull/11. Originally I thought that this was a Travis-CI feature, but have since learned that this is apparently easy - trivial even - to hook up to Buildbot, since it's a simple HTTP API to invoke when a build completes, and there is even some existing buildbot infrastructure (deployed by Django, among others) to automate it. - Instead of having to describe each patch location so that you can comment on it in a single message, if you want to put a comment on a particular part of a diff in a Github pull request, you can just click on it and start typing. - In addition to the diff, it's reasonably easy to see the code in context on the web, which is faster than getting it into one's local development environment. - If a review is successful, instead of having to have a local development environment, a committer can just hit the "approve" button and it's landed immediately.

- Instead of having to read through all history ever to see what's still relevant, a pull request will hide comments that address outdated diffs, allowing the change author to easily see what remains.

These advantages are not comprehensive, but they're the more significant ones I remember from this discussion.

A prerequisite for using Github for code reviews would be using Git rather than Subversion. Luckily there's not much work to do in this area, thanks to Tom's excellent work on the Git import and automatic Github mirror. As a bonus, by using Git instead of Subversion, we can start properly recording merge metadata.

In this discussion, Alex Gaynor pointed out that Django has a hybrid workflow where they still use Trac for bug tracking, and Github for code review. We would therefore *not* need to come up with a way to migrate all of our tickets to Github issues (which seems, oddly, to be fairly unpopular even among those who like github a lot).

What would need to happen in order for this to take place?

1. We'd need some consensus (hence this message). 2. We'd need to update the release process < http://twistedmatrix.com/trac/wiki/ReleaseProcess> and our development documentation < http://twistedmatrix.com/trac/wiki/BasicGuideToContributingCode> to refer to the relevant Git commands rather than Subversion commands. 3. We'd need a redirect from http://twistedmatrix.com/trac/browser/ and http://twistedmatrix.com/trac/changeset/ that would point at < https://github.com/twisted/twisted> and < https://github.com/twisted/twisted/commits/> respectively. 4. We'd need a Github web hook that could poke Buildbot to kick off commits. 5. We'd need Buildbot integration to update Github pull requests with build results when builds complete. 6. We'd need someone to install git rather than bzr on all the buildbots, and update the configuration of the builders to get the code from a git rather than Subversion URL. 7. Someone will need to convert every open ticket in review to a pull request.

I do anticipate some objections.

One objection is that each of the above tasks is going to take some work.

I am fairly confident that some of the people who have educated me here will step forward to volunteer to do it. Please reply to this message if you'd like to volunteer, saying what you'd like to volunteer to do. If not, then I guess that objection stands :-).

Another is that this might not be worth that investment of effort. This is why it was nice to have Alex contributing to the discussion: Django did basically this very change (right down to the "Trac for tickets / Github for pull requests" distinction), at a much higher scale than we have, and as he described it the change was *well* worth it.

Another objection is that Github is proprietary software, and an externally-maintained service that we'd be depending upon.

One solution to the "proprietary software" thing is the availability of the MIT-licensed http://gitlab.org. It's a largely feature-complete clone of Github; if, for some reason, we need to migrate away from Github in a hurry, it will be relatively painless to set up Gitlab instead, and the fact that Git is a DVCS means every contributor will serve as a backup. The main reason I would not suggest just deploying it is that it creates another sticky infrastructure-management problem, and while Braid is great, I'd prefer to avoid creating *more* work in that area. Github also has APIs for literally all of their features, so we can create a backup script.

(Also worth noting: Gitlab is an open-source competitor to Github, but they still trust Github enough to https://github.com/gitlabhq/ host their own development there.)

Finally, my own minor concern: Github has no notion of a "code review" as a unit of work. A pull request is just "open" until it is "closed". Closing pull requests to request changes would be jarring to the cultural norms associated with Github's UI. All the github users I've spoken with, even those who follow processes which are effectively identical to Twisted's, have assured me that this is not really an issue. A code review is "accepted" when you merge it; it's "rejected" if the pull request is still open but has some comments on it. This will make porting over < http://twistedmatrix.com/highscores/> a bit challenging, but I think it would be worth letting that break for the time being.

-glyph

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

in general: +1 for this

Finally, my own minor concern: Github has no notion of a "code review" as a unit of work.  A pull request is just "open" until it is "closed". <snip>

I _think_ the following is true (if so, I find that strange) - pls correct me if I'm wrong: A pull request is not tied to a specific rev, but only to a source repo/branch. While the pull isn't merged (and hence closed), more commits on the source repo/branch can be added. So a pull request is kind of moving target .. GitHub seems to view that as a feature, not bug: """ Pull requests can be sent from any branch or commit but it's recommended that a topic branch be used so that follow-up commits can be pushed to update the pull request if necessary. """ https://help.github.com/articles/using-pull-requests """ It's important to use a new branch for pull requests for several reasons: It allows you to submit multiple pull requests without confusion. The classical Github gotcha is to continue committing to a pull request branch after making the initial request. When these commits are pushed to the remote, they will become part of the original pull request which often ends up conflating unrelated functionality. """ http://codeinthehole.com/writing/pull-requests-and-other-good-practices-for-... However: https://github.com/blog/712-pull-requests-2-0 I'm not sure how to interpret that .. /Tobias

Hi, I think moving to github will be a huge win for the Twisted project, and all the migration/integration issues are manageable. I would recommend you keep two things in mind: (1) I am a member of the FreeBSD project, and am mentoring a Google Summer of Code student. I pushed the student to use the github mirror of the FreeBSD repository: https://github.com/freebsd/freebsd (unsupported officially by the FreeBSD project, but used by a lot of people). github worked great, but the only problem my GSoC student had was a lot of RPC timeout errors when doing "git clone" of the FreeBSD code, which is quite a bit. The student is in New Delhi, India, so I don't know what the networking connectivity to github from around the world is. It was not a complete showstopper, because github allows you to click a link to download a single zip file of the repository, and that worked for him. So keep this in mind if you have a lot of Twisted developers around the world. (2) Make sure you always have a "Plan B"/"Disaster Recovery" plan to get the repo out of github. It's not that critical, but something worth keeping in your back pocket. In the past, project hosting sites like Sourceforge were very popular, and then the fell out of favor for various reasons. Today, github is very popular, but who knows what will happen in future. If github works out in the long term, great, but if in future something bad happens, Twisted should outlive github. -- Craig