[Twisted-Python] git repo maintenance
Hey all, At the sprint today, Glyph mentioned that the github repo hasn't been updated recently and needs someone to maintain it regularly. He also mentioned that pull requests would be accepted from github, at which point I immediately volunteered to keep the repo up to date :-) (I can't bear using svn anymore...) I'll be putting things in place (infrastructure, scripts, etc.) to assist me with this, so let me know if you have any concerns, questions, ideas, etc. Thanks, d
Awesome!!! At what point did this change? ISTR the lack of interest in maintaining the Github account was pretty much a consequence of it seeming pretty pointless: ie it was still easier to just use SVN than to figure out where the parts work together. Will code review also be happening at a pull request level? On Sun, Oct 21, 2012 at 1:24 AM, Duncan McGreggor < oubiwann@twistedmatrix.com> wrote:
-- cheers lvh
On Oct 21, 2012, at 7:02 AM, Laurens Van Houtven <_@lvh.cc> wrote:
Awesome!!!
At what point did this change?
Not sure. I don't know the history of the github repo nor its uses. I was certainly surprised with Glyph said that pull requests would be accepted.
ISTR the lack of interest in maintaining the Github account was pretty much a consequence of it seeming pretty pointless: ie it was still easier to just use SVN than to figure out where the parts work together.
Will code review also be happening at a pull request level?
Here's what I envision... 1) a ticket is created on tm.com/trac for a feature, bug, etc. 2) the developer forks the github repo so they can work in their own space, e.g., creating a branch for the new ticket 3) the ticket is updated with a link to the branch they created on github (e.g., github.com/username/feature-name-1234). 4) when ready for review, the dev rebases against master (which will be syncing with svn every 30 minutes or so), squashing all commits into one and issues a pull request on github 5) the ticket is updated to review status 6) other devs take a look at the diff (git clone twisted repo, git pull dev's branch), with review comments going in the trac ticket 7) dev iterates on feedback, rebasing on master every time 8) once approved, perform final rebase, and ask that your pull request be merged on github 9) someone will then ensure that git master is merged into svn trunk Note that this hasn't been fully tested yet, so one or more parts of this worklfow may have to change... This would allow devs to do 100% of their iteration work in git or hg (hg plays well with github, I hear), basically removing the process itself as a barrier to contributions to Twisted. Currently, my own biggest hurdle to resuming contributions to Twisted is the pain of old tech. I've gotten so used to the luxuries of git, hg, and bzr, that svn feels like walking for two days to pick up something at a store, when I can just drive an hour and be there (and not have to walk back home carrying lumber on my back). Everything else I work on (on a daily basis) uses fast, more efficient code repos and more efficient workflows. If forced to use things that slow me down, I'll just pass and do something that quite honestly is a lot more fun. Hopefully supporting a workflow like this will bring Twisted contributions back into the "fun"realm for more folks... d
On 05:43 pm, duncan.mcgreggor@gmail.com wrote:
This has always been the case. Anyone is free to contribute code in any way they want. (Though, if you contribute code in some way that no one can understand, there may be a delay in the contribution being accepted.) The blocker for people using git for Twisted development has been that no one has wanted to do it badly enough to commit to maintaining the git mirror.
As mentioned above, I have a slightly different perspective on this. :)
At some point *before* svn trunk is updated, Buildbot needs to be able to get the code so it can run tests on all our supported platforms. This is the actually important part of this email, you can ignore the rest if you want but this really needs to happen.
While I'm sympathetic to toolchain woes, I can't help but wonder if you're being really honest here (with yourself, at least). Running "svn diff" may make you feel bad inside, because svn isn't the latest cool toy, but a *hurdle*? It's just difficult to understand.
Hopefully supporting a workflow like this will bring Twisted contributions back into the "fun"realm for more folks...
Yes, hopefully so. Jean-Paul
On Sun, Oct 21, 2012 at 2:48 PM, <exarkun@twistedmatrix.com> wrote:
I'm not sure exactly what you want to hear. The few times I've tried to contributed to Twisted, svn was actually a big barrier. Trying to update my patches so that I'm sure the tests pass on trunk produced mysterious merge conflicts in files I've never touched. Maybe I'm bad at svn, but it's never worked well for me.
-- Jasper
On Oct 21, 2012, at 11:54 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
In case it was actually not clear and you're not just being sarcastic, exarkun was saying that it's difficult to understand that someone with the required expertise to contribute to Twisted in the first place would have trouble with running the command 'svn diff'. 'svn diff' itself is not at all difficult to understand. Since Subversion is effectively the baseline fisher-price "my first version control system", I would assume that anyone who could effectively use Git (which has all the user-interface convenience of an unshielded circular saw) would have no trouble with it, especially with the very basic usage that contributing to Twisted requires. Since you asked, there are two things that I'd like to hear: "Hooray! I will help oubiwann maintain the git(hub) mirror! what would you like me to do?" specific problems that you've had with SVN that we might be able to address in the future or help you with, and not vague bellyaching about how we're not using the thing you happen to like best. If you're going to say the first thing and commit to helping though, please be sure first that you actually have the time and energy to follow through. At this point, the number of people who have appeared, volunteered to maintain a git mirror, done it for five minutes and then disappeared forever, leaving it in a broken, unmaintained state, is in the double digits. (I am starting to wonder if Git gives it users some kind of brain damage that makes a person incapable of meeting commitments.)
The few times I've tried to contributed to Twisted, svn was actually a big barrier. Trying to update my patches so that I'm sure the tests pass on trunk produced mysterious merge conflicts in files I've never touched. Maybe I'm bad at svn, but it's never worked well for me.
Why aren't you just using git for local development then? You don't have commit access, so you should never need to touch an svn client other than git if you don't feel like it. This is not entirely a rhetorical question. We have always tried to be accommodating to DVCS users, providing instructions and repeated requests for both a plain git and/or a github ambassador to keep svn nicely synchronized and reduce the friction required for users of those tools to make contributions. If the documentation we've offered on <http://twistedmatrix.com/trac/wiki/GitMirror> is in any way incorrect or non-optimal, please don't hesitate to say exactly what would be better. If you need wiki edit permission to update the page, I'll gladly give it to you. -glyph
On Mon, Oct 22, 2012 at 12:05 AM, Glyph <glyph@twistedmatrix.com> wrote:
As I said, the issue I had was not "svn diff" -- I had been working on my fix, uploaded it to Trac, and it was a few months or so before someone reviewed it. I forget who it was, but the reviewer prompted me to make a few small style changes, flesh out a testcase, write a .news file, that sort of thing. I updated my source tree to pull in new changes from trunk, to make sure the patch that I had been working on didn't rot. I was frustrated when I got merge conflicts for files that I've never touched before.
I'd love to help out and do things, but unfortunately I don't have the time these days. If you think that we should review code on the Twisted GitHub repository, I'd gladly help out with that. I would help out and review patches on Trac, but it's not as convenient: as far as I'm aware: I can't set Trac up so that I get emails when a patch comes in in a module I'm comfortable with, and I can't get an RSS feed of all the important events in the project I'm aware of. If that exists and I couldn't find it, I'll gladly review patches today.
This is another rhetorical anecdote, I tried using git-svn when I was contributing to PyPy during the SVN days. It just didn't work out. I don't know or can't say the same for bzr-svn or hg-svn or whatever else cross-VCS systems there are, but I was left with a bad taste in my mouth, so I figured it would be more worthwhile to stick with the original source tool. I may just be a drooling moron.
-- Jasper
On Oct 21, 2012, at 9:55 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
If you can replicate the exact branch and changes you had difficulty with, I'd love to hear about the specifics, because this sounds like maybe it's a gotcha we should add to some development documentation. It should not be possible for this to happen with a simple 'svn up' so I want to know what command to tell users they should avoid. Feel free to reply off-list if it gets into fiddly details of subversion that might be off-topic.
You say "rhetorical anecdote", I say "FUD" :). But, perhaps you are more just in the class of users that need somebody like Duncan's help in order to use git against Twisted effectively, rather than the class of users who know the version control tools well enough to be able to provide that help.
I may just be a drooling moron.
I didn't say you were a "moron", being unable to figure out git's UI for something is merely an indication that you probably aren't eligible for the Fields medal; frankly I have a great deal of difficulty with it as well. If I didn't, I might have updated things myself. I hope you'll have a better time once those instructions are updated. -glyph
On Oct 21, 2012, at 11:05 PM, Glyph wrote:
I think there are a few issues with the documentation, from the standpoint of an infrequent contributor: • It appears that you have to go "all in" with svn if you want to contribute, when that isn't actually the case • <http://twistedmatrix.com/trac/wiki/GitMirror> is not terribly easy to find. • It does come up in search, but is otherwise not easily accessible when you're spelunking for info on how to contribute • The page with the links has it as a parenthetical aside <http://twistedmatrix.com/trac/wiki/TwistedDevelopment> • Casual contributors should have a different story than core committers, but the distinction isn't always prominent • <http://twistedmatrix.com/trac/wiki/GitMirror> makes this distinction, which is good • <http://twistedmatrix.com/trac/wiki/BazaarMirror> does not. For what it's worth, I've been using Mercurial patch queues to manage a couple of infrequent, small contributions. It works pretty well and doesn't rely on svn history or an svn bridge. That's not terribly interesting, but the point is that there are probably other workflows that are effective and it would be useful if they could be documented and shared easily. I created a ticket to address some of these points and carry on the docs discussion elsewhere. If it seems like a good idea I'll work on updating the wiki. http://twistedmatrix.com/trac/ticket/6154 Lucas Taylor
On Sun, Oct 21, 2012 at 11:30 AM, <exarkun@twistedmatrix.com> wrote:
I want to use git instead of svn so badly that I can taste it. Hopefully this provides the motivation necessary to maintain the repo :-)
Excellent feedback -- thanks! This is the sort of thing I need in order to plan this out sufficiently...
Well, I viewed git and github as the hipster's latest cool toy for a long time, but once I was force to use it, I was converted. The time I spent wrestling with tool chains and workflows all but disappeared. I never even thought about it anymore. That's the ideal user interface with any tool, as far as I'm concerned. When you can use it to be efficient and don't even think about the tool, where it's just an extension of your natural thought process, that's a tool that has been built correctly. Sadly, that's the opposite experience I have with svn-based branch management :-/ d
On Oct 21, 2012, at 11:57 AM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
For what it's worth, I haven't authored a branch for Twisted with SVN in the last 3 years. I use Bazaar almost exclusively and I get all the fun tools one generally associates with a DVCS, including offline history and a nice graphical revision viewer. (One reason I am concerned about migrating to Git is that this will stop working; in my opinions, Git's offline GUI tools are far worse than Bazaar's - almost as much worse as Launchpad's source browser is than Github.) Similarly, I do code reviews by using a local 'bzr merge'. The only weird workflow which isn't just built in to BZR-SVN is this: <http://twistedmatrix.com/trac/wiki/BazaarMirror#CommittingaBazaarbranchtoaSu...>. This involves rebasing, which makes me a little sad; I wish we could preserve more history, but it works fine. I still land branches on trunk using SVN, but that takes about ten seconds assuming you use Combinator or svnmerge.py, so I don't see that being a big impediment. You only need to worry about that if you have commit access anyway, so that's not most external contributors. If the need to do that to land changes is preventing you from writing them in the first place, just let me know when you have a branch to land, and I'll do it for you. You don't need to use SVN, you haven't needed to use SVN since git-svn came into existence, and we've had several threads on this list before about improving the Git instructions here: <http://twistedmatrix.com/trac/wiki/GitMirror>. If you look at the history for that page, you can see that it is two years old, and originally created by exarkun, who is not a git user himself. So, it strikes me as a bit odd, and I am a bit dismayed that anyone would have been put off of by the need to use Subversion when working with Twisted since you can use Git or Bazaar - or, I assume, Mercurial - right now. I would find it a bit more understandable if you are dismayed by the need to use Trac, which is harder to avoid when working on Twisted. For various reasons - which I hope we don't have to discuss right now - we aren't going to move the project to Github any time soon, so if anyone reading this thread is interested in alleviating some of the pain with Trac, <https://launchpad.net/frack> is a good place to get involved to help out. There are some pretty straightforward bugs that anyone interested could fix there. Frack is already up and running on twistedmatrix.com, albeit at a weird URL. See, for example, <http://twistedmatrix.com/users/frack.twistd/ui/ticket.html?id=1956>. This might be useful because it is a considerably faster ticket-viewer than Trac, do to its somewhat more parsimonious issuance of SQL queries. -glyph
On Sun, Oct 21, 2012 at 11:05 PM, Glyph <glyph@twistedmatrix.com> wrote:
I really wish I could figure out how to do this. The one page on the wiki helps a bit, but is insufficient for those not already familiar with bzr (like me). I've tried setting this up a couple of times, and I always end up just giving up on it. (and no I don't recall exactly why, except for the one time when installing bzr on windows hosed my machine and made it unbootable...not Twisted's fault of course).
Unless someone applied my patch, Combinator won't work on Windows. Also, since the great Divmod site meltdown, there's not good instructions on how to use it (aside from the wayback machine, which is what I've been using). I'd never heard of svnmerge.py until you mentioned it. Are there any instructions on how to use it with Twisted? I'm not finding any, and I'm reluctant to experiment since Twisted has a very specific way on interacting with the SVN repo.
Presumably you could use Mercurial with the hggit plugin against a git repo, but you can't use it directly against the SVN repo, as it has a bad commit in it that hgsubversion totally chokes on: https://bitbucket.org/durin42/hgsubversion/issue/350
Trac has never really bothered me that much. Yes, occasionally it will do something stupid, and it's...not fast, but I guess I'm just used to the pain. I'm looking forward to Frack, though. Kevin Horn
On Oct 22, 2012, at 10:53 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
Let me keep it super short and sweet for you: C:\PROJEC~1> bzr branch http://svn.twistedmatrix.com/bzr/Twisted/trunk Twisted C:\PROJEC~1> cd Twisted as many times as desired, C:\PROJEC~1\TWISTED> hack hack hack C:\PROJEC~1\TWISTED> C:\python27\python.exe bin\trial twisted C:\PROJEC~1\TWISTED> bzr commit Review time! C:\PROJEC~1\TWISTED> bzr send -o - > my.patch Stick my.patch on a ticket and you're good to go. No need for combinator, or virtualenv, or subversion (not even bzr-svn, we did that part for you), or even a C compiler, assuming the feature you're working on doesn't need one. If any of this isn't covered in the wiki page then please update it. And please feel free to ask more questions - if specific version control or workflow issues are preventing anyone from working on Twisted then it is definitely on-topic for this list and I will be happy to provide answers and update development documentation.
(and no I don't recall exactly why, except for the one time when installing bzr on windows hosed my machine and made it unbootable...not Twisted's fault of course).
Can we just agree that you had a virus and that this had nothing to do with anything in this thread? Even if that's not true I think it might be long-term more mentally healthy for you to believe :-)
I guess I should test this out myself at least once, but my understanding is that you just do svnmerge.py --bidirectional -S the-branch-you-want-to-merge in trunk. You can test this out for yourself though, since svnmerge.py is loudly advertised as "commit-free": you always have to do the commit yourself. More info here: <http://www.orcaware.com/svn/wiki/Svnmerge.py>. If we can ever upgrade to a version of SVN on the server that supports merge-tracking, the need for either of these tools will go away since a plain 'svn merge ../branches/branchname' will do the right thing. Any volunteer sysadmins want to take the opportunity to crawl out of the woodwork? (Please?) But again: if you have trouble with this part, please just drop an email to the list and I will do it for you. Do not let this minor step be an impediment to contributing to Twisted.
Sadness. Hopefully this renewed attention will get the mercurial folks to fix this bug sooner? Thanks for filing it.
Great, glad to hear it. By the way, if you've been shying away because it's slow, you should know that we did recently fix a major performance problem with the website. Bots were running amok on some very CPU and memory intensive requests, and robots.txt was not deployed correctly so it wasn't reducing any of the traffic as it should have. At the most recent sprint nobody complained about being unable to work because of the website falling over, which indicates that some of our more dire problems are now under control. -glyph
On Mon, Oct 22, 2012 at 2:22 PM, Glyph <glyph@twistedmatrix.com> wrote:
See that should totally be on a wiki page someplace. I'd be happy to put it up there, but I don't have edit permissions on that page (or any of them?) apparently...
Actually I know exactly why this happened, and it's because Windows has a hard limit on the number of "icon overlays" you can add to Explorer. I installed TortoiseBzr along with bzr, and went over that limit. This by itself is not such a big deal, but it turns out that the failure behavior of this particular issue is NOT BOOTING ANY MORE. (stupid Windows) (FYI, I've since installed bzr again, and have had no similar problems.) (also FYI, I haven't had a virus/spyware on a windows machine that I owned in more than 10 years...<knock on wood>...the trick is renaming the iexplore.exe executable...) But aside from that, the couple of times I've tried messing around with bzr I end up going in circles in their documentation, can't decide what I should be doing, decide I don't have time to mess with it and just go back to SVN (which is painful). Of course by that time I've usually gotten distracted and I never finish whatever it was I was working on... Mostly these days I just bang my head against translating the Lore docs into Sphinx (yes, I'm _still_ working on that). Unless someone applied my patch, Combinator won't work on Windows. Also,
Hmmm...interesting.
I had no idea that Twisted was still using such an old version. I was under the impression that the hackery that Combinator used was incompatible with the new merge tracking and that's why it wasn't being used (because it would screw up those still using Combinator...this assumes there are any). If that's not true, perhaps I can take a look, though I likely wouldn't be able to devote any serious time to it until at least Mid-November.
My _major_ impediments are 2 and 6 years old... :) The above is at most a _minor_ impediment. But even minor impediments can drive away new developers, which I assume is not what we want. Presumably you could use Mercurial with the hggit plugin against a git
De nada...it was entirely self-interest-driven though. :)
I haven't noticed any real problems with it since the grand trac revamp a couple years ago, when Twisted's Trac instance became mostly stable. Granted I'm not in there every day like some are... -- Kevin Horn
On Oct 22, 2012, at 4:54 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
You're a trac admin now. Enjoy; don't mess anything up too bad.
Glad to hear this is still moving along, however glacially.
Nope, Combinator will keep working just fine if we upgrade. Something to do with the subversion database itself, I think.... I don't know the details. I really don't want to get sucked into maintaining that part of the infrastructure; anyone who would agrees with me and would like to keep me focused on the parts of Twisted which actually need my particular skills and knowledge (documentation for Deferreds, especially cancellation; Tubes, reactor improvements) would be advised to volunteer :).
But again: if you have trouble with this part, please just drop an email to the list and I will do it for you. Do not let this minor step be an impediment to contributing to Twisted.
My _major_ impediments are 2 and 6 years old... :) The above is at most a _minor_ impediment. But even minor impediments can drive away new developers, which I assume is not what we want.
Actually, we have tons of patches in the queue to review. The main thing right now is not attracting new developers, but convincing developers who are already active in the project to do code review. If every developer with a commit bit did one code review every week, I believe we'd be in great shape right now in terms of both the number of features being added, bugs being fixed, and the experience of the average new contributor. Of course, that would itself attract more contributors until we reached the equilibrium of tickets hanging out for months with no reviews again ;-). But we'd be getting hundreds more tickets into a state where they can make progress. I think we can all agree, though, that the most frustrating part of this whole process is the waiting. If code reviews were all instantaneous, it would be a lot less grating to have to deal with our respective pet peeves in the version control or ticket tracking infrastructure. So please, if you're qualified to do code reviews for Twisted, hop on #twisted-dev and try to do just one a week. It would make a _huge_ difference.
Let's hope it pays off.
Well, personally, I can tell you it's way better than it was a few months ago ;-). -g
On Sat, Oct 20, 2012 at 4:24 PM, Duncan McGreggor <oubiwann@twistedmatrix.com> wrote:
So, I've got a git repo set up for trunk of svn, and for the life of me, I can't merge it with the git repo on github. As such, my desire is to do a force push. *However* this would break github forks that everyone has made so far, based on twisted/twisted (the complete list is here: https://github.com/twisted/twisted/network/members). Thoughts? Permission to push the button? d
On Mon, Oct 22, 2012 at 10:55 AM, Duncan McGreggor < oubiwann@twistedmatrix.com> wrote:
Based on https://github.com/twisted/twisted/network (That graph indicates that no one has actually done any substantial work based on the old repository.) I'm inclined to say do it. I'm a little concerned about how you ended up with a thing that which wasn't mergeable? Could we start by you documenting how you created this repo? -David
On Oct 22, 2012, at 11:30 AM, David Reid <dreid@dreid.org> wrote:
I'm a little concerned about how you ended up with a thing that which wasn't mergeable? Could we start by you documenting how you created this repo?
Also, how did powdahound create theirs? Understanding the distinction so that other git users know how to create their repos the same way, or someone else wants to take over maintenance, would be _highly_ useful. -g
On Mon, Oct 22, 2012 at 11:30 AM, David Reid <dreid@dreid.org> wrote:
Sure thing. All I did to build a git repo was the usual: git svn clone svn://svn.twistedmatrix.com/svn/Twitsed/trunk twisted All of my attempts to merge were using the local copy created by the branch above (or by creating a clone of it with just one or two revisions beyond what the github repo has, to minimize the commit # and maximize the chance of a successful merge). For instance: * git clone git@github.com:twisted/twisted.git twisted-github; cd twisted-github; git pull ../twisted master (and other variations upon that theme). I've tried using several different merge strategies; either they couldn't complete the merge, or they resulted in gobs of conflicts. d
On Mon, Oct 22, 2012 at 12:01 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
So I just tried something else: git pull -s recursive -Xtheirs ../twisted master since this was run in the github master, the "theirs" is the svn clone master, and forcing the adoption of "theirs" resulted in many fewer conflicts. Hundreds of fewer conflicts, maybe? Looking into this to see if anything was broken by this approach... d
On Mon, Oct 22, 2012 at 12:04 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
It seems that the topfiles were mostly affected by this. Some deleted files git didn't know what do to with. But then I noticed that other files that had been modified weren't included in the change set. *sighs* As a sanity check, I did a local rsync, and sure enough, there are a bunch of files that didn't get merged using the strategy above that were changed in the svn repo. I'll put this part of the effort on hold, and take a look at the repo mirror that Glyph pointed me to in his most recent email... d
On Oct 22, 2012, at 12:01 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
Sure thing. All I did to build a git repo was the usual:
git svn clone svn://svn.twistedmatrix.com/svn/Twitsed/trunk twisted
The spelling error in this command makes me think that maybe you didn't copy the command exactly from wherever you did it? ;-) -g
On Oct 22, 2012, at 12:01 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
... the local copy ...
I should also say that I think it's very important to set up a documented, automated process, ideally on the twistedmatrix.com infrastructure somewhere, that will do the pull and update and stuff. There's already a git mirror on wolfwood in /var/www/git/Twisted. I have no idea why it's not being automatically updated. For example, there's a post-commit hook that updates the bzr mirror, so it's always up to date. See wolfwood:/svn/Twisted/hooks/post-commit for the details of how that works. A post-commit hook might not be appropriate for git; so you might want to queue up changes for a cron job. -glyph
On Mon, Oct 22, 2012 at 12:30 PM, Glyph <glyph@twistedmatrix.com> wrote:
Agreed.
ideally on the twistedmatrix.com infrastructure somewhere, that will do the pull and update and stuff.
Excellent. I'll check to see if I have access to these.
There's already a git mirror on wolfwood in /var/www/git/Twisted.
Now *that's* good information to have ;-)
Sweet. This is the sort of thing I need. d
Awesome!!! At what point did this change? ISTR the lack of interest in maintaining the Github account was pretty much a consequence of it seeming pretty pointless: ie it was still easier to just use SVN than to figure out where the parts work together. Will code review also be happening at a pull request level? On Sun, Oct 21, 2012 at 1:24 AM, Duncan McGreggor < oubiwann@twistedmatrix.com> wrote:
-- cheers lvh
On Oct 21, 2012, at 7:02 AM, Laurens Van Houtven <_@lvh.cc> wrote:
Awesome!!!
At what point did this change?
Not sure. I don't know the history of the github repo nor its uses. I was certainly surprised with Glyph said that pull requests would be accepted.
ISTR the lack of interest in maintaining the Github account was pretty much a consequence of it seeming pretty pointless: ie it was still easier to just use SVN than to figure out where the parts work together.
Will code review also be happening at a pull request level?
Here's what I envision... 1) a ticket is created on tm.com/trac for a feature, bug, etc. 2) the developer forks the github repo so they can work in their own space, e.g., creating a branch for the new ticket 3) the ticket is updated with a link to the branch they created on github (e.g., github.com/username/feature-name-1234). 4) when ready for review, the dev rebases against master (which will be syncing with svn every 30 minutes or so), squashing all commits into one and issues a pull request on github 5) the ticket is updated to review status 6) other devs take a look at the diff (git clone twisted repo, git pull dev's branch), with review comments going in the trac ticket 7) dev iterates on feedback, rebasing on master every time 8) once approved, perform final rebase, and ask that your pull request be merged on github 9) someone will then ensure that git master is merged into svn trunk Note that this hasn't been fully tested yet, so one or more parts of this worklfow may have to change... This would allow devs to do 100% of their iteration work in git or hg (hg plays well with github, I hear), basically removing the process itself as a barrier to contributions to Twisted. Currently, my own biggest hurdle to resuming contributions to Twisted is the pain of old tech. I've gotten so used to the luxuries of git, hg, and bzr, that svn feels like walking for two days to pick up something at a store, when I can just drive an hour and be there (and not have to walk back home carrying lumber on my back). Everything else I work on (on a daily basis) uses fast, more efficient code repos and more efficient workflows. If forced to use things that slow me down, I'll just pass and do something that quite honestly is a lot more fun. Hopefully supporting a workflow like this will bring Twisted contributions back into the "fun"realm for more folks... d
On 05:43 pm, duncan.mcgreggor@gmail.com wrote:
This has always been the case. Anyone is free to contribute code in any way they want. (Though, if you contribute code in some way that no one can understand, there may be a delay in the contribution being accepted.) The blocker for people using git for Twisted development has been that no one has wanted to do it badly enough to commit to maintaining the git mirror.
As mentioned above, I have a slightly different perspective on this. :)
At some point *before* svn trunk is updated, Buildbot needs to be able to get the code so it can run tests on all our supported platforms. This is the actually important part of this email, you can ignore the rest if you want but this really needs to happen.
While I'm sympathetic to toolchain woes, I can't help but wonder if you're being really honest here (with yourself, at least). Running "svn diff" may make you feel bad inside, because svn isn't the latest cool toy, but a *hurdle*? It's just difficult to understand.
Hopefully supporting a workflow like this will bring Twisted contributions back into the "fun"realm for more folks...
Yes, hopefully so. Jean-Paul
On Sun, Oct 21, 2012 at 2:48 PM, <exarkun@twistedmatrix.com> wrote:
I'm not sure exactly what you want to hear. The few times I've tried to contributed to Twisted, svn was actually a big barrier. Trying to update my patches so that I'm sure the tests pass on trunk produced mysterious merge conflicts in files I've never touched. Maybe I'm bad at svn, but it's never worked well for me.
-- Jasper
On Oct 21, 2012, at 11:54 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
In case it was actually not clear and you're not just being sarcastic, exarkun was saying that it's difficult to understand that someone with the required expertise to contribute to Twisted in the first place would have trouble with running the command 'svn diff'. 'svn diff' itself is not at all difficult to understand. Since Subversion is effectively the baseline fisher-price "my first version control system", I would assume that anyone who could effectively use Git (which has all the user-interface convenience of an unshielded circular saw) would have no trouble with it, especially with the very basic usage that contributing to Twisted requires. Since you asked, there are two things that I'd like to hear: "Hooray! I will help oubiwann maintain the git(hub) mirror! what would you like me to do?" specific problems that you've had with SVN that we might be able to address in the future or help you with, and not vague bellyaching about how we're not using the thing you happen to like best. If you're going to say the first thing and commit to helping though, please be sure first that you actually have the time and energy to follow through. At this point, the number of people who have appeared, volunteered to maintain a git mirror, done it for five minutes and then disappeared forever, leaving it in a broken, unmaintained state, is in the double digits. (I am starting to wonder if Git gives it users some kind of brain damage that makes a person incapable of meeting commitments.)
The few times I've tried to contributed to Twisted, svn was actually a big barrier. Trying to update my patches so that I'm sure the tests pass on trunk produced mysterious merge conflicts in files I've never touched. Maybe I'm bad at svn, but it's never worked well for me.
Why aren't you just using git for local development then? You don't have commit access, so you should never need to touch an svn client other than git if you don't feel like it. This is not entirely a rhetorical question. We have always tried to be accommodating to DVCS users, providing instructions and repeated requests for both a plain git and/or a github ambassador to keep svn nicely synchronized and reduce the friction required for users of those tools to make contributions. If the documentation we've offered on <http://twistedmatrix.com/trac/wiki/GitMirror> is in any way incorrect or non-optimal, please don't hesitate to say exactly what would be better. If you need wiki edit permission to update the page, I'll gladly give it to you. -glyph
On Mon, Oct 22, 2012 at 12:05 AM, Glyph <glyph@twistedmatrix.com> wrote:
As I said, the issue I had was not "svn diff" -- I had been working on my fix, uploaded it to Trac, and it was a few months or so before someone reviewed it. I forget who it was, but the reviewer prompted me to make a few small style changes, flesh out a testcase, write a .news file, that sort of thing. I updated my source tree to pull in new changes from trunk, to make sure the patch that I had been working on didn't rot. I was frustrated when I got merge conflicts for files that I've never touched before.
I'd love to help out and do things, but unfortunately I don't have the time these days. If you think that we should review code on the Twisted GitHub repository, I'd gladly help out with that. I would help out and review patches on Trac, but it's not as convenient: as far as I'm aware: I can't set Trac up so that I get emails when a patch comes in in a module I'm comfortable with, and I can't get an RSS feed of all the important events in the project I'm aware of. If that exists and I couldn't find it, I'll gladly review patches today.
This is another rhetorical anecdote, I tried using git-svn when I was contributing to PyPy during the SVN days. It just didn't work out. I don't know or can't say the same for bzr-svn or hg-svn or whatever else cross-VCS systems there are, but I was left with a bad taste in my mouth, so I figured it would be more worthwhile to stick with the original source tool. I may just be a drooling moron.
-- Jasper
On Oct 21, 2012, at 9:55 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
If you can replicate the exact branch and changes you had difficulty with, I'd love to hear about the specifics, because this sounds like maybe it's a gotcha we should add to some development documentation. It should not be possible for this to happen with a simple 'svn up' so I want to know what command to tell users they should avoid. Feel free to reply off-list if it gets into fiddly details of subversion that might be off-topic.
You say "rhetorical anecdote", I say "FUD" :). But, perhaps you are more just in the class of users that need somebody like Duncan's help in order to use git against Twisted effectively, rather than the class of users who know the version control tools well enough to be able to provide that help.
I may just be a drooling moron.
I didn't say you were a "moron", being unable to figure out git's UI for something is merely an indication that you probably aren't eligible for the Fields medal; frankly I have a great deal of difficulty with it as well. If I didn't, I might have updated things myself. I hope you'll have a better time once those instructions are updated. -glyph
On Oct 21, 2012, at 11:05 PM, Glyph wrote:
I think there are a few issues with the documentation, from the standpoint of an infrequent contributor: • It appears that you have to go "all in" with svn if you want to contribute, when that isn't actually the case • <http://twistedmatrix.com/trac/wiki/GitMirror> is not terribly easy to find. • It does come up in search, but is otherwise not easily accessible when you're spelunking for info on how to contribute • The page with the links has it as a parenthetical aside <http://twistedmatrix.com/trac/wiki/TwistedDevelopment> • Casual contributors should have a different story than core committers, but the distinction isn't always prominent • <http://twistedmatrix.com/trac/wiki/GitMirror> makes this distinction, which is good • <http://twistedmatrix.com/trac/wiki/BazaarMirror> does not. For what it's worth, I've been using Mercurial patch queues to manage a couple of infrequent, small contributions. It works pretty well and doesn't rely on svn history or an svn bridge. That's not terribly interesting, but the point is that there are probably other workflows that are effective and it would be useful if they could be documented and shared easily. I created a ticket to address some of these points and carry on the docs discussion elsewhere. If it seems like a good idea I'll work on updating the wiki. http://twistedmatrix.com/trac/ticket/6154 Lucas Taylor
On Sun, Oct 21, 2012 at 11:30 AM, <exarkun@twistedmatrix.com> wrote:
I want to use git instead of svn so badly that I can taste it. Hopefully this provides the motivation necessary to maintain the repo :-)
Excellent feedback -- thanks! This is the sort of thing I need in order to plan this out sufficiently...
Well, I viewed git and github as the hipster's latest cool toy for a long time, but once I was force to use it, I was converted. The time I spent wrestling with tool chains and workflows all but disappeared. I never even thought about it anymore. That's the ideal user interface with any tool, as far as I'm concerned. When you can use it to be efficient and don't even think about the tool, where it's just an extension of your natural thought process, that's a tool that has been built correctly. Sadly, that's the opposite experience I have with svn-based branch management :-/ d
On Oct 21, 2012, at 11:57 AM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
For what it's worth, I haven't authored a branch for Twisted with SVN in the last 3 years. I use Bazaar almost exclusively and I get all the fun tools one generally associates with a DVCS, including offline history and a nice graphical revision viewer. (One reason I am concerned about migrating to Git is that this will stop working; in my opinions, Git's offline GUI tools are far worse than Bazaar's - almost as much worse as Launchpad's source browser is than Github.) Similarly, I do code reviews by using a local 'bzr merge'. The only weird workflow which isn't just built in to BZR-SVN is this: <http://twistedmatrix.com/trac/wiki/BazaarMirror#CommittingaBazaarbranchtoaSu...>. This involves rebasing, which makes me a little sad; I wish we could preserve more history, but it works fine. I still land branches on trunk using SVN, but that takes about ten seconds assuming you use Combinator or svnmerge.py, so I don't see that being a big impediment. You only need to worry about that if you have commit access anyway, so that's not most external contributors. If the need to do that to land changes is preventing you from writing them in the first place, just let me know when you have a branch to land, and I'll do it for you. You don't need to use SVN, you haven't needed to use SVN since git-svn came into existence, and we've had several threads on this list before about improving the Git instructions here: <http://twistedmatrix.com/trac/wiki/GitMirror>. If you look at the history for that page, you can see that it is two years old, and originally created by exarkun, who is not a git user himself. So, it strikes me as a bit odd, and I am a bit dismayed that anyone would have been put off of by the need to use Subversion when working with Twisted since you can use Git or Bazaar - or, I assume, Mercurial - right now. I would find it a bit more understandable if you are dismayed by the need to use Trac, which is harder to avoid when working on Twisted. For various reasons - which I hope we don't have to discuss right now - we aren't going to move the project to Github any time soon, so if anyone reading this thread is interested in alleviating some of the pain with Trac, <https://launchpad.net/frack> is a good place to get involved to help out. There are some pretty straightforward bugs that anyone interested could fix there. Frack is already up and running on twistedmatrix.com, albeit at a weird URL. See, for example, <http://twistedmatrix.com/users/frack.twistd/ui/ticket.html?id=1956>. This might be useful because it is a considerably faster ticket-viewer than Trac, do to its somewhat more parsimonious issuance of SQL queries. -glyph
On Sun, Oct 21, 2012 at 11:05 PM, Glyph <glyph@twistedmatrix.com> wrote:
I really wish I could figure out how to do this. The one page on the wiki helps a bit, but is insufficient for those not already familiar with bzr (like me). I've tried setting this up a couple of times, and I always end up just giving up on it. (and no I don't recall exactly why, except for the one time when installing bzr on windows hosed my machine and made it unbootable...not Twisted's fault of course).
Unless someone applied my patch, Combinator won't work on Windows. Also, since the great Divmod site meltdown, there's not good instructions on how to use it (aside from the wayback machine, which is what I've been using). I'd never heard of svnmerge.py until you mentioned it. Are there any instructions on how to use it with Twisted? I'm not finding any, and I'm reluctant to experiment since Twisted has a very specific way on interacting with the SVN repo.
Presumably you could use Mercurial with the hggit plugin against a git repo, but you can't use it directly against the SVN repo, as it has a bad commit in it that hgsubversion totally chokes on: https://bitbucket.org/durin42/hgsubversion/issue/350
Trac has never really bothered me that much. Yes, occasionally it will do something stupid, and it's...not fast, but I guess I'm just used to the pain. I'm looking forward to Frack, though. Kevin Horn
On Oct 22, 2012, at 10:53 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
Let me keep it super short and sweet for you: C:\PROJEC~1> bzr branch http://svn.twistedmatrix.com/bzr/Twisted/trunk Twisted C:\PROJEC~1> cd Twisted as many times as desired, C:\PROJEC~1\TWISTED> hack hack hack C:\PROJEC~1\TWISTED> C:\python27\python.exe bin\trial twisted C:\PROJEC~1\TWISTED> bzr commit Review time! C:\PROJEC~1\TWISTED> bzr send -o - > my.patch Stick my.patch on a ticket and you're good to go. No need for combinator, or virtualenv, or subversion (not even bzr-svn, we did that part for you), or even a C compiler, assuming the feature you're working on doesn't need one. If any of this isn't covered in the wiki page then please update it. And please feel free to ask more questions - if specific version control or workflow issues are preventing anyone from working on Twisted then it is definitely on-topic for this list and I will be happy to provide answers and update development documentation.
(and no I don't recall exactly why, except for the one time when installing bzr on windows hosed my machine and made it unbootable...not Twisted's fault of course).
Can we just agree that you had a virus and that this had nothing to do with anything in this thread? Even if that's not true I think it might be long-term more mentally healthy for you to believe :-)
I guess I should test this out myself at least once, but my understanding is that you just do svnmerge.py --bidirectional -S the-branch-you-want-to-merge in trunk. You can test this out for yourself though, since svnmerge.py is loudly advertised as "commit-free": you always have to do the commit yourself. More info here: <http://www.orcaware.com/svn/wiki/Svnmerge.py>. If we can ever upgrade to a version of SVN on the server that supports merge-tracking, the need for either of these tools will go away since a plain 'svn merge ../branches/branchname' will do the right thing. Any volunteer sysadmins want to take the opportunity to crawl out of the woodwork? (Please?) But again: if you have trouble with this part, please just drop an email to the list and I will do it for you. Do not let this minor step be an impediment to contributing to Twisted.
Sadness. Hopefully this renewed attention will get the mercurial folks to fix this bug sooner? Thanks for filing it.
Great, glad to hear it. By the way, if you've been shying away because it's slow, you should know that we did recently fix a major performance problem with the website. Bots were running amok on some very CPU and memory intensive requests, and robots.txt was not deployed correctly so it wasn't reducing any of the traffic as it should have. At the most recent sprint nobody complained about being unable to work because of the website falling over, which indicates that some of our more dire problems are now under control. -glyph
On Mon, Oct 22, 2012 at 2:22 PM, Glyph <glyph@twistedmatrix.com> wrote:
See that should totally be on a wiki page someplace. I'd be happy to put it up there, but I don't have edit permissions on that page (or any of them?) apparently...
Actually I know exactly why this happened, and it's because Windows has a hard limit on the number of "icon overlays" you can add to Explorer. I installed TortoiseBzr along with bzr, and went over that limit. This by itself is not such a big deal, but it turns out that the failure behavior of this particular issue is NOT BOOTING ANY MORE. (stupid Windows) (FYI, I've since installed bzr again, and have had no similar problems.) (also FYI, I haven't had a virus/spyware on a windows machine that I owned in more than 10 years...<knock on wood>...the trick is renaming the iexplore.exe executable...) But aside from that, the couple of times I've tried messing around with bzr I end up going in circles in their documentation, can't decide what I should be doing, decide I don't have time to mess with it and just go back to SVN (which is painful). Of course by that time I've usually gotten distracted and I never finish whatever it was I was working on... Mostly these days I just bang my head against translating the Lore docs into Sphinx (yes, I'm _still_ working on that). Unless someone applied my patch, Combinator won't work on Windows. Also,
Hmmm...interesting.
I had no idea that Twisted was still using such an old version. I was under the impression that the hackery that Combinator used was incompatible with the new merge tracking and that's why it wasn't being used (because it would screw up those still using Combinator...this assumes there are any). If that's not true, perhaps I can take a look, though I likely wouldn't be able to devote any serious time to it until at least Mid-November.
My _major_ impediments are 2 and 6 years old... :) The above is at most a _minor_ impediment. But even minor impediments can drive away new developers, which I assume is not what we want. Presumably you could use Mercurial with the hggit plugin against a git
De nada...it was entirely self-interest-driven though. :)
I haven't noticed any real problems with it since the grand trac revamp a couple years ago, when Twisted's Trac instance became mostly stable. Granted I'm not in there every day like some are... -- Kevin Horn
On Oct 22, 2012, at 4:54 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
You're a trac admin now. Enjoy; don't mess anything up too bad.
Glad to hear this is still moving along, however glacially.
Nope, Combinator will keep working just fine if we upgrade. Something to do with the subversion database itself, I think.... I don't know the details. I really don't want to get sucked into maintaining that part of the infrastructure; anyone who would agrees with me and would like to keep me focused on the parts of Twisted which actually need my particular skills and knowledge (documentation for Deferreds, especially cancellation; Tubes, reactor improvements) would be advised to volunteer :).
But again: if you have trouble with this part, please just drop an email to the list and I will do it for you. Do not let this minor step be an impediment to contributing to Twisted.
My _major_ impediments are 2 and 6 years old... :) The above is at most a _minor_ impediment. But even minor impediments can drive away new developers, which I assume is not what we want.
Actually, we have tons of patches in the queue to review. The main thing right now is not attracting new developers, but convincing developers who are already active in the project to do code review. If every developer with a commit bit did one code review every week, I believe we'd be in great shape right now in terms of both the number of features being added, bugs being fixed, and the experience of the average new contributor. Of course, that would itself attract more contributors until we reached the equilibrium of tickets hanging out for months with no reviews again ;-). But we'd be getting hundreds more tickets into a state where they can make progress. I think we can all agree, though, that the most frustrating part of this whole process is the waiting. If code reviews were all instantaneous, it would be a lot less grating to have to deal with our respective pet peeves in the version control or ticket tracking infrastructure. So please, if you're qualified to do code reviews for Twisted, hop on #twisted-dev and try to do just one a week. It would make a _huge_ difference.
Let's hope it pays off.
Well, personally, I can tell you it's way better than it was a few months ago ;-). -g
On Sat, Oct 20, 2012 at 4:24 PM, Duncan McGreggor <oubiwann@twistedmatrix.com> wrote:
So, I've got a git repo set up for trunk of svn, and for the life of me, I can't merge it with the git repo on github. As such, my desire is to do a force push. *However* this would break github forks that everyone has made so far, based on twisted/twisted (the complete list is here: https://github.com/twisted/twisted/network/members). Thoughts? Permission to push the button? d
On Mon, Oct 22, 2012 at 10:55 AM, Duncan McGreggor < oubiwann@twistedmatrix.com> wrote:
Based on https://github.com/twisted/twisted/network (That graph indicates that no one has actually done any substantial work based on the old repository.) I'm inclined to say do it. I'm a little concerned about how you ended up with a thing that which wasn't mergeable? Could we start by you documenting how you created this repo? -David
On Oct 22, 2012, at 11:30 AM, David Reid <dreid@dreid.org> wrote:
I'm a little concerned about how you ended up with a thing that which wasn't mergeable? Could we start by you documenting how you created this repo?
Also, how did powdahound create theirs? Understanding the distinction so that other git users know how to create their repos the same way, or someone else wants to take over maintenance, would be _highly_ useful. -g
On Mon, Oct 22, 2012 at 11:30 AM, David Reid <dreid@dreid.org> wrote:
Sure thing. All I did to build a git repo was the usual: git svn clone svn://svn.twistedmatrix.com/svn/Twitsed/trunk twisted All of my attempts to merge were using the local copy created by the branch above (or by creating a clone of it with just one or two revisions beyond what the github repo has, to minimize the commit # and maximize the chance of a successful merge). For instance: * git clone git@github.com:twisted/twisted.git twisted-github; cd twisted-github; git pull ../twisted master (and other variations upon that theme). I've tried using several different merge strategies; either they couldn't complete the merge, or they resulted in gobs of conflicts. d
On Mon, Oct 22, 2012 at 12:01 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
So I just tried something else: git pull -s recursive -Xtheirs ../twisted master since this was run in the github master, the "theirs" is the svn clone master, and forcing the adoption of "theirs" resulted in many fewer conflicts. Hundreds of fewer conflicts, maybe? Looking into this to see if anything was broken by this approach... d
On Mon, Oct 22, 2012 at 12:04 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
It seems that the topfiles were mostly affected by this. Some deleted files git didn't know what do to with. But then I noticed that other files that had been modified weren't included in the change set. *sighs* As a sanity check, I did a local rsync, and sure enough, there are a bunch of files that didn't get merged using the strategy above that were changed in the svn repo. I'll put this part of the effort on hold, and take a look at the repo mirror that Glyph pointed me to in his most recent email... d
On Oct 22, 2012, at 12:01 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
Sure thing. All I did to build a git repo was the usual:
git svn clone svn://svn.twistedmatrix.com/svn/Twitsed/trunk twisted
The spelling error in this command makes me think that maybe you didn't copy the command exactly from wherever you did it? ;-) -g
On Oct 22, 2012, at 12:01 PM, Duncan McGreggor <duncan.mcgreggor@gmail.com> wrote:
... the local copy ...
I should also say that I think it's very important to set up a documented, automated process, ideally on the twistedmatrix.com infrastructure somewhere, that will do the pull and update and stuff. There's already a git mirror on wolfwood in /var/www/git/Twisted. I have no idea why it's not being automatically updated. For example, there's a post-commit hook that updates the bzr mirror, so it's always up to date. See wolfwood:/svn/Twisted/hooks/post-commit for the details of how that works. A post-commit hook might not be appropriate for git; so you might want to queue up changes for a cron job. -glyph
On Mon, Oct 22, 2012 at 12:30 PM, Glyph <glyph@twistedmatrix.com> wrote:
Agreed.
ideally on the twistedmatrix.com infrastructure somewhere, that will do the pull and update and stuff.
Excellent. I'll check to see if I have access to these.
There's already a git mirror on wolfwood in /var/www/git/Twisted.
Now *that's* good information to have ;-)
Sweet. This is the sort of thing I need. d
participants (10)
-
David Reid -
Duncan M. McGreggor -
Duncan McGreggor
-
Duncan McGreggor -
exarkun＠twistedmatrix.com -
Glyph -
Jasper St. Pierre -
Kevin Horn -
Laurens Van Houtven -
Lucas Taylor