I'm just starting to evaluate Twisted, so apologies if these questions seem naive. It seems to me that the Twisted model for 'reactor' is actually not what you'd expect from the name (at least, if you have a background in other reactor implementations including ACE, basic select etc and Java NIO) since it a) handles queueing outgoing data for you and b) presents incoming data after completion of the IO So its a model that maps well to an AIO implementation so long as inputs from non-blocking sources complete as soon as "some" data is ready, rather than buffer full. This high level approachh should work well on Windows - which is really pleasing. (Albeit there may be buffer starvation issues on all platforms I guess, but that's probably scaled beyond Python ambitions) A quick look at the open bug list shows a lot of noise on the Win32 side though. Is there any reason to have any framework that calls the Win32 select function? And why is the event reactor broken at 64 handles? From what I can see, that's not how to use WaitForMultipleEvents - rather one should regard it as a way to hash completion events from standard overlapped IO requests against 61 of the handles and keep a list of the IOs in progress against each handle. Scales admirably, unlike any attempt to treat the handle array like a select fd_set. Shouldn't broken Win32 implementations be 'retired'? Also - is there any way to determine how much data is queued for write against an ITransport? It appears not, though I would have thought it useful (and, a way to get an event callback when the queued amount falls below a specified threshold) - then in cases where data can be generated on demand we can limit the outgoing buffer usage. James
On Fri, 2008-03-28 at 06:23 +0000, James Mansion wrote:
Shouldn't broken Win32 implementations be 'retired'?
The old I/O Completion Ports reactor (which should be most scalable Win32 reactor) was somewhat buggy. Twisted 8.0, released this week, has a complete rewrite that hopefully is a lot better.
Also - is there any way to determine how much data is queued for write against an ITransport?
You might want to look at the producer/consumer APIs (there's a howto), they let you know at least when the buffer becomes full or becomes empty. The APIs could be improved to provide more info or configurable thresholds, but that hasn't been done yet.
On 01:39 pm, itamar@itamarst.org wrote:
On Fri, 2008-03-28 at 06:23 +0000, James Mansion wrote:
Shouldn't broken Win32 implementations be 'retired'?
These implementations aren't "broken", so much as "limited". Please feel free to contribute patches which reduce these limitations.
The old I/O Completion Ports reactor (which should be most scalable Win32 reactor) was somewhat buggy. Twisted 8.0, released this week, has a complete rewrite that hopefully is a lot better.
It still doesn't support SSL, though: http://twistedmatrix.com/trac/ticket/593 and maybe some other features too (I don't have a Windows handy or the time required to read the code to check - this really ought to be better-documented). The select reactor is also convenient for testing on Windows, because it's the basis of at least one GUI reactor (GTK+) which should also support Win32. Your suggestion for handle buckets in the win32eventreactor is one which has been suggested before and sounds good to me. Nobody has bothered to implement it yet :). However, even in its absence, I don't believe we'll be completely dropping win32eventreactor because it is useful for integrating with programs that do win32 drawing, even if iocpreactor has higher performance and is more scalable.
glyph@divmod.com wrote:
On 01:39 pm, itamar@itamarst.org wrote:
On Fri, 2008-03-28 at 06:23 +0000, James Mansion wrote:
Shouldn't broken Win32 implementations be 'retired'?
These implementations aren't "broken", so much as "limited".
Well, if they don't honour what the documentation says, then they seem broken to me, and the documentation doesn't say you might get an unexpected 64 connection limit. No matter.
Please feel free to contribute patches which reduce these limitations.
The old I/O Completion Ports reactor (which should be most scalable Win32 reactor) was somewhat buggy. Twisted 8.0, released this week, has a complete rewrite that hopefully is a lot better.
It still doesn't support SSL, though:
Need something patterned very much like Len's stuff here http://www.lenholgate.com/archives/000456.html I guess.
and maybe some other features too (I don't have a Windows handy or the time required to read the code to check - this really ought to be better-documented).
The select reactor is also convenient for testing on Windows, because it's the basis of at least one GUI reactor (GTK+) which should also support Win32.
Your suggestion for handle buckets in the win32eventreactor is one which has been suggested before and sounds good to me. Nobody has bothered to implement it yet :). However, even in its absence, I don't believe we'll be completely dropping win32eventreactor because it is useful for integrating with programs that do win32 drawing, even if iocpreactor has higher performance and is more scalable. Well, you need a daemon worker thread to call GetQueuedCompletionStatus(Ex) and enqueue the results to the main thread, but that daemon worker doesn't even need to have the full support of the C library multithreading so long as it just swizzles pointers and calls raw API routines. Its normally more of a gotcha that you might try to submit IO from a pool thread that might reasonably be terminated before the IO completes. Why do you need an event reactor to support win32 applications, except to call MsgWait... rather than GetQueuedCompletionStatus?
On a similar sort of topic - is there a reason to have lots of implementations for POSIX, rather than use libevent or libev? James
On 31 Mar, 10:15 pm, james@mansionfamily.plus.com wrote:
glyph@divmod.com wrote:
These implementations aren't "broken", so much as "limited". Well, if they don't honour what the documentation says, then they seem broken to me, and the documentation doesn't say you might get an unexpected 64 connection limit. No matter.
This is actually a problem for every reactor; handling of the connection limit (and there is a connection limit everywhere, even if it's 65535) is poorly documented and poorly handled. Patches which rectify this situation for any reactor, either from the perspective of docs or code, would of course be appreciated.
It still doesn't support SSL, though:
http://twistedmatrix.com/trac/ticket/593 Need something patterned very much like Len's stuff here http://www.lenholgate.com/archives/000456.html I guess.
The real solution to this problem is really: * Add an SSH IProtocol implementation that provides ITransport, rather than building it into the reactor * Add an API to ITransport to switch protocols, so that the protocol doesn't need to support protocol switching The BIO stuff mentioned on that URL is vaguely correct, but the interesting stuff is exposing it to Python. Now that JP Calderone has taken over maintenance of PyOpenSSL, we might actually have a chance to get that done :). Once you've got an SSL connection that is getting bytes from and delivering bytes to a transport, you don't need to care about any of the win32-specific aspects of that post; you just do the I/O using whatever platform I/O facility you want and the crypto happens completely disconnected from this socket. iocpreactor already implements enough of this to work. Again, if you want to see this happen: write code which does these things and contribute it.
Your suggestion for handle buckets in the win32eventreactor is one which has been suggested before and sounds good to me. Nobody has bothered to implement it yet :). However, even in its absence, I don't believe we'll be completely dropping win32eventreactor because it is useful for integrating with programs that do win32 drawing, even if iocpreactor has higher performance and is more scalable. Well, you need a daemon worker thread to call GetQueuedCompletionStatus(Ex)
(... blah blah win32 API stuff that I don't understand ...)
Why do you need an event reactor to support win32 applications, except to call MsgWait... rather than GetQueuedCompletionStatus?
My answer here, if I even have one, would be horribly vague: something about GDI+, or DirectX, or something. Maybe there's no good reason, I don't know. This level of specificity is probably best served by the issue tracker; file a ticket. And, if you're going to file a ticket, be prepared to actually follow up with an implementation. While we want to maintain support for Windows, the level of energy for doing really interesting Windows-specific stuff in Twisted is, in a word, "low". One thing you might want to know before you file that ticket is that *if* there's a reason for the way things are now, it's because in the Twisted idiom we always make sure GUI stuff and network stuff is happening in the same thread. If one of the approaches that you mentioned requires a dedicated "network" application thread, then that's probably why we aren't doing it.
On a similar sort of topic - is there a reason to have lots of implementations for POSIX, rather than use libevent or libev?
There is more than one answer to this question. Maybe someone will be helpful and turn some of these answers into a FAQ on the wiki: 1) Twisted predates libevent by a few years and libev by many years. One might instead ask why libevent didn't help us develop a C reactor, rather than writing a whole new library. 2) There's a libevent reactor in a branch. Contribute patches to deal with the outstanding review comments: http://twistedmatrix.com/trac/ticket/1930 3) Twisted runs on many platforms, so it supports the lowest common denominator; that is not libevent, it is Python's "select" module. 4) Neither libevent nor libev (to my knowledge) integrates with any GUI event loops. 5) Despite many valid rationalizations for its existence, the code in Twisted was developed organically over many years. The stuff you'll find here is the stuff that people thought was interesting and had time to work on. Strategically standardizing on a single low-level multiplexing mechanism is not something that is particularly fun or rewarding, especially when getting rid of the old code removes value for some users. (Not everyone already has libevent installed.)
Patches which rectify this situation for any reactor, either from the perspective of docs or code, would of course be appreciated. There's no point giving a commitment to doing more than discussing implementation approaches. With the best will in the world its unlikely to get to the top of my pile and
And, if you're going to file a ticket, be prepared to actually follow up with an implementation. Hmm - that's a crap attitude unless you want to deter any concensus
While we want to maintain support for Windows, the level of energy for doing really interesting Windows-specific stuff in Twisted is, in a word, "low".
One thing you might want to know before you file that ticket is that *if* there's a reason for the way things are now, it's because in the Twisted idiom we always make sure GUI stuff and network stuff is happening in the same thread. If one of the approaches that you mentioned requires a dedicated "network" application thread, then that's probably why we aren't doing it. I appreciate that a reactive system will generally want to run its completion callbacks in the main
On a similar sort of topic - is there a reason to have lots of implementations for POSIX, rather than use libevent or libev?
There is more than one answer to this question. Maybe someone will be helpful and turn some of these answers into a FAQ on the wiki:
1) Twisted predates libevent by a few years and libev by many years. One might instead ask why libevent didn't help us develop a C reactor, rather than writing a whole new library. I've no idea, but if you have a separation of a support library with an abstraction that one might expose with SWIG or similar and reuse directly from C(++) - then this is well not actually apparent on the web site. I think it would be a good idea though,and its more
glyph@divmod.com wrote: there's no point living a fantasy. I do need to understand how limited the current implementation is though. formation during design. I know its quite common in open source. :-( thread, but that shouldn't preclude restarting incomplete writes or doing low-level protocol stuff in a thread. likely that I'd be able to contribute something of this sort (eventually) than anything Python-specific. You have a nice model and one that is much more portable than the common reactor idiom.
5) Despite many valid rationalizations for its existence, the code in Twisted was developed organically over many years. The stuff you'll find here is the stuff that people thought was interesting and had time to work on. Strategically standardizing on a single low-level multiplexing mechanism is not something that is particularly fun or rewarding, especially when getting rid of the old code removes value for some users. (Not everyone already has libevent installed.) The curse of volunteer-ware, I guess. Understandable, but a shame in some ways.
On Mon, Mar 31, 2008 at 10:37 PM, James Mansion
glyph@divmod.com wrote: Patches which rectify this situation for any reactor, either from the perspective of docs or code, would of course be appreciated. There's no point giving a commitment to doing more than discussing implementation approaches. With the best will in the world its unlikely to get to the top of my pile and there's no point living a fantasy. [...] And, if you're going to file a ticket, be prepared to actually follow up with an implementation. Hmm - that's a crap attitude unless you want to deter any concensus formation during design. I know its quite common in open source. :-(
Twisted developers' time is as limited as yours is. They're not living in a fantasy either. Resolving a bug includes gathering requirements and building consensus, but building consensus goes much faster if there's an implementation handy to discuss. Even a quick hack is useful as a discussion point. A very common scenario is that a quick hack is eventually refined into a unit tested, UQDS-vetted implementation. However, a hand-waving discussion never is. I certainly understand your frustrations (I've been there myself, many times over, with pretty much every piece of software I've ever developed with). I agree heartily with the point that there is a bug if the software doesn't behave according to documentation, and there is even an argument to be made that the documentation should note known limitations--particularly when, as in this case, they are not secret black knowledge but common knowledge, and doubly when this non-secret non-black knowledge is a stumbling block for so many newbies. Still, things get fixed when someone fixes them. It falls on the person who needs them fixed to do so, no matter whether you're talking about software or rain gutters.
Cory Dodt wrote:
Resolving a bug includes gathering requirements and building consensus, but building consensus goes much faster if there's an implementation handy to discuss. Even a quick hack is useful as a discussion point. A very common scenario is that a quick hack is eventually refined into a unit tested, UQDS-vetted implementation. However, a hand-waving discussion never is.
Call me old-fashioned, but what you are describing is the difference between design-free-hacking followed by iteration, and actually designing something. You know, all that waterfall stuff. I know, its not fashionable right now. Seriously, though, its too late at the review-of-nearly-working-code. There's too much pressure to incrementally fix it, and at least one participant will have a sense of ownership in something that might be a country mile from the best solution. I know its easier to design when you can meet each other and use a white board or just scribble on paper, but its still entirely possible to use words. So - I disagree. I'm quite happy to hand-wave, and to listen to my colleagues' hand-waving. If it communicates design ideas - and requirements - before any wasteful coding, that's good. Honest.
Still, things get fixed when someone fixes them. It falls on the person who needs them fixed to do so, no matter whether you're talking about software or rain gutters.
Hmm. You sure it doesn't happen after the prioritisation meeting and we all get our steer? James
On Apr 2, 2008, at 6:24 PM, James Mansion wrote:
Cory Dodt wrote:
Resolving a bug includes gathering requirements and building consensus, but building consensus goes much faster if there's an implementation handy to discuss. Even a quick hack is useful as a discussion point. A very common scenario is that a quick hack is eventually refined into a unit tested, UQDS-vetted implementation. However, a hand-waving discussion never is.
Call me old-fashioned, but what you are describing is the difference between design-free-hacking followed by iteration, and actually designing something. You know, all that waterfall stuff. I know, its not fashionable right now.
Seriously, though, its too late at the review-of-nearly-working- code. There's too much pressure to incrementally fix it, and at least one participant will have a sense of ownership in something that might be a country mile from the best solution.
I know its easier to design when you can meet each other and use a white board or just scribble on paper, but its still entirely possible to use words.
So - I disagree. I'm quite happy to hand-wave, and to listen to my colleagues' hand-waving. If it communicates design ideas - and requirements - before any wasteful coding, that's good. Honest.
Still, things get fixed when someone fixes them. It falls on the person who needs them fixed to do so, no matter whether you're talking about software or rain gutters.
Hmm. You sure it doesn't happen after the prioritisation meeting and we all get our steer?
Perhaps I'm misinterpreting your meaning, but I think your attitude is way out of line. Free software is possibly the only scenario where you truly get more than you pay for, but that doesn't mean it comes without a cost. One cost is that the project and its community dictate their own development process, and one person has little or no ability to change that inertia. It is the right, possibly even the purpose of the community to grow however it will. I get just as frustrated as the next guy when I need something fixed RFN, but that's the price you pay in exchange for a huge library of high-quality code. For every time I've run into an issue with Twisted that the core devs were too busy to fix, there's 50 other times when Twisted has saved my ass. Wasteful coding? Who do you think you're talking to here? -phil
Phil Christensen wrote:
Wasteful coding? Who do you think you're talking to here? I am suggesting that the '90% coding' suggests to me that design discussion starts when most of a concrete implementation is available. It wasn't your term but you have taken offence at what I said.
Well, I disagree. I find it strange that there should be such a rejection of the idea that one can achieve some degree of concensus on a design approach before coding starts. It seems to me very wasteful if someone should expend time getting an implementation 'mostly there' if in fact there is a devil in the detail that will come up and bite it and invalidate the whole thing - and the only way to do that is to be prepared to discuss abstract design approaches *before* coding. Even if there is no resource available to code it, it can leave a documented concencus that can be used to bootstrap implementation when resource becomes available. There are some aspects of system design relating to distributed and asynchronous systems that are *not* amenable to iterative refinement and refactoring - you can't generally test and refactor your way out of a design that is fundamentally deadlock prone, for example. It may be that the way that people like to 'collaborate' in open source does not generally extend to design review, but that's wasteful and ultimately foolish for exactly the same reasons it is in everyday work, and it suggests a particularly limited view of what 'collaboration' and 'teamwork' means in the context of software development. You can't force wouldbe contributors to do differently, and ultimately you may get results that work, but its very inefficient and there have been some high-profile attempted contributions to (say) the Linux kernel that have been bounced after huge amounts of work, when it would have been apparent that the design approach would never be accepted if it had been reviewed up front. In some respects the BSD projects' approaches (or at least aspirations) make them more attractive. Sometimes, anyway. It would just be nice to find a project where design is welcomed rather than some macho 'code is king' bullshit. Would you be happy to work on a team at work that spends so little time on design? Why should working on a distributed volunteer team be materially different? The question really is whether you want to encourage collaboration of design and design review, or not?
Can we please drop this line of discussion? It's not going to lead anywhere productive. If you want to continue to discuss how Twisted can work better on Windows, *please* do. No one is rejecting that discussion. However, discussions about how free software is a failure or whatever serve no useful purpose. Thank you, Jean-Paul
On Sun, 2008-04-06 at 21:53 +0100, James Mansion wrote:
Well, I disagree. I find it strange that there should be such a rejection of the idea that one can achieve some degree of concensus on a design approach before coding starts.
It depends how complex the problem being solved is. Many of our features start as code-free tickets in the issue tracker. So please do open tickets with suggestions/bug reports even if you don't have code: http://twistedmatrix.com/
On 6 Apr, 08:53 pm, james@mansionfamily.plus.com wrote:
It may be that the way that people like to 'collaborate' in open source does not generally extend to design review, but that's wasteful and ultimately foolish
If you really insist on continuing this debate (I recommend not; I don't see a way it could come to a useful conclusion), I would really appreciate some links to software engineering research which conclusively verifies this claim. As it is, all I've got here is the fact that your experience and knowledge of the field does not overlap mine at all. Design review is, in fact, a waste of time _if you are including people who have not been involved in and thoroughly understand the implementation_. Perhaps putting this in more familiar terms will help you understand why your wisdom is not being met with respect. There is definitely a culture clash here. In most companies, this is reflected in a hierarchy; junior developers are not given senior architect positions and are not included in high- level design reviews. People may be hired directly at senior-level positions, of course, but they are _invited_ to do so; they don't just start showing up one day. And junior developers do sometimes provide design input, but potential hires never do. In Twisted-land, people like myself, JP Calderone, Thomas Herve, and Chris Armstrong can engage in this sort of high-level discussion and review. We frequently do, often in the context of comments on tickets, sometimes at in-person meetings (PyCon was great for this kind of thing). However, we do this because we have "high ranking" positions on the project. We earned these ranks by contributing code. To be blunt, you haven't. Not only that: you specifically began this conversation by saying that you never would. So the analogue in the world of business is: you're not just a guy we haven't "hired" for this project, you're a guy who has already said "I will never work for you under any circumstances". You may understand why we don't invite you to our design meetings, from this perspective. Despite how it may sound, this isn't any insult to you personally. In fact, the decision to spend your Sunday nights doing something worthwhile rather than correcting errors in the finer points of Twisted's recv() error handling on win32 might be an endorsement of your generally sound judgment. I know that sometimes I'd rather have been working out, watching a movie, spending time with my friends, or at least doing something billable during those hours. But, if what you want is people to listen to you on our humble little project, you need to acquire some of the reputation currency that we trade around here. This is compounded by the fact that Twisted, organizationally, has a very wary attitude towards design reviews. Speaking from my own experiences: once one has reached a certain level of abstraction, it is *very* easy to delude oneself into believing that further consideration and discussion is useful work, when in fact all of the assumptions in the discussion are based on a subtly flawed or impossible-to-implement decision. We discuss design and we think about it, but every discussion is followed by a prototype. A design discussion is an unverified hypothesis. There's no point in developing it into a theory until you have some further indication that it might be implemented. And unless you have resources available (i.e. developers, ready and waiting to do more work) to perform the experiment, than brainstorming more hypotheses is a waste of time.
glyph@divmod.com wrote:
A design discussion is an unverified hypothesis. There's no point in developing it into a theory until you have some further indication that it might be implemented. And unless you have resources available (i.e. developers, ready and waiting to do more work) to perform the experiment, than brainstorming more hypotheses is a waste of time.
Well said. Or as I like to put it "not everyone appreciates that computer science is an experimental science." :) Steve
On Wed, Apr 2, 2008 at 6:24 PM, James Mansion
So - I disagree. I'm quite happy to hand-wave, and to listen to my colleagues' hand-waving. If it communicates design ideas - and requirements - before any wasteful coding, that's good. Honest.
"90% code" is cheap and more expressive for communicating design ideas.
On 05:37 am, james@mansionfamily.plus.com wrote:
glyph@divmod.com wrote:
There's no point giving a commitment to doing more than discussing implementation approaches. With the best will in the world its unlikely to get to the top of my pile and there's no point living a fantasy.
I know the feeling.
I do need to understand how limited the current implementation is though.
It seems you've figured out the limitations at this point.
The curse of volunteer-ware, I guess. Understandable, but a shame in some ways.
A few months ago, I would have just said "Oh well", but now, you can do something about it! If you don't have the spare time to contribute an implementation, the Twisted project can now accept tax-deductible donations by way of the Software Freedom Conservancy: you'll notice a Google Checkout link on the front page of twistedmatrix.com. We plan to have most of this money go directly to funding development, so not everything will necessarily be volunteer effort after that.
* James Mansion
It seems to me that the Twisted model for 'reactor' is actually not what you'd expect from the name (at least, if you have a background in other reactor implementations including ACE, basic select etc and Java NIO) since it a) handles queueing outgoing data for you and b) presents incoming data after completion of the IO
I believe what Twisted calls the "reactor" would be better called a "proactor". -- mithrandi, i Ainil en-Balandor, a faer Ambar
participants (9)
-
Cory Dodt -
glyph＠divmod.com -
Itamar Shtull-Trauring -
James Mansion -
Jean-Paul Calderone -
Mike Pelletier -
Phil Christensen -
Stephen Waterbury -
Tristan Seligmann