On Thu, Oct 11, 2012 at 5:18 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Oct 9, 2012 at 11:00 AM, Laurens Van Houtven <_@lvh.cc> wrote:
Oh my me. This is a very long thread that I probably should have replied to a long time ago. This thread is intensely long right now, and tonight is the first chance I've had to try and go through it comprehensively. I'll try to reply to individual points made in the thread -- if I missed yours, please don't be offended, I promise it's my fault :)
No problem, I'm running behind myself...
FYI, I'm the sucker who originally got tricked into starting PEP 3153, aka async-pep.
I suppose that's your pet name for it. :-) For most everyone else it's PEP 3153.
First of all, I'm glad to see that there's some more "let's get that pep along" movement. I tabled it because:
a) I didn't have enough time to contribute, b) a lot of promised contributions ended up not happening when it came down to it, which was incredibly demotivating. The combination of this thread, plus the fact that I was strong armed at Pycon ZA by a bunch of community members that shall not be named (Alex, Armin, Maciej, Larry ;-)) into exploring this thing again.
First of all, I don't feel async-pep is an attempt at twisted light in the stdlib. Other than separation of transport and protocol, there's not really much there that even smells of twisted (especially since right now I'd probably throw consumers/producers out) -- and that separation is simply good practice. Twisted does the same thing, but it didn't invent it. Furthermore, the advantages seem clear: reusability and testability are more than enough for me.
If there's one take away idea from async-pep, it's reusable protocols.
Is there a newer version that what's on http://www.python.org/dev/peps/pep-3153/ ? It seems to be missing any specific proposals, after spending a lot of time giving a rationale and defining some terms. The version on https://github.com/lvh/async-pep doesn't seem to be any more complete.
The PEP should probably be a number of PEPs. At first sight, it seems that this number is at least four:
1. Protocol and transport abstractions, making no mention of asynchronous IO (this is what I want 3153 to be, because it's small, manageable, and virtually everyone appears to agree it's a fantastic idea)
But the devil is in the details. *What* specifically are you proposing? How would you write a protocol handler/parser without any reference to I/O? Most protocols are two-way streets -- you read some stuff, and you write some stuff, then you read some more. (HTTP may be the exception here, if you don't keep the connection open.)
2. A base reactor interface
I agree that this should be a separate PEP. But I do think that in practice there will be dependencies between the different PEPs you are proposing.
3. A way of structuring callbacks: probably deferreds with a built-in inlineCallbacks for people who want to write synchronous-looking code with explicit yields for asynchronous procedures
Your previous two ideas sound like you're not tied to backward compatibility with Tornado and/or Twisted (not even via an adaptation layer). Given that we're talking Python 3.4 here that's fine with me (though I think we should be careful to offer a path forward for those packages and their users, even if it means making changes to the libraries). But Twisted Deferred is pretty arcane, and I would much rather not use it as the basis of a forward-looking design. I'd much rather see what we can mooch off PEP 3148 (Futures).
Could you be more specific? I've never heard Deferreds in particular called "arcane". They're very popular in e.g. the JS world, and possibly elsewhere. Moreover, they're extremely similar to futures, so if one is arcane so is the other. Maybe if you could elaborate on features of their designs that are better/worse? As far as I know, they mostly differ in that: - Callbacks are added in a pipeline, rather than "in parallel" - Deferreds pass in values along the pipeline, rather than self (and have a separate pipeline for error values). Neither is clearly better or more obvious than the other. If anything I generally find deferred composition more useful than deferred tee-ing, so I feel like composition is the correct base operator, but you could pick another. Either way, each is implementable in terms of the other (ish?). The pipeline approach is particularly nice for the errback pipeline, because it allows chained exception (Failure) handling on the deferred to be very simple. The larger issue is that futures don't make chaining easy at all, even if it is theoretically possible. For example, look at the following Twisted code: http://bpaste.net/show/RfEwoaflO0qY76N8NjHx/ , and imagine how that might generalize to more realistic error handling scenarios. The equivalent Futures code would involve creating one Future per callback in the pipeline and manually hooking them up with a special callback that passes values to the next future. And if we add that to the futures API, the API will almost certainly be somewhat similar to what Twisted has with deferreds and chaining and such. So then, equally arcane. To my mind, it is Futures that need to mooch off of Deferreds, not the other way around. Twisted's Deferreds have a lot of history with making asynchronous computation pleasant, and Futures are missing a lot of good tools. -- Devin