[Python-ideas] asyncore: included batteries don't fit

[Devin Jeanpierre]

On Thu, Oct 11, 2012 at 5:18 PM, Guido van Rossum <guido at python.org> wrote:
> On Tue, Oct 9, 2012 at 11:00 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
>> Oh my me. This is a very long thread that I probably should have replied to
>> a long time ago. This thread is intensely long right now, and tonight is the
>> first chance I've had to try and go through it comprehensively. I'll try to
>> reply to individual points made in the thread -- if I missed yours, please
>> don't be offended, I promise it's my fault :)
>
> No problem, I'm running behind myself...
>
>> FYI, I'm the sucker who originally got tricked into starting PEP 3153, aka
>> async-pep.
>
> I suppose that's your pet name for it. :-) For most everyone else it's PEP 3153.
>
>> First of all, I'm glad to see that there's some more "let's get that pep
>> along" movement. I tabled it because:
>>
>> a) I didn't have enough time to contribute,
>> b) a lot of promised contributions ended up not happening when it came down
>> to it, which was incredibly demotivating. The combination of this thread,
>> plus the fact that I was strong armed at Pycon ZA by a bunch of community
>> members that shall not be named (Alex, Armin, Maciej, Larry ;-)) into
>> exploring this thing again.
>>
>> First of all, I don't feel async-pep is an attempt at twisted light in the
>> stdlib. Other than separation of transport and protocol, there's not really
>> much there that even smells of twisted (especially since right now I'd
>> probably throw consumers/producers out) -- and that separation is simply
>> good practice. Twisted does the same thing, but it didn't invent it.
>> Furthermore, the advantages seem clear: reusability and testability are more
>> than enough for me.
>>
>> If there's one take away idea from async-pep, it's reusable protocols.
>
> Is there a newer version that what's on
> http://www.python.org/dev/peps/pep-3153/ ? It seems to be missing any
> specific proposals, after spending a lot of time giving a rationale
> and defining some terms. The version on
> https://github.com/lvh/async-pep doesn't seem to be any more complete.
>
>> The PEP should probably be a number of PEPs. At first sight, it seems that
>> this number is at least four:
>>
>> 1. Protocol and transport abstractions, making no mention of asynchronous IO
>> (this is what I want 3153 to be, because it's small, manageable, and
>> virtually everyone appears to agree it's a fantastic idea)
>
> But the devil is in the details. *What* specifically are you
> proposing? How would you write a protocol handler/parser without any
> reference to I/O? Most protocols are two-way streets -- you read some
> stuff, and you write some stuff, then you read some more. (HTTP may be
> the exception here, if you don't keep the connection open.)
>
>> 2. A base reactor interface
>
> I agree that this should be a separate PEP. But I do think that in
> practice there will be dependencies between the different PEPs you are
> proposing.
>
>> 3. A way of structuring callbacks: probably deferreds with a built-in
>> inlineCallbacks for people who want to write synchronous-looking code with
>> explicit yields for asynchronous procedures
>
> Your previous two ideas sound like you're not tied to backward
> compatibility with Tornado and/or Twisted (not even via an adaptation
> layer). Given that we're talking Python 3.4 here that's fine with me
> (though I think we should be careful to offer a path forward for those
> packages and their users, even if it means making changes to the
> libraries). But Twisted Deferred is pretty arcane, and I would much
> rather not use it as the basis of a forward-looking design. I'd much
> rather see what we can mooch off PEP 3148 (Futures).

Could you be more specific? I've never heard Deferreds in particular
called "arcane". They're very popular in e.g. the JS world, and
possibly elsewhere. Moreover, they're extremely similar to futures, so
if one is arcane so is the other.

Maybe if you could elaborate on features of their designs that are better/worse?

As far as I know, they mostly differ in that:

- Callbacks are added in a pipeline, rather than "in parallel"
- Deferreds pass in values along the pipeline, rather than self (and
have a separate pipeline for error values).

Neither is clearly better or more obvious than the other. If anything
I generally find deferred composition more useful than deferred
tee-ing, so I feel like composition is the correct base operator, but
you could pick another. Either way, each is implementable in terms of
the other (ish?). The pipeline approach is particularly nice for the
errback pipeline, because it allows chained exception (Failure)
handling on the deferred to be very simple. The larger issue is that
futures don't make chaining easy at all, even if it is theoretically
possible.

For example, look at the following Twisted code:
http://bpaste.net/show/RfEwoaflO0qY76N8NjHx/ , and imagine how that
might generalize to more realistic error handling scenarios.

The equivalent Futures code would involve creating one Future per
callback in the pipeline and manually hooking them up with a special
callback that passes values to the next future. And if we add that to
the futures API, the API will almost certainly be somewhat similar to
what Twisted has with deferreds and chaining and such. So then,
equally arcane.

To my mind, it is Futures that need to mooch off of Deferreds, not the
other way around. Twisted's Deferreds have a lot of history with
making asynchronous computation pleasant, and Futures are missing a
lot of good tools.

-- Devin