[Python-ideas] asyncore: included batteries don't fit

[Guido van Rossum]

On Thu, Oct 11, 2012 at 3:42 PM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
> On Thu, Oct 11, 2012 at 5:18 PM, Guido van Rossum <guido at python.org> wrote:
>> [...] Twisted Deferred is pretty arcane, and I would much
>> rather not use it as the basis of a forward-looking design. I'd much
>> rather see what we can mooch off PEP 3148 (Futures).
>
> Could you be more specific? I've never heard Deferreds in particular
> called "arcane". They're very popular in e.g. the JS world,

Really? Twisted is used in the JS world? Or do you just mean the
pervasiveness of callback style async programming? That's one of the
things I am desperately trying to keep out of Python, I find that
style unreadable and unmanageable (whenever I click on a button in a
website and nothing happens I know someone has a bug in their
callbacks). I understand you feel different; but I feel the general
sentiment is that callback-based async programming is even harder than
multi-threaded programming (and nobody is claiming that threads are
easy :-).

> and possibly elsewhere. Moreover, they're extremely similar to futures, so
> if one is arcane so is the other.

I love Futures, they represent a nice simple programming model. But I
especially love that you can write async code using Futures and
yield-based coroutines (what you call inlineCallbacks) and never have
to write an explicit callback function. Ever.

> Maybe if you could elaborate on features of their designs that are better/worse?
>
> As far as I know, they mostly differ in that:
>
> - Callbacks are added in a pipeline, rather than "in parallel"
> - Deferreds pass in values along the pipeline, rather than self (and
> have a separate pipeline for error values).

These two combined are indeed what mostly feels arcane to me.

> Neither is clearly better or more obvious than the other. If anything
> I generally find deferred composition more useful than deferred
> tee-ing, so I feel like composition is the correct base operator, but
> you could pick another.

If you're writing long complicated chains of callbacks that benefit
from these features, IMO you are already doing it wrong. I understand
that this is a matter of style where I won't be able to convince you.
But style is important to me, so let's agree to disagree.

> Either way, each is implementable in terms of
> the other (ish?). The pipeline approach is particularly nice for the
> errback pipeline, because it allows chained exception (Failure)
> handling on the deferred to be very simple. The larger issue is that
> futures don't make chaining easy at all, even if it is theoretically
> possible.

But as soon as you switch from callbacks to yield-based coroutines the
chaining becomes natural, error handling is just a matter of
try/except statements (or not if you want the error to bubble up) and
(IMO) the code becomes much more readable.

> For example, look at the following Twisted code:
> http://bpaste.net/show/RfEwoaflO0qY76N8NjHx/ , and imagine how that
> might generalize to more realistic error handling scenarios.

Looks fine to me. I have a lot of code like that in NDB and it works
great. (Note that NDB's Futures are not the same as PEP 3148 Futures,
although they have some things in common; in particular NDB Futures
are not tied to threads.)

> The equivalent Futures code would involve creating one Future per
> callback in the pipeline and manually hooking them up with a special
> callback that passes values to the next future. And if we add that to
> the futures API, the API will almost certainly be somewhat similar to
> what Twisted has with deferreds and chaining and such. So then,
> equally arcane.

The *implementation* of this stuff in NDB is certainly hairy; I
already posted the link to the code:
http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets.py#349
However, this is internal code and doesn't affect the Future API at
all.

> To my mind, it is Futures that need to mooch off of Deferreds, not the
> other way around. Twisted's Deferreds have a lot of history with
> making asynchronous computation pleasant, and Futures are missing a
> lot of good tools.

I am totally open to learning from Twisted's experience. I hope that
you are willing to share even the end result might not look like
Twisted at all -- after all in Python 3.3 we have "yield from" and
return from a generator and many years of experience with different
styles of async APIs. In addition to Twisted, there's Tornado and
Monocle, and then there's the whole greenlets/gevent and
Stackless/microthreads community that we can't completely ignore. I
believe somewhere is an ideal async architecture, and I hope you can
help us discover it.

(For example, I am very interested in Twisted's experiences writing
real-world performant, robust reactors.)

-- 
--Guido van Rossum (python.org/~guido)