[Python-ideas] asyncore: included batteries don't fit

[Devin Jeanpierre]

First of all, sorry for not snipping the reply I made previously.
Noticed that only after I sent it :(

On Thu, Oct 11, 2012 at 7:37 PM, Guido van Rossum <guido at python.org> wrote:
> On Thu, Oct 11, 2012 at 3:42 PM, Devin Jeanpierre
> <jeanpierreda at gmail.com> wrote:
>> Could you be more specific? I've never heard Deferreds in particular
>> called "arcane". They're very popular in e.g. the JS world,
>
> Really? Twisted is used in the JS world? Or do you just mean the
> pervasiveness of callback style async programming?

Ah, I mean Deferreds. I attended a talk earlier this year all about
deferreds in JS, and not a single reference to Python or Twisted was
made!

These are the examples I remember mentioned in the talk:

- http://api.jquery.com/category/deferred-object/ (not very twistedish
at all, ill-liked by the speaker)
- http://mochi.github.com/mochikit/doc/html/MochiKit/Async.html (maybe
not a good example, mochikit tries to be "python in JS")
- http://dojotoolkit.org/reference-guide/1.8/dojo/Deferred.html
- https://github.com/kriskowal/q (also includes an explanation of why
the author likes deferreds)

There were a few more that the speaker mentioned, but didn't cover.
One of his points was that the various systems of deferreds are subtly
different, some very badly so, and that it was a mess, but that
deferreds were still awesome. JS is a language where async programming
is mainstream, so lots of people try to make it easier, and they all
do it slightly differently.

> That's one of the
> things I am desperately trying to keep out of Python, I find that
> style unreadable and unmanageable (whenever I click on a button in a
> website and nothing happens I know someone has a bug in their
> callbacks). I understand you feel different; but I feel the general
> sentiment is that callback-based async programming is even harder than
> multi-threaded programming (and nobody is claiming that threads are
> easy :-).

:S

There are (at least?) four different styles of asynchronous
computation used in Twisted, and you seem to be confused as to which
ones I'm talking about.

1. Explicit callbacks:

    For example, reactor.callLater(t, lambda: print("woo hoo"))

2. Method dispatch callbacks:

    Similar to the above, the reactor or somebody has a handle on your
object, and calls methods that you've defined when events happen
    e.g. IProtocol's dataReceived method

3. Deferred callbacks:

    When you ask for something to be done, it's set up, and you get an
object back, which you can add a pipeline of callbacks to that will be
called whenever whatever happens
    e.g. twisted.internet.threads.deferToThread(print,
"x").addCallback(print, "x was printed in some other thread!")

4. Generator coroutines

    These are a syntactic wrapper around deferreds. If you yield a
deferred, you will be sent the result if the deferred succeeds, or an
exception if the deferred fails.
    e.g. examples from previous message


I don't see a reason for the first to exist at all, the second one is
kind of nice in some circumstances (see below), but perhaps overused.

I feel like you're railing on the first and second when I'm talking
about the third and fourth. I could be wrong.

>> and possibly elsewhere. Moreover, they're extremely similar to futures, so
>> if one is arcane so is the other.
>
> I love Futures, they represent a nice simple programming model. But I
> especially love that you can write async code using Futures and
> yield-based coroutines (what you call inlineCallbacks) and never have
> to write an explicit callback function. Ever.

The reason explicit non-deferred callbacks are involved in Twisted is
because of situations in which deferreds are not present, because of
past history in Twisted. It is not at all a limitation of deferreds or
something futures are better at, best as I'm aware.

(In case that's what you're getting at.)


Anyway, one big issue is that generator coroutines can't really
effectively replace callbacks everywhere. Consider the GUI button
example you gave. How do you write that as a coroutine?

I can see it being written like this:

    def mycoroutine(gui):
        while True:
            clickevent = yield gui.mybutton1.on_click()
            # handle clickevent

But that's probably worse than using callbacks.

>> Neither is clearly better or more obvious than the other. If anything
>> I generally find deferred composition more useful than deferred
>> tee-ing, so I feel like composition is the correct base operator, but
>> you could pick another.
>
> If you're writing long complicated chains of callbacks that benefit
> from these features, IMO you are already doing it wrong. I understand
> that this is a matter of style where I won't be able to convince you.
> But style is important to me, so let's agree to disagree.

This is more than a matter of style, so at least for now I'd like to
hold off on calling it even.

In my day to day silly, synchronous, python code, I do lots of
synchronous requests. For example, it's not unreasonable for me to
want to load two different files from disk, or make several database
interactions, etc. If I want to make this asynchronous, I have to find
a way to execute multiple things that could hypothetically block, at
the same time. If I can't do that easily, then the asynchronous
solution has failed, because its entire purpose is to do everything
that I do synchronously, except without blocking the main thread.

Here's an example with lots of synchronous requests in Django:

def view_paste(request, filekey):
    try:
        fileinfo= Pastes.objects.get(key=filekey)
    except DoesNotExist:
        t = loader.get_template('pastebin/error.html')
        return HttpResponse(t.render(Context(dict(error='File does not
exist'))))

    f = open(fileinfo.filename)
    fcontents = f.read()
    t = loader.get_template('pastebin/paste.html')
    return HttpResponse(t.render(Context(dict(file=fcontents))))

How many blocking requests are there? Lots. This is, in a word, a
long, complicated chain of synchronous requests. This is also very
similar to what actual django code might look like in some
circumstances. Even if we might think this is unreasonable, some
subset of alteration of this is reasonable. Certainly we should be
able to, say, load multiple (!) objects from the database, and open
the template (possibly from disk), all potentially-blocking
operations.

This is inherently a long, complicated chain of requests, whether we
implement it asynchronously or synchronously, or use Deferreds or
Futures, or write it in Java or Python. Some parts can be done at any
time before the end (loader.get_template(...)), some need to be done
in a certain order, and there's branching depending on what happens in
different cases. In order to even write this code _at all_, we need a
way to chain these IO actions together. If we can't chain them
together, we can't produce that final synthesis of results at the end.

We _need_ a pipeline or something computationally equivalent or more
powerful. Results from past "deferred computations" need to be passed
forward into future "deferred computations", in order to implement
this at all.

This is not a style issue, this is an issue of needing to be able to
solve problems that involve more than one computation where the
results of every computation matters somewhere. It's just that in this
case, some of the computations are computed asynchronously.

> I am totally open to learning from Twisted's experience. I hope that
> you are willing to share even the end result might not look like
> Twisted at all -- after all in Python 3.3 we have "yield from" and
> return from a generator and many years of experience with different
> styles of async APIs. In addition to Twisted, there's Tornado and
> Monocle, and then there's the whole greenlets/gevent and
> Stackless/microthreads community that we can't completely ignore. I
> believe somewhere is an ideal async architecture, and I hope you can
> help us discover it.
>
> (For example, I am very interested in Twisted's experiences writing
> real-world performant, robust reactors.)

For that stuff, you'd have to speak to the main authors of Twisted.
I'm just a twisted user. :(

In the end it really doesn't matter what API you go with. The Twisted
people will wrap it up so that they are compatible, as far as that is
possible.

I hope I haven't detracted too much from the main thrust of the
surrounding discussion. Futures/deferreds are a pretty big tangent, so
sorry. I justified it to myself by figuring that it'd probably come up
anyway, somehow, since these are useful abstractions for asynchronous
programming.

-- Devin