[Python-ideas] asyncore: included batteries don't fit

[Guido van Rossum]

On Sun, Oct 7, 2012 at 6:41 PM, Ben Darnell <ben at bendarnell.com> wrote:
> Hi python-ideas,
>
> I'm jumping in to this thread on behalf of Tornado.

Welcome!

> I think there are
> actually two separate issues here and it's important to keep them
> distinct:  at a low level, there is a need for a standardized event
> loop, while at a higher level there is a question of what asynchronous
> code should look like.

Yes, yes. I tried to bring up thing distinction. I'm glad I didn't
completely fail.

> This thread so far has been more about the latter, but the need for
> standardization is more acute for the core event loop.  I've written a
> bridge between Tornado and Twisted so libraries written for both event
> loops can coexist, but obviously that wouldn't scale if there were a
> proliferation of event loop implementations out there.  I'd be in
> favor of a simple event loop interface in the standard library, with
> reference implementation(s) (select, epoll, kqueue, iocp) and some
> means of configuring the global (or thread-local) singleton.  My
> preference is to keep the interface fairly low-level and close to the
> underlying mechanisms (i.e. like IReactorFDSet instead of
> IReactor{TCP,UDP,SSL,etc}), so that different interfaces like
> Tornado's IOStream or Twisted's protocols can be built on top of it.

As long as it's not so low-level that other people shy away from it.

I also have a feeling that one way or another this will require
cooperation between the Twisted and Tornado developers in order to
come up with a compromise that both are willing to conform to in a
meaningful way. (Unfortunately I don't know how to define "meaningful
way" more precisely here. I guess the idea is that almost all things
*using* an event loop use the standardized abstract API without caring
whether underneath it's Tornado, Twisted, or some simpler thing in the
stdlib.

> As for the higher-level question of what asynchronous code should look
> like, there's a lot more room for spirited debate, and I don't think
> there's enough consensus to declare a One True Way.  Personally, I'm
> -1 on greenlets as a general solution (what if you have to call
> MySQLdb or getaddrinfo?), although they can be useful in particular
> cases to convert well-behaved synchronous code into async (as in
> Motor: http://emptysquare.net/blog/introducing-motor-an-asynchronous-mongodb-driver-for-python-and-tornado/).

Agreed on both counts.

>  I like Futures, though, and I find that they work well in
> asynchronous code.  The use of the result() method to encapsulate both
> successful responses and exceptions is especially nice with generator
> coroutines.

Yay!

> FWIW, here's the interface I'm moving towards for async code.  From
> the caller's perspective, asynchronous functions return a Future (the
> future has to be constructed by hand since there is no Executor
> involved),

Ditto for NDB (though there's a decorator that often takes care of the
future construction).

> and also take an optional callback argument (mainly for
> consistency with currently-prevailing patterns for async code; if the
> callback is given it is simply added to the Future with
> add_done_callback).

That's interesting. I haven't found the need for this yet. Is it
really so common that you can't write this as a Future() constructor
plus a call to add_done_callback()? Or is there some subtle semantic
difference?

> In Tornado the Future is created by a decorator
> and hidden from the asynchronous function (it just sees the callback),

Hm, interesting. NDB goes the other way, the callbacks are mostly used
to make Futures work, and most code (including large swaths of
internal code) uses Futures. I think NDB is similar to monocle here.
In NDB, you can do

  f = <some function returning a Future>
  r = yield f

where "yield f" is mostly equivalent to f.result(), except it gives
better opportunity for concurrency.

> although this relies on some Tornado-specific magic for exception
> handling.  In a coroutine, the decorator recognizes Futures and
> resumes execution when the future is done.  With these decorators
> asynchronous code looks almost like synchronous code, except for the
> "yield" keyword before each asynchronous call.

Yes! Same here.

I am currently trying to understand if using "yield from" (and
returning a value from a generator) will simplify things. For example
maybe the need for a special decorator might go away. But I keep
getting headaches -- perhaps there's a Monad involved. :-)

-- 
--Guido van Rossum (python.org/~guido)