[Python-ideas] asyncore: included batteries don't fit

[Ben Darnell]

Hi python-ideas,

I'm jumping in to this thread on behalf of Tornado.  I think there are
actually two separate issues here and it's important to keep them
distinct:  at a low level, there is a need for a standardized event
loop, while at a higher level there is a question of what asynchronous
code should look like.

This thread so far has been more about the latter, but the need for
standardization is more acute for the core event loop.  I've written a
bridge between Tornado and Twisted so libraries written for both event
loops can coexist, but obviously that wouldn't scale if there were a
proliferation of event loop implementations out there.  I'd be in
favor of a simple event loop interface in the standard library, with
reference implementation(s) (select, epoll, kqueue, iocp) and some
means of configuring the global (or thread-local) singleton.  My
preference is to keep the interface fairly low-level and close to the
underlying mechanisms (i.e. like IReactorFDSet instead of
IReactor{TCP,UDP,SSL,etc}), so that different interfaces like
Tornado's IOStream or Twisted's protocols can be built on top of it.

As for the higher-level question of what asynchronous code should look
like, there's a lot more room for spirited debate, and I don't think
there's enough consensus to declare a One True Way.  Personally, I'm
-1 on greenlets as a general solution (what if you have to call
MySQLdb or getaddrinfo?), although they can be useful in particular
cases to convert well-behaved synchronous code into async (as in
Motor: http://emptysquare.net/blog/introducing-motor-an-asynchronous-mongodb-driver-for-python-and-tornado/).
 I like Futures, though, and I find that they work well in
asynchronous code.  The use of the result() method to encapsulate both
successful responses and exceptions is especially nice with generator
coroutines.

FWIW, here's the interface I'm moving towards for async code.  From
the caller's perspective, asynchronous functions return a Future (the
future has to be constructed by hand since there is no Executor
involved), and also take an optional callback argument (mainly for
consistency with currently-prevailing patterns for async code; if the
callback is given it is simply added to the Future with
add_done_callback).  In Tornado the Future is created by a decorator
and hidden from the asynchronous function (it just sees the callback),
although this relies on some Tornado-specific magic for exception
handling.  In a coroutine, the decorator recognizes Futures and
resumes execution when the future is done.  With these decorators
asynchronous code looks almost like synchronous code, except for the
"yield" keyword before each asynchronous call.

-Ben