On Sun, Oct 7, 2012 at 6:41 PM, Ben Darnell <ben@bendarnell.com> wrote:
Hi python-ideas,
I'm jumping in to this thread on behalf of Tornado.
Welcome!
I think there are actually two separate issues here and it's important to keep them distinct: at a low level, there is a need for a standardized event loop, while at a higher level there is a question of what asynchronous code should look like.
Yes, yes. I tried to bring up thing distinction. I'm glad I didn't completely fail.
This thread so far has been more about the latter, but the need for standardization is more acute for the core event loop. I've written a bridge between Tornado and Twisted so libraries written for both event loops can coexist, but obviously that wouldn't scale if there were a proliferation of event loop implementations out there. I'd be in favor of a simple event loop interface in the standard library, with reference implementation(s) (select, epoll, kqueue, iocp) and some means of configuring the global (or thread-local) singleton. My preference is to keep the interface fairly low-level and close to the underlying mechanisms (i.e. like IReactorFDSet instead of IReactor{TCP,UDP,SSL,etc}), so that different interfaces like Tornado's IOStream or Twisted's protocols can be built on top of it.
As long as it's not so low-level that other people shy away from it. I also have a feeling that one way or another this will require cooperation between the Twisted and Tornado developers in order to come up with a compromise that both are willing to conform to in a meaningful way. (Unfortunately I don't know how to define "meaningful way" more precisely here. I guess the idea is that almost all things *using* an event loop use the standardized abstract API without caring whether underneath it's Tornado, Twisted, or some simpler thing in the stdlib.
As for the higher-level question of what asynchronous code should look like, there's a lot more room for spirited debate, and I don't think there's enough consensus to declare a One True Way. Personally, I'm -1 on greenlets as a general solution (what if you have to call MySQLdb or getaddrinfo?), although they can be useful in particular cases to convert well-behaved synchronous code into async (as in Motor: http://emptysquare.net/blog/introducing-motor-an-asynchronous-mongodb-driver...).
Agreed on both counts.
I like Futures, though, and I find that they work well in asynchronous code. The use of the result() method to encapsulate both successful responses and exceptions is especially nice with generator coroutines.
Yay!
FWIW, here's the interface I'm moving towards for async code. From the caller's perspective, asynchronous functions return a Future (the future has to be constructed by hand since there is no Executor involved),
Ditto for NDB (though there's a decorator that often takes care of the future construction).
and also take an optional callback argument (mainly for consistency with currently-prevailing patterns for async code; if the callback is given it is simply added to the Future with add_done_callback).
That's interesting. I haven't found the need for this yet. Is it really so common that you can't write this as a Future() constructor plus a call to add_done_callback()? Or is there some subtle semantic difference?
In Tornado the Future is created by a decorator and hidden from the asynchronous function (it just sees the callback),
Hm, interesting. NDB goes the other way, the callbacks are mostly used to make Futures work, and most code (including large swaths of internal code) uses Futures. I think NDB is similar to monocle here. In NDB, you can do f = <some function returning a Future> r = yield f where "yield f" is mostly equivalent to f.result(), except it gives better opportunity for concurrency.
although this relies on some Tornado-specific magic for exception handling. In a coroutine, the decorator recognizes Futures and resumes execution when the future is done. With these decorators asynchronous code looks almost like synchronous code, except for the "yield" keyword before each asynchronous call.
Yes! Same here. I am currently trying to understand if using "yield from" (and returning a value from a generator) will simplify things. For example maybe the need for a special decorator might go away. But I keep getting headaches -- perhaps there's a Monad involved. :-) -- --Guido van Rossum (python.org/~guido)