On Sat, Oct 6, 2012 at 3:00 PM, Guido van Rossum <guido@python.org> wrote:
This is an incredibly important discussion.
I would like to contribute despite my limited experience with the various popular options. My own async explorations are limited to the constraints of the App Engine runtime environment, where a rather unique type of reactor is required. I am developing some ideas around separating reactors, futures, and yield-based coroutines, but they take more thinking and probably some experimental coding before I'm ready to write it up in any detail. For a hint on what I'm after, you might read up on monocle (https://github.com/saucelabs/monocle) and my approach to building coroutines on top of Futures (http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets...).
Yield-based coroutines like monocle are the simplest way to do multi-paradigm in the same code. Whether you have a async-style reactor, greenlet-style stack switching, cooperatively scheduled generator trampolines, or just plain blocking threaded sockets; that style works with all of them (the futures and wrapper around everything just looks a little different). That said, it forces everyone to drink the same coroutine-styled kool-aid. That doesn't bother me. But I understand it, and have built similar systems before. I don't have an intuition about whether 3rd parties will like it or will migrate to it. Someone want to ping the Twisted and Tornado folks about it?
In the mean time I'd like to bring up a few higher-order issues:
(1) How importance is it to offer a compatibility path for asyncore? I would have thought that offering an integration path forward for Twisted and Tornado would be more important.
(2) We're at a fork in the road here. On the one hand, we could choose to deeply integrate greenlets/gevents into the standard library. (It's not monkey-patching if it's integrated, after all. :-) I'm not sure how this would work for other implementations than CPython, or even how to address CPython on non-x86 architectures. But users seem to like the programming model: write synchronous code, get async operation for free. It's easy to write protocol parsers that way. On the other hand, we could reject this approach: the integration would never be completely smooth, there's the issue of other implementations and architectures, it probably would never work smoothly even for CPython/x86 when 3rd party extension modules are involved. Callback-based APIs don't have these downsides, but they are harder to program; however we can make programming them easier by using yield-based coroutines. Even Twisted offers those (inline callbacks).
Before I invest much more time in these ideas I'd like to at least have (2) sorted out.
Combining your responses to #1 and now this, are you proposing a path forward for Twisted/Tornado to be greenlets? That's an interesting approach to the problem, though I can see the draw. ;) I have been hesitant on the Twisted side of things for an arbitrarily selfish reason. After 2-3 hours of reading over a codebase (which I've done 5 or 6 times in the last 8 years), I ask myself whether I believe I understand 80+% of how things work; how data flows, how callbacks/layers are invoked, and whether I could add a piece of arbitrary functionality to one layer or another (or to determine the proper layer in which to add the functionality). If my answer is "no", then my gut says "this is probably a bad idea". But if I start figuring out the layers before I've finished my 2-3 hours, and I start finding bugs? Well, then I think it's a much better idea, even if the implementation is buggy. Maybe something like Monocle would be better (considering your favor for that style, it obviously has a leg-up on the competition). I don't know. But if something like Monocle can merge it all together, then maybe I'd be happy. Incidentally, I can think of a few different styles of wrappers that would actually let people using asyncore-derived stuff use something like Monocle. So maybe that's really the right answer? Regards, - Josiah P.S. Thank you for weighing in on this Guido. Even if it doesn't end up the way I had originally hoped, at least now there's discussion.