[Python-ideas] asyncore: included batteries don't fit

[Josiah Carlson]

On Sat, Oct 6, 2012 at 3:00 PM, Guido van Rossum <guido at python.org> wrote:
> This is an incredibly important discussion.
>
> I would like to contribute despite my limited experience with the
> various popular options. My own async explorations are limited to the
> constraints of the App Engine runtime environment, where a rather
> unique type of reactor is required. I am developing some ideas around
> separating reactors, futures, and yield-based coroutines, but they
> take more thinking and probably some experimental coding before I'm
> ready to write it up in any detail. For a hint on what I'm after, you
> might read up on monocle (https://github.com/saucelabs/monocle) and my
> approach to building coroutines on top of Futures
> (http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets.py#349).

Yield-based coroutines like monocle are the simplest way to do
multi-paradigm in the same code. Whether you have a async-style
reactor, greenlet-style stack switching, cooperatively scheduled
generator trampolines, or just plain blocking threaded sockets; that
style works with all of them (the futures and wrapper around
everything just looks a little different).

That said, it forces everyone to drink the same coroutine-styled
kool-aid. That doesn't bother me. But I understand it, and have built
similar systems before. I don't have an intuition about whether 3rd
parties will like it or will migrate to it. Someone want to ping the
Twisted and Tornado folks about it?

> In the mean time I'd like to bring up a few higher-order issues:
>
> (1) How importance is it to offer a compatibility path for asyncore? I
> would have thought that offering an integration path forward for
> Twisted and Tornado would be more important.
>
> (2) We're at a fork in the road here. On the one hand, we could choose
> to deeply integrate greenlets/gevents into the standard library. (It's
> not monkey-patching if it's integrated, after all. :-) I'm not sure
> how this would work for other implementations than CPython, or even
> how to address CPython on non-x86 architectures. But users seem to
> like the programming model: write synchronous code, get async
> operation for free. It's easy to write protocol parsers that way. On
> the other hand, we could reject this approach: the integration would
> never be completely smooth, there's the issue of other implementations
> and architectures, it probably would never work smoothly even for
> CPython/x86 when 3rd party extension modules are involved.
> Callback-based APIs don't have these downsides, but they are harder to
> program; however we can make programming them easier by using
> yield-based coroutines. Even Twisted offers those (inline callbacks).
>
> Before I invest much more time in these ideas I'd like to at least
> have (2) sorted out.

Combining your responses to #1 and now this, are you proposing a path
forward for Twisted/Tornado to be greenlets? That's an interesting
approach to the problem, though I can see the draw. ;)

I have been hesitant on the Twisted side of things for an arbitrarily
selfish reason. After 2-3 hours of reading over a codebase (which I've
done 5 or 6 times in the last 8 years), I ask myself whether I believe
I understand 80+% of how things work; how data flows, how
callbacks/layers are invoked, and whether I could add a piece of
arbitrary functionality to one layer or another (or to determine the
proper layer in which to add the functionality). If my answer is "no",
then my gut says "this is probably a bad idea". But if I start
figuring out the layers before I've finished my 2-3 hours, and I start
finding bugs? Well, then I think it's a much better idea, even if the
implementation is buggy.

Maybe something like Monocle would be better (considering your favor
for that style, it obviously has a leg-up on the competition). I don't
know. But if something like Monocle can merge it all together, then
maybe I'd be happy. Incidentally, I can think of a few different
styles of wrappers that would actually let people using
asyncore-derived stuff use something like Monocle. So maybe that's
really the right answer?

Regards,
 - Josiah

P.S. Thank you for weighing in on this Guido. Even if it doesn't end
up the way I had originally hoped, at least now there's discussion.