[Python-ideas] asyncore: included batteries don't fit

[Guido van Rossum]

On Sat, Oct 6, 2012 at 7:22 PM, Josiah Carlson <josiah.carlson at gmail.com> wrote:
> On Sat, Oct 6, 2012 at 3:00 PM, Guido van Rossum <guido at python.org> wrote:
>> This is an incredibly important discussion.
>>
>> I would like to contribute despite my limited experience with the
>> various popular options. My own async explorations are limited to the
>> constraints of the App Engine runtime environment, where a rather
>> unique type of reactor is required. I am developing some ideas around
>> separating reactors, futures, and yield-based coroutines, but they
>> take more thinking and probably some experimental coding before I'm
>> ready to write it up in any detail. For a hint on what I'm after, you
>> might read up on monocle (https://github.com/saucelabs/monocle) and my
>> approach to building coroutines on top of Futures
>> (http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets.py#349).
>
> Yield-based coroutines like monocle are the simplest way to do
> multi-paradigm in the same code. Whether you have a async-style
> reactor, greenlet-style stack switching, cooperatively scheduled
> generator trampolines, or just plain blocking threaded sockets; that
> style works with all of them (the futures and wrapper around
> everything just looks a little different).

Glad I'm not completely crazy here. :-)

> That said, it forces everyone to drink the same coroutine-styled
> kool-aid. That doesn't bother me. But I understand it, and have built
> similar systems before. I don't have an intuition about whether 3rd
> parties will like it or will migrate to it. Someone want to ping the
> Twisted and Tornado folks about it?

They should be reading this. Or maybe we should bring it up on
python-dev before too long.

>> In the mean time I'd like to bring up a few higher-order issues:
>>
>> (1) How importance is it to offer a compatibility path for asyncore? I
>> would have thought that offering an integration path forward for
>> Twisted and Tornado would be more important.
>>
>> (2) We're at a fork in the road here. On the one hand, we could choose
>> to deeply integrate greenlets/gevents into the standard library. (It's
>> not monkey-patching if it's integrated, after all. :-) I'm not sure
>> how this would work for other implementations than CPython, or even
>> how to address CPython on non-x86 architectures. But users seem to
>> like the programming model: write synchronous code, get async
>> operation for free. It's easy to write protocol parsers that way. On
>> the other hand, we could reject this approach: the integration would
>> never be completely smooth, there's the issue of other implementations
>> and architectures, it probably would never work smoothly even for
>> CPython/x86 when 3rd party extension modules are involved.
>> Callback-based APIs don't have these downsides, but they are harder to
>> program; however we can make programming them easier by using
>> yield-based coroutines. Even Twisted offers those (inline callbacks).
>>
>> Before I invest much more time in these ideas I'd like to at least
>> have (2) sorted out.
>
> Combining your responses to #1 and now this, are you proposing a path
> forward for Twisted/Tornado to be greenlets? That's an interesting
> approach to the problem, though I can see the draw. ;)

Can't tell whether you're serious, but that's not what I meant. Surely
it will never fly for Twisted. Tornado apparently already works with
greenlets (though maybe through a third party hack). But personally
I'd be leaning towards rejecting greenlets, for the same reasons I've
kept the doors tightly shut for Stackless -- I like it fine as a
library, but not as a language feature, because I don't see how it can
be supported on all platforms where Python must be supported.

However I figured that if we define the interfaces well enough, it
might be possible to use (a superficially modified version of)
Twisted's reactors instead of the standard ones, and, orthogonally,
Twisted's deferred's could be wrapped in the standard Futures (or the
other way around?) when used with a non-Twisted reactor. Which would
hopefully open the door for migrating some of their more useful
protocol parsers into the stdlib.

> I have been hesitant on the Twisted side of things for an arbitrarily
> selfish reason. After 2-3 hours of reading over a codebase (which I've
> done 5 or 6 times in the last 8 years), I ask myself whether I believe
> I understand 80+% of how things work; how data flows, how
> callbacks/layers are invoked, and whether I could add a piece of
> arbitrary functionality to one layer or another (or to determine the
> proper layer in which to add the functionality). If my answer is "no",
> then my gut says "this is probably a bad idea". But if I start
> figuring out the layers before I've finished my 2-3 hours, and I start
> finding bugs? Well, then I think it's a much better idea, even if the
> implementation is buggy.

Can't figure what you're implying here. On which side does Twisted fall for you?

> Maybe something like Monocle would be better (considering your favor
> for that style, it obviously has a leg-up on the competition). I don't
> know. But if something like Monocle can merge it all together, then
> maybe I'd be happy.

My worry is that monocle is too simple and does not cater for advanced
needs. It doesn't seem to have caught on much outside the company
where it originated.

> Incidentally, I can think of a few different
> styles of wrappers that would actually let people using
> asyncore-derived stuff use something like Monocle. So maybe that's
> really the right answer?

I still don't really think asyncore is going to be a problem. It can
easily be separated into a reactor and callbacks.

> Regards,
>  - Josiah
>
> P.S. Thank you for weighing in on this Guido. Even if it doesn't end
> up the way I had originally hoped, at least now there's discussion.

Hm, there seemed to be plenty of discussion before...

-- 
--Guido van Rossum (python.org/~guido)