[Python-ideas] asyncore: included batteries don't fit

Sun Oct 7 06:17:23 CEST 2012

On Sat, Oct 6, 2012 at 9:05 PM, Guido van Rossum <guido at python.org> wrote:

> On Sat, Oct 6, 2012 at 7:22 PM, Josiah Carlson <josiah.carlson at gmail.com>
> wrote:
> > On Sat, Oct 6, 2012 at 3:00 PM, Guido van Rossum <guido at python.org>
> wrote:
> >> This is an incredibly important discussion.
> >>
> >> I would like to contribute despite my limited experience with the
> >> various popular options. My own async explorations are limited to the
> >> constraints of the App Engine runtime environment, where a rather
> >> unique type of reactor is required. I am developing some ideas around
> >> separating reactors, futures, and yield-based coroutines, but they
> >> take more thinking and probably some experimental coding before I'm
> >> ready to write it up in any detail. For a hint on what I'm after, you
> >> might read up on monocle (https://github.com/saucelabs/monocle) and my
> >> approach to building coroutines on top of Futures
> >> (
> http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets.py#349
> ).
> >
> > Yield-based coroutines like monocle are the simplest way to do
> > multi-paradigm in the same code. Whether you have a async-style
> > reactor, greenlet-style stack switching, cooperatively scheduled
> > generator trampolines, or just plain blocking threaded sockets; that
> > style works with all of them (the futures and wrapper around
> > everything just looks a little different).
>
> Glad I'm not completely crazy here. :-)
>
> > That said, it forces everyone to drink the same coroutine-styled
> > kool-aid. That doesn't bother me. But I understand it, and have built
> > similar systems before. I don't have an intuition about whether 3rd
> > parties will like it or will migrate to it. Someone want to ping the
> > Twisted and Tornado folks about it?
>
> They should be reading this.

Yup, we are. I've pinged others in the Twisted cabal on this matter, so
hopefully you'll be hearing from one or more of us soon...

d

> Or maybe we should bring it up on
> python-dev before too long.
>
> >> In the mean time I'd like to bring up a few higher-order issues:
> >>
> >> (1) How importance is it to offer a compatibility path for asyncore? I
> >> would have thought that offering an integration path forward for
> >> Twisted and Tornado would be more important.
> >>
> >> (2) We're at a fork in the road here. On the one hand, we could choose
> >> to deeply integrate greenlets/gevents into the standard library. (It's
> >> not monkey-patching if it's integrated, after all. :-) I'm not sure
> >> how this would work for other implementations than CPython, or even
> >> how to address CPython on non-x86 architectures. But users seem to
> >> like the programming model: write synchronous code, get async
> >> operation for free. It's easy to write protocol parsers that way. On
> >> the other hand, we could reject this approach: the integration would
> >> never be completely smooth, there's the issue of other implementations
> >> and architectures, it probably would never work smoothly even for
> >> CPython/x86 when 3rd party extension modules are involved.
> >> Callback-based APIs don't have these downsides, but they are harder to
> >> program; however we can make programming them easier by using
> >> yield-based coroutines. Even Twisted offers those (inline callbacks).
> >>
> >> Before I invest much more time in these ideas I'd like to at least
> >> have (2) sorted out.
> >
> > Combining your responses to #1 and now this, are you proposing a path
> > forward for Twisted/Tornado to be greenlets? That's an interesting
> > approach to the problem, though I can see the draw. ;)
>
> Can't tell whether you're serious, but that's not what I meant. Surely
> it will never fly for Twisted. Tornado apparently already works with
> greenlets (though maybe through a third party hack). But personally
> I'd be leaning towards rejecting greenlets, for the same reasons I've
> kept the doors tightly shut for Stackless -- I like it fine as a
> library, but not as a language feature, because I don't see how it can
> be supported on all platforms where Python must be supported.
>
> However I figured that if we define the interfaces well enough, it
> might be possible to use (a superficially modified version of)
> Twisted's reactors instead of the standard ones, and, orthogonally,
> Twisted's deferred's could be wrapped in the standard Futures (or the
> other way around?) when used with a non-Twisted reactor. Which would
> hopefully open the door for migrating some of their more useful
> protocol parsers into the stdlib.
>
> > I have been hesitant on the Twisted side of things for an arbitrarily
> > selfish reason. After 2-3 hours of reading over a codebase (which I've
> > done 5 or 6 times in the last 8 years), I ask myself whether I believe
> > I understand 80+% of how things work; how data flows, how
> > callbacks/layers are invoked, and whether I could add a piece of
> > arbitrary functionality to one layer or another (or to determine the
> > proper layer in which to add the functionality). If my answer is "no",
> > then my gut says "this is probably a bad idea". But if I start
> > figuring out the layers before I've finished my 2-3 hours, and I start
> > finding bugs? Well, then I think it's a much better idea, even if the
> > implementation is buggy.
>
> Can't figure what you're implying here. On which side does Twisted fall
> for you?
>
> > Maybe something like Monocle would be better (considering your favor
> > for that style, it obviously has a leg-up on the competition). I don't
> > know. But if something like Monocle can merge it all together, then
> > maybe I'd be happy.
>
> My worry is that monocle is too simple and does not cater for advanced
> needs. It doesn't seem to have caught on much outside the company
> where it originated.
>
> > Incidentally, I can think of a few different
> > styles of wrappers that would actually let people using
> > asyncore-derived stuff use something like Monocle. So maybe that's
> > really the right answer?
>
> I still don't really think asyncore is going to be a problem. It can
> easily be separated into a reactor and callbacks.
>
> > Regards,
> >  - Josiah
> >
> > P.S. Thank you for weighing in on this Guido. Even if it doesn't end
> > up the way I had originally hoped, at least now there's discussion.
>
> Hm, there seemed to be plenty of discussion before...
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121006/75271366/attachment.html>