[Python-ideas] The async API of the future: Reactors

[Calvin Spealman]

On Fri, Oct 12, 2012 at 3:32 PM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Oct 12, 2012 at 11:33 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Fri, 12 Oct 2012 11:13:23 -0700
>> Guido van Rossum <guido at python.org> wrote:
>>> OTOH someone else might prefer a buffered stream
>>> abstraction that just keeps filling its read buffer (and draining its
>>> write buffer) using level-triggered callbacks, at least up to a
>>> certain buffer size -- we have to be robust here and make it
>>> impossible for an evil client to fill up all our memory without our
>>> approval!
>>
>> I'd like to know what a sane buffered API for non-blocking I/O may look
>> like, because right now it doesn't seem to make a lot of sense. At
>> least this bug is tricky to resolve:
>> http://bugs.python.org/issue13322
>
> Good question. It actually depends quite a bit on whether you have an
> event loop or not -- with the help of an event loop, you can have a
> level-triggered callback that fills the buffer behind your back (up to
> a given limit, at which point it should unregister the I/O object);
> that bug seems to be about a situation without an event loop, where
> you can't do that. Also the existing io module design never
> anticipated cooperation with an event loop.
>
>>> - There's an abstract Reactor class and an abstract Async I/O object
>>> class. To get a reactor to call you back, you must give it an I/O
>>> object, a callback, and maybe some more stuff. (I have gone back and
>>> like passing optional args for the callback, rather than requiring
>>> lambdas to create closures.) Note that the callback is *not* a
>>> designated method on the I/O object!
>>
>> Why isn't it? In practice, you need several callbacks: in Twisted
>> parlance, you have dataReceived but also e.g. ConnectionLost
>> (depending on the transport, you may even imagine other callbacks, for
>> example for things happening on the TLS layer?).
>
> Yes, but I really want to separate the callbacks from the object, so
> that I don't have to inherit from an I/O object class -- asyncore
> requires this and IMO it's wrong. It also makes it harder to use the
> same callback code with different types of I/O objects.

Why is subclassing a problem? It can be overused, but seems the right
thing to do in this case. You want a protocol that responds to new data by
echoing and tells the user when the connection was terminated? It makes
sense that this is a subclass: a special case of some class that handles the
base behavior.

What if this was just an optional way and we could also provide a helper to
attach handlers to the base class instance without subclassing it? The function
registering it could take keyword arguments mapping additional event->callbacks
to the object.

>>> - In systems supporting file descriptors, there's a reactor
>>> implementation that knows how to use select/poll/etc., and there are
>>> concrete I/O object classes that wrap file descriptors. On Windows,
>>> those would only be socket file descriptors. On Unix, any file
>>> descriptor would do.
>>
>> Windows *is* able to do async I/O on things other than sockets (see the
>> discussion about IOCP). It's just that the Windows implementation of
>> select() (the POSIX function call) is limited to sockets.
>
> I know, but IOCP is currently not supported in the stdlib. I expect
> that on Windows, to use IOCP, you'd need to use a different reactor
> implementation and a different I/O object than the vanilla fd-based
> ones. My design is actually *inspired* by the desire to support this
> cleanly.
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy