[Python-ideas] The async API of the future: Reactors

[Guido van Rossum]

On Fri, Oct 12, 2012 at 11:33 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 12 Oct 2012 11:13:23 -0700
> Guido van Rossum <guido at python.org> wrote:
>> OTOH someone else might prefer a buffered stream
>> abstraction that just keeps filling its read buffer (and draining its
>> write buffer) using level-triggered callbacks, at least up to a
>> certain buffer size -- we have to be robust here and make it
>> impossible for an evil client to fill up all our memory without our
>> approval!
>
> I'd like to know what a sane buffered API for non-blocking I/O may look
> like, because right now it doesn't seem to make a lot of sense. At
> least this bug is tricky to resolve:
> http://bugs.python.org/issue13322

Good question. It actually depends quite a bit on whether you have an
event loop or not -- with the help of an event loop, you can have a
level-triggered callback that fills the buffer behind your back (up to
a given limit, at which point it should unregister the I/O object);
that bug seems to be about a situation without an event loop, where
you can't do that. Also the existing io module design never
anticipated cooperation with an event loop.

>> - There's an abstract Reactor class and an abstract Async I/O object
>> class. To get a reactor to call you back, you must give it an I/O
>> object, a callback, and maybe some more stuff. (I have gone back and
>> like passing optional args for the callback, rather than requiring
>> lambdas to create closures.) Note that the callback is *not* a
>> designated method on the I/O object!
>
> Why isn't it? In practice, you need several callbacks: in Twisted
> parlance, you have dataReceived but also e.g. ConnectionLost
> (depending on the transport, you may even imagine other callbacks, for
> example for things happening on the TLS layer?).

Yes, but I really want to separate the callbacks from the object, so
that I don't have to inherit from an I/O object class -- asyncore
requires this and IMO it's wrong. It also makes it harder to use the
same callback code with different types of I/O objects.

>> - In systems supporting file descriptors, there's a reactor
>> implementation that knows how to use select/poll/etc., and there are
>> concrete I/O object classes that wrap file descriptors. On Windows,
>> those would only be socket file descriptors. On Unix, any file
>> descriptor would do.
>
> Windows *is* able to do async I/O on things other than sockets (see the
> discussion about IOCP). It's just that the Windows implementation of
> select() (the POSIX function call) is limited to sockets.

I know, but IOCP is currently not supported in the stdlib. I expect
that on Windows, to use IOCP, you'd need to use a different reactor
implementation and a different I/O object than the vanilla fd-based
ones. My design is actually *inspired* by the desire to support this
cleanly.

-- 
--Guido van Rossum (python.org/~guido)