On Fri, Oct 12, 2012 at 11:33 AM, Antoine Pitrou solipsis@pitrou.net wrote:
On Fri, 12 Oct 2012 11:13:23 -0700 Guido van Rossum guido@python.org wrote:
OTOH someone else might prefer a buffered stream abstraction that just keeps filling its read buffer (and draining its write buffer) using level-triggered callbacks, at least up to a certain buffer size -- we have to be robust here and make it impossible for an evil client to fill up all our memory without our approval!
I'd like to know what a sane buffered API for non-blocking I/O may look like, because right now it doesn't seem to make a lot of sense. At least this bug is tricky to resolve: http://bugs.python.org/issue13322
Good question. It actually depends quite a bit on whether you have an event loop or not -- with the help of an event loop, you can have a level-triggered callback that fills the buffer behind your back (up to a given limit, at which point it should unregister the I/O object); that bug seems to be about a situation without an event loop, where you can't do that. Also the existing io module design never anticipated cooperation with an event loop.
- There's an abstract Reactor class and an abstract Async I/O object
class. To get a reactor to call you back, you must give it an I/O object, a callback, and maybe some more stuff. (I have gone back and like passing optional args for the callback, rather than requiring lambdas to create closures.) Note that the callback is *not* a designated method on the I/O object!
Why isn't it? In practice, you need several callbacks: in Twisted parlance, you have dataReceived but also e.g. ConnectionLost (depending on the transport, you may even imagine other callbacks, for example for things happening on the TLS layer?).
Yes, but I really want to separate the callbacks from the object, so that I don't have to inherit from an I/O object class -- asyncore requires this and IMO it's wrong. It also makes it harder to use the same callback code with different types of I/O objects.
- In systems supporting file descriptors, there's a reactor
implementation that knows how to use select/poll/etc., and there are concrete I/O object classes that wrap file descriptors. On Windows, those would only be socket file descriptors. On Unix, any file descriptor would do.
Windows *is* able to do async I/O on things other than sockets (see the discussion about IOCP). It's just that the Windows implementation of select() (the POSIX function call) is limited to sockets.
I know, but IOCP is currently not supported in the stdlib. I expect that on Windows, to use IOCP, you'd need to use a different reactor implementation and a different I/O object than the vanilla fd-based ones. My design is actually *inspired* by the desire to support this cleanly.