[Python-ideas] The async API of the future

[Glyph]

On Oct 20, 2012, at 4:26 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Glyph wrote:
> 
>> The main interfaces you need are here:
>> 
>> <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.ITransport.html>
>> <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IProtocol.html>
>> <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IConsumer.html>
>> <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IProducer.html>
> 
> These don't look anywhere near adequate to me. How do I make
> a sendmsg() call on a unix-domain socket and pass access rights
> over it? How do I do a readdir() on a file descriptor representing
> a directory? Etc.

You don't.  Notice I didn't even include basic datagram transports in those interfaces, and those are relatively straightforward compared to your questions.

Antoine already cited the answer to your first question - on POSIX-y operating systems, you add another interface, something like <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IUNIXTransport.html#sendFileDescriptor>.

Except, the first question doesn't even make sense as posed on Windows.  Windows doesn't have any support for AF_UNIX/AF_LOCAL; if you want to send a "file descriptor" (read: object handle) to another process, you have to have either use DuplicateHandle or WSADuplicateSocket.  Note that these work differently, so it depends which type of "file descriptor" you're trying to pass; if you are passing a socket you need a PID, if you're passing an anonymous pipe you need a process handle.

The second question doesn't make sense anywhere.  readdir() is blocking, always and forever; opendir() doesn't take flags and you can't effectively or portably set O_NONBLOCK on a directory descriptor with ioctls.  All filesystem operations also block, for all practical purposes.  So, really your question reverts to "how does one integrate a thread pool with an event loop", to which the answer is <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorThreads.html>.

Of course, all of these operations _can_ be made 'really' non-blocking with sufficient terrifyingly deep platform-specific knowledge.  A DIR* is just a file descriptor, eventually, and readdir() is eventually some kind of syscall on it.  POSIX AIO operations might be used to read without blocking[1].

However, trying to get consensus about, standardize, and implement every possible I/O operation before implementing the core I/O loop interface for sockets and timed calls is pretty extreme cart-before-horse-putting.  People implemented gazillions of amazing networking applications in Python over the past two decades, despite the fact that it only even got sendmsg support recently.  Heck, even Twisted's support of sendmsg and file-descriptor sending is relatively recent.  I realize that this list is dedicated to the discussion of all proposals regardless of how radical and insane they might be, but that doesn't mean that *every* proposal must have its scope expanded until it is as radical and insane as possible.

The fact that Twisted has so many separate interfaces is not a coincidence.  We took an explicitly layered approach to building the main loop, so that anyone who needed an esoteric I/O operation could always write their own platform-specific handler.  Anyone building an application that requires that layer will probably need to write something specific to their operating system and their preferred framework (Twisted, Tornado, the stdlib loop whether it's based on asyncore or not, etc).

I believe that trying to cover every case in advance so they won't have to use an escape hatch and then not providing such an escape hatch is just going to make the API huge and confusing for newcomers and frustratingly limiting for people with really advanced use-cases.

-glyph

[1]: Actually, no, they can't: <http://stackoverflow.com/questions/87892/what-is-the-status-of-posix-asynchronous-i-o-aio?rq=1>.  But maybe one day this would be consistently implemented.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121021/8b2562ad/attachment.html>