[Python-ideas] An alternate approach to async IO

[Trent Nelson]

On Wed, Nov 28, 2012 at 02:52:56PM -0800, Guido van Rossum wrote:
>    Well, okay, please go benchmark something and don't let my ignorance of
>    async I/O on Windows discourage me.

    Great!

> (I suppose you've actually written code like this in C or C++ so you
> know it all works?)

    Not recently :-)

    (I have some helpers though: http://trent.snakebite.net/books.jpg)

>    It still looks to me like you'll have a hard time keeping 16 cores busy if
>    the premise is that you're doing *some* processing in Python (as opposed
>    to the rather unlikely use case of backing up 1GB files),

    Yeah it won't take much for Python to become the single-core
    bottleneck once this is in place.  But we can explore ways to
    address this down the track.  (I still think multiprocessing
    might be a viable approach for map/reducing all the events
    over multiple cores.)

>    but it also looks to me that, if your approach works, it could be
>    sliced into (e.g.) a Twisted reactor easily without changing
>    Twisted's high-level interfaces in any way.

    It's almost ridiculous how well suited this approach would be for
    Twisted.  Which is funny, given their aversion to Windows ;-)
    I spent a few days reviewing their overall code/approach when I
    was looking at iocpreactor, and it definitely influenced the
    approach I'm proposing.

    I haven't mentioned all the other async Windows ops (especially all
    the new ones in 8) that this approach would also support.  Juicy
    stuff like async accept, getaddrinfo, etc.  That would slot straight
    in just as nicely:

        if event.type == EventType.GetAddrInfoComplete:
            ...
        elif event.type == EventType.AcceptComplete:
            ...

>    Do you have an implementation for the "interlocked list" that you mention?

    Do MSDN docs count? ;-)

    http://msdn.microsoft.com/en-us/library/windows/desktop/ms684121(v=vs.85).aspx

    It's a pretty simple API.  Shouldn't be too hard to replicate with
    gcc/clang primitives.

        Trent.