[Python-Dev] PyParallel: alternate async I/O and GIL removal

[Trent Nelson]

Hi folks,

    Video of the presentation I gave last weekend at PyData NYC
    regarding PyParallel just went live: https://vimeo.com/79539317

    Slides are here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores-1

    The work was driven by the async I/O discussions around this time
    last year on python-ideas.  That resulted in me sending this:

        http://markmail.org/thread/kh3qgjbydvxt3exw#query:+page:1+mid:arua62vllzugjy2v+state:results

    ....where I attempted to argue that there was a better way of
    doing async I/O on Windows than the status quo of single-threaded,
    non-blocking I/O with an event multiplex syscall.

    I wasn't successful in convincing anyone at the time; I had no code
    to back it up and I didn't articulate my plans for GIL removal at
    the time either (figuring the initial suggestion would be met with
    enough scepticism as is).

    So, in the video above, I spend a lot of time detailing how IOCP
    works on Windows, how it presents us with a better environment than
    UNIX for doing asynchronous I/O, and how it paired nicely with the
    other work I did on coming up with a way for multiple threads to
    execute simultaneously across all cores without introducing any
    speed penalties.

    I'm particularly interested to hear if the video/slides helped
    UNIX-centric people gain a better understanding of how Windows does
    IOCP and why it would be preferable when doing async I/O.

    The reverse is also true: if you still think single-threaded, non-
    blocking synchronous I/O via kqueue/epoll is better than the
    approach afforded by IOCP, I'm interested in hearing why.

    As crazy as it sounds, my long term goal would be to try and
    influence Linux and BSD kernels to implement thread-agnostic I/O
    support such that an IOCP-like mechanism could be exposed; Solaris
    and AIX already do this via event ports and AIX's verbatim copy of
    Windows' IOCP API.

    (There is some promising work already being done on Linux; see
     recent MegaPipe paper for an example.)

    Regards,

        Trent.