[Python-ideas] [Python-Dev] A bit about the GIL

[Trent Nelson]

On Mon, Apr 01, 2013 at 04:29:59PM -0700, Alfredo Solano Martínez wrote:
> >     I've been making pretty good progress with my pyparallel work.  See
> >     the initial slides here:
> >
> >         http://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async
> 
> Really interesting stuff, thanks for the link. Is there a video of the
> presentation available?

    Just the slides for now, unfortunately.

> >    I've since set up a separate mailing list for it here:
> >
> >       http://lists.snakebite.net/mailman/listinfo/pyparallel/
> >
> >    And switched to bitbucket.org for the primary repo (I still commit
> >    to hg.python.org/sandbox/trent too, though):
> >
> >        https://bitbucket.org/tpn/pyparallel
> 
> Will take a look at all that. Any ETA for the Linux version?

    Nothing formal, no.  The plan is to work out all the kinks on
    Windows, then step back, figure out the best way to abstract the
    API, then attack the POSIX implementation.

    (There are two aspects to the work; the parallel stuff, which is the
     changes to the interpreter to allow multiple threads to run CPython
     internals concurrently, and the async stuff, which will be heavily
     tied to the best IO multiplex option on the underlying platform
     (IOCP on AIX, event ports on Solaris, kqueue on *BSD, epoll on
     Linux, poll on everything else).  The parallel stuff is pretty
     platform agnostic, which is nice.  (Aside from the thread/register
     trick; but it appears as though most contemporary ISAs have some
     way of doing the same thing.))

> >     You might find this interesting, too:
> >
> >         http://hg.python.org/sandbox/trent.peps/file/6de5ed566af9/pep-async.txt
> 
> There's a lot of nice ideas there. It reminded me of the typical MPI
> workflow, with the main thread as the master process (the GIL acting
> as the barrier)  doing the scatter and gather to the other processes.
> I really liked the part about not doing any reference counting and
> just nuking everything after its done (it's the only way to be sure)

    You know, I re-read that PEP last night for the first time since I
    wrote it.  I found it quite amusing -- some things are completely
    wacky, but quite a lot of it is pretty close to how everything is
    now.  I had zero experience with CPython nitty-gritty internals when
    I wrote that, which is pretty evident from some of the things I'm
    suggesting.

    The "no refcounting and nuke everything when done" aspect has worked
    surprisingly well.  Shared-nothing code executing in a parallel
    thread absolutely flies.  Mallocs are basically free, frees are
    no-ops, no reference counting and no garbage collection; everything
    gets released in a single call when we're done.

    Factor in the heap snapshot/reset/rollback stuff in the IO loop and
    it's extremely cache friendly too.  Definitely very pleased with how
    all of that stuff is shaping up.

> Alfredo

    Regards,

        Trent.