[Python-Dev] Slides from today's parallel/async Python talk

Trent Nelson trent at snakebite.org
Fri Apr 5 15:12:23 CEST 2013


On Thu, Apr 04, 2013 at 11:53:01PM -0700, Charles-Fran?ois Natali wrote:
> Hello,
> 
> >> async.submit_work(func, args, kwds, callback=None, errback=None)
> >>
> >> How do you implement arguments passing and return value?
> >>
> >> e.g. let's say I pass a list as argument: how do you iterate on the
> >> list from the worker thread without modifying the backing objects for
> >> refcounts (IIUC you use a per-thread heap and don't do any
> >> refcounting).
> >
> >     Correct, nothing special is done for the arguments (apart from
> >     incref'ing them in the main thread before kicking off the parallel
> >     thread (then decref'ing them in the main thread once we're sure the
> >     parallel thread has finished)).
> 
> IIUC you incref the argument from the main thread before publishing it
> to the worker thread: but what about containers like list? How do you
> make sure the refcounts of the elements don't get deallocated while
> the worker thread iterates?

    Ah, so, all of my examples were missing async.run().  They should
    have looked like this:

        async.submit_work(foo)
        async.submit_work(bar)
        async.run()

    async.run() is called from the main thread, with the GIL held, and
    it blocks until all parallel threads (well, parallel contexts, to be
    exact) have completed.  The parallel 'work' doesn't actually start
    until async.run() is called either.  (That's completely untrue at
    the moment; async.submit_work(foo) will execute foo() in a parallel
    thread immediately.  Fixing that is on the todo list.)

    With only parallel threads running, no main-thread objects could
    ever be deallocated*, as no decref'ing is ever done.

    [*]: unless you went out of your way to delete/deallocate main
    thread objects via the @async.call_from_main_thread facility.  At
    the moment, that's firmly in the category of "Don't Do That".

    (And, thinking about it a little more, I guess I could augment the
     ceval loop in such a way that in order for the main thread to run
     things scheduled via @async.call_from_main_thread, all parallel
     threads need to be suspended.  Or I could just freeze/thaw them
     (although I don't know if there are POSIX counterparts to those
     Windows methods).  That would definitely impede performance, but
     it would assure data integrity.  Perhaps it should be enabled by
     default, with the option to disable it for consenting adults.)

> More generally, how do you deal with non-local objects?

    Read-only ops against non-local (main-thread) objects from parallel
    threads are free, which is nice.  Things get tricky when you try to
    mutate main-thread objects from parallel threads.  That's where all
    the context persistence, interlocked data types, object protection
    etc stuff comes in.

    Is... that what you mean by how do I deal with non-local objects?
    I took a guess ;-)

    Regards,

        Trent.


More information about the Python-Dev mailing list