[Python-Dev] Slides from today's parallel/async Python talk
trent at snakebite.org
Fri Apr 5 15:12:23 CEST 2013
On Thu, Apr 04, 2013 at 11:53:01PM -0700, Charles-Fran?ois Natali wrote:
> >> async.submit_work(func, args, kwds, callback=None, errback=None)
> >> How do you implement arguments passing and return value?
> >> e.g. let's say I pass a list as argument: how do you iterate on the
> >> list from the worker thread without modifying the backing objects for
> >> refcounts (IIUC you use a per-thread heap and don't do any
> >> refcounting).
> > Correct, nothing special is done for the arguments (apart from
> > incref'ing them in the main thread before kicking off the parallel
> > thread (then decref'ing them in the main thread once we're sure the
> > parallel thread has finished)).
> IIUC you incref the argument from the main thread before publishing it
> to the worker thread: but what about containers like list? How do you
> make sure the refcounts of the elements don't get deallocated while
> the worker thread iterates?
Ah, so, all of my examples were missing async.run(). They should
have looked like this:
async.run() is called from the main thread, with the GIL held, and
it blocks until all parallel threads (well, parallel contexts, to be
exact) have completed. The parallel 'work' doesn't actually start
until async.run() is called either. (That's completely untrue at
the moment; async.submit_work(foo) will execute foo() in a parallel
thread immediately. Fixing that is on the todo list.)
With only parallel threads running, no main-thread objects could
ever be deallocated*, as no decref'ing is ever done.
[*]: unless you went out of your way to delete/deallocate main
thread objects via the @async.call_from_main_thread facility. At
the moment, that's firmly in the category of "Don't Do That".
(And, thinking about it a little more, I guess I could augment the
ceval loop in such a way that in order for the main thread to run
things scheduled via @async.call_from_main_thread, all parallel
threads need to be suspended. Or I could just freeze/thaw them
(although I don't know if there are POSIX counterparts to those
Windows methods). That would definitely impede performance, but
it would assure data integrity. Perhaps it should be enabled by
default, with the option to disable it for consenting adults.)
> More generally, how do you deal with non-local objects?
Read-only ops against non-local (main-thread) objects from parallel
threads are free, which is nice. Things get tricky when you try to
mutate main-thread objects from parallel threads. That's where all
the context persistence, interlocked data types, object protection
etc stuff comes in.
Is... that what you mean by how do I deal with non-local objects?
I took a guess ;-)
More information about the Python-Dev