[Python-Dev] Slides from today's parallel/async Python talk

Trent Nelson trent at snakebite.org
Thu Apr 4 22:04:41 CEST 2013


Hi Charles-François,

On Thu, Apr 04, 2013 at 01:18:58AM -0700, Charles-François Natali wrote:
> Just a quick implementation question (didn't have time to read through
> all your emails :-)
> 
> async.submit_work(func, args, kwds, callback=None, errback=None)
> 
> How do you implement arguments passing and return value?
>
> e.g. let's say I pass a list as argument: how do you iterate on the
> list from the worker thread without modifying the backing objects for
> refcounts (IIUC you use a per-thread heap and don't do any
> refcounting).

    Correct, nothing special is done for the arguments (apart from
    incref'ing them in the main thread before kicking off the parallel
    thread (then decref'ing them in the main thread once we're sure the
    parallel thread has finished)).

> Same thing for return value, how do you pass it to the
> callback?

    For submit_work(), you can't :-)  In fact, an exception is raised if
    the func() or callback() or errback() attempts to return a non-None
    value.

    It's worth noting that I eventually plan to have the map/reduce-type
    functionality (similar to what multiprocessing offers) available via
    a separate 'parallel' façade.  This will be geared towards programs
    that are predominantly single-threaded, but have lots of data that
    can be processed in parallel at various points.

    Now, with that being said, there are a few options available at the
    moment if you want to communicate stuff from parallel threads back
    to the main thread.  Originally, you could do something like this:

        d = async.dict()
        def foo():
            d['foo'] = async.rdtsc()
        def bar():
            d['bar'] = async.rdtsc()

        async.submit_work(foo)
        async.submit_work(bar)

    But I recently identified a few memory-management flaws with that
    approach (I'm still on the fence with this issue... initially I was
    going to drop all support, but I've since had ideas to address the
    memory issues, so, we'll see).

    There's also this option:

        d = dict()

        @async.call_from_main_thread_and_wait
        def store(k, v):
            d[str(k)] = str(v)

        def foo():
            store('foo', async.rdtsc())

        def bar():
            store('bar', async.rdtsc())

        async.submit_work(foo)
        async.submit_work(bar)

    (Not a particularly performant option though; the main-thread
     instantly becomes the bottleneck.)

    Post-PyCon, I've been working on providing new interlocked data
    types that are specifically designed to bridge the parallel/main-
    thread divide:

        xl = async.xlist()
        def foo():
            xl.push(async.rdtsc())
        def bar():
            xl.push(async.rdtsc())

        async.submit_work(foo)
        async.submit_work(bar)

        while True:
            x = xl.pop()
            if not x:
                break
            process(x)

    What's interesting about xlist() is that it takes ownership of the
    parallel objects being pushed onto it.  That is, it basically clones
    them, using memory allocated from its own internal heap (allowing
    the parallel-thread's context heap to be freed, which is desirable).

    The push/pop operations are interlocked at the C level, which
    obviates the need for any explicit locking.

    I've put that work on hold for now though; I want to finish the
    async client/server stuff (it's about 60-70% done) first.  Once
    that's done, I'll tackle the parallel.*-type façade.

        Trent.


More information about the Python-Dev mailing list