[Python-ideas] [Python-Dev] PyParallel: alternate async I/O and GIL removal

[Nick Coghlan]

On 18 Nov 2013 08:35, "Trent Nelson" <trent at snakebite.org> wrote:
>
>     So, on the second iteration, I came up with some new concepts;
>     context persistence and object promotion.  A main-thread list or
>     dict could be async protected such that this would work:
>
>         # main thread
>         d1 = {}
>         l1 = []
>         async.protect(d1)
>         async.protect(l1)
>         # this would also work
>         d2 = async.dict()
>         l2 = async.list()
>
>         # fyi: async.rdtsc() returns a PyLong wrapped
>         # version of the CPU TSC; handy for generating
>         # non-interned objects allocated from a parallel
>         # context
>
>         def callback(name):
>             d1[name] = async.rdtsc()
>             l1.append(async.rdtsc())
>
>         async.submit_work(callback, 'foo')
>         async.submit_work(callback, 'bar')
>         async.submit_work(callback, 'moo')
>         async.run()
>
>     That actually works; the async.protect() call intercepts the
>     object's tp_as_mapping and tp_as_sequence fields and redirects them
>     to thread-safe versions that use read/write locks.  It also toggles
>     a persistence bit on both the parallel context and the parallel long
>     object, such that reference counting *is* actually enabled on it
>     once it's back in the main thread -- when the ref count drops to 0,
>     we check to see if it's an object that's been persisted, and if so,
>     we decref the original context -- when the context's refcount gets
>     to 0, only *then* do we free it.
>
>         (I also did some stuff where you could promote simple objects
>         where it made sense -- i.e. there's no need to keep a 4k context
>         around if the end result was a scalar that could be represented
>         in 50-200 bytes; just memcpy it from the main thread ("promote
>         it to a main thread object with reference counting") and free
>         the context.)

Sweet, this is basically the Rust memory model, which is the direction I'd
hoped you would end up going with this (hence why I was asking if you had
looked into the details of Rust at the PyCon language summit).

For anyone that hasn't looked at Rust, all variables are thread local by
default. There are then two mechanisms for sharing with other threads:
ownership transfer and promotion to the shared heap.

All of this is baked into the compiler, so things like trying to access an
object after sending it to another thread trigger a compile error.

PyParallel has the additional complication of remaining compatible with
standard code that assumes shared memory by default when running in serial
mode, but that appears to be a manageable problem.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20131118/316fe84d/attachment.html>