[Python-ideas] Python and Concurrency

[Jim Jewett]

On 3/22/07, Ron Adam <rrr at ronadam.com> wrote:
> Josiah Carlson wrote:
> > Ron Adam <rrr at ronadam.com> wrote:
> >> Josiah Carlson wrote:

> >>> On the upside, this is possible (change the PyObject_HEAD macro,
> >>> PyINCREF, PyDECREF, remove the GIL), but the amount of work

The zillions of details are the same details you have to consider when
writing a concurrent database.  Having python store its objects
externally and retrieve them when needed (vs allocating memory and
pointers) would be a huge one-time loss, but it might eventually be
worthwhile.

PyPy calls that "external database" an "object_space", so the language
support is already there, and still will be when the hardware makes it
worthwhile.


> (1)
>     xyz = vector(r)
>     forall coords as c:     # parallel modify coords 'inplace' with body
>         c = rotate(c, xyz)
>
>     * 'with' syntax form because 'c' does not outlive the body.


Why not just keep using

    xyz = vector(r)
    rotate(xyz)

and let the compiler take care of it?

At most, we would want a way of marking objects as "read-only" or
callables as "instant", so that the compiler would know it doesn't
have to worry about the definition of "len" changing mid-stream.  (Ron
suggests something similar at the end of the message, and Talin's
Transactional metaclass is related.)

> >> * Generators with the semantics of calculating first and waiting on 'yield'
> >> for 'next', so the value is immediately returned. (depends on CPU load)

On its own, this just makes response time worse.  That said,
generators are also callables, and the compiler might do a better job
if it knew that it wouldn't have to worry about external
redefinitions.

There may also be some value in some sort of "idletasks" abstraction
that says "hey, go ahead and precompute this, but only if you've got
nothing better to do."  Garbage collection could certainly benefit
from this, and translating (or duplicating) string representations
into multiple encodings.

> In the video he also talked about avoiding locks.  I was thinking a more
> limited function object (for concurrent uses only) might be used that has:

>     * no global keyword
>     * only access to external immutable objects

Or at least, it doesn't mutate them itself, and it doesn't promise to
use newer versions that get created after the call begins.

>     * no closures.

This level of information should be useful.

But in practice, most of my functions already meet this definition.
(In theory, they access builtins that could be replaced, etc.)  I
wouldn't want to mark them all by hand.  I'm still inclined to trust
the compiler, and just accept that it may eventually be the PyPy
translator rather than the CPython interpreter.

-jJ