[concurrency] Issues with the current concurrency mechanisms in Python

[Mike Meyer]

On Wed, 15 Feb 2012 13:11:27 +0200
Christopher Reay <christopherreay at gmail.com> wrote:
> Have you checked out the stuff on STM going on in pypy-dev?

I've seen some of the comments on it, and have checking on what's
really going on in the list of things to do.

> If you think it is in topic, Ill go and try and dig out the STM thread from
> pypy an post it somewhere

I certainly think it is, *especially* if they're going to expose STM
at the Python level. Adding better tools for dealing with concurrency
in general is the fourth large goal. It may be the hardest to deal
with, as it certainly involves language changes. But it may be
required for some of the others as well.

> > 2) Concurrent access to mutable objects is *much*, *much* harder than
> >   memory management.
> > 3) Threading (whether done with OS or userland threads) expose every object
> >   in the program to concurrent access.
> This is in contrast with what you mentioned here, about concurrent access
> to objects, where, in the standard threading/lock model, all mutations are
> carried out in a non-deterministic space (which is generally not what
> people want. It can take developers some time to learn and understand that
> they are operating in a non-deterministic space which isnt convergent with
> their imagination of what "should" be happening).

Right. STM is one of the possible fixes. I'm used to seeing it with
*other* tools as well, though.

> As far as I can dig it, the requirement for managing concurrent access to
> objects within a threaded application is some kind of reflection of the
> requirement for sharing concurrent access to objects across processes.

IIUC, you're right. Concurrent access is a PITA to deal with - at
least if you have to lock everything by hand. It doesn't matter how
that happens, whether it's because threading exposes everything, or
you're using explicitly shared objects between processes.

Threading makes it worse because *everything* is shared, so you have
to worry about it everywhere. I've seen a couple of cases where module
A decides it wants to use threads for something, then module B starts
having concurrency problems because it's not thread safe.

> The issues here are that threading (certainly for Python) is
> fundamentally limited to running on a single core, and can require
> dancing around the GIL if multiple processes are accessing our
> python state concurrently.

This I either don't parse, or don't agree with. Python code can't run
concurrently with other Python code - the GIL prevents it. PyPy with
STM or Jython (which uses fine-grained locks) and C extensions in
cPython that release the GIL can run concurrently, potentially on
multiple cores.

     <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org