[Python-ideas] Fwd: Concurrent safety?

[Nick Coghlan]

On Tue, Nov 1, 2011 at 6:01 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>  > I've identified the problem I want to solve: I want to make
>  > concurrent use of python objects "safe by default",
>
> But that's not what you've proposed, AIUI.  You've proposed making
> concurrent use *safer*, but not yet *safe*.  That's quite different
> from the analogy with automatic memory management, where the
> programmer can't do anything dangerous with pointers (because they
> can't do anything at all).  The analogous model for concurrency is
> processes, it seems to me.  (I don't have a desperate need for high-
> performance concurrency, so I take no position on processes + message
> passing vs. threads + shared resources.)

Guido and python-dev in general *have* effectively taken a position on
that, though (mainly due to Global Interpreter Lock discussions).

1. Even for threads, the recommended approach is to use queue.Queue to
avoid the common concurrency issues (such as race conditions and
deadlock) associated with explicit locking
2. In Python 3, concurrent.futures offers an even *safer* interface
and higher level interface for many concurrent workloads
3. If you use multiple processes and serialised messages, or higher
level APIs like concurrent.futures, you can not only scale to multiple
cores, but also to multiple *machines*.

This has led to a quite deserved reputation for being intolerant of
changes that claim to make multithreaded development "better", but
only at the expense of making single-threaded development worse.
That's actually one of the more interesting aspects of PyPy's
experiments with software transactional memory - the sophistication of
their toolchain means that they can make the STM feature optional at
the translation stage, without resorting to ugly #ifdef hackery the
way we would need to in order to make such a feature similarly
optional in CPython.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia