[Python-Dev] Pythonic concurrency

[Paul Moore]

On 9/30/05, Jim Jewett <jimjjewett at gmail.com> wrote:
> Bruce Eckel wrote:
>
> > 3) Tasks are cheap enough that I can make
> > thousands of them, ...
>
> > 4) Tasks are "self-guarding," so they prevent
> > other tasks from interfering with them. The
> > only way tasks can communicate with each
> > other is through some kind of formal
> > mechanism (something queue-ish,
> > I'd imagine).
>
> I think these two are the hardest to reconcile.

Agreed. I think the biggest problem is that (4) is too strong. At the
OS level, certain issues (such as the current directory, or the stdio
streams) are per-process, so that taking (4) at face value requires
multiprocessing, which violates (3)...

If you constrain (4), you can get something much more effective. For
example, if you accept that certain things are volatile (start with
OS-specified per-process stuff, plus the Python builtins, maybe) then
it's much easier to produce solutions.

I don't think that shared state is that big an issue per se - after
all, we're all used to the idea that the contents of a file might
change "behind our backs". The real issue is not having a *clearly
defined* shared state.

The big problem with threads is that the shared state is *everything*.
There's no partitioning at all. Every thread-based abstraction is
based around threads voluntarily restricting themselves to a limited
set of "safe" communication operations, and otherwise scrupulously
avoiding each other's space ("Don't sit there, that's auntie Mary's
chair"...)

If we had an abstraction (multiple interpreters, anyone?) which still
had a shared state, but a much smaller one, which was defined clearly,
such that "unsafe" operations were easy to identify, then day-to-day
concurrent programming would be a lot easier. Yes, it's still possible
to break things, but people can do that already by hacking about with
builtins, or abusing the introspection API, and it's not made Python
unusable, because they generally don't...

This seems to me to be a perfect case for a "Practicality beats
Purity" approach.

Paul.