[Python-ideas] Python and Concurrency

[Neil Toronto]

Talin wrote:
> One thing that is important to understand is that I'm not talking about 
> "automatic parallelization" where the compiler automatically figures out 
> what parts can be parallelized. That would require so much change to the 
> Python language as to make it no longer Python.
>   
...


> I am not even necessarily talking about changing the Python language 
> (although certainly the internals of the interpreter will need to be 
> changed.) The same language can be used to describe the same kinds of 
> problems and operations, but the implications of those language elements 
> will change. This is analogous to the fact that these massively 
> multicore CPUs 10 years from now will most likely be using the same 
> instruction sets as today - but that does not mean that programming as a 
> discipline will anything like what it is now.
>   

I'm not convinced you wouldn't have to change Python. After dorking 
around online for years, I've *finally* found someone who put into 
math-talk my biggest problem with current programming paradigms and how 
they relate to concurrency:

http://alarmingdevelopment.org/index.php?p=5

I don't agree with everything in the post, but this part I do:

"Most programming languages adopt a control flow model of execution, 
mirroring the hardware, in which there is an execution pointer flowing 
through the program. The primary reason for this is to permit 
side-effects to be ordered by the programmer. The problem is that 
interdependencies between side-effects are naturally a partial order, 
while control flow establishes a total (linear) order. This means that 
the actual design exists only in the programmer’s mind. It is up to the 
programmer to mentally compile (by a topological sort) these implicit 
dependencies into a total order expressed in a control flow. Whenever 
the program’s control flow is to be changed, the implicit 
interdependencies encoded into it must be remembered or guessed at in 
order to maintain them. Obviously the language should allow the partial 
order to be explicitly specified, and a compiler should take care of 
working out an execution schedule for it."

There's an elephant-in-the-living-room UI problem, here: how would one 
go about extracting a partial order from a programmer? A text editor is 
fine for a total order, but I can't think of how I'd use one non-messily 
to define a partial order. How about a Gantt chart for a partial order, 
or some other kind of dependency diagram? How would you make it as easy 
to use as a text editor? The funny thing is, once you solve this 
problem, it may even be *easier* to program this way, because rather 
than maintaining the partial order in your head (or inferring it from a 
total order in the code), it'd be right in front of you.

There's no reason a program with partial flow control couldn't have very 
Python-like syntax. After reading this, though, which formalized what 
I've long felt is the biggest problem with concurrent programming, I'd 
have to say it'd definitely not be Python itself.

For the record, I disagree strongly with the "let's keep concurrency in 
the libraries" idea. I want to program in *Python*, dangit. Or at least 
something that feels a lot like it.

Neil