Josiah Carlson wrote:
Ron Adam <rrr@ronadam.com> wrote:
Josiah Carlson wrote:
But it's not about threads, it is about concurrent execution of code (which threads in Python do not allow). The only way to allow this is to basically attach a re-entrant lock on every single Python object (depending on the platform, perhaps 12 bytes minimum for count, process, thread). The sheer volume of the number of acquire/release cycles during execution is staggering (think about the number of incref/decref operations), and the increase in size of every object by around 12 bytes is not terribly appealing.
On the upside, this is possible (change the PyObject_HEAD macro, PyINCREF, PyDECREF, remove the GIL), but the amount of work necessary to actually make it happen is huge, and it would likely result in negative performance until sheer concurrency wins out over the acquire/release overhead. It seems to me some types of operations are more suited for concurrent operations than others, so maybe new objects that are designed to be naturally usable in this way could help. Or maybe there's a way to lock groups of objects at the same time by having them share a lock if they are related?
That is a fine-grained vs. coarse-grained locking argument. There is literature.
On the other hand, there is also the pi-calculus: http://scienceblogs.com/goodmath/2007/03/an_experiment_with_calculus_an_1.ph...
Of course, the pi-calculus is not reasonable unless one starts with a lambda calculus and decides to modify it (parallel lisp?), so it isn't all that applicable here.
Interesting, but I think you are correct, pi-calculus is probably better suited to a special purpose library. Here's some more complete examples of what I was thinking. From a python programmers point of view. The underlying implementation would need to be worked out of course, either with locks, or messages, or by limiting access to mutable objects in some other way.
Thinking out loud of ways a python program may use concurrent processing:
* Applying a single function concurrently over a list. (A more limited function object might make this easier.)
(1) xyz = vector(r) forall coords as c: # parallel modify coords 'inplace' with body c = rotate(c, xyz) * 'with' syntax form because 'c' does not outlive the body.
* Feeding a single set of arguments concurrently over a list of callables.
(2) x = 42 result = [None] * len(funcs) forall (funcs, result) as (f, r): r = f(x)
* Generators with the semantics of calculating first and waiting on 'yield' for 'next', so the value is immediately returned. (depends on CPU load)
(3) This corresponds nicely to the suggested use of 'future' in the video. def foo(x=42): # Starts when process is created instead of waiting # for f.next() call to start. while 1: future yield x x += 42 f = foo() # start a concurrent process result = f.wait() # waits here for value if it's not ready.
* Listcomps that perform the same calculation on each item may be a natural multi-processing structure.
(4) result = [x = x**2 forall x in values]
These examples are all what are generally referred to as "embarassingly parallel" in literature. One serious issue with programming parallel algorithms generally is that not all algorithms are necessarily parallelizable. Some are, certainly, but not all. The task is to discover those alternate algorithms that *are* parallelizable in such a way to offer gains that are "worth it".
Yes, they are obvious, but why not start with the obvious? It's what most users will understand first and it may be the less obvious stuff can be put in terms of the more obvious "embarrassingly parallel" things.
Personally, I think that if there were a *cheap* IPC to make cross-process calls not too expensive, many of the examples above that you talk about would be handled easily.
In the video he also talked about avoiding locks. I was thinking a more limited function object (for concurrent uses only) might be used that has: * no global keyword * only access to external immutable objects * no closures. In other words these need to pass 'values' explicitly at call time and return time. (or 'future yield' time) Ron