Talin wrote:

My overall goal here is to be able to continue writing programs in Python 10 years from now, not just as a hobby, but as part of my professional work. If Python is able to leverage the power of the CPUs that are being created at that time, I will be able to make a strong case for doing so. On the other hand, if I have a 128-core CPU on my desk, and Python is only able to utilize 1/128th of that power without resorting to tedious calculations of race conditions and deadlocks, then its likely that my Python programming will be relegated to the role of a hobby.

-- Talin

A Very nice introduction Talin. I will certainly look at the video tomorrow morning. Thanks. My first thought is it's not quite as bad as it seems, because any third party extensions will be able to use the remaining 127/128th of the power. ;-) You would need to subtract any CPU's used by the OS and other concurrent processes. (These would probably continue to use more resources as well.) I wonder if some of cpu's would be definable for special purposes or not? Maybe 64 of them set aside for simultaneous parallel calculations? There may be a way to express to the OS that a particular process on a data structure be carried out as 'Wide' as possible. ('Narrow' as possible being on a single CPU in a single Thread.) It might be vary nice for these 'wide' structures to have their own API as well. All speculation of course, ;-) Cheers, Ron

"Lucio Torre" wrote:

...

I'm sure that there's a lot more like that out there. However, there is also a lot of stuff out there that is *superficially* similar to what I am talking about, and I want to make the distinction clear. For example, any discussion of concurrency in Python will naturally raise the topic of both IPython and Stackless. However, IPython (from what I understand) is really about distributed computing and not so much about fine-grained concurrency; And Stackless (from what I understand) is really about coroutines or continuations, which is a different kind of concurrency. Unless I am mistaken (and I very well could be) neither of these are immediately applicable to the problem of authoring Python programs for multi-core CPUs, but I think that both of them contain valuable ideas that are worth discussing.

From what i understand, i think that the main contribution of the stackless aproach to concurrency is microthreads: The ability to have lots and lots of cheap threads. If you want to program for some huge amount of cores, you will have to have even more threads than cores you have today.

But it's not about threads, it is about concurrent execution of code (which threads in Python do not allow). The only way to allow this is to basically attach a re-entrant lock on every single Python object (depending on the platform, perhaps 12 bytes minimum for count, process, thread). The sheer volume of the number of acquire/release cycles during execution is staggering (think about the number of incref/decref operations), and the increase in size of every object by around 12 bytes is not terribly appealing. On the upside, this is possible (change the PyObject_HEAD macro, PyINCREF, PyDECREF, remove the GIL), but the amount of work necessary to actually make it happen is huge, and it would likely result in negative performance until sheer concurrency wins out over the acquire/release overhead. - Josiah

Josiah Carlson wrote:

"Lucio Torre" wrote:

...

...

I'm sure that there's a lot more like that out there. However, there is also a lot of stuff out there that is *superficially* similar to what I am talking about, and I want to make the distinction clear. For example, any discussion of concurrency in Python will naturally raise the topic of both IPython and Stackless. However, IPython (from what I understand) is really about distributed computing and not so much about fine-grained concurrency; And Stackless (from what I understand) is really about coroutines or continuations, which is a different kind of concurrency. Unless I am mistaken (and I very well could be) neither of these are immediately applicable to the problem of authoring Python programs for multi-core CPUs, but I think that both of them contain valuable ideas that are worth discussing. From what i understand, i think that the main contribution of the stackless aproach to concurrency is microthreads: The ability to have lots and lots of cheap threads. If you want to program for some huge amount of cores, you will have to have even more threads than cores you have today.

But it's not about threads, it is about concurrent execution of code (which threads in Python do not allow). The only way to allow this is to basically attach a re-entrant lock on every single Python object (depending on the platform, perhaps 12 bytes minimum for count, process, thread). The sheer volume of the number of acquire/release cycles during execution is staggering (think about the number of incref/decref operations), and the increase in size of every object by around 12 bytes is not terribly appealing.

On the upside, this is possible (change the PyObject_HEAD macro, PyINCREF, PyDECREF, remove the GIL), but the amount of work necessary to actually make it happen is huge, and it would likely result in negative performance until sheer concurrency wins out over the acquire/release overhead.

It seems to me some types of operations are more suited for concurrent operations than others, so maybe new objects that are designed to be naturally usable in this way could help. Or maybe there's a way to lock groups of objects at the same time by having them share a lock if they are related? I imagine there will be some low level C support that could be used transparently, such as copying large areas of memory with multiple CPU's. These may even be the existing C copy functions reimplemented to take advantage of multiple CPU environments so new versions of python may have limited use of this even if no support is explicitly added. Thinking out loud of ways a python program may use concurrent processing: * Applying a single function concurrently over a list. (A more limited function object might make this easier.) * Feeding a single set of arguments concurrently over a list of callables. * Generators with the semantics of calculating first and waiting on 'yield' for 'next', so the value is immediately returned. (depends on CPU load) * Listcomps that perform the same calculation on each item may be a natural multi-processing structure. Ron

On 3/22/07, Jason Orendorff wrote:

On 3/22/07, Talin wrote:

...

I would like to direct your attention to a presentation that was given by Mr. Herb Sutter on the subject of concurrency support in programming languages. It's a long video, but I would urge you to watch at least the first five minutes of it:

http://video.google.com/videoplay?docid=7625918717318948700&q=herb+sutter

Thanks for the excellent summary.

I won't try to summarize Brendan Eich's February blog entry on this topic, because it's short enough and worth reading:

http://weblogs.mozillazine.org/roadmap/archives/2007/02/threads_suck.html

Right. I saw Sutter give this talk (or a very similar one) in Oxford last April, and I'm thoroughly unconvinced that Python is doomed unless it adds more thread support. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Ron Adam wrote:

Thinking out loud of ways a python program may use concurrent processing:

* Applying a single function concurrently over a list. * Feeding a single set of arguments concurrently over a list of callables. * Generators with the semantics of calculating first and waiting on 'yield' * Listcomps that perform the same calculation on each item

The solution is obvious -- just add a 'par' statement. Then-we-can-call-it-Poccam-ly, Greg

Josiah Carlson wrote:

Ron Adam wrote:

...

Josiah Carlson wrote:

...

But it's not about threads, it is about concurrent execution of code (which threads in Python do not allow). The only way to allow this is to basically attach a re-entrant lock on every single Python object (depending on the platform, perhaps 12 bytes minimum for count, process, thread). The sheer volume of the number of acquire/release cycles during execution is staggering (think about the number of incref/decref operations), and the increase in size of every object by around 12 bytes is not terribly appealing.

On the upside, this is possible (change the PyObject_HEAD macro, PyINCREF, PyDECREF, remove the GIL), but the amount of work necessary to actually make it happen is huge, and it would likely result in negative performance until sheer concurrency wins out over the acquire/release overhead. It seems to me some types of operations are more suited for concurrent operations than others, so maybe new objects that are designed to be naturally usable in this way could help. Or maybe there's a way to lock groups of objects at the same time by having them share a lock if they are related?

That is a fine-grained vs. coarse-grained locking argument. There is literature.

On the other hand, there is also the pi-calculus: http://scienceblogs.com/goodmath/2007/03/an_experiment_with_calculus_an_1.ph...

Of course, the pi-calculus is not reasonable unless one starts with a lambda calculus and decides to modify it (parallel lisp?), so it isn't all that applicable here.

Interesting, but I think you are correct, pi-calculus is probably better suited to a special purpose library. Here's some more complete examples of what I was thinking. From a python programmers point of view. The underlying implementation would need to be worked out of course, either with locks, or messages, or by limiting access to mutable objects in some other way.

...

Thinking out loud of ways a python program may use concurrent processing:

* Applying a single function concurrently over a list. (A more limited function object might make this easier.)

(1) xyz = vector(r) forall coords as c: # parallel modify coords 'inplace' with body c = rotate(c, xyz) * 'with' syntax form because 'c' does not outlive the body.

...

* Feeding a single set of arguments concurrently over a list of callables.

(2) x = 42 result = [None] * len(funcs) forall (funcs, result) as (f, r): r = f(x)

...

* Generators with the semantics of calculating first and waiting on 'yield' for 'next', so the value is immediately returned. (depends on CPU load)

(3) This corresponds nicely to the suggested use of 'future' in the video. def foo(x=42): # Starts when process is created instead of waiting # for f.next() call to start. while 1: future yield x x += 42 f = foo() # start a concurrent process result = f.wait() # waits here for value if it's not ready.

...

* Listcomps that perform the same calculation on each item may be a natural multi-processing structure.

(4) result = [x = x**2 forall x in values]

These examples are all what are generally referred to as "embarassingly parallel" in literature. One serious issue with programming parallel algorithms generally is that not all algorithms are necessarily parallelizable. Some are, certainly, but not all. The task is to discover those alternate algorithms that *are* parallelizable in such a way to offer gains that are "worth it".

Yes, they are obvious, but why not start with the obvious? It's what most users will understand first and it may be the less obvious stuff can be put in terms of the more obvious "embarrassingly parallel" things.

Personally, I think that if there were a *cheap* IPC to make cross-process calls not too expensive, many of the examples above that you talk about would be handled easily.

In the video he also talked about avoiding locks. I was thinking a more limited function object (for concurrent uses only) might be used that has: * no global keyword * only access to external immutable objects * no closures. In other words these need to pass 'values' explicitly at call time and return time. (or 'future yield' time) Ron

On 3/22/07, Ron Adam wrote:

Josiah Carlson wrote:

...

Ron Adam wrote:

...

Josiah Carlson wrote:

...

...

...

On the upside, this is possible (change the PyObject_HEAD macro, PyINCREF, PyDECREF, remove the GIL), but the amount of work

The zillions of details are the same details you have to consider when writing a concurrent database. Having python store its objects externally and retrieve them when needed (vs allocating memory and pointers) would be a huge one-time loss, but it might eventually be worthwhile. PyPy calls that "external database" an "object_space", so the language support is already there, and still will be when the hardware makes it worthwhile.

(1) xyz = vector(r) forall coords as c: # parallel modify coords 'inplace' with body c = rotate(c, xyz)

* 'with' syntax form because 'c' does not outlive the body.

Why not just keep using xyz = vector(r) rotate(xyz) and let the compiler take care of it? At most, we would want a way of marking objects as "read-only" or callables as "instant", so that the compiler would know it doesn't have to worry about the definition of "len" changing mid-stream. (Ron suggests something similar at the end of the message, and Talin's Transactional metaclass is related.)

...

...

* Generators with the semantics of calculating first and waiting on 'yield' for 'next', so the value is immediately returned. (depends on CPU load)

On its own, this just makes response time worse. That said, generators are also callables, and the compiler might do a better job if it knew that it wouldn't have to worry about external redefinitions. There may also be some value in some sort of "idletasks" abstraction that says "hey, go ahead and precompute this, but only if you've got nothing better to do." Garbage collection could certainly benefit from this, and translating (or duplicating) string representations into multiple encodings.

In the video he also talked about avoiding locks. I was thinking a more limited function object (for concurrent uses only) might be used that has:

* no global keyword * only access to external immutable objects

Or at least, it doesn't mutate them itself, and it doesn't promise to use newer versions that get created after the call begins.

* no closures.

This level of information should be useful. But in practice, most of my functions already meet this definition. (In theory, they access builtins that could be replaced, etc.) I wouldn't want to mark them all by hand. I'm still inclined to trust the compiler, and just accept that it may eventually be the PyPy translator rather than the CPython interpreter. -jJ

Thinking more about this, it seems to me that discussions of syntax for doing parallel operations and nifty classes for synchronization are a bit premature. The real question, it seems to me, is how to get Python to operate concurrently at all. Python's GIL cannot be removed without going through and fixing a thousand different places where different threads might access shared variables within the interpreter. Clearly, that's not the kind of work that is going to be done in a month or less. It might be useful to decompose that task further into some digestible chunks. One of these chunks is garbage collection. It seems to me that reference counting as it exists today would is a serious hindrance to concurrency, because it requires writing to an object each time you create a new reference. Instead, it should be possible to pass a reference to an object between threads, without actually modifying the object unless one of the threads actually changes an attribute. There are a number of papers on concurrent garbage collection out there on the web that might serve as a useful starting point. Of course, the .Net CLR and Java VM already have collectors of this type, so maybe those versions of Python already get this for free. I also wonder what other things that the GIL is protecting can be broken out as large, coherent chunks. -- Talin

Aahz wrote:

On Sun, Mar 25, 2007, Talin wrote:

...

Thinking more about this, it seems to me that discussions of syntax for doing parallel operations and nifty classes for synchronization are a bit premature. The real question, it seems to me, is how to get Python to operate concurrently at all.

Maybe that's what it seems to you; to others of us who have been looking at this problem for a while, the real question is how to get a better multi-process control and IPC library in Python, preferably one that is cross-platform. You can investigate that right now, and you don't even need to discuss it with other people.

(Despite my oft-stated fondness for threading, I do recognize the problems with threading, and if there were a way to make processes as simple as threads from a programming standpoint, I'd be much more willing to push processes.)

I generally agree. XML-RPC works pretty well in this regard, though as I talked about a couple months ago, it's transport format encoding and decoding result in overhead during calling that leaves something to be desired. - Josiah

Talin wrote:

Thinking more about this, it seems to me that discussions of syntax for doing parallel operations and nifty classes for synchronization are a bit premature.

Yes, but it may help to have a few possible "end user" examples in mind while you work on the problem. The point isn't the exact syntax, but the uses that they imply and what other requirements these examples need in order to work.

The real question, it seems to me, is how to get Python to operate concurrently at all.

Yes, and I agree that it would be better to split the problem into smaller chunks. Maybe start by finding some "easier" to do simple cases first.

Python's GIL cannot be removed without going through and fixing a thousand different places where different threads might access shared variables within the interpreter. Clearly, that's not the kind of work that is going to be done in a month or less.

Here is an older discussion on removing the GIL and multiprocessing vs threading. Maybe it will be of help? http://mail.python.org/pipermail/python-dev/2005-September/056423.html

It might be useful to decompose that task further into some digestible chunks. One of these chunks is garbage collection. It seems to me that reference counting as it exists today would is a serious hindrance to concurrency, because it requires writing to an object each time you create a new reference. Instead, it should be possible to pass a reference to an object between threads, without actually modifying the object unless one of the threads actually changes an attribute.

I'm not familiar with the details of python's garbage collecting yet. I was thinking the problem may be simplified by having a single mutable container-cell object. But that wouldn't be enough because in python it isn't just a problem of a mutable object changing, but one of a names reference to an object being changed. Could an explicit name space for sharing in threads and processes help?

There are a number of papers on concurrent garbage collection out there on the web that might serve as a useful starting point. Of course, the .Net CLR and Java VM already have collectors of this type, so maybe those versions of Python already get this for free.

I also wonder what other things that the GIL is protecting can be broken out as large, coherent chunks.

I have no idea, ... Ron

Talin wrote:

Aahz wrote:

...

On Sun, Mar 25, 2007, Talin wrote:

...

Thinking more about this, it seems to me that discussions of syntax for doing parallel operations and nifty classes for synchronization are a bit premature. The real question, it seems to me, is how to get Python to operate concurrently at all.

Maybe that's what it seems to you; to others of us who have been looking at this problem for a while, the real question is how to get a better multi-process control and IPC library in Python, preferably one that is cross-platform. You can investigate that right now, and you don't even need to discuss it with other people.

If you mean some sort of inter-process messaging system, there are a number that already exist; I'd look at IPython and py-globus for starters.

My feeling is that while such an approach is vastly easier from the standpoint of Python developers, and may be easier from the standpoint of a typical Python programmer, it doesn't actually solve the problem that I'm attempting to address, which is figuring out how to write client-side software that dynamically scales to the number of processors on the system.

At some point either the user or the system (Python) needs to figure out that splitting up a sequential task into multiple parallel tasks is productive. On the system end of things, that isn't easy. How much money and time has been poured into C/C++ compiler development, and about all they can auto parallelize (via vector operations) are things like: for (i=0;i

My view is that while the number of algorithms that we have that can be efficiently parallelized in a fine-grained threading environment is small (compared to the total number of strictly sequential algorithms), the number of algorithms that can be adapted to heavy-weight, coarse-grained processes is much smaller still.

For example, it is easy to imagine a quicksort routine where different threads are responsible for sorting various sub-partitions of the array. If this were to be done via processes, the overhead of marshalling and unmarshalling the array elements would completely swamp the benefits of making it concurrent.

But that algorithm wouldn't be used for sorting data on multiple processors. A variant of mergesort would be used (distribute blocks equally to processors, sort them individually, merge the results - in parallel). But again, all of this relies on two things: 1. a method for executing multiple streams of instructions simultaneously 2. a method of communication between the streams of instructions Without significant work, #1 isn't possible using threads in Python. It is trivial using processes. Without work, #2 isn't "fast" using processes in Python. It is trivial using threads. But here's the thing: with work, #2 can be made fast. Using unix domain sockets (on linux, 3.4 ghz P4 Xeons, DDR2-PC4200 memory (you can get 50% faster memory nowadays)), I've been able to push 400 megs/second between processes. Maybe anonymous or named pipes, or perhaps a shared mmap with some sort of synchronization would allow for IPC to be cross platform and just about as fast. The reason that I (and perhaps others) have been pushing for IPC is because it is easier to solve than the removal of Pythons threading limitations, with many of the same benefits, and even a few extra (being able to distribute processes across different machines). - Josiah

On 3/28/07, Greg Ewing wrote:

I was thinking about thread killing, and why we think it's okay to kill OS processes but not threads.

[Suggestion for a Queue variant that knows when one end is dead] I think the bigger problem is when threads don't restrict themselves to queues, but just use the same memory directly. If a thread dies in the middle of an "atomic" action, other threads will see corrupted memory. If a process dies then, nobody else would be able to see the memory anyhow. What we really need is a Task object that treats shared memory (perhaps with a small list of specified exceptions) as immutable. If you're willing to rely on style guidelines, then you can already get this today. If you want safety, and efficiency ... that may be harder to do as an addon. -jJ

Jim Jewett wrote:

On 3/28/07, Greg Ewing wrote:

...

I was thinking about thread killing, and why we think it's okay to kill OS processes but not threads.

[Suggestion for a Queue variant that knows when one end is dead]

I think the bigger problem is when threads don't restrict themselves to queues, but just use the same memory directly. If a thread dies in the middle of an "atomic" action, other threads will see corrupted memory. If a process dies then, nobody else would be able to see the memory anyhow.

What we really need is a Task object that treats shared memory (perhaps with a small list of specified exceptions) as immutable.

I think a set of all of these as tools would be good. They are all different parts of the same elephant. And I don't see why it needs to be a single unified thing. * A 'better' task object for easily creating tasks. + We have a threading object now. (Needs improving.) * A message mechanism (Que-pipe) for getting the status of a task. + In and out message ques for communicating with a thread. + A way to wait on task events (messages) nicely. + A way for exceptions to propagate out of task objects. * Shared memory - + Prevent names from being rebound + Prevent objects from being altered (It isn't just about objects being immutable, but also about name's not being rebound to other objects. Unless you want to pass objects references for every object to every task? Or you trust that any imported code will play nice.) Would the following be a good summery of the issues? (Where the terms used mean) frozen: object can't be altered while frozen locked: name can't be rebound to another object Threading issues: (In order of difficulty) 1. Pass immutable objects back and forth. + Works now. 2. Share immutable objects by-ref. + Works now. 3. Pass mutable "deep" copies back and forth. ? Works now. (but not for all objects?) 4. Pass frozen mutable objects. - Needs freezable/unfreezable mutable objects. (Not the same as making an immutable copy.) 5. Share immutable object in a shared name space. - Needs name locking. 6. Share "frozen" mutable objects in shared name space. - Needs name locking - Needs freezable mutable objects 7. pass mutable objects. - Has conflicts with shared access. 8. Share mutable object by-ref. - Has conflicts. (same as #7.) 9. Share mutable object in shared name space. - Needs name locking. - Has conflicts. If we can make the first 6 of these work, that may be enough. 7,8 and 9 have to do with race conditions and other simultaneous data access issues. Name locking might work like this: Doing 'lock <name>' and 'unlock <name>' could just move the name to and from a locked name space in the same scope. Attempting to rebind a name while it's in a locked name space would raise an exception. The point is, rather than attach a lock to each individual name, it may be much easier, faster, and save memory by having a new name spaces for this. You could also pass a locked-name-space reference to a task all at once like we pass locals or globals now. It doesn't identify who locked what, so unlocking a name used by a thread would be in the "if it hurts, don't do that" category. If locking or unlocking a name in outside scopes is disallowed, then knowing who locked what won't be a problem. Freezing would be some way of preventing an object from being changed. It isn't concerned with the name it is referenced with. And it should not require copying the object. Part of that may be made easier by locking names.(?) Frozen objects may be useful in other ways besides threading.(?) Names locked to immutable objects act like constants so they may have other uses as well. Cheers, Ron

If you're willing to rely on style guidelines, then you can already get this today.

If you want safety, and efficiency ... that may be harder to do as an addon.

Jim Jewett wrote:

On 3/29/07, Ron Adam wrote:

...

Jim Jewett wrote:

...

What we really need is a Task object that treats shared memory (perhaps with a small list of specified exceptions) as immutable.

...

* A 'better' task object for easily creating tasks. + We have a threading object now. (Needs improving.)

But the task isn't in any way restricted. Brett's security sandbox might be a useful starting point, if it is fast enough. Otherwise, we'll probably need to stick with microthreading to get things small enough to contain.

...

* Shared memory - + Prevent names from being rebound + Prevent objects from being altered

I had thought of the names as being part of a shared dictionary. (Of course, immutable dictionaries aren't really available out-of-the-box now, and I'm not sure I would trust the supposed immutability of anything that wasn't proxied.)

Not all that different. The immutable dictionary would be an implantation detail of a locked name space I think. I'm wondering if there might be a way to have an inheritance by container relationship, where certain characteristics can be acquired from parent containers. Not exactly the same as class inheritance.

...

frozen: object can't be altered while frozen locked: name can't be rebound to another object

...

3. Pass mutable "deep" copies back and forth. ? Works now. (but not for all objects?)

Well, anything that can be deep-copied -- unless you also want the mutations to be collected back into a single location.

It would need to be able to make a round trip.

...

4. Pass frozen mutable objects. - Needs freezable/unfreezable mutable objects. (Not the same as making an immutable copy.)

And there is where it starts to fall apart. Though if you look at the pypy dict and interpreter optimizations, they have started to deal with it through versioning types.

I didn't find anything about "versioning" at these links. Did I miss it?

http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html#multi...

http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html#id23

_Ron

Neil Toronto wrote: (I'm going to rearrange your post so that my reply flows a bit better)

There's no reason a program with partial flow control couldn't have very Python-like syntax. After reading this, though, which formalized what I've long felt is the biggest problem with concurrent programming, I'd have to say it'd definitely not be Python itself.

It depends on what operations one wants to support. "Apply this function to all of this data" is easy. To say things like 'line x depends on line x-2, and line x+1 depends on line x-1, and line x+2 depends on line x and x+1, certainly that is not easy. But I question the purpose of being able to offer up that kind of information (in Python specifically). Presumably it is so that those tasks that don't depend on each other could be executed in parallel; but unless you have a method by which parallel execution is fast (or at least faster than just doing it in series), it's not terribly useful (especially if those operations are data structure manipulations that need to be propagated back to the 'main process').

There's an elephant-in-the-living-room UI problem, here: how would one go about extracting a partial order from a programmer? A text editor is fine for a total order, but I can't think of how I'd use one non-messily to define a partial order. How about a Gantt chart for a partial order, or some other kind of dependency diagram? How would you make it as easy to use as a text editor? The funny thing is, once you solve this problem, it may even be *easier* to program this way, because rather than maintaining the partial order in your head (or inferring it from a total order in the code), it'd be right in front of you.

Generally, the standard way of defining a partial order is via dependency graph. Unfortunately, breaking blocks of code into a dependency graph (or partial-order control-flow) tends to make the code hard to understand. I know there are various tools that use this particular kind of method, but those that I have seen leave much to be desired. Alternatively, there is a huge amount of R&D that has gone into C/C++ compilers to extract this information automatically from source code, and even more on the hardware end of things to automatically extract this information from machine code as it executes. Unfortunately, due to Python's dynamic nature, even something as simple as 'i += 0' can lead to all sorts of underlying system changes, and we may not be able to reliably extract this information (though PyPy with the LLVM backend may offer opportunities here).

For the record, I disagree strongly with the "let's keep concurrency in the libraries" idea. I want to program in *Python*, dangit. Or at least something that feels a lot like it.

And leaving concurrency in a library allows Python to stay Python. For certain tasks, one merely needs parallel variants of currently existing Python functions/operations. Take Google's MapReduce [1], which applies a function to a large number of data elements in parallel, then combines the results of those computations. While it is not universal, it can do certain operations quickly. Other tasks merely require the execution of *some function* while *some other function* is executing. Free threading, and various ways of allowing concurrent thread execution has been offered, but the more I read about the Processing package, the more I like it. These options don't offer a solution to what you seem to be wanting; an easy definition of partial order on code to be executed in Python. However, without language-level support for something like... exec lines in *block in *parallel: i += 1 j += fcn(foo) bar = fcn2(bar) ...I don't see how it is possible. Then again, I'm not sure I completely agree with Mr. Edwards or yourself in that being able to state partial ordering will offer improvements over the status quo. Then again, I tend to not worry about the blocks of 3-4 lines that aren't dependent on one another, as much as the entire function suite returning what I intended it to. - Josiah [1] http://labs.google.com/papers/mapreduce.html

On 4/2/07, Neil Toronto wrote:

I don't agree with everything in the post, but this part I do:

[...] It is up to the programmer to mentally compile (by a topological sort) these implicit dependencies into a total order expressed in a control flow.

The fancy phrase "topological sort" here obscures that this "compilation" is something humans are good at. We do it all the time. We make plans and carry them out. -j

Jim Jewett wrote:

On 4/1/07, Ron Adam wrote:

...

Jim Jewett wrote:

...

...

And there is where it starts to fall apart. Though if you look at the pypy dict and interpreter optimizations, they have started to deal with it through versioning types.

I didn't find anything about "versioning" at these links. Did I miss it?

...

http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html#multi...

...

...

http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html#id23

Sorry; I wasn't pointing to enough of the document.

http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html

discussed versions under method caching, just above the Interpreter Optimizations section.

Thanks, Found it. (Been busy with other things.) I may also depend on what abstraction level is desired. A high level of abstraction would hide all of this under the covers and it would be done transparently. A lower level would provide the tools needed to do it with, but also have the property of "if it hurts, don't do that." Ron