Talin wrote:
Thinking more about this, it seems to me that discussions of syntax for doing parallel operations and nifty classes for synchronization are a bit premature.
Yes, but it may help to have a few possible "end user" examples in mind while you work on the problem. The point isn't the exact syntax, but the uses that they imply and what other requirements these examples need in order to work.
The real question, it seems to me, is how to get Python to operate concurrently at all.
Yes, and I agree that it would be better to split the problem into smaller chunks. Maybe start by finding some "easier" to do simple cases first.
Python's GIL cannot be removed without going through and fixing a thousand different places where different threads might access shared variables within the interpreter. Clearly, that's not the kind of work that is going to be done in a month or less.
Here is an older discussion on removing the GIL and multiprocessing vs threading. Maybe it will be of help? http://mail.python.org/pipermail/python-dev/2005-September/056423.html
It might be useful to decompose that task further into some digestible chunks. One of these chunks is garbage collection. It seems to me that reference counting as it exists today would is a serious hindrance to concurrency, because it requires writing to an object each time you create a new reference. Instead, it should be possible to pass a reference to an object between threads, without actually modifying the object unless one of the threads actually changes an attribute.
I'm not familiar with the details of python's garbage collecting yet. I was thinking the problem may be simplified by having a single mutable container-cell object. But that wouldn't be enough because in python it isn't just a problem of a mutable object changing, but one of a names reference to an object being changed. Could an explicit name space for sharing in threads and processes help?
There are a number of papers on concurrent garbage collection out there on the web that might serve as a useful starting point. Of course, the .Net CLR and Java VM already have collectors of this type, so maybe those versions of Python already get this for free.
I also wonder what other things that the GIL is protecting can be broken out as large, coherent chunks.
I have no idea, ... Ron