[Python-ideas] Python and Concurrency

Ron Adam rrr at ronadam.com
Sun Mar 25 19:13:34 CEST 2007


Talin wrote:
 > Thinking more about this, it seems to me that discussions of syntax for
 > doing parallel operations and nifty classes for synchronization are a
 > bit premature.

Yes, but it may help to have a few possible "end user" examples in mind 
while you work on the problem.  The point isn't the exact syntax, but the 
uses that they imply and what other requirements these examples need in 
order to work.


 >The real question, it seems to me, is how to get Python
 > to operate concurrently at all.

Yes, and I agree that it would be better to split the problem into smaller 
chunks.  Maybe start by finding some "easier" to do simple cases first.


 > Python's GIL cannot be removed without going through and fixing a
 > thousand different places where different threads might access shared
 > variables within the interpreter. Clearly, that's not the kind of work
 > that is going to be done in a month or less.

Here is an older discussion on removing the GIL and multiprocessing vs 
threading.  Maybe it will be of help?

http://mail.python.org/pipermail/python-dev/2005-September/056423.html


 > It might be useful to decompose that task further into some digestible
 > chunks. One of these chunks is garbage collection. It seems to me that
 > reference counting as it exists today would is a serious hindrance to
 > concurrency, because it requires writing to an object each time you
 > create a new reference. Instead, it should be possible to pass a
 > reference to an object between threads, without actually modifying the
 > object unless one of the threads actually changes an attribute.

I'm not familiar with the details of python's garbage collecting yet.

I was thinking the problem may be simplified by having a single mutable 
container-cell object.  But that wouldn't be enough because in python it 
isn't just a problem of a mutable object changing, but one of a names 
reference to an object being changed.

Could an explicit name space for sharing in threads and processes help?


 > There are a number of papers on concurrent garbage collection out there
 > on the web that might serve as a useful starting point. Of course, the
 > .Net CLR and Java VM already have collectors of this type, so maybe
 > those versions of Python already get this for free.
 >
 > I also wonder what other things that the GIL is protecting can be broken
 > out as large, coherent chunks.

I have no idea,
   ... Ron




More information about the Python-ideas mailing list