SMP with Python - trying to unlock 2th CPU

[Johannes Nix]

Hi,

I am using Python on a dual processor workstation for digital audio
processing. My programm is now processing about 1/4 real time, and I
want to speed it up by using both processors. (Without much effort and
considering that one week of programming does the equivalent of at
least four months work on a five-processor TMSC40 DSP system, this
seems really good!)

I tried to create a second thread and connect the two threads using
the Queue module. (Acutually, there are more threads because a Tkinter
plot widget visualizes data). However, I was puzzled that my
program went slower, not faster :-(.

Reading the docs I learned that the Python interpreter uses a Big
Global Lock always when manipulating a Python object. This would mean
that only one thread can run at the same time, and only one processor
is used. (In other words, my second CPU is locked up).

Now, which may be the best way to speed up things? It would be
possible to implement some operations as parallel C threads, but I
would like to take this step later, as this is a lot work.

Another possible solution would be to connect two independent Python
interpreters via Unix pipes, this fits to the current structure of the
algorithm but may turn out too inflexible. The great drawback of this
solution is that only a data flow in one direction is practical.
Also, the amount of data moved is quite large.

It seems possible to use also memory-mapped files using the mmap
module, but this is probably much more complicated. Also, there is the
overhead of serializing data for transmission. The best solution would
be perhaps if I could create a Numpy array object which shares memory,
and has a fixed memory location. Then both processes could share some
data, e.g. by such functions:

def get_shared_memory_array(shmid)
	....

def create_shared_memory_array(shmid,array)
	.....

At Vaults fo Parnassus, there lurks a shared memory module around
(http://sirac.inrialpes.fr/~marangoz/python/shm/)

Does anyone has experiences with using SMP with Python, and which
other solutions may be practical?


And, is it realistic that the Big Global Lock gets ever removed, for
example replaced by many individual locks protecting every Python
object?  

(Sounds like Linux' 1.0. "Global Spinlock")


seeking-the-unlocking-key-solution'ly,

Johannes