[SciPy-User] Multiprocessing and shared memory

[Felix Schlesinger]

> Felix Schlesinger skrev:
>> I have been working on an application using scipy that solves a highly
>> parallel problem. To avoid the GIL in python I used to multiprocessing
>> package. The main issue I ran into is shared memory.
>
> http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip
>
> Basically it uses named shared memory (Sys V IPC on Unix) as buffer. The
> ndarray is pickled by its kernel name, the buffer is not copied. Thus
> you can quickly communicate shared memory ndarrays between processes
> (using multiprocessing.Queue).

> numpy.memmap uses BSD memap, not System V IPC. That means the shared
> segment has no name, so it must be created in the parent prior to forking.

I am fine with passing the shared array at fork, so this is not much
of an issue, but I am not sure I follow anyway: scipy.memmap is
constructed based on an open file handle. Couldn't I just pass that
filehandle along a queue if needed?
Again, not really what I am worrying about. The memory leaks are much
more serious. I am not quite sure calling sys.exit is safe either. It
might affect module level objects in the main process (buffers, etc.).
The details are tricky as always.

> There is a multicore branch of numpy. I have never used it.

I'll look into that, but I did not find much documentation.

> Also note that you can use Cython with normal Python threads, and
> release the GIL when working with the ndarrays in Cython. Cython has a
> special syntax for numpy arrays. This is what I currently do, and why I
> more or less lost my interest in shared memory. The GIL is only a
> problem if you don't release it. But in Cython you can just do:
>
> with nogil:
>    <suite>

Well this only works if the parallel operation does not touch any
python objects, right? Otherwise it would reenter the interpreter
without the GIL and not be thread-safe anymore. In other words the
whole algorithm would have to be moved into cython. Is this approach
stable in your experience? It sounds like it would be easy to create
race-conditions to me.

Felix