[Numpy-discussion] Shared memory ndarrays (update)

Sturla Molden sturla at molden.no
Mon Apr 11 15:29:35 EDT 2011


Den 11.04.2011 01:20, skrev Sturla Molden:
>
> Changes:
>
> - 64-bit support.
> - Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
> - Added a global lock as there are callbacks to Python (the GIL is not 
> sufficient serialization).

I will also add a barrier synchronization primitive to this (as it is 
very useful for numerical computing, more so than mutexes or semaphores) 
and a work scheduler, so it will be easy to distribute work between the 
processes.

The barrier is implemented with atomic read/writes on top of shared 
memory, so it can be sent over multiprocessing.Queue.

It is also possible to implement locks, semaphores, events, etc., the 
same way.

Observe that the synchronization primitives in multiprocessing, e.g. 
multiprocessing.RLock, cannot be sent over a Queue. Named shared memory 
takes that restriction away. Thus, objects that are pickled for 
multiprocessing.Queue can contain locks, events, barriers, etc.

Sturla



More information about the NumPy-Discussion mailing list