[Numpy-discussion] Shared memory ndarrays (update)

Sturla Molden sturla at molden.no
Sun Apr 10 19:20:39 EDT 2011

Here is an update for the shared memory arrays that Gaël and I wrote two 
years ago. They are NumPy arrays referencing shared memory, and IPC 
using multiprocessing.Queue is possible by monkey patching how ndarrays 
are pickled.


     import numpy as np
     import sharedmem as sm

     shared_array = sm.zeros(n)

I.e. the only difference from ndarrays is that pickle.dumps and 
multiprocessing.Queue do not make a copy of the buffer, and that 
allocated memory is shared between professes (e.g. created with os.fork, 
subprocess or multiprocessing.)

A named memory map of the paging file is used on Windows. Unix System V 
IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance).


- 64-bit support.
- Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
- Added a global lock as there are callbacks to Python (the GIL is not 
sufficient serialization).

I need help with testing, particularly on Linux / Apple and with the 
most recent NumPy.

I'm an idiot with build tools, hence no setup.py. Invoke Cython and then 
cc. Compile sharedmemory_sysv.pyx for Linux/Unix or 
sharedmemory_sysv.pyx and ntqueryobject.c for Windows.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: sharedmem.zip
Type: application/x-zip-compressed
Size: 8794 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/a6b16acb/attachment.bin>

More information about the NumPy-Discussion mailing list