[Numpy-discussion] Shared memory ndarrays (update)

[Sturla Molden]

Here is an update for the shared memory arrays that Gaël and I wrote two 
years ago. They are NumPy arrays referencing shared memory, and IPC 
using multiprocessing.Queue is possible by monkey patching how ndarrays 
are pickled.

Usage:

     import numpy as np
     import sharedmem as sm

     shared_array = sm.zeros(n)

I.e. the only difference from ndarrays is that pickle.dumps and 
multiprocessing.Queue do not make a copy of the buffer, and that 
allocated memory is shared between professes (e.g. created with os.fork, 
subprocess or multiprocessing.)

A named memory map of the paging file is used on Windows. Unix System V 
IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance).

Changes:

- 64-bit support.
- Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
- Added a global lock as there are callbacks to Python (the GIL is not 
sufficient serialization).

I need help with testing, particularly on Linux / Apple and with the 
most recent NumPy.

I'm an idiot with build tools, hence no setup.py. Invoke Cython and then 
cc. Compile sharedmemory_sysv.pyx for Linux/Unix or 
sharedmemory_sysv.pyx and ntqueryobject.c for Windows.


Regards,
Sturla
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sharedmem.zip
Type: application/x-zip-compressed
Size: 8794 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/a6b16acb/attachment.bin>