[Numpy-discussion] Shared memory ndarrays (update)
sturla at molden.no
Sun Apr 10 19:20:39 EDT 2011
Here is an update for the shared memory arrays that Gaël and I wrote two
years ago. They are NumPy arrays referencing shared memory, and IPC
using multiprocessing.Queue is possible by monkey patching how ndarrays
import numpy as np
import sharedmem as sm
shared_array = sm.zeros(n)
I.e. the only difference from ndarrays is that pickle.dumps and
multiprocessing.Queue do not make a copy of the buffer, and that
allocated memory is shared between professes (e.g. created with os.fork,
subprocess or multiprocessing.)
A named memory map of the paging file is used on Windows. Unix System V
IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance).
- 64-bit support.
- Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
- Added a global lock as there are callbacks to Python (the GIL is not
I need help with testing, particularly on Linux / Apple and with the
most recent NumPy.
I'm an idiot with build tools, hence no setup.py. Invoke Cython and then
cc. Compile sharedmemory_sysv.pyx for Linux/Unix or
sharedmemory_sysv.pyx and ntqueryobject.c for Windows.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 8794 bytes
Desc: not available
More information about the NumPy-Discussion