Den 11.04.2011 01:20, skrev Sturla Molden:

I'm an idiot with build tools, hence no setup.py. Invoke Cython and then cc. Compile sharedmemory_sysv.pyx for Linux/Unix or sharedmemory_sysv.pyx and ntqueryobject.c for Windows.

Eh, that is sharedmemory_win.pyx and ntqueryobject.c for Windows :-) Sturla

On Mon, Apr 11, 2011 at 7:05 AM, Gael Varoquaux wrote:

Hey Sturla,

It's really great that you are still working on that. I'll test the code under Linux.

The scipy community has moved to github. If I create a repository under github and put the code on it, would you use it? If I find time, I'll add a setup.py.

Gaël

Hi, just wanted to say that I find this module, and shared memory in general very interesting, because I work with very large image data sets. I have a non python question: for Java there seems to exist a module/package/class called nio http://download.oracle.com/javase/1.4.2/docs/api/java/nio/MappedByteBuffer.h... public abstract class MappedByteBuffer extends ByteBuffer A direct byte buffer whose content is a memory-mapped region of a file. Could this be used to share memory between Java applications and numpy ? Just hoping that someone here knows Java much better than me. - Sebastian Haase

Hi everyone, I was looking up the options that are available for shared memory arrays and this thread came up at the right time. The doc says that multiprocessing .Array(...) gives a shared memory array. But from the code it seems to me that it is actually using a mmap. Is that correct a correct assessment, and if so, is there any advantage in using multiprocessing.Array(...) over simple numpy mmaped arrays. Regards srean

"Shared memory" is memory mapping from the paging file (i.e. RAM), not a file on disk. They can have a name or be anonymous. I have explained why we need named shared memory before. If you didn't understand it, try to pass an instance of |multiprocessing.Array over | |multiprocessing.Queue. |Sturla Den 11.04.2011 20:11, skrev srean:

Apologies for adding to my own post. |multiprocessing.Array(...) uses an anonymous mmapped file. I am not sure if that means it is resident on RAM or the swap device. But my original question remains, what are the pros and cons of using it versus numpy mmapped arrays. If ||multiprocessing.Arrayis indeed resident in memory (subject to swapping of course) that would still be advatageous compared to a file mapped from a on-disk filesystem.|

On Mon, Apr 11, 2011 at 12:42 PM, srean mailto:srean.list@gmail.com> wrote:

Hi everyone,

I was looking up the options that are available for shared memory arrays and this thread came up at the right time. The doc says that|multiprocessing.Array(...) gives a shared memory array. But from the code it seems to me that it is actually using a mmap. Is that correct a correct assessment, and if so, is there any advantage in using ||multiprocessing.Array(...)over simple numpy mmaped arrays.

Regards srean |||

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Got you and thanks a lot for the explanation. I am not using Queues so I think I am safe for the time being. Given that you have worked a lot on these issues, would you recommend plain mmapped numpy arrays over multiprocessing.Array Thanks again -- srean On Mon, Apr 11, 2011 at 1:36 PM, Sturla Molden wrote:

"Shared memory" is memory mapping from the paging file (i.e. RAM), not a file on disk. They can have a name or be anonymous. I have explained why we need named shared memory before. If you didn't understand it, try to pass an instance of multiprocessing.Array over multiprocessing.Queue.

Sturla

Den 11.04.2011 21:15, skrev srean:

Got you and thanks a lot for the explanation. I am not using Queues so I think I am safe for the time being. Given that you have worked a lot on these issues, would you recommend plain mmapped numpy arrays over |multiprocessing.Array| | With multiprocessing you must use multiprocessing.Array.

With os.fork you can use either. Sturla |