[Numpy-discussion] numpy/Windows shared arrays between processes?

[Sebastian Haase]

Hi!
I was in fact experimenting with this. The solution seemed to lie in
"simple" memmap as it is implemented in Windows:

import numpy as N
def arrSharedMemory(shape, dtype, tag="PriithonSharedMemory"):
    """
    Windows only !
    share memory between different processes if same `tag` is used.
    """
    itemsize = N.dtype(dtype).itemsize
    count = N.product(shape)
    size =  count * itemsize

    import mmap
    sharedmem = mmap.mmap(0, size, tag)
    a=N.frombuffer(sharedmem, dtype, count)
    a.shape = shape
    return a

For explaintion look up the microsoft site for the mmap documentation.
And/or the Python-doc for mmap.
(( I have to mention, that I could crash a process while testing this ... ))

If anyone here would know an equivalent way of doing this on
Linux/OS-X  we were back to a cross-platfrom function.



Hope this helps,
Sebastian Haase

On 10/9/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> Ray S wrote:
> > Is anyone sharing arrays between processes on Windows?
> > I tried compiling the posh sources (once, so far) with the new MS
> > toolkit and failed...
> > What other solutions are in use?
> >
> > Have a second process create an array view from an address would
> > suffice for this particular purpose. I could pass the address as a
> > parameter of the second process's argv.
> >
> > I've also tried things like
> > pb=pythonapi.PyBuffer_FromReadWriteMemory(9508824, 9*sizeof(c_int))
> > N.frombuffer(pb, N.int32)
> > which fails since pb is and int. What are my options?
> >
> (disclaimer: I know nothing about windows idiosyncraties)
>
> Could not this be because you compiled the posh sources with a
> compiler/runtime which is different than the other extensions and python
> interpreter ? I don't know the details, but since most of the posix
> functions related to files and processes are broken beyond despair in
> windows, and in particular, many posix handles cannot cross dll
> boundaries compiled by different compilers, I would not be surprised if
> this cause some trouble.
>
> The fact that POSH is said to be posix-only on python.org
> (http://wiki.python.org/moin/ParallelProcessing) would imply that people
> do not care much about windows, too (but again, this is just from
> reading what posh is about; I have never used it personnally).
>
> cheers,
>
> David
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>