[Numpy-discussion] numpy/Windows shared arrays between processes?

[David Cournapeau]

Ray S wrote:
> Thanks all:
>
> At 10:00 AM 10/10/2007, Robert Kern wrote:
>  >Something like the following should suffice (untested, though 
> I've >done similar things with ctypes before):
>
> I tested, successfully:
> """ nFromAddress.py
> """
>
> def fromaddress(address, dtype, shape, strides=None):
>      """ Create a numpy array from an integer address, a dtype
>      or dtype string, a shape tuple, and possibly strides.
>      """
>      import numpy
>      # Make sure our dtype is a dtype, not just "f" or whatever.
>      dtype = numpy.dtype(dtype)
>
>      class Dummy(object):
>          pass
>      d = Dummy()
>      d.__array_interface__ = dict(
>          data = (address, False),
>          typestr = dtype.str,
>          descr = dtype.descr,
>          shape = shape,
>          strides = strides,
>          version = 3,
>      )
>      return numpy.asarray(d)
>
> ## Numeric example, with address kludge
> import Numeric, numpy, ctypes, string
> a0 = Numeric.zeros((10000), Numeric.Int16)
> nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16)
> tmp=(ctypes.c_long*1)(0)
> ctypes.memmove(tmp, nAddress+8, 4)
> nAddress = tmp[0]
> a1 = fromaddress(nAddress, numpy.int16, (10000,)) ## explicit type
> a0[0] = 5
> print a1[0]
>
> ## numpy example
> a2 = numpy.zeros(10000, numpy.int16)
> nAddress = a2.__array_interface__['data'][0]
> nType = a2.__array_interface__['typestr']
> nShape = a2.__array_interface__['shape']
> a3 = fromaddress(nAddress, nType, nShape)
> a2[0] = 5
> print a3[0]
>
> So, now with little effort the relevant info can be passed over 
> pipes, shared memory, etc. and shares/views created in other 
> processes, since they are not objects but ints and strings.
>
>  >David Cournapeau Wrote:
>  >Basically, you cannot expect file descriptors (or even file 
> handles: >the
>  >standard ones from C library fopen) to cross dll boundaries if 
> the >dll do not have exactly the same runtime.
>
> It sounds like there is a general dilemma: no one with Python 2.4 or 
> 2.5 can reliably expect to compile extensions/modules if they did not 
> install the 7.1 compiler in time.
Well, in theory you could: 'just' recompile python. The problem is not 
the compiler as such, but the C runtime. I don't see any solution to 
this situation, unfortunately; if MS decides to ship a broken libc, it 
is hard to get around that in a portable way.

For files (I don't know any other problems, but this certainly do not 
mean they do not exist), the only way I know is to use the win32 files 
handles. At least, it works in C (I had similar problems when dealing 
with tmp files on win32). To do it directly in python, you may need 
pywin32 specific functions (I cannot remember them on the top of my head).

> The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), 
> with associated upgrade issues.
But what if the next MS compiler has again broken libc implementation ? 
(Incidently, VS2005 was not used for python2.5 for even more broken libc 
than in 2003): 
http://mail.python.org/pipermail/python-dev/2006-April/064126.html

cheers,

David