Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Thanks all: At 10:00 AM 10/10/2007, Robert Kern wrote:
Something like the following should suffice (untested, though I've >done similar things with ctypes before):
David Cournapeau Wrote: Basically, you cannot expect file descriptors (or even file handles: >the standard ones from C library fopen) to cross dll boundaries if
I tested, successfully: """ nFromAddress.py """ def fromaddress(address, dtype, shape, strides=None): """ Create a numpy array from an integer address, a dtype or dtype string, a shape tuple, and possibly strides. """ import numpy # Make sure our dtype is a dtype, not just "f" or whatever. dtype = numpy.dtype(dtype) class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d) ## Numeric example, with address kludge import Numeric, numpy, ctypes, string a0 = Numeric.zeros((10000), Numeric.Int16) nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16) tmp=(ctypes.c_long*1)(0) ctypes.memmove(tmp, nAddress+8, 4) nAddress = tmp[0] a1 = fromaddress(nAddress, numpy.int16, (10000,)) ## explicit type a0[0] = 5 print a1[0] ## numpy example a2 = numpy.zeros(10000, numpy.int16) nAddress = a2.__array_interface__['data'][0] nType = a2.__array_interface__['typestr'] nShape = a2.__array_interface__['shape'] a3 = fromaddress(nAddress, nType, nShape) a2[0] = 5 print a3[0] So, now with little effort the relevant info can be passed over pipes, shared memory, etc. and shares/views created in other processes, since they are not objects but ints and strings. the >dll do not have exactly the same runtime. It sounds like there is a general dilemma: no one with Python 2.4 or 2.5 can reliably expect to compile extensions/modules if they did not install the 7.1 compiler in time. The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), with associated upgrade issues. It would be nice if the build bots could also compile suggested modules/extentions. Thanks again, Ray
Thanks all:
At 10:00 AM 10/10/2007, Robert Kern wrote:
Something like the following should suffice (untested, though I've >done similar things with ctypes before):
I tested, successfully: """ nFromAddress.py """
def fromaddress(address, dtype, shape, strides=None): """ Create a numpy array from an integer address, a dtype or dtype string, a shape tuple, and possibly strides. """ import numpy # Make sure our dtype is a dtype, not just "f" or whatever. dtype = numpy.dtype(dtype)
class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d)
## Numeric example, with address kludge import Numeric, numpy, ctypes, string a0 = Numeric.zeros((10000), Numeric.Int16) nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16) tmp=(ctypes.c_long*1)(0) ctypes.memmove(tmp, nAddress+8, 4) nAddress = tmp[0] a1 = fromaddress(nAddress, numpy.int16, (10000,)) ## explicit type a0[0] = 5 print a1[0]
## numpy example a2 = numpy.zeros(10000, numpy.int16) nAddress = a2.__array_interface__['data'][0] nType = a2.__array_interface__['typestr'] nShape = a2.__array_interface__['shape'] a3 = fromaddress(nAddress, nType, nShape) a2[0] = 5 print a3[0]
So, now with little effort the relevant info can be passed over pipes, shared memory, etc. and shares/views created in other processes, since they are not objects but ints and strings.
David Cournapeau Wrote: Basically, you cannot expect file descriptors (or even file handles: >the standard ones from C library fopen) to cross dll boundaries if the >dll do not have exactly the same runtime.
It sounds like there is a general dilemma: no one with Python 2.4 or 2.5 can reliably expect to compile extensions/modules if they did not install the 7.1 compiler in time. Well, in theory you could: 'just' recompile python. The problem is not
The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), with associated upgrade issues. But what if the next MS compiler has again broken libc implementation ? (Incidently, VS2005 was not used for python2.5 for even more broken libc
Ray S wrote: the compiler as such, but the C runtime. I don't see any solution to this situation, unfortunately; if MS decides to ship a broken libc, it is hard to get around that in a portable way. For files (I don't know any other problems, but this certainly do not mean they do not exist), the only way I know is to use the win32 files handles. At least, it works in C (I had similar problems when dealing with tmp files on win32). To do it directly in python, you may need pywin32 specific functions (I cannot remember them on the top of my head). than in 2003): http://mail.python.org/pipermail/python-dev/2006-April/064126.html cheers, David
The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), with associated upgrade issues. But what if the next MS compiler has again broken libc implementation ? (Incidently, VS2005 was not used for python2.5 for even more broken libc than in 2003): http://mail.python.org/pipermail/python-dev/2006-April/064126.html
I don't what he meant by a broken libc, if it is the fact that there is a lot of deprecated standard functions, I don't call it broken (besides, this deprecation follows a technical paper that describe the new safe functions, although it does not deprecate these functions). Matthieu
Matthieu Brucher wrote:
> The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), > with associated upgrade issues. But what if the next MS compiler has again broken libc implementation ? (Incidently, VS2005 was not used for python2.5 for even more broken libc than in 2003): http://mail.python.org/pipermail/python-dev/2006-April/064126.html
I don't what he meant by a broken libc, if it is the fact that there is a lot of deprecated standard functions, I don't call it broken (besides, this deprecation follows a technical paper that describe the new safe functions, although it does not deprecate these functions).
If unilaterally deprecating standard functions which exist for years is not broken, I really wonder what is :) cheers, David
I don't what he meant by a broken libc, if it is the fact that there is a lot of deprecated standard functions, I don't call it broken (besides, this deprecation follows a technical paper that describe the new safe functions, although it does not deprecate these functions). If unilaterally deprecating standard functions which exist for years is not broken, I really wonder what is :)
They are deprecated (although a simple flag can get rid of those deprecation) not removed. Besides, the deprecated functions are in fact functions that can lead to security issues (for the first time Microsoft did something not completely stupid about this topic), so telling that the programmer should not use them but more secure one may be seen as a good idea (from a certain point of view). Matthieu
Matthieu Brucher wrote:
> I don't what he meant by a broken libc, if it is the fact that there > is a lot of deprecated standard functions, I don't call it broken > (besides, this deprecation follows a technical paper that describe the > new safe functions, although it does not deprecate these functions). If unilaterally deprecating standard functions which exist for years is not broken, I really wonder what is :)
They are deprecated (although a simple flag can get rid of those deprecation) not removed.
Besides, the deprecated functions are in fact functions that can lead to security issues This is far from obvious. Actually, numerous people (who I trust more
deprecated means that they will disappear soon, and (worse), people will chose to use the non deprecated, hence programming non portable code (those functions are neither portable or standard; different OS have different implementation). than MS) have raised their concern on those functions: they add complexity for no obvious advantage. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1106.txt cheers, Ddavid
participants (3)
-
David Cournapeau -
Matthieu Brucher -
Ray S