Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
At 05:22 AM 10/9/2007, David Cournapeau wrote:
Could not this be because you compiled the posh sources with a compiler/runtime which is different than the other extensions and python interpreter ?
It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and anyone else who didn't download it in time) are now seemingly SOL since it is no longer available. I saw much discussion of this as well, but even 2.5 is now "fixed" on 7.1 and reports of compiling distutil modules with the new MS SDK and having them work at all with 2.4 were very mixed. I also tried GCC and had a litany of other errors with the posh.
Sebastian Haase added: I was in fact experimenting with this. The solution seemed to lie in "simple" memmap as it is implemented in Windows:
<snip> I had just found and started to write some tests with that MS function. If I can truly write to the array in one process and instantly read it in the other I'll be happy. Did you find that locks or semaphores were needed?
(( I have to mention, that I could crash a process while testing this ... ))
That was one of my first results! I also found that using ctypes to create arrays from the other process's address and laying a numpy array on top was prone to that in experimentation. But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array). Thanks, Ray Schumacher
On 10/9/07, Ray Schumacher <subscriber100@rjs.org> wrote:
At 05:22 AM 10/9/2007, David Cournapeau wrote:
Could not this be because you compiled the posh sources with a compiler/runtime which is different than the other extensions and python interpreter ?
It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and anyone else who didn't download it in time) are now seemingly SOL since it is no longer available. I saw much discussion of this as well, but even 2.5 is now "fixed" on 7.1 and reports of compiling distutil modules with the new MS SDK and having them work at all with 2.4 were very mixed. I also tried GCC and had a litany of other errors with the posh.
Sebastian Haase added: I was in fact experimenting with this. The solution seemed to lie in "simple" memmap as it is implemented in Windows:
<snip> I had just found and started to write some tests with that MS function. If I can truly write to the array in one process and instantly read it in the other I'll be happy. Did you find that locks or semaphores were needed?
Maybe that's why it crashed ;-) !? But for simple use it seems fine.
(( I have to mention, that I could crash a process while testing this ... ))
That was one of my first results! I also found that using ctypes to create arrays from the other process's address and laying a numpy array on top was prone to that in experimentation. But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array).
I assume this is a different issue, but haven't looked into it yet. -Sebastian
Ray Schumacher wrote:
At 05:22 AM 10/9/2007, David Cournapeau wrote:
Could not this be because you compiled the posh sources with a compiler/runtime which is different than the other extensions and python interpreter ?
It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and anyone else who didn't download it in time) are now seemingly SOL since it is no longer available. I saw much discussion of this as well, but even 2.5 is now "fixed" on 7.1 and reports of compiling distutil modules with the new MS SDK and having them work at all with 2.4 were very mixed. I also tried GCC and had a litany of other errors with the posh.
Basically, you cannot expect file descriptors (or even file handles: the standard ones from C library fopen) to cross dll boundaries if the dll do not have exactly the same runtime. The only solution is to convert your file descriptor to the non standard HANDLE things from the Win32 API, and pass this at the API level. Now, I have no idea if this works also for mmap files; maybe this is broken too. cheers, David
participants (3)
-
David Cournapeau -
Ray Schumacher -
Sebastian Haase