copying ctypes arrays to numarray?
I'm posting here to see if numarray has a method that could work like array.buffer_info(). numarray.info() returns a text output that can't be used like the array method is to memmove() between ctypes and Python arrays without parsing, apparently. ctypes thread with Thomas Heller below. His main question: "Maybe you should ask to make sure that there's no way to copy between objects implementing the buffer protocol with some Python function that I do not know about?"
On Wed, 15 Dec 2004 20:16:07 -0800, RJ <rays@san.rr.com> wrote:
I'm posting here to see if numarray has a method that could work like array.buffer_info(). numarray.info() returns a text output that can't be used like the array method is to memmove() between ctypes and Python arrays without parsing, apparently.
ctypes thread with Thomas Heller below. His main question: "Maybe you should ask to make sure that there's no way to copy between objects implementing the buffer protocol with some Python function that I do not know about?"
I just tried some things:
import ctypes a = (ctypes.c_int * 5)() a[0] = 1; a[1] = 2; a[2] = 3; a[3] = 4; a[4] = 5 list(a) [1, 2, 3, 4, 5] import numarray
buf = numarray.zeros(shape=20, type='i4') buf array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) buf[2:7] = a buf array([0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
temp = numarray.array(sequence=buffer(a), shape=5, type='i4') temp array([1, 2, 3, 4, 5]) temp._data <read-only buffer for 0x00FA9230, ptr 0x00BFA668, size 20 at 0x010D6DA0> buffer(a) <read-only buffer for 0x00FA9230, ptr 0x00BFA668, size 20 at 0x00FAF600>
a[2] = 10 temp array([ 1, 2, 10, 4, 5])
The first block just creates the ctypes data. The second block uses slice assignment to copy the data from the ctypes array into the numarray.array. The third block uses the buffer interface to create a numarray.array which points to the same memory location as the ctypes array. The forth block illustrates that it's really the same memory. You have to benchmark which one of the two solutions is the better one for you. Regards, Florian Schulze
I'm re-posting this here to see if anyone can shed light on numarray.info() behavior, and why assignment from a (ctypes.c_long * 2000)() to a numarray is so much slower than a memmove(). I was surprised that assignment wasn't faster and that numarray assignment was consistently ~.5% faster than Numeric. The N vs. n memmove() flip-flopped for fastest, with array.array always slower. After some discussion, I made a better direct comparison of ctypes and memmove, map etc., using all the methods suggested so far. I think it's fairly valid.(?) Run on a 2.4GHz WinXP, Py2.3, a number of times. Parsing numarray.info() was a royal pain, it writes directly to stdout! I.e.:
inf = src.info() class: <class 'numarray.numarraycore.NumArray'> shape: (32,) strides: (8,) byteoffset: 0 bytestride: 8 itemsize: 8 aligned: 1 contiguous: 1 data: <memory at 00815950 with size:256 held by object 00815930 aliasing object 00000000> byteorder: little byteswap: 0 type: Float64 inf type(inf) <type 'NoneType'>
Since the ctype does support slicing, I was considering leaving the data in the device driver buffer (~1MB circular) and poking into it, but memmove is so much faster than slicing the ctype, I'm doing memmove()s to numarray. I presume that I should check for numarray.iscontiguous( ) or is_c_array( ) first to be safe... Results and code below. Thanks to all for the help, Ray
python test.py Array address 11443208 n address 17039424 N address 21102656 for loop 2027.0 us ea map 1781.0 us ea slice 1704.0 us ea assign N 1244.0 us ea assign n 1242.0 us ea memmove 4.3831 us ea memmove N 3.4773 us ea memmove n 3.4803 us ea
_____________________________________________________ ## test.py import array import numarray import ctypes import time import string import StringIO import sys buf = (ctypes.c_long * 2000)() Array = array.array("l", [0]*10000) n = numarray.zeros((1000000), numarray.Int32) N = numarray.zeros((1000000), numarray.Int32) #!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!! # n.info() writes directly to stdout! stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo n.info() ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo N.info() Ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! print 'Array address', Array.buffer_info()[0] ninfo = string.split(ninfo) nAddress = int(ninfo[20], 16) print 'n address', nAddress Ninfo = string.split(Ninfo) NAddress = int(Ninfo[20], 16) print 'N address', NAddress t0 = time.clock() for loop in range(1000): for i in range(2000): n[loop+i] = buf[i] print 'for loop ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = map(None, buf) print 'map ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = buf[0:2000] print 'slice ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(10000): N[loop:loop+2000] = buf print 'assign N', round((time.clock()-t0)*100), 'us ea' t0 = time.clock() for loop in range(10000): n[loop:loop+2000] = buf print 'assign n', round((time.clock()-t0)*100), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+Array.buffer_info()[0], buf, 2000) print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+NAddress, buf, 2000) print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+nAddress, buf, 2000) print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea'
I agree that numarray's .info() would be better as a string, but I've been leery of changing it. Before we go to far with this, try: repr(src._data) to get a string like: "<memory at 00815950 with size:256 held by object 00815930 aliasing object 00000000>" Note: this technique depends on the buffer object used as the basis for the array. Here the repr of numarray's memory object is giving useful information. Other buffers, e.g. a string, might not. It looks to me like you forgot to import Numeric so I don't think there's any numarray:Numeric comparison going on... either that or I just don't understand what you're to compare. To be safe, check the .is_c_array() method. Regards, Todd On Thu, 2004-12-16 at 13:33 -0800, rays wrote:
I'm re-posting this here to see if anyone can shed light on numarray.info() behavior, and why assignment from a (ctypes.c_long * 2000)() to a numarray is so much slower than a memmove().
I was surprised that assignment wasn't faster and that numarray assignment was consistently ~.5% faster than Numeric. The N vs. n memmove() flip-flopped for fastest, with array.array always slower.
After some discussion, I made a better direct comparison of ctypes and memmove, map etc., using all the methods suggested so far. I think it's fairly valid.(?) Run on a 2.4GHz WinXP, Py2.3, a number of times.
Parsing numarray.info() was a royal pain, it writes directly to stdout! I.e.:
inf = src.info() class: <class 'numarray.numarraycore.NumArray'> shape: (32,) strides: (8,) byteoffset: 0 bytestride: 8 itemsize: 8 aligned: 1 contiguous: 1 data: <memory at 00815950 with size:256 held by object 00815930 aliasing object 00000000> byteorder: little byteswap: 0 type: Float64 inf type(inf) <type 'NoneType'>
Since the ctype does support slicing, I was considering leaving the data in the device driver buffer (~1MB circular) and poking into it, but memmove is so much faster than slicing the ctype, I'm doing memmove()s to numarray. I presume that I should check for numarray.iscontiguous( ) or is_c_array( ) first to be safe...
Results and code below.
Thanks to all for the help, Ray
python test.py Array address 11443208 n address 17039424 N address 21102656 for loop 2027.0 us ea map 1781.0 us ea slice 1704.0 us ea assign N 1244.0 us ea assign n 1242.0 us ea memmove 4.3831 us ea memmove N 3.4773 us ea memmove n 3.4803 us ea
_____________________________________________________
## test.py import array import numarray import ctypes import time import string import StringIO import sys
buf = (ctypes.c_long * 2000)() Array = array.array("l", [0]*10000) n = numarray.zeros((1000000), numarray.Int32) N = numarray.zeros((1000000), numarray.Int32)
#!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!! # n.info() writes directly to stdout! stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo n.info() ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout
stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo N.info() Ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
print 'Array address', Array.buffer_info()[0]
ninfo = string.split(ninfo) nAddress = int(ninfo[20], 16) print 'n address', nAddress Ninfo = string.split(Ninfo) NAddress = int(Ninfo[20], 16) print 'N address', NAddress
t0 = time.clock() for loop in range(1000): for i in range(2000): n[loop+i] = buf[i] print 'for loop ', round((time.clock()-t0)*1000), 'us ea'
t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = map(None, buf) print 'map ', round((time.clock()-t0)*1000), 'us ea'
t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = buf[0:2000] print 'slice ', round((time.clock()-t0)*1000), 'us ea'
t0 = time.clock() for loop in range(10000): N[loop:loop+2000] = buf print 'assign N', round((time.clock()-t0)*100), 'us ea'
t0 = time.clock() for loop in range(10000): n[loop:loop+2000] = buf print 'assign n', round((time.clock()-t0)*100), 'us ea'
t0 = time.clock() for loop in range(10000): ctypes.memmove(10+Array.buffer_info()[0], buf, 2000) print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea'
t0 = time.clock() for loop in range(10000): ctypes.memmove(10+NAddress, buf, 2000) print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea'
t0 = time.clock() for loop in range(10000): ctypes.memmove(10+nAddress, buf, 2000) print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea'
Doh. Thanks Todd, At 04:55 PM 12/16/2004 -0500, Todd Miller wrote:
try: repr(src._data) ... Other buffers, e.g. a string, might not.
Which is fine for this app, so I will.
It looks to me like you forgot to import Numeric
Yep. I screwed up the import statements - I'll re-do it tomorrow. In that light, it is interesting that the assign N were always longer than the n ones...
To be safe, check the .is_c_array() method.
Will do. I am also a little concerned about the address changing in a long-running process (literally, weeks), so I might check that too. Also, why is assignment from a (ctypes.c_long * 2000)() to a numarray so much slower than a memmove()? Does the assign make a temp buffer? Ray
On Thu, 2004-12-16 at 20:04, RJ wrote:
Doh. Thanks Todd,
At 04:55 PM 12/16/2004 -0500, Todd Miller wrote:
try: repr(src._data) ... Other buffers, e.g. a string, might not.
Which is fine for this app, so I will.
It looks to me like you forgot to import Numeric
Yep. I screwed up the import statements - I'll re-do it tomorrow. In that light, it is interesting that the assign N were always longer than the n ones...
To be safe, check the .is_c_array() method.
Will do. I am also a little concerned about the address changing in a long-running process (literally, weeks), so I might check that too.
As long as you let the NumArray allocate its own memory, it will allocate a numarray.memory object and those don't move. If you resize the array, that's a different story...
Also, why is assignment from a (ctypes.c_long * 2000)() to a numarray so much slower than a memmove()? Does the assign make a temp buffer?
For the assignment to numarray, I think the ctype is opaque and as a result Python uses the sequence protocol; this is a huge amount of work compared to a tight C loop moving bytes. I think every 4 bytes the ctype is doing a method call and building a Python long, numarray is doing a method call and extracting a Python long, then the Python long is destructed. I may be missing something, but the comparisons of for loop, map, and slice look like they're only doing 1000 moves but the array assigments are doing 10000, so I think there's another factor of 10 missing in that comparison.
At 09:53 AM 12/17/2004 -0500, Todd Miller wrote:
I may be missing something, but the comparisons of for loop, map, and slice look like they're only doing 1000 moves but the array assigments are doing 10000, so I think there's another factor of 10 missing in that comparison.
The multiplier in the print statement compensates to still give microseconds per move; I couldn't bear to wait for 10,000 slow iterations. I'll re-do the test shortly in any case. Ray Secret anti-spam filter-passing text. Include with reply: qwertyuiop
participants (4)
-
Florian Schulze -
rays -
RJ
-
Todd Miller