If I have an array N:
N = Numeric.zeros((1000,), Numeric.Float) repr(N.__copy__)
'<built-in method __copy__ of array object at 0x00809F00>'
What is the actual address of the first element? Or, as an offset from the object?
numarray gives us that:
N = numarray.zeros((1000,), numarray.Float) N.info()
class: <class 'numarray.numarraycore.NumArray'> shape: (1000,) strides: (8,) byteoffset: 0 bytestride: 8 itemsize: 8 aligned: 1 contiguous: 1 data: <memory at 009d67b8 with size:8000 held by object 009d6798 aliasing object 00000000> byteorder: little byteswap: 0 type: Float64
In numarray, the offset appears to be 20. If I try to use memmove() to fill a Numeric array it faults when using an offset of 20...
Ray
On Mon, 2005-01-31 at 17:04 -0800, Ray S wrote:
If I have an array N:
N = Numeric.zeros((1000,), Numeric.Float) repr(N.__copy__)
'<built-in method __copy__ of array object at 0x00809F00>'
What is the actual address of the first element?
In C, look at a->data.
Or, as an offset from the object?
This doesn't really make sense given the layout of the objects and how the memory is stored. You should probably read over the section on the C-API in the Numeric and numarray manuals. The Numeric manual, here, is probably the best place to start because it is the simplest:
http://www.pfdubois.com/numpy/numpy.pdf
The numarray manual, here:
http://prdownloads.sourceforge.net/numpy/numarray-1.1.pdf?download
has a section on a Numeric compatibility layer which is very similar.
numarray gives us that:
N = numarray.zeros((1000,), numarray.Float) N.info()
class: <class 'numarray.numarraycore.NumArray'> shape: (1000,) strides: (8,) byteoffset: 0 bytestride: 8 itemsize: 8 aligned: 1 contiguous: 1 data: <memory at 009d67b8 with size:8000 held by object 009d6798 aliasing object 00000000> byteorder: little byteswap: 0 type: Float64
In numarray, the offset appears to be 20. If I try to use memmove() to fill a Numeric array it faults when using an offset of 20...
The offset of 0x20 is unique to the numarray.memory memory object. It has nothing to do with Numeric arrays so it could only work by accident.
Note that the "data" section of numarray's info() is referring to a buffer object which is stored in the _data attribute. I just changed the info() to avoid this confusion in the future by renaming "data" to "buffer". From the overall perspective of the array object, the "working data pointer" (a->data in C) is the sum of the byteoffset and _data base pointer.
In response to this post, I modified info() to the following:
import numarray as na a = na.arange(10) a.info()
class: <class 'numarray.numarraycore.NumArray'> shape: (10,) strides: (4,) byteoffset: 0 bytestride: 4 itemsize: 4 aligned: 1 contiguous: 1 buffer: <memory at 0xb7c07f68 with size:0x00000028 held by object 0xb7f8d5a0 aliasing object 0x00000000> fragile data pointer: 0xb7c07f68 (DEBUG ONLY) byteorder: 'little' byteswap: 0 type: Int32
I think the "fragile data pointer" is generally useful information, but not completely dependable so I gave it the garish name it has. Comments?
Regards, Todd
Thanks Todd, Travis,
Yes, Travis, I _was_ using ctypes' memmove() with numarray to great benefit. I had also understood that the address of the numarray data was stable.
Originally, I was trying to move data from a hardware A/D driver DLL call to Python as quickly as possible and did some benchmarking, so as per Thomas Heller's suggestion on the ctypes list I used memmove(): http://sourceforge.net/mailarchive/forum.php?thread_id=6166311&forum_id=... from ctypes import * import array src = array.array("i", range(32)) dst = (c_int * 32)() memmove(dst, *src.buffer_info()) memmove() is orders of magnitude faster than map() or assignments. I then tested and found numarray faster for the rest of the tasks (FFT etc.) http://sourceforge.net/mailarchive/forum.php?thread_id=6205658&forum_id=... and so parsed the info() output and plugged the data[0] address into the call.
Thomas Heller (in the above) suggested he make a mod to ctypes to accept Python objects which implement the buffer interface as function parameters, which would allow Numeric use once implemented. I like numarray's breadth of methods so I used it at the time, but in another try at speed-squeezing yesterday afternoon I found Numeric's FFT and subtraction to be >30% faster in this case, so I switched that code over (this increase includes the use of map() with Numeric to read the A/D in another thread). Using memmove() with Numeric would speed up the reader thread once again.
At 08:06 AM 2/1/2005 -0500, Todd Miller wrote:
What is the actual address of the first element?
In C, look at a->data.
I had read the Numeric API and looked at the PyObject structure, as Travis then suggested, but my question then is: if the offset from the "array object at 0x..." (object address value) to the array[0] address is not fixed and must be read from the pointer in the PyObject structure, can we get that pointer's value directly from Python or ctypes? ctypes pointer() "allows" poking in and directly reading memory: "It is also possible to use indexes different from 0, but you must know what you're doing when you use this: You access or change arbitrary memory locations when you do this." http://starship.python.net/crew/theller/ctypes/tutorial.html So, could I use ctypes' pointer(offset) to read the structure's pointer-to-data[0]-value and then use it in memmove()? I'll cross-post this to ctypes-users...
I think the "fragile data pointer" is generally useful information, but not completely dependable so I gave it the garish name it has. Comments?
It's quite reasonable. Is the data pointer value really often changed for contiguous arrays in small stand-alone Python apps? If so, I may stick with map() to avoid having to re-read the pointer before each data buffer transfer.
If anyone is interested I could post some relevant code snips from the office...
Thanks for the help, Ray
-
Ray S -
RJ -
Todd Miller