Buffers and pointers (Py3.4)
rgaddi at highlandtechnology.invalid
Mon Apr 17 13:58:31 EDT 2017
So long as we're all feeling so concerned about speed lately...
I've got a piece of hardware that allows me to call a C library that
calls an ioctl() that kicks off a DMA transfer and moves data between
process memory and external hardware. I would like a way to turn an
arbitrary object implementing the buffer protocol into a pointer so that
I can pass that pointer to the library to be passed to the hardware to
stream my data.
The obvious question on your minds is why on earth am I doing this.
Example case: my hardware is being used as a bridge to talk to a 16
channel transient digitizer, acquiring data at 500 ksps per channel. I
would like my Python application to be able to acquire, say, 10M samples
per channel. I can have a list of 16 10M NumPy arrays, and every 5 ms I
can request a single DMA transfer to grab all the new data that's been
acquired and push it, zero-copy, into my arrays before the hardware
FIFOs get full and I start losing data.
If I knew the buffers would always be NumPy arrays I could use
ndarray.ctypes.data to get the pointer. Always ctypes buffers, I could
just use ctypes.addressof. In either case I can flip them into
memoryviews, but nothing seems to give me a pointer from that.
Bytearrays, array.arrarys..., I've got nothin'.
If I were writing this as a C extension, getting that information from
any buffer object would be trivial, but that changes my project from a
pure Python wrapper using ctypes to a mixed language project with
compile dependencies and mental headaches.
buffertype = c_uint8 * size
return addressof(buffertype.from_buffer(buf, offset))
works but is inefficient and woefully inelegant. I saw some discussion
on the ctypes bug tracker https://bugs.python.org/issue27274 about this
nearly a year ago, but no actual action.
I really want to take all of this low-level horribleness, wrap it up
into a higher-level API, and never be subject to the gory details again.
This pointer thing is about the last hangup. Ideas?
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
More information about the Python-list