NumPy frombuffer giving nonsense values when reading C float array on Windows
sth
urschrei at gmail.com
Tue Jul 26 14:31:23 EDT 2016
On Tuesday, 26 July 2016 19:10:46 UTC+1, eryk sun wrote:
> On Tue, Jul 26, 2016 at 12:06 PM, sth wrote:
> > I'm using ctypes to interface with a binary which returns a void pointer (ctypes c_void_p) to a nested 64-bit float array:
>
> If this comes from a function result, are you certain that its restype
> is ctypes.c_void_p? I commonly see typos here such as setting
> "restypes" instead of "restype".
>
> > [[1.0, 2.0], [3.0, 4.0], … ]
> > then return the pointer so it can be freed
> >
> > I'm using the following code to de-reference it:
> >
> > # a 10-element array
> > shape = (10, 2)
> > array_size = np.prod(shape)
> > mem_size = 8 * array_size
> > array_str = ctypes.string_at(ptr, mem_size)
> > # convert to NumPy array,and copy to a list
> > ls = np.frombuffer(array_str, dtype="float64", count=array_size).reshape(shape).tolist()
> > # return pointer so it can be freed
> > drop_array(ptr)
> > return ls
> >
> > This works correctly and consistently on Linux and OSX using NumPy 1.11.0, but fails on
> > Windows 32 bit and 64-bit about 50% of the time, returning nonsense values. Am I doing
> > something wrong? Is there a better way to do this?
>
> numpy.ctypeslib facilitates working with ctypes functions, pointers
> and arrays via the factory functions as_array, as_ctypes, and
> ndpointer.
>
> ndpointer creates a c_void_p subclass that overrides the default
> from_param method to allow passing arrays as arguments to ctypes
> functions and also implements the _check_retval_ hook to automatically
> convert a pointer result to a numpy array.
>
> The from_param method validates an array argument to ensure it has the
> proper data type, shape, and memory layout. For example:
>
> g = ctypes.CDLL(None) # Unix only
> Base = np.ctypeslib.ndpointer(dtype='B', shape=(4,))
>
> # strchr example
> g.strchr.argtypes = (Base, ctypes.c_char)
> g.strchr.restype = ctypes.c_char_p
>
> d = np.array(list(b'012\0'), dtype='B')
> e = np.array(list(b'0123\0'), dtype='B') # wrong shape
>
> >>> g.strchr(d, b'0'[0])
> b'012'
> >>> g.strchr(e, b'0'[0])
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ctypes.ArgumentError: argument 1: <class 'TypeError'>:
> array must have shape (4,)
>
> The _check_retval_ hook of an ndpointer calls numpy.array on the
> result of a function. Its __array_interface__ property is used to
> create a copy with the defined data type and shape. For example:
>
> g.strchr.restype = Base
>
> >>> d.ctypes._as_parameter_ # source address
> c_void_p(24657952)
> >>> a = g.strchr(d, b'0'[0])
> >>> a
> array([48, 49, 50, 0], dtype=uint8)
> >>> a.ctypes._as_parameter_ # it's a copy
> c_void_p(19303504)
>
> As a copy, the array owns its data:
>
> >>> a.flags
> C_CONTIGUOUS : True
> F_CONTIGUOUS : True
> OWNDATA : True
> WRITEABLE : True
> ALIGNED : True
> UPDATEIFCOPY : False
>
> You can subclass the ndpointer type to have _check_retval_ instead
> return a view of the result (i.e. copy=False), which may be desirable
> for a large result array but probably isn't worth it for small arrays.
> For example:
>
> class Result(Base):
> @classmethod
> def _check_retval_(cls, result):
> return np.array(result, copy=False)
>
> g.strchr.restype = Result
>
> >>> a = g.strchr(d, b'0'[0])
> >>> a.ctypes._as_parameter_ # it's NOT a copy
> c_void_p(24657952)
>
> Because it's not a copy, the array view doesn't own the data, but note
> that it's not a read-only view:
>
> >>> a.flags
> C_CONTIGUOUS : True
> F_CONTIGUOUS : True
> OWNDATA : False
> WRITEABLE : True
> ALIGNED : True
> UPDATEIFCOPY : False
The restype is a ctypes Structure instance with a single __fields__ entry (coords), which is a Structure with two fields (len and data) which are the FFI array's length and the void pointer to its memory:
https://github.com/urschrei/pypolyline/blob/master/pypolyline/util.py#L109-L117
I'm only half-following your explanation of how ctypeslib works, but it seems clear that I'm doing something wrong.
More information about the Python-list
mailing list