[Numpy-discussion] Using np.frombuffer and cffi.buffer on array of C structs (problem with struct member padding)

[Allan Haldane]

On 01/31/2018 02:06 AM, Joe wrote:
> Does someone know of a function or a convenient way to automatically
> derive a dtype object from a C typedef struct string or a cffi.typeof()?

I remember when I wrote those docs over a year ago, I searched for an 
established way to do this but didn't find one. If you find/come up with 
one, let us know and I might add a pointer or description of it in the docs.

Searching just now, I came across this recent gist:

https://gist.github.com/inactivist/4ef7058c2132fa16759d

So it looks like we can do something like:

     # typemap from C type to numpy type may need some work...
     typemap = {'char': 'byte', 'short': 'short', 'int': 'intc',
                'long long': 'longlong', 'size_t': 'intp',
                'float': 'float', 'double': 'double'}

     def extract_field_info(tp):
         fields = []
         for field, fieldtype in tp.fields:
             if fieldtype.type.kind == 'primitive':
                 format = typemap[fieldtype.type.cname]
             else:
                 format = extract_field_info(fieldtype.type)
             fields.append((field, format, fieldtype.offset))
         return fields

      from cffi import FFI
      ffi = FFI()
      ffi.cdef("struct foo { int a; char b; double d;};")
      tp = ffi.typeof('struct foo')

      names, formats, offsets = zip(*extract_field_info(tp))
      np.dtype({'names': names, 'formats': formats, 'offsets': offsets})
      # compare to: np.dtype('i4,i1,f8', align=True)

I just made that up now and it may be buggy and have missing corner 
cases. But if you agree it works, maybe we can mention it in the 
structured array docs instead of the vague sentence there now, that 
"some work may be needed to obtain correspondence with the C struct".

Allan

> Am 27.01.2018 10:30 schrieb Joe:
>> Thanks for your help on this! This solved my issue.
>>
>>
>> Am 25.01.2018 um 19:01 schrieb Allan Haldane:
>>> There is a new section discussing alignment in the numpy 1.14 structured
>>> array docs, which has some hints about interfacing with C structs.
>>>
>>> These new 1.14 docs are not online yet on scipy.org, but in the meantime
>>>   you can view them here:
>>> https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-alignment 
>>>
>>>
>>> (That links specifically to the discussion of alignments and padding).
>>>
>>> Allan
>>>
>>> On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:
>>>>
>>>>>
>>>>> The numpy dtype constructor takes an “align” keyword that will pad it
>>>>> for you.
>>>>
>>>> https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html 
>>>>
>>>>
>>>> -CHB
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion