[Numpy-discussion] constructing record dtypes from the c-api

[Jason Newton]

Hi folks,

The moderator for the ML approved my subscription so I can now post
this back in the numpy list rather than scipy.  Apologies for the
duplicate/cross posting.


I was trying to figure out how to make a dtype for a c-struct on the
c-side and storing that in some boost python libraries I'm making.

Imagine the following c-struct, greatly simplified of course from the
real ones I need to expose:

struct datum{
    double position[3];
    float velocity[3];
    int32_t d0;
    uint64_t time_of_receipt;
};


How would you make the dtype/PyArray_Descr for  this?

I have as a point of reference compound types in HDF for similar
struct descriptions (h5py makes these nearly 1:1 and converts back and
forth to dtypes and hdf5 types, it uses Cython to accomplish this) -
but I don't want to bring in hdf for this task - I'm not sure how well
the offsets would go over in that translation to h5py too.

Proper/best solutions would make use of offsetof as we insert entries
to the dtype/PyArray_Descr.  It's fine if you do this in straight C -
I just need to figure out how to accomplish this in a practical way.

The language I'm working in is C++11.  The end goal is probably going
to be to create a helper infrastructure to allow this to be made
automatically-ish provided implementation of a [static] visitor
pattern in C++.  The idea is to make numpy compatible c++ POD types
rather than use Boost.Python wrapped proxies for every object which
will cut down on some complicated and time consuming code (both of
computer and developer) when ndarrays are what's called for.

Related - would one work with record dtypes passed to C?  How would
one lookup fields and offsets within a structure?

Thanks for any advisement!

-Jason