[Numpy-discussion] record arrays with char*?

Tue Feb 11 11:57:05 EST 2014

Thanks for the answers! My responses are inline.

On Tue, Feb 11, 2014 at 8:36 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Mon, Feb 10, 2014 at 10:43 PM, Christopher Jordan-Squire
> <cjordan1 at uw.edu> wrote:
>>
>> I'm trying to wrap some C code using cython. The C code can take
>> inputs in two modes: dense inputs and sparse inputs. For dense inputs
>> the array indexing is naive. I have wrappers for that. In the sparse
>> case the matrix entries are typically indexed via names. So, for
>> example, the library documentation includes this as input you could
>> give:
>>
>> struct
>> {
>>  char* ind;
>>  double val, wght;
>> } data[] = { {"camera", 15, 2}, {"necklace", 100, 20}, {"vase", 90, 20},
>>                  {"pictures", 60, 30}, {"tv", 40, 40}, {"video", 15, 30}};
>>
>> At the C level, data is passed to the function by directly giving its
>> address. (i.e. the C function takes as an argument (unsigned long)
>> data, casting the data pointer to an int)
>
>
> wow -- that's prone to error! but I"m still not sure which pointer you're
> talking about -- a pointer to this struct?
>

I mean data. data is an array of structs, but in C that's
(essentially) the same as a pointer to a struct. So data is the
pointer I'm referring to.

>> I'd like to create something similar using record arrays, such as
>>
>> np.array([("camera", 15, 2), ("necklace", 100, 20), ... ],
>> dtype='object,f8,f8').
>
>
>
>>
>> Unfortunately this fails because
>> (1) In cython I need to determine the address of the first element and
>> I can't take the address of a an input whose type I don't know (the
>> exact type will vary on the application, so more or fewer fields may
>> be in the C struct)
>
>
> still a bit confused, but if this is types as an array in Cython, you should
> be abel to do somethign like:
>
> &the_array[i]
>
> to get the address of the ith element.
>

To do that I need to be able to tell cython the type of the memory
view I give. There are very few examples for non-primitive arrays in
cython, so I'm not sure what that would look like. Or at least I
*think* I need to do that, based on the cython errors I'm getting.

>> (2) I don't think a python object type is what I want--I need a char*
>> representation of the string. (Unfortunately I can't test this because
>> I haven't solved (1) -- how do you pass a record array around in
>> cython and/or take its address?)
>
>
> well, and object type will give you a pointer to a pyobject. If you know for
> sure that that pyobject is a string object (probably want a bytes object --
> you son't want unicode here), then you should be abel to get the address of
> the underlying char array. But that would require passing something
> different off to the C code that the address of that element.
>
> You could use an unsigned long for that first field, as you are assuming
> that in the C code anyway  but I don't hink there is a way in numpy to set
> that to a pointer to a char allocated elsewhere -- where would it be
> allocated?
>

The strings are char*, the unsigned long cast was simply to cast data
to a memory address. (The specific setup of the library was passing a
string with the memory location in hexadecimal. This seems weird, but
it's because the C functions being called are an intermediary to a
simple (third-party) virtual machine running a program compiled from
another language. It doesn't deal with pointers directly, so the other
language is just passed the memory address, in hex and as a string,
for where the data resides. Along with the number of elements in the
block of memory pointed to.)

> So I would give up on expecting to store the struct directly in numpy array,
> and rather, put something reasonable (maybe what you have above) in the
> numpy array, and build the C struct you need from that rather than passing a
> pointer in directly.
>

That sounds reasonable. I really wanted to avoid this because, as I
mentioned above, I'm just trying to generate the data in numpy and
pass it to this virtual machine running a program compiled from
another language. The form of the struct depends on what was done in
the other language. It could easily have more or fewer fields, have
the fields reordered, etc.. I wanted to avoid having to write wrappers
for all such possibilities and instead use numpy record arrays as
mapping exactly to C structs. But if that's really the only way to go
then I guess I'll have to write struct wrappers in cython.

> -Chris
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>