[Numpy-discussion] ragged array implimentation

[Jeff Whitaker]

On 3/7/11 11:42 AM, Christopher Barker wrote:
> On 3/7/11 9:33 AM, Francesc Alted wrote:
>> A Monday 07 March 2011 18:28:11 Christopher Barker escrigué:
>>> I'm setting out to write some code to access and work with ragged
>>> arrays stored in netcdf files. It dawned on me that ragged arrays
>>> are not all that uncommon, so I'm wondering if any of you have any
>>> code you've developed that I could learn-from borrow from, etc.
>> A list of numpy arrays would not be enough?  Or you want something more
>> specific?
> Maybe that would, but in mapping to the netcdf data model, I'm thinking
> more like a big 1-d numpy array, with a way to index into it. Also, that
> would allow you to do some math with the arrays, if the broad casting
> made sense, anyway.
>
> But now that you've entered the conversation, does HDF and/or pytables
> have a standard way of dealing with this?
>
> On 3/7/11 9:37 AM, Jeff Whitaker wrote:
>> Chris:  The netcdf4-python modules reads netcdf vlen arrays
> arethose a netcdf4 feature?
Chris:

Yes, although I don't think many people are using it.

See section 10 in 
http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4-module.html.

> So far, I'm still workign with netcdf3 --
> though this could be a compelling reason to move on!
>
> We've talked about this some on the CF list, and I don't think anyone
> brought that up.
>
>> and returns
>> numpy object arrays, where the elements of the object arrays are
>> themselves 1d numpy arrays. I don't think there is any other way to do
>> it.  In your example, the 'ragged' array would be a 1d numpy array with
>> dtype='O', and the individual elements would be 1d numpy arrays with
>> dtype=int.  Of course, these arrays are very awkward to deal with and
>> operations will be slow.
> Depends some of the operations, but yes.
>
> That still may be the best option, but I'm exploring others.
>
> is a "vlen array" stored contiguously in netcdf?
Probably, although I don't know for sure what the underlying HDF5 layer 
is doing.

-Jeff
> -Chris
>
>
>
>


-- 
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/PSD  R/PSD1        Email  : Jeffrey.S.Whitaker at noaa.gov
325 Broadway                Office : Skaggs Research Cntr 1D-113
Boulder, CO, USA 80303-3328 Web    : http://tinyurl.com/5telg