[Numpy-discussion] fast numpy i/o

[Christopher Barker]

Robert Kern wrote:
> https://raw.github.com/numpy/numpy/master/doc/neps/npy-format.txt

Just a note. From that doc:

"""
     HDF5 is a complicated format that more or less implements
     a hierarchical filesystem-in-a-file.  This fact makes satisfying
     some of the Requirements difficult.  To the author's knowledge, as
     of this writing, there is no application or library that reads or
     writes even a subset of HDF5 files that does not use the canonical
     libhdf5 implementation.
"""

I'm pretty sure that the NetcdfJava libs, developed by Unidata, use 
their own home-grown code. netcdf4 is built on HDF5, so that qualifies 
as "a library that reads or writes a subset of HDF5 files". Perhaps 
there are lessons to be learned there. (too bad it's Java)

"""
     Furthermore, by
     providing the first non-libhdf5 implementation of HDF5, we would
     be able to encourage more adoption of simple HDF5 in applications
     where it was previously infeasible because of the size of the
     library.
"""

I suppose this point is still true -- a C lib that supported a subset of 
hdf would be nice.

That being said, I like the simplicity of the .npy format, and I don't 
know that anyone wants to take any of this on anyway.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov