[SciPy-User] HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

[John]

Back and forth can be good, as it brings out the 'juicy' details...
let's just not throwing darts ;)

I've used pytables, and it's been really useful. I guess, I was more
or less trying to find out from the Scipy users who are potentially
working with satellite data, what their tool chains are. Because there
are several different formats distributed HDF-EOS, HDF4, HDF5, etc.
It's a pain, honestly.

Luckily, the support of folks like Francesc and Jeff Whitaker and the
crew at PyNIO seem to making things easier...

The feedback so far has been really helpful. Thanks!

On Mon, Nov 1, 2010 at 11:13 PM, Zachary Pincus <zachary.pincus at yale.edu> wrote:
> Ack -- I didn't mean to set off a big back and forth! I'd only wanted
> to convey that h5py and pytables seem to serve different purposes: one
> a "simple and thin" pythonic wrapper around the official hdf5
> libraries, and the other seeking to provide some value-add on top of
> that.
>
> I guess I got something of the rationale for using h5py over pytables
> wrong -- but there is some rationale, right? What is that?
>
>
> On Nov 1, 2010, at 6:02 PM, Francesc Alted wrote:
>
>> A Monday 01 November 2010 21:19:02 Robert Kern escrigué:
>>> On Mon, Nov 1, 2010 at 15:01, Dav Clark <dav at alum.mit.edu> wrote:
>>>> On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:
>>>>> Where pytables tries to present its own interface, h5py just gives
>>>>> you the hdf5 file. This means that pytables can do a lot of neat
>>>>> things (like the indexed searching), but it also means that (at
>>>>> least last I checked) pytables isn't the best tool for reading in
>>>>> hdf5 files not created by pytables -- for that, you'd want h5py.
>>>>
>>>> Every time I've had an issue with pytables reading a non-pytables
>>>> created file, I've submitted a bug and it got fixed usually in a
>>>> few days. At the time, I was using HDF5 as a transfer layer
>>>> between matlab's rudimentary hdf5 support and python w/ pytables.
>>>> (Thanks Francesc!)
>>>
>>> I just wanted to add that in my experience, you can read just about
>>> any HDF5 file with PyTables except for a few with some more exotic
>>> features.
>>
>> Let me chime in just to try to clarify couple of things.  First, both
>> PyTables and h5py can read most of the HDF5 files out there, but
>> none of
>> them has *complete* support for HDF5 files (implementing complete
>> support for the whole HDF5 standard is really a tough task).  In
>> addition, the last time that I checked this (about one year ago, so
>> things might have changed since then), PyTables can read (and create)
>> HDF5 files that h5py cannot; and the contrary is true too.
>>
>>> If you absolutely need to write an HDF5 file according to a
>>> strict standard without any extra bits, you may need h5py. However,
>>> many other readers of your standard probably won't care about the
>>> extra bits PyTables includes.
>>
>> I suppose that the 'extra bits' you are referring to are the HDF5
>> attributes that complement HDF5 nodes as metainfo.  Let me say that
>> most
>> of these attributes are not PyTables-specific, but those used in the
>> high-level API of HDF5 (http://www.hdfgroup.org/HDF5/doc/HL/).
>> Anyway,
>> as I said many times, if these attributes are causing some trouble to
>> the user (they should not), you can always disable its creation by
>> setting the PYTABLES_SYS_ATTRS parameter to false during the opening
>> of
>> a file (or, if you like this to be permanent, in the
>> tables/parameters.py).  For more info about this, see:
>>
>> http://www.pytables.org/docs/manual/apc.html#id364726
>>
>>> You just have to be a little bit
>>> careful to make sure that you aren't relying on any PyTables
>>> features, like Python-pickled attributes.
>>
>> PyTables only uses pickle when trying to save attributes that are not
>> supported by HDF5 (with the exception of unicode strings that should
>> be
>> implemented soon in PyTables).  For example, if you try to save a list
>> as an attribute:
>>
>> node.attrs.my_attr = [1,2,[3,4]]
>>
>> as such a list cannot be represented by HDF5 natively, PyTables
>> chooses
>> to pickle it and save it.  During retrieval, the pickle is
>> automatically
>> detected and unpickled before being returned to the user.  Of course,
>> you will not be able to read such attributes with a non-Python
>> application.  And, although I consider this like a feature, I can
>> understand that this might be considered as a bug by others (but I
>> have
>> to say that very few PyTables users, if any at all, has ever
>> complained
>> about this 'feature'/'bug').
>>
>> Hope this helps clarifying some points,
>>
>> --
>> Francesc Alted
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Configuration
``````````````````````````
Plone 2.5.3-final,
CMF-1.6.4,
Zope (Zope 2.9.7-final, python 2.4.4, linux2),
Python 2.6
PIL 1.1.6
Mailman 2.1.9
Postfix 2.4.5
Procmail v3.22 2001/09/10
Basemap: 1.0
Matplotlib: 1.0.0