[SciPy-User] Maximum file size for .npz format?
Chris Barker
Chris.Barker at noaa.gov
Fri Mar 12 13:42:21 EST 2010
Paul Anton Letnes wrote:
> On 12. mars 2010, at 09.29, Gökhan Sever wrote:
>> I've experienced similar issues too, but I moved to NetCDF. The
>> only
>> disadvantage was that I did not find any python modules that work well
>> _and_ support numpy. Hence, I am considering moving to HDF5. Which
>> python module would people here recommend? (Or, alternatively, did I
>> miss a great netCDF python module that someone could tell me about?)
yes, numpy support is critical -- why anyone would write a netcdf
wrapper and not use numpy is beyond me.
>> There is http://code.google.com/p/netcdf4-python/
>>
>> I know netcdf4 is a subset of HDF5. What advantages there to use HDF5 not NetCDF4 ?
The way I think about it is that netcdf is a more structured subset of
hdf5. If the structure imposed by netcdf works well for your needs, it's
a good option. There are also things like the CF metadata standard that
make it easier to exchange data with others.
However if you are using it only to save and reload data for your own
app -- pytables may be a better bet.
I've found the netcdf4-python package to be robust and have a nice
interface -- and certainly works well with numpy. My only build issue
with it are actually with getting hd5 built right, the netcdf part has
been easier, and the python bindings very easy -- at least on OS-X.
Windows support is not in as good shape, though if you don't' need
opendap support, I think there are Windows binaries for netcdf4 you
could use.
Also, I think the python bindings still support netcd3, if you don't
need the extra stuff netcdf4 gives you (you may, if you need big files
-- not sure about that).
If netcdf seems like the right way to go for you, then I'm sure you can
get netcdf4-python working -- and Jeff Whitaker is very helpful if you
have trouble.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker at noaa.gov
More information about the SciPy-User
mailing list