[SciPy-User] Maximum file size for .npz format?

Chris Barker Chris.Barker at noaa.gov
Fri Mar 12 13:42:21 EST 2010


Paul Anton Letnes wrote:
> On 12. mars 2010, at 09.29, Gökhan Sever wrote:
>> I've experienced similar issues too, but I moved to NetCDF. The
>> only
>> disadvantage was that I did not find any python modules that work well
>> _and_ support numpy. Hence, I am considering moving to HDF5. Which
>> python module would people here recommend? (Or, alternatively, did I
>> miss a great netCDF python module that someone could tell me about?)

yes, numpy support is critical -- why anyone would write a netcdf 
wrapper and not use numpy is beyond me.

>> There is http://code.google.com/p/netcdf4-python/
>>
>> I know netcdf4 is a subset of HDF5. What advantages there to use HDF5 not NetCDF4 ?

The way I think about it is that netcdf is a more structured subset of 
hdf5. If the structure imposed by netcdf works well for your needs, it's 
a good option. There are also things like the CF metadata standard that 
make it easier to exchange data with others.

However if you are using it only to save and reload data for your own 
app --  pytables may be a better bet.

I've found the netcdf4-python package to be robust and have a nice 
interface -- and certainly works well with numpy. My only build issue 
with it are actually with getting hd5 built right, the netcdf part has 
been easier, and the python bindings very easy -- at least on OS-X. 
Windows support is not in as good shape, though if you don't' need 
opendap support, I think there are Windows binaries for netcdf4 you 
could use.

Also, I think the python bindings still support netcd3, if you don't 
need the extra stuff netcdf4 gives you (you may, if you need big files 
-- not sure about that).

If netcdf seems like the right way to go for you, then I'm sure you can 
get netcdf4-python working -- and Jeff Whitaker is very helpful if you 
have trouble.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



More information about the SciPy-User mailing list