what python module to modify NetCDF data?
Dear all, I want to change some variable values in a series of NetCDF file. Did anybody else did this before using python? Now I use pupynere for reading data from NetCDF files and making plots. but the document of pupynere for writing data to NetCDF file is quite simple and I still feel difficult to do this with pupynere. the NetCDF file I want to change is a global data (0.5X0.5d resolution, 360X720grid with 12 time steps) and have approx. 10 variables. I just want to change some points for a specific variable for all 12 time steps. I know it's possible use NCO ncap2 utility to do the job. but now I have some problem in using ncap2 within a shell script. I guess there is some easy way to use some python module to do the job? like mainly altering the data that need to change while let the others remaining intact? Any idea will be greatly appreciated. I will all a good weekend, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************
Hi, I have never seen myself a NetCDF file but if your NetCDF file is using HDF5 as format (possible since NetCDF 4 if I am not mistaken), you should be able to use h5py or PyTables to access and or modify it. Best regards, Armando Quoting Chao YUE <chaoyuejoy@gmail.com>:
Dear all,
I want to change some variable values in a series of NetCDF file. Did anybody else did this before using python? Now I use pupynere for reading data from NetCDF files and making plots. but the document of pupynere for writing data to NetCDF file is quite simple and I still feel difficult to do this with pupynere.
the NetCDF file I want to change is a global data (0.5X0.5d resolution, 360X720grid with 12 time steps) and have approx. 10 variables. I just want to change some points for a specific variable for all 12 time steps. I know it's possible use NCO ncap2 utility to do the job. but now I have some problem in using ncap2 within a shell script. I guess there is some easy way to use some python module to do the job? like mainly altering the data that need to change while let the others remaining intact?
Any idea will be greatly appreciated. I will all a good weekend,
Chao
-- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************
Quoting Chao YUE <chaoyuejoy@gmail.com>:
Dear all,
I want to change some variable values in a series of NetCDF file. Did anybody else did this before using python? Now I use pupynere for reading data from NetCDF files and making plots. but the document of pupynere for writing data to NetCDF file is quite simple and I still feel difficult to do this with pupynere.
the NetCDF file I want to change is a global data (0.5X0.5d resolution, 360X720grid with 12 time steps) and have approx. 10 variables. I just want to change some points for a specific variable for all 12 time steps. I know it's possible use NCO ncap2 utility to do the job. but now I have some problem in using ncap2 within a shell script. I guess there is some easy way to use some python module to do the job? like mainly altering the data that need to change while let the others remaining intact?
Any idea will be greatly appreciated. I will all a good weekend,
Chao
Hi. Have a look to [1] and [2]. [1] http://code.google.com/p/netcdf4-python/ [2] http://www.scipy.org/doc/api_docs/SciPy.io.netcdf.html
Quoting Chao YUE <chaoyuejoy@gmail.com>:
Dear all,
I want to change some variable values in a series of NetCDF file. Did anybody else did this before using python? Now I use pupynere for reading data from NetCDF files and making plots.
but
the document of pupynere for writing data to NetCDF file is quite simple and I still feel difficult to do this with pupynere.
the NetCDF file I want to change is a global data (0.5X0.5d resolution, 360X720grid with 12 time steps) and have approx. 10 variables. I just want to change some points for a specific variable for all 12 time steps. I know it's possible use NCO ncap2 utility to do the job. but now I have some problem in using ncap2 within a shell script. I guess there is some easy way to use some python module to do the job?
On Saturday, October 8, 2011, Kiko <kikocorreoso@gmail.com> wrote: like
mainly altering the data that need to change while let the others remaining intact?
Any idea will be greatly appreciated. I will all a good weekend,
Chao
Hi.
Have a look to [1] and [2].
[1] http://code.google.com/p/netcdf4-python/ [2] http://www.scipy.org/doc/api_docs/SciPy.io.netcdf.html
Just a caveat about scipy.io.netcdf, it only reads and writes. It does not modify. So, what one has to do with that module is to load up all data, modify some of it, and then save it all back to a new file. Ben Root
On 8 oct. 11, at 17:57, Vicente Sole wrote:
I have never seen myself a NetCDF file but if your NetCDF file is using HDF5 as format (possible since NetCDF 4 if I am not mistaken), you should be able to use h5py or PyTables to access and or modify it.
I haven't tried this, but I don't think it's a good plan. PyTables can read arbitrary HDF5 files, but writes only a subset. The HDF5-based netCDF 4 format is also a subset of valid HDF5 files. It is unlikely that those two subsets are compatible, meaning that sooner or later PyTables will produce a file that netCDF 4 will not accept. h5py would be a better choice for an HDF5-based approach. However, there are two reasons for using a specific netCDF interface rather than HDF5: 1) There are still plenty of non-HDF5 netCDF files around, and netCDF4 continues to support them. 2) The netCDF interface is simpler than the HDF5 interface, and therefore easier to use, even if that difference is much less important in Python than in C or Fortran. A full netCDF interface has been available for many years as part of the ScientificPython package: http://dirac.cnrs-orleans.fr/ScientificPython/ScientificPythonManual/ Recent versions fully support netCDF4, including the HDF5-based formats. There are other Python interfaces to the netCDF libraries, but I haven't used them. Konrad.
Thanks for all for very useful discussions. I am just on the beginning of modeling and for now I only use netCDF4 data. I think it might be a good idea to try netcdf4-python<http://code.google.com/p/netcdf4-python/>first. I think whitaker will continue to develop this package. But it gave me a good idea what python package people are using for netCDF data. Chao 2011/10/9 Konrad Hinsen <konrad.hinsen@fastmail.net>
On 8 oct. 11, at 17:57, Vicente Sole wrote:
I have never seen myself a NetCDF file but if your NetCDF file is using HDF5 as format (possible since NetCDF 4 if I am not mistaken), you should be able to use h5py or PyTables to access and or modify it.
I haven't tried this, but I don't think it's a good plan. PyTables can read arbitrary HDF5 files, but writes only a subset. The HDF5-based netCDF 4 format is also a subset of valid HDF5 files. It is unlikely that those two subsets are compatible, meaning that sooner or later PyTables will produce a file that netCDF 4 will not accept. h5py would be a better choice for an HDF5-based approach.
However, there are two reasons for using a specific netCDF interface rather than HDF5:
1) There are still plenty of non-HDF5 netCDF files around, and netCDF4 continues to support them. 2) The netCDF interface is simpler than the HDF5 interface, and therefore easier to use, even if that difference is much less important in Python than in C or Fortran.
A full netCDF interface has been available for many years as part of the ScientificPython package:
http://dirac.cnrs-orleans.fr/ScientificPython/ScientificPythonManual/
Recent versions fully support netCDF4, including the HDF5-based formats.
There are other Python interfaces to the netCDF libraries, but I haven't used them.
Konrad.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************
On 10/8/11 7:45 AM, Chao YUE wrote:
I want to change some variable values in a series of NetCDF file. Did anybody else did this before using python?
I like the netcdf4 package -- very powerful and a pretty nice numpy-compatible API: http://code.google.com/p/netcdf4-python/ Check out the "examples" and "utils" directories for some code to take a look at. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (6)
-
Benjamin Root -
Chao YUE -
Chris Barker -
Kiko -
Konrad Hinsen -
Vicente Sole