MFDatasets and NetCDF4
Hi, I hope this is the right place to ask this. I've found the MFDataset works well in reading NetCDF3 files, but it appears that it doesn't work at present for NetCDF4 files. Is this an inherent problem with the NetCDF4 file structure, or would it be possible to implement the MFDataset for NetCDF4 files sometime? It would be very useful. --George Nurser.
Hi George, On Fri, Sep 25, 2009 at 6:55 AM, George Nurser <gnurser@googlemail.com>wrote:
Hi, I hope this is the right place to ask this. I've found the MFDataset works well in reading NetCDF3 files, but it appears that it doesn't work at present for NetCDF4 files.
It works on my side for netCDF4 files. What error are you getting ?
Is this an inherent problem with the NetCDF4 file structure, or would it be possible to implement the MFDataset for NetCDF4 files sometime? It would be very useful.
From the docstring:
Datasets must be in C{NETCDF4_CLASSIC, NETCDF3_CLASSIC or NETCDF3_64BIT} format (C{NETCDF4} Datasets won't work). I suspect your files are not in CLASSIC mode. NETCDF4 datasets are allowed to have a more complex hierarchy than the CLASSIC mode, and I think this is what makes concatenation difficult to implement. That is, there would be no simple rule to determine which fields should be concatenated. David
--George Nurser. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
2009/9/25 David Huard <david.huard@gmail.com>:
Hi George,
On Fri, Sep 25, 2009 at 6:55 AM, George Nurser <gnurser@googlemail.com> wrote:
Hi, I hope this is the right place to ask this. I've found the MFDataset works well in reading NetCDF3 files, but it appears that it doesn't work at present for NetCDF4 files.
It works on my side for netCDF4 files. What error are you getting ?
Is this an inherent problem with the NetCDF4 file structure, or would it be possible to implement the MFDataset for NetCDF4 files sometime? It would be very useful.
From the docstring:
Datasets must be in C{NETCDF4_CLASSIC, NETCDF3_CLASSIC or NETCDF3_64BIT} format (C{NETCDF4} Datasets won't work).
I suspect your files are not in CLASSIC mode.
I haven't tried it yet:) I was just trusting the documentation (which I've found to be very good). Doing ncdump -k on my files gives 'netCDF-4' so I guess the files are indeed not in CLASSIC mode.
NETCDF4 datasets are allowed to have a more complex hierarchy than the CLASSIC mode, and I think this is what makes concatenation difficult to implement. That is, there would be no simple rule to determine which fields should be concatenated.
David
Ok. I suspected it was something like that. If the netCDF-4 CLASSIC mode allows data compression and chunking, which I think it is supposed to do, perhaps I can persuade the guys running the model to produce files in netCDF-4 CLASSIC mode. Many thanks. George.
Hello list, I just thought I'd point out a difference between 'import numarray' and 'import numpy.numarray' . Consider the following In [1]: from numpy.numarray import * In [2]: d = array((1,2,3,4)) In [3]: f = reshape(d,(2,2)) In [4]: print f [[1 2] [3 4]] In [5]: f.transpose() Out[5]: array([[1, 3], [2, 4]]) In [6]: print f [[1 2] [3 4]] Now in pure numarray, f would have changed to the transposed form, so that the output from [5] and [6] would match, and be different from that of [4]. (I don't have numarray installed myself but a workmates computer, and examples on the web have the usage f.transpose() . Now, In [7]: f = f.transpose() In [8]: print f [[1 3] [2 4]] as expected. I mention this because I think that it is worth knowing having lost a LOT of time to it. Is it worth filing as a bug report? Michael Walker Plant Modelling Group CIRAD, Montpellier 04 67 61 57 27
Mon, 28 Sep 2009 10:07:47 +0200, Michael.Walker wrote: [clip]
In [7]: f = f.transpose()
In [8]: print f [[1 3] [2 4]]
as expected. I mention this because I think that it is worth knowing having lost a LOT of time to it. Is it worth filing as a bug report?
Yes. It indeed seems that in numarray, transpose() transposes the array in-place. This could maybe be fixed by a new numarray-emulating ndarray subclass. The tricky problem then is that some functions don't, IIRC, preserve subclasses, which may lead to surprises. (Anyway, these should be fixed at some point...) At the least, we should write a well-visible "differences to numarray" document that explains all differences and known bugs. -- Pauli Virtanen
On 09/28/2009 03:15 AM, Pauli Virtanen wrote:
Mon, 28 Sep 2009 10:07:47 +0200, Michael.Walker wrote: [clip]
In [7]: f = f.transpose()
In [8]: print f [[1 3] [2 4]]
as expected. I mention this because I think that it is worth knowing having lost a LOT of time to it. Is it worth filing as a bug report?
Yes. It indeed seems that in numarray, transpose() transposes the array in-place.
This could maybe be fixed by a new numarray-emulating ndarray subclass. The tricky problem then is that some functions don't, IIRC, preserve subclasses, which may lead to surprises. (Anyway, these should be fixed at some point...)
At the least, we should write a well-visible "differences to numarray" document that explains all differences and known bugs.
This is not a bug! This specific difference between numpy and numarray is documented on the 'converting from numarray' page: http://www.scipy.org/Converting_from_numarray What actually is incorrect is that the numpy.numarray.transpose has the same docstring as numpy.transpose. So it would be very helpful to first correct the numpy.array.transpose documentation. A larger goal would be to correctly document all the numpy.numarray and numpy.numeric functions as these should not be linked to the similar numpy functions. If these are identical then it should state that, what differences exist and then refer to equivalent numpy page for example, numpy.numarray.matrixmultiply and numpy.dot. Also, the documentation for these numpy.numarray and numpy.numeric functions should state that these are mainly included for compatibility reasons and may be removed at a future date. Bruce
Mon, 28 Sep 2009 09:29:30 -0500, Bruce Southey wrote: [clip]
This is not a bug! This specific difference between numpy and numarray is documented on the 'converting from numarray' page: http://www.scipy.org/Converting_from_numarray
Oh. I completely missed that page. Now, it should just be transferred to the main documentation. Also, it might be possible to make numpy.numarray.ndarray different from numpy.ndarray. But I doubt this is high priority -- it may be more efficient just to document the fact.
What actually is incorrect is that the numpy.numarray.transpose has the same docstring as numpy.transpose. So it would be very helpful to first correct the numpy.array.transpose documentation.
numpy.numarray.transpose is numpy.transpose, so fixing this would involve implementing the numarray-style transpose, too. -- Pauli Virtanen
On 09/28/2009 03:15 AM, Pauli Virtanen wrote:
Mon, 28 Sep 2009 10:07:47 +0200, Michael.Walker wrote: [clip]
In [7]: f = f.transpose()
In [8]: print f [[1 3] [2 4]]
as expected. I mention this because I think that it is worth knowing having lost a LOT of time to it. Is it worth filing as a bug report?
This is not a bug! This specific difference between numpy and numarray is documented on the 'converting from numarray' page: http://www.scipy.org/Converting_from_numarray
I am referring to the behaviour of numpy.numarray.transpose() being that of numpy.transpose() instead of numarray.transpose. One expects that numpy.numarray would function as numarray, for the purpose of backwards compatability. So, is this worth filing a bug report for? Michael Walker Plant Modelling Group CIRAD, Montpellier 04 67 61 57 27
Tue, 29 Sep 2009 11:08:42 +0200, Michael.Walker wrote: [clip]
I am referring to the behaviour of numpy.numarray.transpose() being that of numpy.transpose() instead of numarray.transpose. One expects that
You probably mean the transpose methods numpy.numarray.ndarray.transpose and numarray.ndarray.transpose. The transpose functions function identically.
numpy.numarray would function as numarray, for the purpose of backwards compatability. So, is this worth filing a bug report for?
There is no harm in creating a bug ticket. This is at least a documentation issue even in the case the main problem becomes a wontfix. -- Pauli Virtanen
participants (5)
-
Bruce Southey -
David Huard -
George Nurser -
Michael.Walker＠sophia.inria.fr -
Pauli Virtanen