GDF PR--call for some feedback
Hi all, I have a work-in-progress PR in the hopper regarding GDF reading and writing: https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing-... What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it. However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt). The two main changes are: 1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes). 2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units. Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now. I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted. Best, John -- John ZuHone Postdoctoral Researcher NASA/Goddard Space Flight Center jzuhone@gmail.com john.zuhone@nasa.gov
Hi John,
I'm not averse to the changes in the GDF standard you propose. I think now
that yt-3.0 is incorporating units as cleanly as it is, this makes sense to
allow the format to specify an arbitary unittype instead of requiring cgs,
since it can be so easily modified within current/future versions of yt.
I honestly don't know how many people are using GDF, but it seems like
these changes would break backwards-compatibility. Is it possible for us
to retain that compatibility by automatically running "field_to_cgs" on GDF
files which do not possess the HDF5 attributes specifying the units? Or
perhaps since yt is the main source of GDF files, we can just move forward
without having to worry about too much backwards-compatibility issues?
Anyway, this seems like a positive change. +1 to continue on this.
Cameron
On Sat, Jul 12, 2014 at 11:48 AM, John Zuhone
Hi all,
I have a work-in-progress PR in the hopper regarding GDF reading and writing:
https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing-...
What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it.
However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt).
The two main changes are:
1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes).
2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units.
Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now.
I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted.
Best,
John
-- John ZuHone
Postdoctoral Researcher NASA/Goddard Space Flight Center
jzuhone@gmail.com john.zuhone@nasa.gov _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
On 12.07.2014 20:48, John Zuhone wrote:
Hi all,
I have a work-in-progress PR in the hopper regarding GDF reading and writing:
https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing-...
What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it.
However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt).
The two main changes are:
1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes).
2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units.
Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now.
I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted.
Hi John, I'm not sure I understand. Currently fields *are* stored in arbitrary units. That's why "field_to_cgs" was necessary to make yt use them in a consistent fashion. However, now I think that with YTArray around we can go one step further and just directly feed constructor with `field_units`. "dataset_units" would be then unnecessary and each field could have separate unit system e.g. density in cm^-3, velocity in km/s etc. "field_to_cgs" could stay for backwards compatibility sake. This would require some additional work with things that are not pure data and also have units, like box size, current time etc. Possibly we'd need a way to define units in GDF file for all those people still using "dinosaurs" or "leagues" Cheers, Kacper
Hi Kacper,
I'm not sure I understand. Currently fields *are* stored in arbitrary units. That's why "field_to_cgs" was necessary to make yt use them in a consistent fashion.
As it stood, the GDF frontend in 3.0 was not doing anything with the units. It was returning everything as dimensionless. Also, the field units were being stored in LaTeX, so I don't know if it was able to read them.
However, now I think that with YTArray around we can go one step further and just directly feed constructor with `field_units`. "dataset_units" would be then unnecessary and each field could have separate unit system e.g. density in cm^-3, velocity in km/s etc. "field_to_cgs" could stay for backwards compatibility sake.
Sounds good to me, but see below about "dataset_units."
This would require some additional work with things that are not pure data and also have units, like box size, current time etc. Possibly we'd need a way to define units in GDF file for all those people still using "dinosaurs" or "leagues"
But this is exactly what "dataset_units" is for, not for the fields themselves but for those parameters. That way we can set the attributes that hang off the dataset directly, e.g., ds.length_unit. John
On 12.07.2014 23:46, John ZuHone wrote:
Hi Kacper,
I'm not sure I understand. Currently fields *are* stored in arbitrary units. That's why "field_to_cgs" was necessary to make yt use them in a consistent fashion.
As it stood, the GDF frontend in 3.0 was not doing anything with the units. It was returning everything as dimensionless. Also, the field units were being stored in LaTeX, so I don't know if it was able to read them.
It certainly wasn't. I was merely suggesting an extension to the existing format. I think YTArray is able to spit out nice LaTeX representation of its units (maybe Nathan will correct me here), so current function of "field_units" is redundant.
However, now I think that with YTArray around we can go one step further and just directly feed constructor with `field_units`. "dataset_units" would be then unnecessary and each field could have separate unit system e.g. density in cm^-3, velocity in km/s etc. "field_to_cgs" could stay for backwards compatibility sake.
Sounds good to me, but see below about "dataset_units."
This would require some additional work with things that are not pure data and also have units, like box size, current time etc. Possibly we'd need a way to define units in GDF file for all those people still using "dinosaurs" or "leagues"
But this is exactly what "dataset_units" is for, not for the fields themselves but for those parameters. That way we can set the attributes that hang off the dataset directly, e.g., ds.length_unit.
OK, that makes sense. Cheers, Kacper
John _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
On Sun, Jul 13, 2014 at 12:59 AM, Kacper Kowalik
On 12.07.2014 23:46, John ZuHone wrote:
Hi Kacper,
I'm not sure I understand. Currently fields *are* stored in arbitrary units. That's why "field_to_cgs" was necessary to make yt use them in a consistent fashion.
As it stood, the GDF frontend in 3.0 was not doing anything with the units. It was returning everything as dimensionless. Also, the field units were being stored in LaTeX, so I don't know if it was able to read them.
It certainly wasn't. I was merely suggesting an extension to the existing format. I think YTArray is able to spit out nice LaTeX representation of its units (maybe Nathan will correct me here), so current function of "field_units" is redundant.
For some YTArray named arr: arr.units.latex_representation()
However, now I think that with YTArray around we can go one step further and just directly feed constructor with `field_units`. "dataset_units" would be then unnecessary and each field could have separate unit system e.g. density in cm^-3, velocity in km/s etc. "field_to_cgs" could stay for backwards compatibility sake.
Sounds good to me, but see below about "dataset_units."
This would require some additional work with things that are not pure data and also have units, like box size, current time etc. Possibly we'd need a way to define units in GDF file for all those people still using "dinosaurs" or "leagues"
But this is exactly what "dataset_units" is for, not for the fields
themselves but for those parameters. That way we can set the attributes that hang off the dataset directly, e.g., ds.length_unit.
OK, that makes sense. Cheers, Kacper
John _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi John, I just looked over the GDF format spec you linked to. It's a nice
goal to have a general format, and I know from previous attempts that it
can be hard to fit everyone in. Perhaps for the next spec, you might
consider making ``refine_by`` two-dimensional, with one dimension referring
to the level index and the other referring to the dimension. I think that
this would be more general. (Although I'll note that in Maestro, we always
ignore the second dimension and just assume all coordinate directions jump
by the same amount).
On Sat, Jul 12, 2014 at 2:48 PM, John Zuhone
Hi all,
I have a work-in-progress PR in the hopper regarding GDF reading and writing:
https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing-...
What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it.
However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt).
The two main changes are:
1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes).
2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units.
Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now.
I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted.
Best,
John
-- John ZuHone
Postdoctoral Researcher NASA/Goddard Space Flight Center
jzuhone@gmail.com john.zuhone@nasa.gov _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Michael Zingale Associate Professor Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 *phone*: 631-632-8225 *e-mail*: Michael.Zingale@stonybrook.edu *web*: http://www.astro.sunysb.edu/mzingale
Hi John,
On Sat, Jul 12, 2014 at 1:48 PM, John Zuhone
Hi all,
I have a work-in-progress PR in the hopper regarding GDF reading and writing:
https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing-...
What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it.
However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt).
The two main changes are:
1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes).
2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units.
I'm in favor of both of these things, along with MikeZ's suggestion about refine by. I think we should call this a 1.1 standard, and then we can evaluate a few other big holes with a 2.0 in the future. We should also probably allow units like B-field, velocity, etc, as these have in the past had odd, non-conforming properties.
Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now.
I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted.
Best,
John
-- John ZuHone
Postdoctoral Researcher NASA/Goddard Space Flight Center
jzuhone@gmail.com john.zuhone@nasa.gov _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
After looking at this some more I've discovered a couple of other wrinkles.
My first thought was to set the dataset units (length, time, mass, etc.) for backup GDF files to be equal to that of the parent dataset, so there's consistency there. However, I then realized that most fields written to backup files would be in cgs, not code units. If any of these fields have aliases (like "density", "velocity_x", etc.), then they will get converted "to cgs" using the conversion factors even if they don't need to be (e.g., they already are in cgs).
So, the backup file could have code units equal to cgs, which would relieve this problem but would leave its code units inconsistent with that of the parent file, or we could find some way to keep the underlying code units of the files the same but prevent this conversion from happening (either by removing the aliases altogether from the GDF frontend or some other procedure).
Would appreciate feedback on this.
On Jul 14, 2014, at 1:49 PM, Matthew Turk
Hi John,
On Sat, Jul 12, 2014 at 1:48 PM, John Zuhone
wrote: Hi all,
I have a work-in-progress PR in the hopper regarding GDF reading and writing:
https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing-...
What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it.
However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt).
The two main changes are:
1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes).
2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units.
I'm in favor of both of these things, along with MikeZ's suggestion about refine by. I think we should call this a 1.1 standard, and then we can evaluate a few other big holes with a 2.0 in the future.
We should also probably allow units like B-field, velocity, etc, as these have in the past had odd, non-conforming properties.
Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now.
I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted.
Best,
John
-- John ZuHone
Postdoctoral Researcher NASA/Goddard Space Flight Center
jzuhone@gmail.com john.zuhone@nasa.gov _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hello,
Time to drag this up from the dead...
I am currently in the process of making my simulation code write GDF files,
and also write a frontend to read them and provide some extra goodies for
our code.
Having had a look over this thread and the associated PR, the units top
level block sounds useful. Are there plans to add it to the standard
document? Is it documented anywhere else?
Stuart
On 15 July 2014 23:35, John ZuHone
After looking at this some more I've discovered a couple of other wrinkles.
My first thought was to set the dataset units (length, time, mass, etc.) for backup GDF files to be equal to that of the parent dataset, so there's consistency there. However, I then realized that most fields written to backup files would be in cgs, not code units. If any of these fields have aliases (like "density", "velocity_x", etc.), then they will get converted "to cgs" using the conversion factors even if they don't need to be (e.g., they already are in cgs).
So, the backup file could have code units equal to cgs, which would relieve this problem but would leave its code units inconsistent with that of the parent file, or we could find some way to keep the underlying code units of the files the same but prevent this conversion from happening (either by removing the aliases altogether from the GDF frontend or some other procedure).
Would appreciate feedback on this.
On Jul 14, 2014, at 1:49 PM, Matthew Turk
wrote: Hi John,
On Sat, Jul 12, 2014 at 1:48 PM, John Zuhone
wrote: Hi all,
I have a work-in-progress PR in the hopper regarding GDF reading and writing:
https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing-...
What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it.
However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt).
The two main changes are:
1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes).
2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units.
I'm in favor of both of these things, along with MikeZ's suggestion about refine by. I think we should call this a 1.1 standard, and then we can evaluate a few other big holes with a 2.0 in the future.
We should also probably allow units like B-field, velocity, etc, as these have in the past had odd, non-conforming properties.
Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now.
I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted.
Best,
John
-- John ZuHone
Postdoctoral Researcher NASA/Goddard Space Flight Center
jzuhone@gmail.com john.zuhone@nasa.gov _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (8)
-
Cameron Hummels
-
John Zuhone
-
John ZuHone
-
Kacper Kowalik
-
Matthew Turk
-
Michael Zingale
-
Nathan Goldbaum
-
Stuart Mumford