Hello YT users, A couple of years ago, Sam Skillman (email chain attached) asked about support for saving derived fields to the disk, so we don't have to re-compute expensive fields over and over again. Did this functionality ever make it into yt? I'm working on something where this would be very useful, so if not I might have a go at implementing myself. Thanks, Andrew Myers ---------- Forwarded message ---------- From: Sam Skillman <samskillman@gmail.com> Date: Tue, Jan 11, 2011 at 9:40 AM Subject: [Yt-dev] saving derived fields To: yt-dev@lists.spacepope.org Hi all, I've started dealing with a lot of time-consuming derived fields and am wondering if there is already support for writing derived fields back to the raw data files (e.g. .cpu files). If not, any thoughts on doing so? Sam _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Oh man, that Sam Skillman from a couple of years ago really dropped the ball. I did not end up giving it a go or report back! That said, I think Matt's suggestion still holds up. One other option would be to use (or modify) the write_to_gdf functionality, though this would duplicate the data. So to reuse some bits... If you give this a shot and test it, let us know how it goes, and feel free to pull request if it works out. Sam On Tue, Feb 5, 2013 at 12:50 PM, Andrew Myers <atmyers@berkeley.edu> wrote:
Hello YT users,
A couple of years ago, Sam Skillman (email chain attached) asked about support for saving derived fields to the disk, so we don't have to re-compute expensive fields over and over again. Did this functionality ever make it into yt? I'm working on something where this would be very useful, so if not I might have a go at implementing myself.
Thanks, Andrew Myers
---------- Forwarded message ---------- From: Sam Skillman <samskillman@gmail.com> Date: Tue, Jan 11, 2011 at 9:40 AM Subject: [Yt-dev] saving derived fields To: yt-dev@lists.spacepope.org
Hi all,
I've started dealing with a lot of time-consuming derived fields and am wondering if there is already support for writing derived fields back to the raw data files (e.g. .cpu files). If not, any thoughts on doing so?
Sam
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Sam, Thanks for the info! I'll give it a try and report back if I get something working. -Andrew On Tue, Feb 5, 2013 at 12:12 PM, Sam Skillman <samskillman@gmail.com> wrote:
Oh man, that Sam Skillman from a couple of years ago really dropped the ball. I did not end up giving it a go or report back!
That said, I think Matt's suggestion still holds up.
One other option would be to use (or modify) the write_to_gdf functionality, though this would duplicate the data.
So to reuse some bits... If you give this a shot and test it, let us know how it goes, and feel free to pull request if it works out.
Sam
On Tue, Feb 5, 2013 at 12:50 PM, Andrew Myers <atmyers@berkeley.edu>wrote:
Hello YT users,
A couple of years ago, Sam Skillman (email chain attached) asked about support for saving derived fields to the disk, so we don't have to re-compute expensive fields over and over again. Did this functionality ever make it into yt? I'm working on something where this would be very useful, so if not I might have a go at implementing myself.
Thanks, Andrew Myers
---------- Forwarded message ---------- From: Sam Skillman <samskillman@gmail.com> Date: Tue, Jan 11, 2011 at 9:40 AM Subject: [Yt-dev] saving derived fields To: yt-dev@lists.spacepope.org
Hi all,
I've started dealing with a lot of time-consuming derived fields and am wondering if there is already support for writing derived fields back to the raw data files (e.g. .cpu files). If not, any thoughts on doing so?
Sam
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hey Andrew et al, In the intervening time, I've had a chance ot think about this more too. I think that Sam's idea of using the write_gdf is a good one. In fact, I would go so far as to say that this is the *right* way to do it, and to allow individuals to affiliate a "backup" GDF file with the dataset. Then inside the IO functionality, if a given field does *not* exist, it could simply be read from the GDF file. This can likely be done already by generating the derived fields and writing out a GDF file, as noted in an earlier thread. But longer term, using the Stream frontend in a more complex way, we can apply fallbacks -- have a "Merged" output that includes the Orion data *and* the GDF, but the two files exist separately. This would require not using load_unifrom_grid or load_amr_grids, but instead creating a Stream hierarchy manually. -Matt On Tue, Feb 5, 2013 at 12:53 PM, Andrew Myers <atmyers@berkeley.edu> wrote:
Hi Sam,
Thanks for the info! I'll give it a try and report back if I get something working.
-Andrew
On Tue, Feb 5, 2013 at 12:12 PM, Sam Skillman <samskillman@gmail.com> wrote:
Oh man, that Sam Skillman from a couple of years ago really dropped the ball. I did not end up giving it a go or report back!
That said, I think Matt's suggestion still holds up.
One other option would be to use (or modify) the write_to_gdf functionality, though this would duplicate the data.
So to reuse some bits... If you give this a shot and test it, let us know how it goes, and feel free to pull request if it works out.
Sam
On Tue, Feb 5, 2013 at 12:50 PM, Andrew Myers <atmyers@berkeley.edu> wrote:
Hello YT users,
A couple of years ago, Sam Skillman (email chain attached) asked about support for saving derived fields to the disk, so we don't have to re-compute expensive fields over and over again. Did this functionality ever make it into yt? I'm working on something where this would be very useful, so if not I might have a go at implementing myself.
Thanks, Andrew Myers
---------- Forwarded message ---------- From: Sam Skillman <samskillman@gmail.com> Date: Tue, Jan 11, 2011 at 9:40 AM Subject: [Yt-dev] saving derived fields To: yt-dev@lists.spacepope.org
Hi all,
I've started dealing with a lot of time-consuming derived fields and am wondering if there is already support for writing derived fields back to the raw data files (e.g. .cpu files). If not, any thoughts on doing so?
Sam
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Andrew submitted a first pass at this: https://bitbucket.org/yt_analysis/yt/pull-request/416/saving-derived-fields-... I think I'm on board with John's comment: this should be broadly available. Great work, Andrew! On Tue, Feb 5, 2013 at 4:39 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hey Andrew et al,
In the intervening time, I've had a chance ot think about this more too. I think that Sam's idea of using the write_gdf is a good one. In fact, I would go so far as to say that this is the *right* way to do it, and to allow individuals to affiliate a "backup" GDF file with the dataset. Then inside the IO functionality, if a given field does *not* exist, it could simply be read from the GDF file.
This can likely be done already by generating the derived fields and writing out a GDF file, as noted in an earlier thread. But longer term, using the Stream frontend in a more complex way, we can apply fallbacks -- have a "Merged" output that includes the Orion data *and* the GDF, but the two files exist separately. This would require not using load_unifrom_grid or load_amr_grids, but instead creating a Stream hierarchy manually.
-Matt
On Tue, Feb 5, 2013 at 12:53 PM, Andrew Myers <atmyers@berkeley.edu> wrote:
Hi Sam,
Thanks for the info! I'll give it a try and report back if I get something working.
-Andrew
On Tue, Feb 5, 2013 at 12:12 PM, Sam Skillman <samskillman@gmail.com> wrote:
Oh man, that Sam Skillman from a couple of years ago really dropped the ball. I did not end up giving it a go or report back!
That said, I think Matt's suggestion still holds up.
One other option would be to use (or modify) the write_to_gdf functionality, though this would duplicate the data.
So to reuse some bits... If you give this a shot and test it, let us know how it goes, and feel free to pull request if it works out.
Sam
On Tue, Feb 5, 2013 at 12:50 PM, Andrew Myers <atmyers@berkeley.edu> wrote:
Hello YT users,
A couple of years ago, Sam Skillman (email chain attached) asked about support for saving derived fields to the disk, so we don't have to re-compute expensive fields over and over again. Did this functionality ever make it into yt? I'm working on something where this would be very useful, so if not I might have a go at implementing myself.
Thanks, Andrew Myers
---------- Forwarded message ---------- From: Sam Skillman <samskillman@gmail.com> Date: Tue, Jan 11, 2011 at 9:40 AM Subject: [Yt-dev] saving derived fields To: yt-dev@lists.spacepope.org
Hi all,
I've started dealing with a lot of time-consuming derived fields and am wondering if there is already support for writing derived fields back to the raw data files (e.g. .cpu files). If not, any thoughts on doing so?
Sam
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (3)
-
Andrew Myers -
Matthew Turk -
Sam Skillman