Registering external data as a Field of an existing dataset?
Hello,
I wanted to ask if yt supports the registration of external data as a field of an existing dataset.
In case it helps clarify my question, I just wanted to briefly elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.
I'm pretty confident that the answer to my question is "no," but I wanted to be sure before I rolled my own solution. Thank you for your help.
Best, Matthew Abruzzo
Hi Matthew,
There is no frontend-agnostic support for this, but I believe it has been implemented in a few cases. If the data in the external files has the same layout as the native data, this should be quite straightforward to implement. One would have to add machinery to search for these side-car files and add their contents to the field list, and then to know which files to open to get the associated fields. The difficulty would come in if the field data had a different shape than the native data, as this would confuse the selector functions. If what you have meets these criteria, then I'm happy to discuss in more detail.
Britton
On Mon, Apr 26, 2021 at 7:53 PM Matthew Abruzzo matthewabruzzo@gmail.com wrote:
Hello,
I wanted to ask if yt supports the registration of external data as a field of an existing dataset.
In case it helps clarify my question, I just wanted to briefly elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.
I'm pretty confident that the answer to my question is "no," but I wanted to be sure before I rolled my own solution. Thank you for your help.
Best, Matthew Abruzzo _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com
Hi Britton,
Thanks for the response. This is really helpful! I haven't actually written the code to generate this external field data yet, but I could definitely package it in the same shape as the native data.
I looked through the source code and it seems like this requires 2 main changes:
1. Tweak the field_list held in the index attribute of a Dataset instance. (Out of curiosity, will things break if the field_list changes after the Dataset instance has been fully initialized? Or does the _detect_output_fields method of the frontend's Index subclass need to be directly modified?)
2. Tweak the implementation of the frontend's subclass of BaseIOHandler such that the _read_fluid_selection method reads the field data from the sidecar files alongside the native data.
Am I missing any other obvious steps? Thanks again for your help.
-Matthew
On Wed, Apr 28, 2021 at 4:42 AM Britton Smith brittonsmith@gmail.com wrote:
Hi Matthew,
There is no frontend-agnostic support for this, but I believe it has been implemented in a few cases. If the data in the external files has the same layout as the native data, this should be quite straightforward to implement. One would have to add machinery to search for these side-car files and add their contents to the field list, and then to know which files to open to get the associated fields. The difficulty would come in if the field data had a different shape than the native data, as this would confuse the selector functions. If what you have meets these criteria, then I'm happy to discuss in more detail.
Britton
On Mon, Apr 26, 2021 at 7:53 PM Matthew Abruzzo matthewabruzzo@gmail.com wrote:
Hello,
I wanted to ask if yt supports the registration of external data as a field of an existing dataset.
In case it helps clarify my question, I just wanted to briefly elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.
I'm pretty confident that the answer to my question is "no," but I wanted to be sure before I rolled my own solution. Thank you for your help.
Best, Matthew Abruzzo _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewabruzzo@gmail.com
Hi Matthew,
One possibility you could consider is -- if the field matches 1-to-1 with an existing grid or particle structure -- is using a derived field, setting the validators to NeedsGridType, and then having it access the data.id to figure out which grid it is and returning the appropriately shaped field itself.
(Incidentally your use case is what the Stream handler was originally designed for -- and then the usage for loading arbitrary datasets took over its use patterns!)
-Matt
On Fri, Apr 30, 2021 at 4:51 PM Matthew Abruzzo matthewabruzzo@gmail.com wrote:
Hi Britton,
Thanks for the response. This is really helpful! I haven't actually written the code to generate this external field data yet, but I could definitely package it in the same shape as the native data.
I looked through the source code and it seems like this requires 2 main changes:
Tweak the field_list held in the index attribute of a Dataset instance. (Out of curiosity, will things break if the field_list changes after the Dataset instance has been fully initialized? Or does the _detect_output_fields method of the frontend's Index subclass need to be directly modified?)
Tweak the implementation of the frontend's subclass of BaseIOHandler such that the _read_fluid_selection method reads the field data from the sidecar files alongside the native data.
Am I missing any other obvious steps? Thanks again for your help.
-Matthew
On Wed, Apr 28, 2021 at 4:42 AM Britton Smith brittonsmith@gmail.com wrote:
Hi Matthew,
There is no frontend-agnostic support for this, but I believe it has been implemented in a few cases. If the data in the external files has the same layout as the native data, this should be quite straightforward to implement. One would have to add machinery to search for these side-car files and add their contents to the field list, and then to know which files to open to get the associated fields. The difficulty would come in if the field data had a different shape than the native data, as this would confuse the selector functions. If what you have meets these criteria, then I'm happy to discuss in more detail.
Britton
On Mon, Apr 26, 2021 at 7:53 PM Matthew Abruzzo matthewabruzzo@gmail.com wrote:
Hello,
I wanted to ask if yt supports the registration of external data as a field of an existing dataset.
In case it helps clarify my question, I just wanted to briefly elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.
I'm pretty confident that the answer to my question is "no," but I wanted to be sure before I rolled my own solution. Thank you for your help.
Best, Matthew Abruzzo _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewabruzzo@gmail.com
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com
Hi Matthew,
I was just going through my inbox and I realized I never followed up on this. Sorry about that.
The two steps you outline should about do it. For step 1, I think modifying _detect_output_fields is the way to go. Modifying the field_list after it has been built will not be sufficient to make the fields fully known to the dataset.
Britton
On Sat, May 1, 2021 at 12:46 AM Matthew Turk matthewturk@gmail.com wrote:
Hi Matthew,
One possibility you could consider is -- if the field matches 1-to-1 with an existing grid or particle structure -- is using a derived field, setting the validators to NeedsGridType, and then having it access the data.id to figure out which grid it is and returning the appropriately shaped field itself.
(Incidentally your use case is what the Stream handler was originally designed for -- and then the usage for loading arbitrary datasets took over its use patterns!)
-Matt
On Fri, Apr 30, 2021 at 4:51 PM Matthew Abruzzo matthewabruzzo@gmail.com wrote:
Hi Britton,
Thanks for the response. This is really helpful! I haven't actually
written the code to generate this external field data yet, but I could definitely package it in the same shape as the native data.
I looked through the source code and it seems like this requires 2 main
changes:
- Tweak the field_list held in the index attribute of a Dataset
instance. (Out of curiosity, will things break if the field_list changes after the Dataset instance has been fully initialized? Or does the _detect_output_fields method of the frontend's Index subclass need to be directly modified?)
- Tweak the implementation of the frontend's subclass of BaseIOHandler
such that the _read_fluid_selection method reads the field data from the sidecar files alongside the native data.
Am I missing any other obvious steps? Thanks again for your help.
-Matthew
On Wed, Apr 28, 2021 at 4:42 AM Britton Smith brittonsmith@gmail.com
wrote:
Hi Matthew,
There is no frontend-agnostic support for this, but I believe it has
been implemented in a few cases. If the data in the external files has the same layout as the native data, this should be quite straightforward to implement. One would have to add machinery to search for these side-car files and add their contents to the field list, and then to know which files to open to get the associated fields. The difficulty would come in if the field data had a different shape than the native data, as this would confuse the selector functions. If what you have meets these criteria, then I'm happy to discuss in more detail.
Britton
On Mon, Apr 26, 2021 at 7:53 PM Matthew Abruzzo <
matthewabruzzo@gmail.com> wrote:
Hello,
I wanted to ask if yt supports the registration of external data as a
field of an existing dataset.
In case it helps clarify my question, I just wanted to briefly
elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.
I'm pretty confident that the answer to my question is "no," but I
wanted to be sure before I rolled my own solution. Thank you for your help.
Best, Matthew Abruzzo _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewabruzzo@gmail.com
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com
participants (3)
-
Britton Smith -
Matthew Abruzzo -
Matthew Turk