Hi Britton, Thanks for the response. This is really helpful! I haven't actually written the code to generate this external field data yet, but I could definitely package it in the same shape as the native data. I looked through the source code and it seems like this requires 2 main changes: 1. Tweak the field_list held in the index attribute of a Dataset instance. (Out of curiosity, will things break if the field_list changes after the Dataset instance has been fully initialized? Or does the _detect_output_fields method of the frontend's Index subclass need to be directly modified?) 2. Tweak the implementation of the frontend's subclass of BaseIOHandler such that the _read_fluid_selection method reads the field data from the sidecar files alongside the native data. Am I missing any other obvious steps? Thanks again for your help. -Matthew On Wed, Apr 28, 2021 at 4:42 AM Britton Smith wrote:

Hi Matthew,

There is no frontend-agnostic support for this, but I believe it has been implemented in a few cases. If the data in the external files has the same layout as the native data, this should be quite straightforward to implement. One would have to add machinery to search for these side-car files and add their contents to the field list, and then to know which files to open to get the associated fields. The difficulty would come in if the field data had a different shape than the native data, as this would confuse the selector functions. If what you have meets these criteria, then I'm happy to discuss in more detail.

Britton

On Mon, Apr 26, 2021 at 7:53 PM Matthew Abruzzo wrote:

...

Hello,

I wanted to ask if yt supports the registration of external data as a field of an existing dataset.

In case it helps clarify my question, I just wanted to briefly elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.

I'm pretty confident that the answer to my question is "no," but I wanted to be sure before I rolled my own solution. Thank you for your help.

Best, Matthew Abruzzo _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com

_______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewabruzzo@gmail.com

Hi Matthew, I was just going through my inbox and I realized I never followed up on this. Sorry about that. The two steps you outline should about do it. For step 1, I think modifying _detect_output_fields is the way to go. Modifying the field_list after it has been built will not be sufficient to make the fields fully known to the dataset. Britton On Sat, May 1, 2021 at 12:46 AM Matthew Turk wrote:

Hi Matthew,

One possibility you could consider is -- if the field matches 1-to-1 with an existing grid or particle structure -- is using a derived field, setting the validators to NeedsGridType, and then having it access the data.id to figure out which grid it is and returning the appropriately shaped field itself.

(Incidentally your use case is what the Stream handler was originally designed for -- and then the usage for loading arbitrary datasets took over its use patterns!)

-Matt

On Fri, Apr 30, 2021 at 4:51 PM Matthew Abruzzo wrote:

...

Hi Britton,

Thanks for the response. This is really helpful! I haven't actually

written the code to generate this external field data yet, but I could definitely package it in the same shape as the native data.

...

I looked through the source code and it seems like this requires 2 main

changes:

...

1. Tweak the field_list held in the index attribute of a Dataset

instance. (Out of curiosity, will things break if the field_list changes after the Dataset instance has been fully initialized? Or does the _detect_output_fields method of the frontend's Index subclass need to be directly modified?)

...

2. Tweak the implementation of the frontend's subclass of BaseIOHandler

such that the _read_fluid_selection method reads the field data from the sidecar files alongside the native data.

...

Am I missing any other obvious steps? Thanks again for your help.

-Matthew

On Wed, Apr 28, 2021 at 4:42 AM Britton Smith

...

...

Hi Matthew,

There is no frontend-agnostic support for this, but I believe it has

been implemented in a few cases. If the data in the external files has the same layout as the native data, this should be quite straightforward to implement. One would have to add machinery to search for these side-car files and add their contents to the field list, and then to know which files to open to get the associated fields. The difficulty would come in if

...

...

Britton

On Mon, Apr 26, 2021 at 7:53 PM Matthew Abruzzo <

matthewabruzzo@gmail.com> wrote:

...

...

Hello,

I wanted to ask if yt supports the registration of external data as a

field of an existing dataset.

...

In case it helps clarify my question, I just wanted to briefly

elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because

wrote: the field data had a different shape than the native data, as this would confuse the selector functions. If what you have meets these criteria, then I'm happy to discuss in more detail. this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.

...

...

...

I'm pretty confident that the answer to my question is "no," but I

wanted to be sure before I rolled my own solution. Thank you for your help.

...

Best, Matthew Abruzzo _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com

_______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewabruzzo@gmail.com

_______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com

---

yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com