Hi Matthew,

I was just going through my inbox and I realized I never followed up on this. Sorry about that.

The two steps you outline should about do it. For step 1, I think modifying _detect_output_fields is the way to go. Modifying the field_list after it has been built will not be sufficient to make the fields fully known to the dataset.

Britton

On Sat, May 1, 2021 at 12:46 AM Matthew Turk <matthewturk@gmail.com> wrote:
Hi Matthew,

One possibility you could consider is -- if the field matches 1-to-1
with an existing grid or particle structure -- is using a derived
field, setting the validators to NeedsGridType, and then having it
access the data.id to figure out which grid it is and returning the
appropriately shaped field itself.

(Incidentally your use case is what the Stream handler was originally
designed for -- and then the usage for loading arbitrary datasets took
over its use patterns!)

-Matt

On Fri, Apr 30, 2021 at 4:51 PM Matthew Abruzzo
<matthewabruzzo@gmail.com> wrote:
>
> Hi Britton,
>
> Thanks for the response. This is really helpful!  I haven't actually written the code to generate this external field data yet, but I could definitely package it in the same shape as the native data.
>
> I looked through the source code and it seems like this requires 2 main changes:
>
> 1. Tweak the field_list held in the index attribute of a Dataset instance. (Out of curiosity, will things break if the field_list changes after the Dataset instance has been fully initialized? Or does the _detect_output_fields method of the frontend's Index subclass need to be directly modified?)
>
> 2. Tweak the implementation of the frontend's subclass of BaseIOHandler such that the _read_fluid_selection method reads the field data from the sidecar files alongside the native data.
>
> Am I missing any other obvious steps? Thanks again for your help.
>
> -Matthew
>
> On Wed, Apr 28, 2021 at 4:42 AM Britton Smith <brittonsmith@gmail.com> wrote:
>>
>> Hi Matthew,
>>
>> There is no frontend-agnostic support for this, but I believe it has been implemented in a few cases. If the data in the external files has the same layout as the native data, this should be quite straightforward to implement. One would have to add machinery to search for these side-car files and add their contents to the field list, and then to know which files to open to get the associated fields. The difficulty would come in if the field data had a different shape than the native data, as this would confuse the selector functions. If what you have meets these criteria, then I'm happy to discuss in more detail.
>>
>> Britton
>>
>> On Mon, Apr 26, 2021 at 7:53 PM Matthew Abruzzo <matthewabruzzo@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I wanted to ask if yt supports the registration of external data as a field of an existing dataset.
>>>
>>> In case it helps clarify my question, I just wanted to briefly elaborate on what I'm looking to do. I'm trying to characterize the turbulent velocities in some high resolution unigrid simulations. Because this involves convolution, I need to compute the turbulent velocity outside of the yt infrastructure (this seems beyond the scope of what a user-defined derived field can support). It would be convenient to be able to load the results back into yt to use the existing analysis machinery (e.g. for phase plots and slices). I have thought about using the "Generic Unigrid Data" frontend, but the simulation's size unfortunately makes that approach somewhat intractable.
>>>
>>> I'm pretty confident that the answer to my question is "no," but I wanted to be sure before I rolled my own solution. Thank you for your help.
>>>
>>> Best,
>>> Matthew Abruzzo
>>> _______________________________________________
>>> yt-users mailing list -- yt-users@python.org
>>> To unsubscribe send an email to yt-users-leave@python.org
>>> https://mail.python.org/mailman3/lists/yt-users.python.org/
>>> Member address: brittonsmith@gmail.com
>>
>> _______________________________________________
>> yt-users mailing list -- yt-users@python.org
>> To unsubscribe send an email to yt-users-leave@python.org
>> https://mail.python.org/mailman3/lists/yt-users.python.org/
>> Member address: matthewabruzzo@gmail.com
>
> _______________________________________________
> yt-users mailing list -- yt-users@python.org
> To unsubscribe send an email to yt-users-leave@python.org
> https://mail.python.org/mailman3/lists/yt-users.python.org/
> Member address: matthewturk@gmail.com
_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org
https://mail.python.org/mailman3/lists/yt-users.python.org/
Member address: brittonsmith@gmail.com