Re: Saving halo profiles as a single dataset
Britton, Thank you - I will experiment and report performance. I think requiring the same number of bins in all profiles is a reasonable restriction, even if it increases the output size significantly. If someone is concerned with keeping the output size minimized, they can always write halos within several mass ranges into separate outputs with numbers of bins fixed within a single output but varied between mass ranges, and that can be accomplished with the existing functionality straightforwardly. n On 06/11/2018 11:31 AM, Britton Smith wrote:
Hi Nick,
Great question, I can see how this could become an issue when halo catalogs get large.
I've sketched out a way to do this below that may be slightly hacky, but will get the job done. http://paste.yt-project.org/show/24/
In the above, I create a new callback that attaches the profiles hanging off the halo object to the halo catalog itself, then combine them into single arrays at the end and save using yt.save_as_dataset. This will work fine if the profiles all have the same number of bins. If not, it would probably be better to write to an hdf5 file by hand with a single hdf5 group per profile. That said, HDF5 performance is known to degrade when the number of groups in a file gets large. In any case, hopefully, something like the above will work.
Perhaps in the future, we can make some modifications to the actual code to do something like this if it seems to work well.
Britton
On Wed, Jun 6, 2018 at 4:02 AM Nick Gnedin
mailto:gnedin@fnal.gov> wrote: I would like to save halo profiles for a large simulation. I understand that I can use a simple callback like this:
hc.add_callback("save_profiles", storage="virial_profiles", output_dir="profiles")
The problem however is that the call back saves each profiles as a separate h5 file, and for a large simulation the number of files may be prohibitive. Instead, I would like to save all of the profiles in a single fileset, ideally as attributes to halo objects.
For example, inside my profiling callback I can do the following:
def hprof_function(halo): ... pv = yt.create_profile(...) setattr(halo,"profs",pv)
However, hc.create() function does not save all the attributes for halo objects, so halo.profs would not be saved.
What would be a proper yt way of storing all the profiles as a single dataset?
Thank you,
n
_______________________________________________ yt-users mailing list -- yt-users@python.org mailto:yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org mailto:yt-users-leave@python.org
participants (1)
-
Nick Gnedin