Hi all, I am trying to run halo finding on a dataset, using hop via yt. Im getting weird results in particular slowdowns and a weird message. Up to here everything looks ok: P002 yt : [INFO ] 2023-10-27 04:31:09,841 Parsing outputs Calling regroup... No minimum group size specified. Assuming 10 particles. ngroups = 53540 and then it takes a while and finally I see something like: P002 yt : [INFO ] 2023-10-25 11:52:40,201 Saving 1 halos: ./info_000065/info_000065.2.h5. and the output file is very large (saving particles by defaut). So Im guessing that somehow the regroup routine is making one huge halo regrouping the 53450 groups... which may be why it is so slow…. Do you know if there is any way to control mingroupsize from yt? It does not seem like this is a parameter of the fof halo finder method that can be tuned from yt. Also, I have no idea if that would fix the problem. Here Im using parallel (8 proc) but the problem is the same with 1 proc. NB: this is ramses, everything looks fine on the ramses test data (output_00101). Any help would be welcome! Cheers, Pierre ============================================================ Pierre OCVIRK Observatoire Astronomique de Strasbourg 11 rue de l'Universite 67000 STRASBOURG FRANCE tel: +33 (0)3 68 85 24 40 fax: +33 (0)3 68 85 24 25 pierre.ocvirk@astro.unistra.fr ============================================================
Hi Pierre, On Fri, Oct 27, 2023 at 4:15 AM ocvirk <pierre.ocvirk@astro.unistra.fr> wrote:
Hi all,
I am trying to run halo finding on a dataset, using hop via yt.
Im getting weird results in particular slowdowns and a weird message. Up to here everything looks ok:
P002 yt : [INFO ] 2023-10-27 04:31:09,841 Parsing outputs Calling regroup... No minimum group size specified. Assuming 10 particles. ngroups = 53540
and then it takes a while and finally I see something like:
P002 yt : [INFO ] 2023-10-25 11:52:40,201 Saving 1 halos: ./info_000065/info_000065.2.h5.
and the output file is very large (saving particles by defaut).
So Im guessing that somehow the regroup routine is making one huge halo regrouping the 53450 groups... which may be why it is so slow….
Yup, I think this is what's happening.
Do you know if there is any way to control mingroupsize from yt? It does not seem like this is a parameter of the fof halo finder method that can be tuned from yt.
I'm not sure that mingroupsize would have the impact you're looking for -- it's set rather low, so I don't think that the behavior you're seeing is actually reliant on mingroupsize, as with this setting it should only cut off halos of size 2-9 particles. You *can* control the density threshold, but I'm not really sure why it would be converting 53540 groups into a single halo. Can you look at the total particle count in the final halo versus in the initial simulation?
Also, I have no idea if that would fix the problem. Here Im using parallel (8 proc) but the problem is the same with 1 proc.
NB: this is ramses, everything looks fine on the ramses test data (output_00101).
Any help would be welcome!
Cheers,
Pierre
============================================================ Pierre OCVIRK Observatoire Astronomique de Strasbourg 11 rue de l'Universite 67000 STRASBOURG FRANCE tel: +33 (0)3 68 85 24 40 fax: +33 (0)3 68 85 24 25
pierre.ocvirk@astro.unistra.fr ============================================================
_______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com
Hi Pierre, Is this is a zoom simulation by any chance? I've experienced issues with them in the past using yt hop and fof. Britton On Fri, Oct 27, 2023 at 10:44 AM Matthew Turk <matthewturk@gmail.com> wrote:
Hi Pierre,
On Fri, Oct 27, 2023 at 4:15 AM ocvirk <pierre.ocvirk@astro.unistra.fr> wrote:
Hi all,
I am trying to run halo finding on a dataset, using hop via yt.
Im getting weird results in particular slowdowns and a weird message. Up to here everything looks ok:
P002 yt : [INFO ] 2023-10-27 04:31:09,841 Parsing outputs Calling regroup... No minimum group size specified. Assuming 10 particles. ngroups = 53540
and then it takes a while and finally I see something like:
P002 yt : [INFO ] 2023-10-25 11:52:40,201 Saving 1 halos: ./info_000065/info_000065.2.h5.
and the output file is very large (saving particles by defaut).
So Im guessing that somehow the regroup routine is making one huge halo regrouping the 53450 groups... which may be why it is so slow….
Yup, I think this is what's happening.
Do you know if there is any way to control mingroupsize from yt? It does not seem like this is a parameter of the fof halo finder method that can be tuned from yt.
I'm not sure that mingroupsize would have the impact you're looking for -- it's set rather low, so I don't think that the behavior you're seeing is actually reliant on mingroupsize, as with this setting it should only cut off halos of size 2-9 particles.
You *can* control the density threshold, but I'm not really sure why it would be converting 53540 groups into a single halo. Can you look at the total particle count in the final halo versus in the initial simulation?
Also, I have no idea if that would fix the problem. Here Im using parallel (8 proc) but the problem is the same with 1 proc.
NB: this is ramses, everything looks fine on the ramses test data (output_00101).
Any help would be welcome!
Cheers,
Pierre
============================================================ Pierre OCVIRK Observatoire Astronomique de Strasbourg 11 rue de l'Universite 67000 STRASBOURG FRANCE tel: +33 (0)3 68 85 24 40 fax: +33 (0)3 68 85 24 25
pierre.ocvirk@astro.unistra.fr ============================================================
_______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com
_______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: brittonsmith@gmail.com
participants (3)
-
Britton Smith -
Matthew Turk -
ocvirk