Nathan,
I added rockstar to my yt installation, and it works, but only one snapshot at a time. I.e, when I try running the halo finder in the loop:
for set in sets: d = yt.load(set) hc = HaloCatalog(data_ds=d,finder_method='rockstar',finder_kwargs={'dm_only':True,'particle_type':'N-BODY'}) hc.create() if(yt.is_root()): print('Memory: ',yt.funcs.get_memory_usage())
it runs once, but at the second loop iteration gives the following error. I checked the memory use, it is very small for this small simulation, so the segfault is most likely the result of the network failure.
This was run with 1 reader and 1 writer on a single node:
mpirun -n 3 python3 hfc-rs.py
but I also tried more ranks, the error occurs independently of the number of ranks used.
This is not critical, since I can run the script for each snapshot separately, but it would be nice to be able to run all of them at once. Perhaps, some network connections are not properly cleaned - I heard that rockstar does something weird with networking.
Nick
------------- error messages ------------------------------ ... P001 yt : [WARNING ] 2018-11-21 10:54:13,109 Total Particle Count: 1.342e+08 [ 0s] Accepting connections...
[ 0s] Accepted all reader / writer connections.
[Warning] Network IO Failure (PID 146476): Connection reset by peer [Warning] Network IO Failure (PID 146476): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused [Warning] Network IO Failure (PID 146477): Connection reset by peer [Warning] Network IO Failure (PID 146477): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused
===================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 139 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ===================================================================================== APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
On 11/19/18 4:12 PM, Nathan Goldbaum wrote:
Hi Nick,
See the installation docs here, in particular the section titled “installing support for the rockstar halo finder”:
http://yt-project.org/doc/installing.html https://urldefense.proofpoint.com/v2/url?u=http-3A__yt-2Dproject.org_doc_installing.html&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=_JCa1ouo3f2lussZXM6IaQ-5vPchVRlGDcaCjIYhX08&e=
Nathan
On Mon, Nov 19, 2018 at 2:47 PM Nick Gnedin <gnedin@fnal.gov mailto:gnedin@fnal.gov> wrote:
Britton, How would I install Rockstar for yt to use? Your webpage mentions install-script.sh, but my from-source installation lacks it: ~/ART/yt >ls appveyor.yml CONTRIBUTING.rst MANIFEST.in setup.cfg yt.egg-info benchmarks COPYING.txt nose.cfg setupext.py build CREDITS __pycache__ setup.py CITATION doc README.md tests clean.sh docker scripts yt I downloaded it as the help recommends, but then the script tries to download and install yet another yt installation. I need to use my clone, since I made some changes to it that I haven't yet submitted in a pull request. On 10/5/2018 12:23 PM, Britton Smith wrote: > Hi Nick, > > Ok, good luck. It should be noted that the version of Rockstar that runs > with yt is not the latest version of Rockstar, known as > Rockstar-Galaxies. yt's Rockstar does not support multi-mass > simulations. You may also find different i/o performance between the two > versions, so it's worth experimenting, if both options are available. > > Britton > > On Fri, Oct 5, 2018 at 10:10 AM Nick Gnedin <gnedin@fnal.gov <mailto:gnedin@fnal.gov> > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>> wrote: > > > Britton, > > Thank you very much, I will try Rockstar first. > > n > > > On 10/5/2018 7:06 PM, Britton Smith wrote: > > HI Nick, > > > > Sadly, yt's FoF and HOP finders don't output member particle > information > > for the halos, so it might not be possible to use those catalogs > to make > > merger trees. ytree's merger-tree creation works with FoF catalogs > > generated by the inline Gadget halo finder, for which yt supports > > extracting member particle information. If you're looking to make > > merger-trees, the best route currently is probably > > Rockstar/consistent-trees, the outputs of which can be loaded by > ytree. > > > > It is probably a week's worth of work to fix up yt's HOP/FoF > finders to > > output particle information and then to add support for loading that > > into yt. > > > > Britton > > > > On Fri, Oct 5, 2018 at 6:07 AM Nick Gnedin <gnedin@fnal.gov <mailto:gnedin@fnal.gov> > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>> > > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov> <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>>> wrote: > > > > > > Folks, > > > > I am confused about how to compute merger trees with yt. > Documentation > > > (https://yt-project.org/doc/analyzing/analysis_modules/halo_merger_tree.html?highlight=merger%20tree <https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_analyzing_analysis-5Fmodules_halo-5Fmerger-5Ftree.html-3Fhighlight-3Dmerger-2520tree&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=GWJcpe_ZhtqzA_FqiMTZ-U_LtrLxBH3BxJUHmd-6aR0&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_analyzing_analysis-5Fmodules_halo-5Fmerger-5Ftree.html-3Fhighlight-3Dmerger-2520tree&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=JJEEm3ckzl6tozYzLjLZC1jZ2haYyYXfmwlk8jQHsFQ&e=> > > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_analyzing_analysis-5Fmodules_halo-5Fmerger-5Ftree.html-3Fhighlight-3Dmerger-2520tree&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=zeRRll5QvyPrOQIZHiEPcx7nJk84AGxCHVY_xIV4OZ8&s=JcesVppFFSFwh2Hj4CdmDhGC6m2PJwj6igNfcC-vblA&e=>) > > > > refers to ytree package, but that package documentation seems to > > indicate that only FoF or Subfind halo catalogs can be > converted into a > > merger tree, while the halo finder in yt is HOP. > > > > I have run HOP for several snapshots from a simulation. What > would you > > recommend to use to make a merger tree from such a data set? > > > > Thank you, > > > > Nick Gnedin > > > > _______________________________________________ > > yt-users mailing list -- yt-users@python.org <mailto:yt-users@python.org> > <mailto:yt-users@python.org <mailto:yt-users@python.org>> > > <mailto:yt-users@python.org <mailto:yt-users@python.org> <mailto:yt-users@python.org <mailto:yt-users@python.org>>> > > To unsubscribe send an email to yt-users-leave@python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org <mailto:yt-users-leave@python.org>> > > <mailto:yt-users-leave@python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org <mailto:yt-users-leave@python.org>>> > > > > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=JxDe9bt4H1-GgpuLF6UKKquta2OoQMf6wg_5UT2ZIEY&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=7HqId4jOunbIbaVHaKY4JleZDtHhnw3aE3vjpm9nHrQ&e=>> > > > Virus-free. www.avg.com <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=FyB2aYEgQceoGXXGcAe8qjdYVoXOPRzn727ksPokV_0&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=OWEryJt_rkdbS3lzMTRsm3xr1DgdeiTVC9BD5GHtASw&e=> > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=JxDe9bt4H1-GgpuLF6UKKquta2OoQMf6wg_5UT2ZIEY&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=7HqId4jOunbIbaVHaKY4JleZDtHhnw3aE3vjpm9nHrQ&e=>> > > > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > _______________________________________________ yt-users mailing list -- yt-users@python.org <mailto:yt-users@python.org> To unsubscribe send an email to yt-users-leave@python.org <mailto:yt-users-leave@python.org>
This error comes from Rockstar rather than yt, unfortunately I don’t know enough sbout rockstar to help here.
On Wed, Nov 21, 2018 at 12:12 PM Nick Gnedin ngnedin@gmail.com wrote:
Nathan,
I added rockstar to my yt installation, and it works, but only one snapshot at a time. I.e, when I try running the halo finder in the loop:
for set in sets: d = yt.load(set) hc =
HaloCatalog(data_ds=d,finder_method='rockstar',finder_kwargs={'dm_only':True,'particle_type':'N-BODY'}) hc.create() if(yt.is_root()): print('Memory: ',yt.funcs.get_memory_usage())
it runs once, but at the second loop iteration gives the following error. I checked the memory use, it is very small for this small simulation, so the segfault is most likely the result of the network failure.
This was run with 1 reader and 1 writer on a single node:
mpirun -n 3 python3 hfc-rs.py
but I also tried more ranks, the error occurs independently of the number of ranks used.
This is not critical, since I can run the script for each snapshot separately, but it would be nice to be able to run all of them at once. Perhaps, some network connections are not properly cleaned - I heard that rockstar does something weird with networking.
Nick
------------- error messages ------------------------------ ... P001 yt : [WARNING ] 2018-11-21 10:54:13,109 Total Particle Count: 1.342e+08 [ 0s] Accepting connections...
[ 0s] Accepted all reader / writer connections.
[Warning] Network IO Failure (PID 146476): Connection reset by peer [Warning] Network IO Failure (PID 146476): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused [Warning] Network IO Failure (PID 146477): Connection reset by peer [Warning] Network IO Failure (PID 146477): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused
===================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 139 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================== APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
On 11/19/18 4:12 PM, Nathan Goldbaum wrote:
Hi Nick,
See the installation docs here, in particular the section titled “installing support for the rockstar halo finder”:
https://urldefense.proofpoint.com/v2/url?u=http-3A__yt-2Dproject.org_doc_ins...
Nathan
On Mon, Nov 19, 2018 at 2:47 PM Nick Gnedin <gnedin@fnal.gov mailto:gnedin@fnal.gov> wrote:
Britton, How would I install Rockstar for yt to use? Your webpage mentions install-script.sh, but my from-source installation lacks it: ~/ART/yt >ls appveyor.yml CONTRIBUTING.rst MANIFEST.in setup.cfg yt.egg-info benchmarks COPYING.txt nose.cfg setupext.py build CREDITS __pycache__ setup.py CITATION doc README.md tests clean.sh docker scripts yt I downloaded it as the help recommends, but then the script tries to download and install yet another yt installation. I need to use my clone, since I made some changes to it that I
haven't
yet submitted in a pull request. On 10/5/2018 12:23 PM, Britton Smith wrote: > Hi Nick, > > Ok, good luck. It should be noted that the version of Rockstar that runs > with yt is not the latest version of Rockstar, known as > Rockstar-Galaxies. yt's Rockstar does not support multi-mass > simulations. You may also find different i/o performance between the two > versions, so it's worth experimenting, if both options are
available.
> > Britton > > On Fri, Oct 5, 2018 at 10:10 AM Nick Gnedin <gnedin@fnal.gov <mailto:gnedin@fnal.gov> > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>> wrote: > > > Britton, > > Thank you very much, I will try Rockstar first. > > n > > > On 10/5/2018 7:06 PM, Britton Smith wrote: > > HI Nick, > > > > Sadly, yt's FoF and HOP finders don't output member
particle
> information > > for the halos, so it might not be possible to use those catalogs > to make > > merger trees. ytree's merger-tree creation works with FoF catalogs > > generated by the inline Gadget halo finder, for which yt supports > > extracting member particle information. If you're looking to make > > merger-trees, the best route currently is probably > > Rockstar/consistent-trees, the outputs of which can be loaded by > ytree. > > > > It is probably a week's worth of work to fix up yt's
HOP/FoF
> finders to > > output particle information and then to add support for loading that > > into yt. > > > > Britton > > > > On Fri, Oct 5, 2018 at 6:07 AM Nick Gnedin <gnedin@fnal.gov <mailto:gnedin@fnal.gov> > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>> > > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov> <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>>> wrote: > > > > > > Folks, > > > > I am confused about how to compute merger trees with
yt.
> Documentation > > > (
https://yt-project.org/doc/analyzing/analysis_modules/halo_merger_tree.html?... < https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_an... < https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_an...
> > > <
https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_an...
) > > > > refers to ytree package, but that package documentation seems to > > indicate that only FoF or Subfind halo catalogs can be > converted into a > > merger tree, while the halo finder in yt is HOP. > > > > I have run HOP for several snapshots from a simulation. What > would you > > recommend to use to make a merger tree from such a data set? > > > > Thank you, > > > > Nick Gnedin > > > > _______________________________________________ > > yt-users mailing list -- yt-users@python.org mailto:yt-users@python.org > <mailto:yt-users@python.org mailto:yt-users@python.org> > > <mailto:yt-users@python.org mailto:yt-users@python.org <mailto:yt-users@python.org mailto:yt-users@python.org>> > > To unsubscribe send an email to yt-users-leave@python.org mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org mailto:yt-users-leave@python.org> > > <mailto:yt-users-leave@python.org mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org mailto:yt-users-leave@python.org>> > > > > > > > <
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&... < https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign...
> <
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign...
> > > Virus-free. www.avg.com <
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ...
> <
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ...
> > > > <
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&... < https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign...
> <
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign...
> > > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > _______________________________________________ yt-users mailing list -- yt-users@python.org <mailto:yt-users@python.org> To unsubscribe send an email to yt-users-leave@python.org <mailto:yt-users-leave@python.org>