Nathan,
I added rockstar to my yt installation, and it works, but only one snapshot at a time. I.e, when I try running the halo finder in the loop:
for set in sets:
d = yt.load(set)
hc =
HaloCatalog(data_ds=d,finder_method='rockstar',finder_kwargs={'dm_only':True,'particle_type':'N-BODY'}) hc.create() if(yt.is_root()): print('Memory: ',yt.funcs.get_memory_usage())
it runs once, but at the second loop iteration gives the following error. I checked the memory use, it is very small for this small simulation, so the segfault is most likely the result of the network failure.
This was run with 1 reader and 1 writer on a single node:
mpirun -n 3 python3 hfc-rs.py
but I also tried more ranks, the error occurs independently of the number of ranks used.
This is not critical, since I can run the script for each snapshot separately, but it would be nice to be able to run all of them at once. Perhaps, some network connections are not properly cleaned - I heard that rockstar does something weird with networking.
Nick
------------- error messages ------------------------------ ... P001 yt : [WARNING ] 2018-11-21 10:54:13,109 Total Particle Count: 1.342e+08 [ 0s] Accepting connections...
[ 0s] Accepted all reader / writer connections.
[Warning] Network IO Failure (PID 146476): Connection reset by peer [Warning] Network IO Failure (PID 146476): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused [Warning] Network IO Failure (PID 146477): Connection reset by peer [Warning] Network IO Failure (PID 146477): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused
===================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 139 = CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
On 11/19/18 4:12 PM, Nathan Goldbaum wrote:
Hi Nick,
See the installation docs here, in particular the section titled “installing support for the rockstar halo finder”:
http://yt-project.org/doc/installing.html https://urldefense.proofpoint.com/v2/url?u=http-3A__yt-2Dproject.org_doc_ins...
Nathan
On Mon, Nov 19, 2018 at 2:47 PM Nick Gnedin <gnedin@fnal.gov
mailto:gnedin@fnal.gov> wrote:
Britton,
How would I install Rockstar for yt to use? Your webpage mentions
install-script.sh, but my from-source installation lacks it:
~/ART/yt >ls
appveyor.yml CONTRIBUTING.rst MANIFEST.in setup.cfg yt.egg-info
benchmarks COPYING.txt nose.cfg setupext.py
build CREDITS __pycache__ setup.py
CITATION doc README.md tests
clean.sh docker scripts yt
I downloaded it as the help recommends, but then the script tries to
download and install yet another yt installation.
I need to use my clone, since I made some changes to it that I haven't
yet submitted in a pull request.
On 10/5/2018 12:23 PM, Britton Smith wrote:
> Hi Nick,
>
> Ok, good luck. It should be noted that the version of Rockstar
that runs
> with yt is not the latest version of Rockstar, known as
> Rockstar-Galaxies. yt's Rockstar does not support multi-mass
> simulations. You may also find different i/o performance between
the two
> versions, so it's worth experimenting, if both options are available.
>
> Britton
>
> On Fri, Oct 5, 2018 at 10:10 AM Nick Gnedin <gnedin@fnal.gov
<mailto:gnedin@fnal.gov>
> <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>> wrote:
>
>
> Britton,
>
> Thank you very much, I will try Rockstar first.
>
> n
>
>
> On 10/5/2018 7:06 PM, Britton Smith wrote:
> > HI Nick,
> >
> > Sadly, yt's FoF and HOP finders don't output member particle
> information
> > for the halos, so it might not be possible to use those
catalogs
> to make
> > merger trees. ytree's merger-tree creation works with FoF
catalogs
> > generated by the inline Gadget halo finder, for which yt
supports
> > extracting member particle information. If you're looking
to make
> > merger-trees, the best route currently is probably
> > Rockstar/consistent-trees, the outputs of which can be
loaded by
> ytree.
> >
> > It is probably a week's worth of work to fix up yt's HOP/FoF
> finders to
> > output particle information and then to add support for
loading that
> > into yt.
> >
> > Britton
> >
> > On Fri, Oct 5, 2018 at 6:07 AM Nick Gnedin
<gnedin@fnal.gov <mailto:gnedin@fnal.gov>
> <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>
> > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>
<mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>>> wrote:
> >
> >
> > Folks,
> >
> > I am confused about how to compute merger trees with yt.
> Documentation
> >
>
(https://yt-project.org/doc/analyzing/analysis_modules/halo_merger_tree.html?highlight=merger%20tree
<https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_analyzing_analysis-5Fmodules_halo-5Fmerger-5Ftree.html-3Fhighlight-3Dmerger-2520tree&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=GWJcpe_ZhtqzA_FqiMTZ-U_LtrLxBH3BxJUHmd-6aR0&e=>
<https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_analyzing_analysis-5Fmodules_halo-5Fmerger-5Ftree.html-3Fhighlight-3Dmerger-2520tree&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=JJEEm3ckzl6tozYzLjLZC1jZ2haYyYXfmwlk8jQHsFQ&e=>
> >
>
<https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_analyzing_analysis-5Fmodules_halo-5Fmerger-5Ftree.html-3Fhighlight-3Dmerger-2520tree&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=zeRRll5QvyPrOQIZHiEPcx7nJk84AGxCHVY_xIV4OZ8&s=JcesVppFFSFwh2Hj4CdmDhGC6m2PJwj6igNfcC-vblA&e=>)
> >
> > refers to ytree package, but that package
documentation seems to
> > indicate that only FoF or Subfind halo catalogs can be
> converted into a
> > merger tree, while the halo finder in yt is HOP.
> >
> > I have run HOP for several snapshots from a
simulation. What
> would you
> > recommend to use to make a merger tree from such a
data set?
> >
> > Thank you,
> >
> > Nick Gnedin
> >
> > _______________________________________________
> > yt-users mailing list -- yt-users@python.org
<mailto:yt-users@python.org>
> <mailto:yt-users@python.org <mailto:yt-users@python.org>>
> > <mailto:yt-users@python.org
<mailto:yt-users@python.org> <mailto:yt-users@python.org
<mailto:yt-users@python.org>>>
> > To unsubscribe send an email to
yt-users-leave@python.org <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
<mailto:yt-users-leave@python.org>>
> > <mailto:yt-users-leave@python.org
<mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
<mailto:yt-users-leave@python.org>>>
> >
> >
> >
>
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=JxDe9bt4H1-GgpuLF6UKKquta2OoQMf6wg_5UT2ZIEY&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=7HqId4jOunbIbaVHaKY4JleZDtHhnw3aE3vjpm9nHrQ&e=>>
>
> > Virus-free. www.avg.com
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=FyB2aYEgQceoGXXGcAe8qjdYVoXOPRzn727ksPokV_0&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=OWEryJt_rkdbS3lzMTRsm3xr1DgdeiTVC9BD5GHtASw&e=>
>
> >
>
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=9Opb8GqEXKKzYt5xhjf2oLw6RNp3vyfctFqa1diXc7k&s=JxDe9bt4H1-GgpuLF6UKKquta2OoQMf6wg_5UT2ZIEY&e=>
>
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsignature-3Futm-5Fmedium-3Demail-26utm-5Fsource-3Dlink-26utm-5Fcampaign-3Dsig-2Demail-26utm-5Fcontent-3Demailclient&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=7jSLZBQB_2gqGQ3elA922w&m=ln3f04sr7kK9aACk-8EcI3uMd-86E1rnTd5rJ2BW5bg&s=7HqId4jOunbIbaVHaKY4JleZDtHhnw3aE3vjpm9nHrQ&e=>>
>
> >
> >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
_______________________________________________
yt-users mailing list -- yt-users@python.org
<mailto:yt-users@python.org>
To unsubscribe send an email to yt-users-leave@python.org
<mailto:yt-users-leave@python.org>
This error comes from Rockstar rather than yt, unfortunately I don’t know enough sbout rockstar to help here.
On Wed, Nov 21, 2018 at 12:12 PM Nick Gnedin ngnedin@gmail.com wrote:
>
Nathan,
I added rockstar to my yt installation, and it works, but only one snapshot at a time. I.e, when I try running the halo finder in the loop:
for set in sets:
d = yt.load(set)
hc =
HaloCatalog(data_ds=d,finder_method='rockstar',finder_kwargs={'dm_only':True,'particle_type':'N-BODY'}) hc.create() if(yt.is_root()): print('Memory: ',yt.funcs.get_memory_usage())
it runs once, but at the second loop iteration gives the following error. I checked the memory use, it is very small for this small simulation, so the segfault is most likely the result of the network failure.
This was run with 1 reader and 1 writer on a single node:
mpirun -n 3 python3 hfc-rs.py
but I also tried more ranks, the error occurs independently of the number of ranks used.
This is not critical, since I can run the script for each snapshot separately, but it would be nice to be able to run all of them at once. Perhaps, some network connections are not properly cleaned - I heard that rockstar does something weird with networking.
Nick
------------- error messages ------------------------------ ... P001 yt : [WARNING ] 2018-11-21 10:54:13,109 Total Particle Count: 1.342e+08 [ 0s] Accepting connections...
[ 0s] Accepted all reader / writer connections.
[Warning] Network IO Failure (PID 146476): Connection reset by peer [Warning] Network IO Failure (PID 146476): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused [Warning] Network IO Failure (PID 146477): Connection reset by peer [Warning] Network IO Failure (PID 146477): Success [Network] Packet receive retry count at: 1 [Warning] Connection attempt 1 to nanna.uchicago.edu:59013 failed: : Connection refused
===================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 139 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================== APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
On 11/19/18 4:12 PM, Nathan Goldbaum wrote:
Hi Nick,
See the installation docs here, in particular the section titled “installing support for the rockstar halo finder”:
http://yt-project.org/doc/installing.html < https://urldefense.proofpoint.com/v2/url?u=http-3A__yt-2Dproject.org_doc_ins...
Nathan
On Mon, Nov 19, 2018 at 2:47 PM Nick Gnedin <gnedin@fnal.gov
mailto:gnedin@fnal.gov> wrote:
Britton,
How would I install Rockstar for yt to use? Your webpage mentions
install-script.sh, but my from-source installation lacks it:
~/ART/yt >ls
appveyor.yml CONTRIBUTING.rst MANIFEST.in setup.cfg yt.egg-info
benchmarks COPYING.txt nose.cfg setupext.py
build CREDITS __pycache__ setup.py
CITATION doc README.md tests
clean.sh docker scripts yt
I downloaded it as the help recommends, but then the script tries to
download and install yet another yt installation.
I need to use my clone, since I made some changes to it that I
haven't yet submitted in a pull request.
On 10/5/2018 12:23 PM, Britton Smith wrote:
> Hi Nick,
>
> Ok, good luck. It should be noted that the version of Rockstar
that runs
> with yt is not the latest version of Rockstar, known as
> Rockstar-Galaxies. yt's Rockstar does not support multi-mass
> simulations. You may also find different i/o performance between
the two
> versions, so it's worth experimenting, if both options are
available. >
> Britton
>
> On Fri, Oct 5, 2018 at 10:10 AM Nick Gnedin <gnedin@fnal.gov
<mailto:gnedin@fnal.gov>
> <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>> wrote:
>
>
> Britton,
>
> Thank you very much, I will try Rockstar first.
>
> n
>
>
> On 10/5/2018 7:06 PM, Britton Smith wrote:
> > HI Nick,
> >
> > Sadly, yt's FoF and HOP finders don't output member
particle
> information
> > for the halos, so it might not be possible to use those
catalogs
> to make
> > merger trees. ytree's merger-tree creation works with FoF
catalogs
> > generated by the inline Gadget halo finder, for which yt
supports
> > extracting member particle information. If you're looking
to make
> > merger-trees, the best route currently is probably
> > Rockstar/consistent-trees, the outputs of which can be
loaded by
> ytree.
> >
> > It is probably a week's worth of work to fix up yt's
HOP/FoF
> finders to
> > output particle information and then to add support for
loading that
> > into yt.
> >
> > Britton
> >
> > On Fri, Oct 5, 2018 at 6:07 AM Nick Gnedin
<gnedin@fnal.gov <mailto:gnedin@fnal.gov>
> <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>
> > <mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>
<mailto:gnedin@fnal.gov <mailto:gnedin@fnal.gov>>>> wrote:
> >
> >
> > Folks,
> >
> > I am confused about how to compute merger trees with
yt.
> Documentation
> >
>
(
https://yt-project.org/doc/analyzing/analysis_modules/halo_merger_tree.html?... https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_a... < https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_an...
> >
>
<
https://urldefense.proofpoint.com/v2/url?u=https-3A__yt-2Dproject.org_doc_an... )
> >
> > refers to ytree package, but that package
documentation seems to
> > indicate that only FoF or Subfind halo catalogs can be
> converted into a
> > merger tree, while the halo finder in yt is HOP.
> >
> > I have run HOP for several snapshots from a
simulation. What
> would you
> > recommend to use to make a merger tree from such a
data set?
> >
> > Thank you,
> >
> > Nick Gnedin
> >
> > _______________________________________________
> > yt-users mailing list -- yt-users@python.org
<mailto:yt-users@python.org>
> <mailto:yt-users@python.org <mailto:yt-users@python.org>>
> > <mailto:yt-users@python.org
<mailto:yt-users@python.org> <mailto:yt-users@python.org
<mailto:yt-users@python.org>>>
> > To unsubscribe send an email to
yt-users-leave@python.org <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
<mailto:yt-users-leave@python.org>>
> > <mailto:yt-users-leave@python.org
<mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
<mailto:yt-users-leave@python.org>>>
> >
> >
> >
>
<
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&... < https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign...
>
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign... > >
> > Virus-free. www.avg.com
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ...
>
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com&d=DwMFaQ...
>
> >
>
<
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&... < https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign...
>
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.avg.com_email-2Dsign... > >
> >
> >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
_______________________________________________
yt-users mailing list -- yt-users@python.org
<mailto:yt-users@python.org>
To unsubscribe send an email to yt-users-leave@python.org
<mailto:yt-users-leave@python.org>
-
Nathan Goldbaum -
Nick Gnedin