Registration open: LSSTC workshop "Data Visualization and exploration in the LSST Era" NCSA, June 19-21, 2018
by Matthew Turk
Hi folks, I am passing on the following message about an upcoming
workshop. This workshop will feature a track about yt, including
tutorials both basic and advanced, and I'm pretty excited about the
breadth of topics that will be covered.
--
Dear All,
We are pleased to announce that our LSSTC workshop "Data Visualization
and exploration in the LSST Era" will be held at the National Center
for Supercomputing Applications (NCSA) at the University of Illinois
at Urbana-Champaign from June 19 to June 21, 2018. The 3-day workshop
is open to all interested attendees. However, preference will be given
to early career scientists given the nature of the workshop format.
Thanks to support from the LSST Enabling Science Group and NCSA we are
glad to waive the registration fee and to provide some travel support.
We welcome proposed contributions for short talks and 1 hour
hands-on-tutorials on the topics listed on the website.
The registration is now open and the deadline is June 1st, 2018.
For more information please visit the workshop website at
http://www.ncsa.illinois.edu/Conferences/LSST18/
We hope to see you there!
On behalf of the SOC:
Matias Carrasco Kind, University of Illinois at Urbana-Champaign
Donna Cox, University of Illinois at Urbana-Champaign
Leanne Guy, LSST
Gilbert Holder, University of Illinois at Urbana-Champaign
Xin Liu, University of Illinois at Urbana-Champaign
Felipe Menanteau, University of Illinois at Urbana-Champaign
William O'Mullane, LSST
Laura Trouille, Adler Planetarium
Matt Turk, University of Illinois at Urbana-Champaign
8 months, 2 weeks
hide phaseplot colorbar
by Stephanie Tonnesen
Hi yt-users!
I would like to make phaseplots without the colorbar. I tried to just use
the
plot.hide_colorbar()
command, but apparently PhasePlot has no attribute hide_colorbar()
Looking at the docs, perhaps what I have to do it use PhasePlotMPL? Is
there an example using this somewhere in the docs?
Thanks in advance for any help/direction!!
Stephanie
--
Dr. Stephanie Tonnesen
Associate Research Scientist
CCA, Flatiron Institute
New York, NY
stonnes(a)gmail.com
8 months, 3 weeks
Fwd: clump find
by Britton Smith
Returning this conversation back to the list.
---------- Forwarded message ----------
From: Le Dan <le.dan33(a)yahoo.com>
Date: Tue, Mar 20, 2018 at 11:47 AM
Subject: Re: [yt-users] clump find
To: Britton Smith <brittonsmith(a)gmail.com>
my code looks like:
import matplotlib
matplotlib.use('Agg')
import yt
import matplotlib.pyplot as plt
import numpy as np
nar = np.array
import time
import copy
from yt.analysis_modules.level_sets.api import *
grav_min = []
x_min = []
y_min = []
z_min = []
loc = []
loc_2 = []
leaf_indices = []
nbins = 20
bin_high = 5e6
bin_low_init=0.1
def new_grav_potential(field,data):
grav_old= data['gravitational_potential']
grav_max = data.get_field_parameter('max_grav')
grav_new = -grav_old+grav_max
return grav_new
yt.add_field('new_grav_potential',function=new_grav_potential,units =
'erg/g',validators = [yt.ValidateParameter('max_grav')])
for frame in [120]:
n_cl = 0
fname = 'DD%04d/data%04d'%(frame,frame)
ds = yt.load(fname)
ad = ds.all_data()
a = ad['gravitational_potential'].max()
b = -ad['gravitational_potential']+a
ad.set_field_parameter('max_grav',a)
master_clump = Clump(ad,('gas','new_grav_potential'))
master_clump.add_validator("min_cells", 8)
c_min = ad["gas", "new_grav_potential"].min()
c_max = ad["gas", "new_grav_potential"].max()
step = 100
* find_clumps(master_clump, c_min, c_max, step)*
# Write a text file of only the leaf nodes.
write_clumps(master_clump,0, "%s_clumps.txt" % ds)
leaf_clumps = get_lowest_clumps(master_clump)
proj = yt.ProjectionPlot(ds,2,('gas','new_grav_potential'),center =
'c',width=(20,'kpc'),field_parameters={'grav_max':a})
proj.annotate_clumps(leaf_clumps)
proj.save('clumps')
The error occurs in the find_clumps function. The error I am receiving is:
yt.fields.field_exceptions.NeedsParameter: (['max_grav'])
Dan Le
On Tuesday, March 20, 2018 1:29 PM, Britton Smith <brittonsmith(a)gmail.com>
wrote:
Hi Dan Le,
Can you post the code you've written and the error you're seeing?
Britton
On Tue, Mar 20, 2018 at 9:57 AM, Le Dan via yt-users <yt-users(a)python.org>
wrote:
Hi all,
I've created a new field and trying to get clump find to work on that
field. When running my program, I received the error saying that a
parameter used to define my new field could not be accessed by the
function, find_clumps(master_clump, c_min, c_max, step). How would I go
about getting rid of this error?
Dan Le
______________________________ _________________
yt-users mailing list -- yt-users(a)python.org
To unsubscribe send an email to yt-users-leave(a)python.org
8 months, 3 weeks
clump find
by Le Dan
Hi all,
I've created a new field and trying to get clump find to work on that field. When running my program, I received the error saying that a parameter used to define my new field could not be accessed by the function, find_clumps(master_clump, c_min, c_max, step). How would I go about getting rid of this error?
Dan Le
8 months, 3 weeks
Supernovae with Ramses and YT
by Matthieu Tricottet
Hello everyone!
I'm simulating supernovae explosions with Ramses. I'm at the very very
beginning of it. My first runs give nothing, but I'm unsure whether it's
because I've misused Ramses or because I don't use YT properly to see
results.
My config file is this one:
https://expirebox.com/download/59d92f446fea392dc9a6189603f1dc34.html
Result files are here:
https://expirebox.com/download/6bd562f210199b6b8581525c84d86eff.html
When I try slices or projections, all I can see is that:
https://ibb.co/jbeg9c
I've loaded it with
import yt
ds = yt.load("my_path/output_00001/info_00001.txt")
And then, tried things like yt.SlicePlot(ds, 'x', "density") or
yt.ProjectionPlot(ds,'x','Density') (I've tried other axis too), but I
can't get anything else that the image I've showed.
Could someone help me to determine if the problem comes from my basic first
simulation (Sedov supernovae phase) or from my Yt plots? I would really
appreciate some help!
Thank you very much
8 months, 4 weeks
Memory and high level covering grids
by Rachel Smullen
Hi all,
I've been running into this problem for a while now, but I've reached a
point where I can't avoid it any longer.
I have a 6 level AMR simulation (run with Orion, but that's probably
irrelevant to my problem) that has a level 0 size of (256)^3. I would like
to extract a covering grid for a small subset at level 4 (for the full
volume, that equates to a (4096)^3 grid). The chunk that I want has an
equivalent size at level 4 of 2048x2048x1536.
Basic setup: I'm running on a node with 16 cores and 256 GB of memory. I
have tried both yt version 3.4 and 3.5 (updated this morning).
I've tried
level = 4
ds = yt.load(filename)
cg = ds.covering_grid(level, ds.domain_left_edge, ds.domain_dimensions
* ds.refine_by**level)
result = cg["density"][1024:3072,1434:3482,512:2048]
This results in the error
File
"/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/data_containers.py",
line 282, in __getitem__
self.get_data(f)
File
"/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py",
line 641, in get_data
if len(fill) > 0: self._fill_fields(fill)
File
"/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py",
line 690, in _fill_fields
for field in fields]
File
"/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py",
line 690, in <listcomp>
for field in fields]
MemoryError
Looking at the memory buildup during execution, it appears that yt is
loading densities for the entire volume to then return my chunk to me. I
have more than enough memory on the machine (the job exits at ~70GB usage),
so I think this is a Python limit on array allocations.
I have also tried creating a box region in the area that I want, but
covering grids cannot be run on regions. Thus, I tried saving my region
as a dataset, but it reloads with incorrect limits and sizes:
box = ds.box([xmin,ymin,zmin], [xmax,ymax,zmax])
box.save_as_dataset(filename = 'box', fields = (('gas','density')))
ds2 = yt.load('box.h5')
print( ds2.domain_left_edge, ds2.domain_dimensions, ds2.refine_by )
cg = ds2.covering_grid(level, ds2.domain_left_edge,
ds2.domain_dimensions * ds2.refine_by**level)
This gives
[ -7.72500000e+18 -7.72500000e+18 -7.72500000e+18] code_length [2 2 2] 2
instead of
[-1.93125e+18 -1.15875e+18 -2.896875e+18] [128 128 96] 2 (my
expectation)
So, to summarize my questions:
1. Is there an easy way for me to directly get my chunk at level 4
resolution with yt as it stands now?
2. Otherwise, is there an easy Python fix to allow the array memory
allocation to be larger?
3. If none of the above, how can I save my region with the correct
properties so I can reload my chunk as a smaller dataset?
Thanks so much for your help!
Rachel Smullen
Graduate Student
Steward Observatory
University of Arizona
9 months
Re: MPI scaling of HOP
by Nick Gnedin
Sure, I am happy to be a guinea pig.
I will need to install that clone on BW, so far I was using the
system-wide version of yt.
On 3/13/2018 6:06 PM, Britton Smith wrote:
> Hi Nick,
>
> I just issued a pull request that should cut out the largest scaling
> bottleneck in the yt FOF/HOP halo finders. The PR can be found here:
> https://github.com/yt-project/yt/pull/1724
>
> As noted in the description, the one change to the results is that the
> final halo catalog will no longer be sorted by mass when it's written.
> Would it be possible for you to test out this modification?
>
> Britton
>
> On Tue, Mar 13, 2018 at 3:11 PM, Britton Smith <brittonsmith(a)gmail.com
> <mailto:brittonsmith@gmail.com>> wrote:
>
> Hi Nick,
>
> Sorry, I've not seen that error before.
>
> It looks like the very latest Rockstar supports ART natively, but it
> may have the same issue. That code is here:
> https://bitbucket.org/pbehroozi/rockstar-galaxies
> <https://bitbucket.org/pbehroozi/rockstar-galaxies>
>
> Britton
>
> On Tue, Mar 13, 2018 at 2:21 PM, Nick Gnedin <ngnedin(a)gmail.com
> <mailto:ngnedin@gmail.com>> wrote:
>
>
> Britton,
>
> That helps, and Rockstar starts and loads data, but now it gives
> an internal error:
>
> P001 yt : [INFO ] 2018-03-13 16:03:11,005 Created 2048
> chunks for ARTIO
> P001 yt : [WARNING ] 2018-03-13 16:05:09,506 Total Particle
> Count: 1.342e+08
> [Error] Couldn't open # dsname index
> :strict! (Err: Servname not supported for ai_socktype)
>
> That error comes from the function default_addrinfo(...) in
> rockstar source code, file inet/socket.c.
>
> I don't know how well you guys know rockstar - let me know if
> you think this is the dead end.
>
> n
>
>
>
> On 03/13/2018 03:18 PM, Britton Smith wrote:
>
> Hi Nick,
>
> You might need to add the "num_readers" and "num_writers"
> keywords to this line:
> rhf = RockstarHaloFinder(d, num_readers=8, num_writers=8)
> The readers are the i/o nodes and the writers are the actual
> halo finding instances. 8 for each is a good place to
> start. When you run, you'll need to have (num_readers +
> num_writers + 1) MPI processes, where the extra one is for
> the server.
>
> Hopefully, that takes care of it.
>
> Britton
>
>
>
> On Tue, Mar 13, 2018 at 12:43 PM, Nick Gnedin
> <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com>
> <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>>> wrote:
>
>
> Nathan,
>
> Thank you. I followed the online instructions and now
> Rockstar
> starts, but gives me an error, irrespective of whether
> I call it
> directly or via a HaloCatalog object:
>
> ---------- code ---------
> d =
> yt.load(root+"/rei20_a"+aexp+"/rei20_a"+aexp+".art")
> rhf = RockstarHaloFinder(d)
> rhf.run()
> -------------------------
>
> Traceback (most recent call last):
> File "hfc.py", line 15, in <module>
> rhf = RockstarHaloFinder(d)
> File
>
> "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py",
> line 230, in __init__
> self.pool, self.workgroup = self.runner.setup_pool()
> File
>
> "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py",
> line 105, in setup_pool
> (self.num_writers, "writers") ]
> File
>
> "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py",
> line 400, in from_sizes
> pool.add_workgroup(size, name = name)
> File
>
> "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py",
> line 368, in add_workgroup
> group = self.comm.comm.Get_group().Incl(ranks)
> AttributeError: 'NoneType' object has no attribute
> 'Get_group'
>
>
> On 03/13/2018 02:01 PM, Nathan Goldbaum wrote:
>
>
>
> On Tue, Mar 13, 2018 at 1:52 PM, Nick Gnedin
> <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com>
> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com>> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com>
> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com>>>> wrote:
>
>
> Britton,
>
> I am trying to run Rockstar, but it does not
> seem to be
> packaged
> with yt by default, import fails:
>
> Traceback (most recent call last):
> File "hfc.py", line 3, in <module>
> from
> yt.analysis_modules.halo_finding.rockstar.api import
> RockstarHaloFinder
> File
>
> "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/api.py",
> line 16, in <module>
> from .rockstar import RockstarHaloFinder
> File
>
> "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/rockstar.py",
> line 28, in <module>
> from . import rockstar_interface
> ImportError: cannot import name
> 'rockstar_interface'
>
>
> I think I am using the latest versions of
> python and yt:
>
>
> yt module located at:
>
> /home/gnedin/anaconda3/lib/python3.6/site-packages
>
> The current version of yt is:
>
> ---
> Version = 3.4.1
> ---
>
>
> It is not packaged with yt out of the box. The
> easiest way to
> get a copy of yt built with the rockstar bindings
> is to install
> yt with the install script. You'll need to modify
> the script so
> that INST_ROCKSTAR=1 once you've downloaded it.
>
> Altenatively you can manually build yt and rockstar
> following
> the instructions in the docs here:
>
> http://yt-project.org/docs/dev/installing.html#installing-support-for-the...
> <http://yt-project.org/docs/dev/installing.html#installing-support-for-the...>
>
> <http://yt-project.org/docs/dev/installing.html#installing-support-for-the...
> <http://yt-project.org/docs/dev/installing.html#installing-support-for-the...>>
>
> The rockstar halo finder is licensed under GPLv3 so
> unfortunately we can't distribute it with the yt
> binaries on
> pypi or conda-forge without changing their license
> as well.
>
>
> n
>
>
> On 03/13/2018 12:44 PM, Britton Smith wrote:
>
> Hi Nick,
>
> Thanks for your report. Your timing data
> confirms my
> suspicion
> about which part of the code isn't
> scaling. The
> rejoining of
> the halo list after the halo finder is run
> makes heavy
> use of
> MPI broadcast calls. Reworking this
> shouldn't be too
> difficult,
> just a question of someone finding the
> time. If anyone is
> interested in trying to fix this, I can
> direct them to the
> places that need the attention.
>
> Nick, in the mean time, you might try the
> Rockstar halo
> finder
> (either the one built-in to yt or the
> standalone),
> which scales
> quite well. The output from both Rockstar
> versions is
> loadable
> with yt.
>
> Britton
>
> On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin
> <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com>
> <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>>
> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com>>>
> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com>>
>
> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com>>>>>
>
> wrote:
>
>
> This is just a notice to the developer.
>
> I have run a HOP halo finder for a
> large ART
> simulation (1024^3
> particles) on BlueWaters with the
> following code:
>
> import yt
> from
> yt.analysis_modules.halo_analysis.api import
> HaloCatalog
> yt.enable_parallelism()
> path =
> "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D"
> aexps = [ "0.1280", "0.1203",
> "0.1115", "0.1002",
> "0.0907" ]
> for aexp in aexps:
> d = yt.load(path+"/rei40_a"+aexp
> +"/rei40_a"+aexp+".art")
> hc =
>
> HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"})
> hc.create()
>
> Because of memory constraints, I have
> to run it on
> at least
> 4 MPI
> ranks, and I noticed that yt
> implementation of HOP
> does not
> scale -
> a 4-rank job takes 14.5 hours and an
> 8-rank one
> takes 15.75
> hours.
> Surely, halo finding for billion
> particles should
> scale
> better than
> that.
>
> Here is some timing info (I can
> provide a full log
> if you care)
>
> 4-rank run:
> P000 yt : [INFO ] 2018-03-11
> 17:31:25,531
> Parameters: ...
> P000 yt : [INFO ] 2018-03-11
> 18:36:49,781
> Initializing
> HOP [1h]
> P002 yt : [INFO ] 2018-03-11
> 22:14:15,486 Parsing
> outputs [3.5h]
> P000 yt : [INFO ] 2018-03-12
> 08:06:38,231
> Saving halo
> ... [10h]
>
> 8-rank run:
> P000 yt : [INFO ] 2018-03-10
> 21:03:27,226
> Parameters: ...
> P000 yt : [INFO ] 2018-03-10
> 21:43:52,543
> Initializing
> HOP [0.75h]
> P005 yt : [INFO ] 2018-03-10
> 23:52:10,389 Parsing
> outputs [2h]
> P000 yt : [INFO ] 2018-03-11
> 12:43:46,645
> Saving halo
> ... [12.5h *]
>
> * - does not scale at all.
>
> n
>
>
> _______________________________________________
> yt-users mailing list --
> yt-users(a)python.org <mailto:yt-users@python.org>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>>>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>>>>
> To unsubscribe send an email to
> yt-users-leave(a)python.org <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>>>
>
>
> _______________________________________________
> yt-users mailing list -- yt-users(a)python.org
> <mailto:yt-users@python.org>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>>>
> To unsubscribe send an email to
> yt-users-leave(a)python.org <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>>
>
>
> _______________________________________________
> yt-users mailing list -- yt-users(a)python.org
> <mailto:yt-users@python.org>
> <mailto:yt-users@python.org <mailto:yt-users@python.org>>
> To unsubscribe send an email to
> yt-users-leave(a)python.org <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>
>
>
>
>
9 months
Re: MPI scaling of HOP
by Nick Gnedin
Britton,
That helps, and Rockstar starts and loads data, but now it gives an
internal error:
P001 yt : [INFO ] 2018-03-13 16:03:11,005 Created 2048 chunks for ARTIO
P001 yt : [WARNING ] 2018-03-13 16:05:09,506 Total Particle Count:
1.342e+08
[Error] Couldn't open # dsname index
:strict! (Err: Servname not supported for ai_socktype)
That error comes from the function default_addrinfo(...) in rockstar
source code, file inet/socket.c.
I don't know how well you guys know rockstar - let me know if you think
this is the dead end.
n
On 03/13/2018 03:18 PM, Britton Smith wrote:
> Hi Nick,
>
> You might need to add the "num_readers" and "num_writers" keywords to
> this line:
> rhf = RockstarHaloFinder(d, num_readers=8, num_writers=8)
> The readers are the i/o nodes and the writers are the actual halo
> finding instances. 8 for each is a good place to start. When you run,
> you'll need to have (num_readers + num_writers + 1) MPI processes, where
> the extra one is for the server.
>
> Hopefully, that takes care of it.
>
> Britton
>
>
> On Tue, Mar 13, 2018 at 12:43 PM, Nick Gnedin <ngnedin(a)gmail.com
> <mailto:ngnedin@gmail.com>> wrote:
>
>
> Nathan,
>
> Thank you. I followed the online instructions and now Rockstar
> starts, but gives me an error, irrespective of whether I call it
> directly or via a HaloCatalog object:
>
> ---------- code ---------
> d = yt.load(root+"/rei20_a"+aexp+"/rei20_a"+aexp+".art")
> rhf = RockstarHaloFinder(d)
> rhf.run()
> -------------------------
>
> Traceback (most recent call last):
> File "hfc.py", line 15, in <module>
> rhf = RockstarHaloFinder(d)
> File
> "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py",
> line 230, in __init__
> self.pool, self.workgroup = self.runner.setup_pool()
> File
> "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py",
> line 105, in setup_pool
> (self.num_writers, "writers") ]
> File
> "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py",
> line 400, in from_sizes
> pool.add_workgroup(size, name = name)
> File
> "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py",
> line 368, in add_workgroup
> group = self.comm.comm.Get_group().Incl(ranks)
> AttributeError: 'NoneType' object has no attribute 'Get_group'
>
>
> On 03/13/2018 02:01 PM, Nathan Goldbaum wrote:
>
>
>
> On Tue, Mar 13, 2018 at 1:52 PM, Nick Gnedin <ngnedin(a)gmail.com
> <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com
> <mailto:ngnedin@gmail.com>>> wrote:
>
>
> Britton,
>
> I am trying to run Rockstar, but it does not seem to be
> packaged
> with yt by default, import fails:
>
> Traceback (most recent call last):
> File "hfc.py", line 3, in <module>
> from yt.analysis_modules.halo_finding.rockstar.api import
> RockstarHaloFinder
> File
>
> "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/api.py",
> line 16, in <module>
> from .rockstar import RockstarHaloFinder
> File
>
> "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/rockstar.py",
> line 28, in <module>
> from . import rockstar_interface
> ImportError: cannot import name 'rockstar_interface'
>
>
> I think I am using the latest versions of python and yt:
>
>
> yt module located at:
> /home/gnedin/anaconda3/lib/python3.6/site-packages
>
> The current version of yt is:
>
> ---
> Version = 3.4.1
> ---
>
>
> It is not packaged with yt out of the box. The easiest way to
> get a copy of yt built with the rockstar bindings is to install
> yt with the install script. You'll need to modify the script so
> that INST_ROCKSTAR=1 once you've downloaded it.
>
> Altenatively you can manually build yt and rockstar following
> the instructions in the docs here:
>
> http://yt-project.org/docs/dev/installing.html#installing-support-for-the...
> <http://yt-project.org/docs/dev/installing.html#installing-support-for-the...>
>
> The rockstar halo finder is licensed under GPLv3 so
> unfortunately we can't distribute it with the yt binaries on
> pypi or conda-forge without changing their license as well.
>
>
> n
>
>
> On 03/13/2018 12:44 PM, Britton Smith wrote:
>
> Hi Nick,
>
> Thanks for your report. Your timing data confirms my
> suspicion
> about which part of the code isn't scaling. The
> rejoining of
> the halo list after the halo finder is run makes heavy
> use of
> MPI broadcast calls. Reworking this shouldn't be too
> difficult,
> just a question of someone finding the time. If anyone is
> interested in trying to fix this, I can direct them to the
> places that need the attention.
>
> Nick, in the mean time, you might try the Rockstar halo
> finder
> (either the one built-in to yt or the standalone),
> which scales
> quite well. The output from both Rockstar versions is
> loadable
> with yt.
>
> Britton
>
> On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin
> <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com>
> <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>>
> <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>
>
> <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>>>>
> wrote:
>
>
> This is just a notice to the developer.
>
> I have run a HOP halo finder for a large ART
> simulation (1024^3
> particles) on BlueWaters with the following code:
>
> import yt
> from yt.analysis_modules.halo_analysis.api import
> HaloCatalog
> yt.enable_parallelism()
> path = "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D"
> aexps = [ "0.1280", "0.1203", "0.1115", "0.1002",
> "0.0907" ]
> for aexp in aexps:
> d = yt.load(path+"/rei40_a"+aexp
> +"/rei40_a"+aexp+".art")
> hc =
>
> HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"})
> hc.create()
>
> Because of memory constraints, I have to run it on
> at least
> 4 MPI
> ranks, and I noticed that yt implementation of HOP
> does not
> scale -
> a 4-rank job takes 14.5 hours and an 8-rank one
> takes 15.75
> hours.
> Surely, halo finding for billion particles should
> scale
> better than
> that.
>
> Here is some timing info (I can provide a full log
> if you care)
>
> 4-rank run:
> P000 yt : [INFO ] 2018-03-11 17:31:25,531
> Parameters: ...
> P000 yt : [INFO ] 2018-03-11 18:36:49,781
> Initializing
> HOP [1h]
> P002 yt : [INFO ] 2018-03-11 22:14:15,486 Parsing
> outputs [3.5h]
> P000 yt : [INFO ] 2018-03-12 08:06:38,231
> Saving halo
> ... [10h]
>
> 8-rank run:
> P000 yt : [INFO ] 2018-03-10 21:03:27,226
> Parameters: ...
> P000 yt : [INFO ] 2018-03-10 21:43:52,543
> Initializing
> HOP [0.75h]
> P005 yt : [INFO ] 2018-03-10 23:52:10,389 Parsing
> outputs [2h]
> P000 yt : [INFO ] 2018-03-11 12:43:46,645
> Saving halo
> ... [12.5h *]
>
> * - does not scale at all.
>
> n
>
> _______________________________________________
> yt-users mailing list -- yt-users(a)python.org
> <mailto:yt-users@python.org>
> <mailto:yt-users@python.org <mailto:yt-users@python.org>>
> <mailto:yt-users@python.org
> <mailto:yt-users@python.org> <mailto:yt-users@python.org
> <mailto:yt-users@python.org>>>
> To unsubscribe send an email to
> yt-users-leave(a)python.org <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>>
>
>
> _______________________________________________
> yt-users mailing list -- yt-users(a)python.org
> <mailto:yt-users@python.org>
> <mailto:yt-users@python.org <mailto:yt-users@python.org>>
> To unsubscribe send an email to yt-users-leave(a)python.org
> <mailto:yt-users-leave@python.org>
> <mailto:yt-users-leave@python.org
> <mailto:yt-users-leave@python.org>>
>
>
> _______________________________________________
> yt-users mailing list -- yt-users(a)python.org
> <mailto:yt-users@python.org>
> To unsubscribe send an email to yt-users-leave(a)python.org
> <mailto:yt-users-leave@python.org>
>
>
9 months
Error in Setting Particle Filter in EAGLE simulation
by Stephanie Ho
Hi,
I tried to set a particle filter with EAGLE and got an error. Here is a simple example, which uses the EAGLE dataset from yt sample data.
import yt
from yt.data_objects.particle_filters import add_particle_filter
eagle_fpath = './data/snapshot_028_z000p000/snap_028_z000p000.0.hdf5'
unit_base = {'UnitLength_in_cm' : 3.085678e+24,
'UnitMass_in_g' : 1.989e+43,
'UnitVelocity_in_cm_per_s' : 100000.0}
ds = yt.load(eagle_fpath,unit_base=unit_base)
ad = ds.all_data()
def cold_gas(pfilter, data):
temperature = data[pfilter.filtered_type, "Temperature"]
filter = (temperature.in_units('K') <= 1e5)
return filter
add_particle_filter("gas_cold", function=cold_gas, filtered_type='PartType0', requires=["Temperature"])
ds.add_particle_filter('gas_cold')
Below shows the error.
UnboundLocalError Traceback (most recent call last)
/Users/stephaniehho/EAGLE_sim/sho/test_with_yt/test_z000p000/particle_filter_simple.py in <module>()
20
21 add_particle_filter("gas_cold", function=cold_gas, filtered_type='PartType0', requires=["Temperature"])
---> 22 ds.add_particle_filter('gas_cold')
/Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in add_particle_filter(self, filter)
673 if f is None:
674 return False
--> 675 used = self._setup_filtered_type(f)
676 if used:
677 filter = f
/Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in _setup_filtered_type(self, filter)
714 if filter.name not in self.filtered_particle_types:
715 self.filtered_particle_types.append(filter.name)
--> 716 new_fields = self._setup_particle_types([filter.name])
717 deps, _ = self.field_info.check_derived_fields(new_fields)
718 self.field_dependencies.update(deps)
/Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in _setup_particle_types(self, ptypes)
723 if ptypes is None: ptypes = self.ds.particle_types_raw
724 for ptype in set(ptypes):
--> 725 df += self._setup_particle_type(ptype)
726 return df
727
/Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in _setup_particle_type(self, ptype)
889 def _setup_particle_type(self, ptype):
890 orig = set(self.field_info.items())
--> 891 self.field_info.setup_particle_fields(ptype)
892 return [n for n, v in set(self.field_info.items()).difference(orig)]
893
/Users/stephaniehho/yt-git/yt/frontends/owls/fields.pyc in setup_particle_fields(self, ptype)
111
112 super(OWLSFieldInfo,self).setup_particle_fields(
--> 113 ptype, num_neighbors=self._num_neighbors, ftype=ftype)
114
115
UnboundLocalError: local variable 'ftype' referenced before assignment
What I find is that, this problem seems to be temporarily solved if I add en else case at line 111 of the code: https://github.com/yt-project/yt/blob/master/yt/frontends/owls/fields.py,
else:
ftype=ptype
But this will raise the questions of, what are ftype and ptype designed for at the first place, or will this temporary fix cause other errors that I have not yet encountered.
Thanks,
Stephanie
9 months
