yt-users March 2018

yt-users@python.org

19 participants
22 discussions

by david collins

Hi, Everybody! Does anyone out there have a technique for getting the variance out of a profile object? A profile object is good at getting <X> vs. B, I'd then like to get < (X - <X>)^2 > vs B. Matt and I had spittballed the possibility some time ago, but I was wondering if anyone out there had successfully done it. Thanks, d. -- Sent from my computer.

1 year, 3 months

3-D Mass power spectra

by 梁家豪

Dear yt Can current yt calculate 3-D Mass power spectra? I checked the website but I didn't find any information. I think calculating 3-D Mass power spectra is a very useful for cosmological simulations. So I guess maybe yt supports this function now....? Thanks in advance

2 years, 1 month

Registration open: LSSTC workshop "Data Visualization and exploration in the LSST Era" NCSA, June 19-21, 2018

by Matthew Turk

Hi folks, I am passing on the following message about an upcoming workshop. This workshop will feature a track about yt, including tutorials both basic and advanced, and I'm pretty excited about the breadth of topics that will be covered. -- Dear All, We are pleased to announce that our LSSTC workshop "Data Visualization and exploration in the LSST Era" will be held at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign from June 19 to June 21, 2018. The 3-day workshop is open to all interested attendees. However, preference will be given to early career scientists given the nature of the workshop format. Thanks to support from the LSST Enabling Science Group and NCSA we are glad to waive the registration fee and to provide some travel support. We welcome proposed contributions for short talks and 1 hour hands-on-tutorials on the topics listed on the website. The registration is now open and the deadline is June 1st, 2018. For more information please visit the workshop website at http://www.ncsa.illinois.edu/Conferences/LSST18/ We hope to see you there! On behalf of the SOC: Matias Carrasco Kind, University of Illinois at Urbana-Champaign Donna Cox, University of Illinois at Urbana-Champaign Leanne Guy, LSST Gilbert Holder, University of Illinois at Urbana-Champaign Xin Liu, University of Illinois at Urbana-Champaign Felipe Menanteau, University of Illinois at Urbana-Champaign William O'Mullane, LSST Laura Trouille, Adler Planetarium Matt Turk, University of Illinois at Urbana-Champaign

3 years, 4 months

hide phaseplot colorbar

by Stephanie Tonnesen

Hi yt-users! I would like to make phaseplots without the colorbar. I tried to just use the plot.hide_colorbar() command, but apparently PhasePlot has no attribute hide_colorbar() Looking at the docs, perhaps what I have to do it use PhasePlotMPL? Is there an example using this somewhere in the docs? Thanks in advance for any help/direction!! Stephanie -- Dr. Stephanie Tonnesen Associate Research Scientist CCA, Flatiron Institute New York, NY stonnes(a)gmail.com

3 years, 4 months

Down Sampling data set

by Kuo-Chuan Pan

Dear all, Is there a simple way to create a data set with lower AMR levels? something like: ds = yt.load(fname, max_level=3) I knew that there is an example to down sample for volume rendering with AMRKDTree, but I don't know how to create a downsample data set which can be analysis and plot with SlicePlot, for example. http://yt-project.org/docs/dev/cookbook/complex_plots.html#downsampling-dat… Many thanks, Kuo-Chuan

3 years, 4 months

Fwd: clump find

by Britton Smith

Returning this conversation back to the list. ---------- Forwarded message ---------- From: Le Dan <le.dan33(a)yahoo.com> Date: Tue, Mar 20, 2018 at 11:47 AM Subject: Re: [yt-users] clump find To: Britton Smith <brittonsmith(a)gmail.com> my code looks like: import matplotlib matplotlib.use('Agg') import yt import matplotlib.pyplot as plt import numpy as np nar = np.array import time import copy from yt.analysis_modules.level_sets.api import * grav_min = [] x_min = [] y_min = [] z_min = [] loc = [] loc_2 = [] leaf_indices = [] nbins = 20 bin_high = 5e6 bin_low_init=0.1 def new_grav_potential(field,data): grav_old= data['gravitational_potential'] grav_max = data.get_field_parameter('max_grav') grav_new = -grav_old+grav_max return grav_new yt.add_field('new_grav_potential',function=new_grav_potential,units = 'erg/g',validators = [yt.ValidateParameter('max_grav')]) for frame in [120]: n_cl = 0 fname = 'DD%04d/data%04d'%(frame,frame) ds = yt.load(fname) ad = ds.all_data() a = ad['gravitational_potential'].max() b = -ad['gravitational_potential']+a ad.set_field_parameter('max_grav',a) master_clump = Clump(ad,('gas','new_grav_potential')) master_clump.add_validator("min_cells", 8) c_min = ad["gas", "new_grav_potential"].min() c_max = ad["gas", "new_grav_potential"].max() step = 100 * find_clumps(master_clump, c_min, c_max, step)* # Write a text file of only the leaf nodes. write_clumps(master_clump,0, "%s_clumps.txt" % ds) leaf_clumps = get_lowest_clumps(master_clump) proj = yt.ProjectionPlot(ds,2,('gas','new_grav_potential'),center = 'c',width=(20,'kpc'),field_parameters={'grav_max':a}) proj.annotate_clumps(leaf_clumps) proj.save('clumps') The error occurs in the find_clumps function. The error I am receiving is: yt.fields.field_exceptions.NeedsParameter: (['max_grav']) Dan Le On Tuesday, March 20, 2018 1:29 PM, Britton Smith <brittonsmith(a)gmail.com> wrote: Hi Dan Le, Can you post the code you've written and the error you're seeing? Britton On Tue, Mar 20, 2018 at 9:57 AM, Le Dan via yt-users <yt-users(a)python.org> wrote: Hi all, I've created a new field and trying to get clump find to work on that field. When running my program, I received the error saying that a parameter used to define my new field could not be accessed by the function, find_clumps(master_clump, c_min, c_max, step). How would I go about getting rid of this error? Dan Le ______________________________ _________________ yt-users mailing list -- yt-users(a)python.org To unsubscribe send an email to yt-users-leave(a)python.org

3 years, 4 months

clump find

by Le Dan

Hi all, I've created a new field and trying to get clump find to work on that field. When running my program, I received the error saying that a parameter used to define my new field could not be accessed by the function, find_clumps(master_clump, c_min, c_max, step). How would I go about getting rid of this error? Dan Le

3 years, 4 months

Supernovae with Ramses and YT

by Matthieu Tricottet

Hello everyone! I'm simulating supernovae explosions with Ramses. I'm at the very very beginning of it. My first runs give nothing, but I'm unsure whether it's because I've misused Ramses or because I don't use YT properly to see results. My config file is this one: https://expirebox.com/download/59d92f446fea392dc9a6189603f1dc34.html Result files are here: https://expirebox.com/download/6bd562f210199b6b8581525c84d86eff.html When I try slices or projections, all I can see is that: https://ibb.co/jbeg9c I've loaded it with import yt ds = yt.load("my_path/output_00001/info_00001.txt") And then, tried things like yt.SlicePlot(ds, 'x', "density") or yt.ProjectionPlot(ds,'x','Density') (I've tried other axis too), but I can't get anything else that the image I've showed. Could someone help me to determine if the problem comes from my basic first simulation (Sedov supernovae phase) or from my Yt plots? I would really appreciate some help! Thank you very much

3 years, 4 months

Memory and high level covering grids

by Rachel Smullen

Hi all, I've been running into this problem for a while now, but I've reached a point where I can't avoid it any longer. I have a 6 level AMR simulation (run with Orion, but that's probably irrelevant to my problem) that has a level 0 size of (256)^3. I would like to extract a covering grid for a small subset at level 4 (for the full volume, that equates to a (4096)^3 grid). The chunk that I want has an equivalent size at level 4 of 2048x2048x1536. Basic setup: I'm running on a node with 16 cores and 256 GB of memory. I have tried both yt version 3.4 and 3.5 (updated this morning). I've tried level = 4 ds = yt.load(filename) cg = ds.covering_grid(level, ds.domain_left_edge, ds.domain_dimensions * ds.refine_by**level) result = cg["density"][1024:3072,1434:3482,512:2048] This results in the error File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/data_containers.py", line 282, in __getitem__ self.get_data(f) File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py", line 641, in get_data if len(fill) > 0: self._fill_fields(fill) File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py", line 690, in _fill_fields for field in fields] File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py", line 690, in <listcomp> for field in fields] MemoryError Looking at the memory buildup during execution, it appears that yt is loading densities for the entire volume to then return my chunk to me. I have more than enough memory on the machine (the job exits at ~70GB usage), so I think this is a Python limit on array allocations. I have also tried creating a box region in the area that I want, but covering grids cannot be run on regions. Thus, I tried saving my region as a dataset, but it reloads with incorrect limits and sizes: box = ds.box([xmin,ymin,zmin], [xmax,ymax,zmax]) box.save_as_dataset(filename = 'box', fields = (('gas','density'))) ds2 = yt.load('box.h5') print( ds2.domain_left_edge, ds2.domain_dimensions, ds2.refine_by ) cg = ds2.covering_grid(level, ds2.domain_left_edge, ds2.domain_dimensions * ds2.refine_by**level) This gives [ -7.72500000e+18 -7.72500000e+18 -7.72500000e+18] code_length [2 2 2] 2 instead of [-1.93125e+18 -1.15875e+18 -2.896875e+18] [128 128 96] 2 (my expectation) So, to summarize my questions: 1. Is there an easy way for me to directly get my chunk at level 4 resolution with yt as it stands now? 2. Otherwise, is there an easy Python fix to allow the array memory allocation to be larger? 3. If none of the above, how can I save my region with the correct properties so I can reload my chunk as a smaller dataset? Thanks so much for your help! Rachel Smullen Graduate Student Steward Observatory University of Arizona

3 years, 4 months

Re: MPI scaling of HOP

by Nick Gnedin

Sure, I am happy to be a guinea pig. I will need to install that clone on BW, so far I was using the system-wide version of yt. On 3/13/2018 6:06 PM, Britton Smith wrote: > Hi Nick, > > I just issued a pull request that should cut out the largest scaling > bottleneck in the yt FOF/HOP halo finders. The PR can be found here: > https://github.com/yt-project/yt/pull/1724 > > As noted in the description, the one change to the results is that the > final halo catalog will no longer be sorted by mass when it's written. > Would it be possible for you to test out this modification? > > Britton > > On Tue, Mar 13, 2018 at 3:11 PM, Britton Smith <brittonsmith(a)gmail.com > <mailto:brittonsmith@gmail.com>> wrote: > > Hi Nick, > > Sorry, I've not seen that error before. > > It looks like the very latest Rockstar supports ART natively, but it > may have the same issue. That code is here: > https://bitbucket.org/pbehroozi/rockstar-galaxies > <https://bitbucket.org/pbehroozi/rockstar-galaxies> > > Britton > > On Tue, Mar 13, 2018 at 2:21 PM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com>> wrote: > > > Britton, > > That helps, and Rockstar starts and loads data, but now it gives > an internal error: > > P001 yt : [INFO ] 2018-03-13 16:03:11,005 Created 2048 > chunks for ARTIO > P001 yt : [WARNING ] 2018-03-13 16:05:09,506 Total Particle > Count: 1.342e+08 > [Error] Couldn't open # dsname index > :strict! (Err: Servname not supported for ai_socktype) > > That error comes from the function default_addrinfo(...) in > rockstar source code, file inet/socket.c. > > I don't know how well you guys know rockstar - let me know if > you think this is the dead end. > > n > > > > On 03/13/2018 03:18 PM, Britton Smith wrote: > > Hi Nick, > > You might need to add the "num_readers" and "num_writers" > keywords to this line: > rhf = RockstarHaloFinder(d, num_readers=8, num_writers=8) > The readers are the i/o nodes and the writers are the actual > halo finding instances. 8 for each is a good place to > start. When you run, you'll need to have (num_readers + > num_writers + 1) MPI processes, where the extra one is for > the server. > > Hopefully, that takes care of it. > > Britton > > > > On Tue, Mar 13, 2018 at 12:43 PM, Nick Gnedin > <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>>> wrote: > > > Nathan, > > Thank you. I followed the online instructions and now > Rockstar > starts, but gives me an error, irrespective of whether > I call it > directly or via a HaloCatalog object: > > ---------- code --------- > d = > yt.load(root+"/rei20_a"+aexp+"/rei20_a"+aexp+".art") > rhf = RockstarHaloFinder(d) > rhf.run() > ------------------------- > > Traceback (most recent call last): > File "hfc.py", line 15, in <module> > rhf = RockstarHaloFinder(d) > File > > "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 230, in __init__ > self.pool, self.workgroup = self.runner.setup_pool() > File > > "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 105, in setup_pool > (self.num_writers, "writers") ] > File > > "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", > line 400, in from_sizes > pool.add_workgroup(size, name = name) > File > > "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", > line 368, in add_workgroup > group = self.comm.comm.Get_group().Incl(ranks) > AttributeError: 'NoneType' object has no attribute > 'Get_group' > > > On 03/13/2018 02:01 PM, Nathan Goldbaum wrote: > > > > On Tue, Mar 13, 2018 at 1:52 PM, Nick Gnedin > <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>>> wrote: > > > Britton, > > I am trying to run Rockstar, but it does not > seem to be > packaged > with yt by default, import fails: > > Traceback (most recent call last): > File "hfc.py", line 3, in <module> > from > yt.analysis_modules.halo_finding.rockstar.api import > RockstarHaloFinder > File > > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/api.py", > line 16, in <module> > from .rockstar import RockstarHaloFinder > File > > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 28, in <module> > from . import rockstar_interface > ImportError: cannot import name > 'rockstar_interface' > > > I think I am using the latest versions of > python and yt: > > > yt module located at: > > /home/gnedin/anaconda3/lib/python3.6/site-packages > > The current version of yt is: > > --- > Version = 3.4.1 > --- > > > It is not packaged with yt out of the box. The > easiest way to > get a copy of yt built with the rockstar bindings > is to install > yt with the install script. You'll need to modify > the script so > that INST_ROCKSTAR=1 once you've downloaded it. > > Altenatively you can manually build yt and rockstar > following > the instructions in the docs here: > > http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r… > <http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r…> > > <http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r… > <http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r…>> > > The rockstar halo finder is licensed under GPLv3 so > unfortunately we can't distribute it with the yt > binaries on > pypi or conda-forge without changing their license > as well. > > > n > > > On 03/13/2018 12:44 PM, Britton Smith wrote: > > Hi Nick, > > Thanks for your report. Your timing data > confirms my > suspicion > about which part of the code isn't > scaling. The > rejoining of > the halo list after the halo finder is run > makes heavy > use of > MPI broadcast calls. Reworking this > shouldn't be too > difficult, > just a question of someone finding the > time. If anyone is > interested in trying to fix this, I can > direct them to the > places that need the attention. > > Nick, in the mean time, you might try the > Rockstar halo > finder > (either the one built-in to yt or the > standalone), > which scales > quite well. The output from both Rockstar > versions is > loadable > with yt. > > Britton > > On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin > <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>> > > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>>>> > > wrote: > > > This is just a notice to the developer. > > I have run a HOP halo finder for a > large ART > simulation (1024^3 > particles) on BlueWaters with the > following code: > > import yt > from > yt.analysis_modules.halo_analysis.api import > HaloCatalog > yt.enable_parallelism() > path = > "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D" > aexps = [ "0.1280", "0.1203", > "0.1115", "0.1002", > "0.0907" ] > for aexp in aexps: > d = yt.load(path+"/rei40_a"+aexp > +"/rei40_a"+aexp+".art") > hc = > > HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"}) > hc.create() > > Because of memory constraints, I have > to run it on > at least > 4 MPI > ranks, and I noticed that yt > implementation of HOP > does not > scale - > a 4-rank job takes 14.5 hours and an > 8-rank one > takes 15.75 > hours. > Surely, halo finding for billion > particles should > scale > better than > that. > > Here is some timing info (I can > provide a full log > if you care) > > 4-rank run: > P000 yt : [INFO ] 2018-03-11 > 17:31:25,531 > Parameters: ... > P000 yt : [INFO ] 2018-03-11 > 18:36:49,781 > Initializing > HOP [1h] > P002 yt : [INFO ] 2018-03-11 > 22:14:15,486 Parsing > outputs [3.5h] > P000 yt : [INFO ] 2018-03-12 > 08:06:38,231 > Saving halo > ... [10h] > > 8-rank run: > P000 yt : [INFO ] 2018-03-10 > 21:03:27,226 > Parameters: ... > P000 yt : [INFO ] 2018-03-10 > 21:43:52,543 > Initializing > HOP [0.75h] > P005 yt : [INFO ] 2018-03-10 > 23:52:10,389 Parsing > outputs [2h] > P000 yt : [INFO ] 2018-03-11 > 12:43:46,645 > Saving halo > ... [12.5h *] > > * - does not scale at all. > > n > > > _______________________________________________ > yt-users mailing list -- > yt-users(a)python.org <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>> > <mailto:yt-users@python.org > <mailto:yt-users@python.org> <mailto:yt-users@python.org > <mailto:yt-users@python.org>>> > <mailto:yt-users@python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>> <mailto:yt-users@python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>>>> > To unsubscribe send an email to > yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>>>> > > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>> > <mailto:yt-users@python.org > <mailto:yt-users@python.org> <mailto:yt-users@python.org > <mailto:yt-users@python.org>>> > To unsubscribe send an email to > yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>>> > > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org <mailto:yt-users@python.org>> > To unsubscribe send an email to > yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > > > >

3 years, 4 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

yt-users March 2018