Mailman 3 March 2018 - yt-users

Memory and high level covering grids
by Rachel Smullen March 15, 2018

March 15, 2018

Hi all, I've been running into this problem for a while now, but I've reached a point where I can't avoid it any longer. I have a 6 level AMR simulation (run with Orion, but that's probably irrelevant to my problem) that has a level 0 size of (256)^3. I would like to extract a covering grid for a small subset at level 4 (for the full volume, that equates to a (4096)^3 grid). The chunk that I want has an equivalent size at level 4 of 2048x2048x1536. Basic setup: I'm running on a node with 16 cores and 256 GB of memory. I have tried both yt version 3.4 and 3.5 (updated this morning). I've tried level = 4 ds = yt.load(filename) cg = ds.covering_grid(level, ds.domain_left_edge, ds.domain_dimensions * ds.refine_by**level) result = cg["density"][1024:3072,1434:3482,512:2048] This results in the error File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/data_containers.py", line 282, in __getitem__ self.get_data(f) File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py", line 641, in get_data if len(fill) > 0: self._fill_fields(fill) File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py", line 690, in _fill_fields for field in fields] File "/home/u14/rsmullen/anaconda3/lib/python3.5/site-packages/yt/data_objects/construction_data_containers.py", line 690, in <listcomp> for field in fields] MemoryError Looking at the memory buildup during execution, it appears that yt is loading densities for the entire volume to then return my chunk to me. I have more than enough memory on the machine (the job exits at ~70GB usage), so I think this is a Python limit on array allocations. I have also tried creating a box region in the area that I want, but covering grids cannot be run on regions. Thus, I tried saving my region as a dataset, but it reloads with incorrect limits and sizes: box = ds.box([xmin,ymin,zmin], [xmax,ymax,zmax]) box.save_as_dataset(filename = 'box', fields = (('gas','density'))) ds2 = yt.load('box.h5') print( ds2.domain_left_edge, ds2.domain_dimensions, ds2.refine_by ) cg = ds2.covering_grid(level, ds2.domain_left_edge, ds2.domain_dimensions * ds2.refine_by**level) This gives [ -7.72500000e+18 -7.72500000e+18 -7.72500000e+18] code_length [2 2 2] 2 instead of [-1.93125e+18 -1.15875e+18 -2.896875e+18] [128 128 96] 2 (my expectation) So, to summarize my questions: 1. Is there an easy way for me to directly get my chunk at level 4 resolution with yt as it stands now? 2. Otherwise, is there an easy Python fix to allow the array memory allocation to be larger? 3. If none of the above, how can I save my region with the correct properties so I can reload my chunk as a smaller dataset? Thanks so much for your help! Rachel Smullen Graduate Student Steward Observatory University of Arizona

2 1

Re: MPI scaling of HOP
by Nick Gnedin March 13, 2018

March 13, 2018

Sure, I am happy to be a guinea pig. I will need to install that clone on BW, so far I was using the system-wide version of yt. On 3/13/2018 6:06 PM, Britton Smith wrote: > Hi Nick, > > I just issued a pull request that should cut out the largest scaling > bottleneck in the yt FOF/HOP halo finders. The PR can be found here: > https://github.com/yt-project/yt/pull/1724 > > As noted in the description, the one change to the results is that the > final halo catalog will no longer be sorted by mass when it's written. > Would it be possible for you to test out this modification? > > Britton > > On Tue, Mar 13, 2018 at 3:11 PM, Britton Smith <brittonsmith(a)gmail.com > <mailto:brittonsmith@gmail.com>> wrote: > > Hi Nick, > > Sorry, I've not seen that error before. > > It looks like the very latest Rockstar supports ART natively, but it > may have the same issue. That code is here: > https://bitbucket.org/pbehroozi/rockstar-galaxies > <https://bitbucket.org/pbehroozi/rockstar-galaxies> > > Britton > > On Tue, Mar 13, 2018 at 2:21 PM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com>> wrote: > > > Britton, > > That helps, and Rockstar starts and loads data, but now it gives > an internal error: > > P001 yt : [INFO ] 2018-03-13 16:03:11,005 Created 2048 > chunks for ARTIO > P001 yt : [WARNING ] 2018-03-13 16:05:09,506 Total Particle > Count: 1.342e+08 > [Error] Couldn't open # dsname index > :strict! (Err: Servname not supported for ai_socktype) > > That error comes from the function default_addrinfo(...) in > rockstar source code, file inet/socket.c. > > I don't know how well you guys know rockstar - let me know if > you think this is the dead end. > > n > > > > On 03/13/2018 03:18 PM, Britton Smith wrote: > > Hi Nick, > > You might need to add the "num_readers" and "num_writers" > keywords to this line: > rhf = RockstarHaloFinder(d, num_readers=8, num_writers=8) > The readers are the i/o nodes and the writers are the actual > halo finding instances. 8 for each is a good place to > start. When you run, you'll need to have (num_readers + > num_writers + 1) MPI processes, where the extra one is for > the server. > > Hopefully, that takes care of it. > > Britton > > > > On Tue, Mar 13, 2018 at 12:43 PM, Nick Gnedin > <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>>> wrote: > > > Nathan, > > Thank you. I followed the online instructions and now > Rockstar > starts, but gives me an error, irrespective of whether > I call it > directly or via a HaloCatalog object: > > ---------- code --------- > d = > yt.load(root+"/rei20_a"+aexp+"/rei20_a"+aexp+".art") > rhf = RockstarHaloFinder(d) > rhf.run() > ------------------------- > > Traceback (most recent call last): > File "hfc.py", line 15, in <module> > rhf = RockstarHaloFinder(d) > File > > "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 230, in __init__ > self.pool, self.workgroup = self.runner.setup_pool() > File > > "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 105, in setup_pool > (self.num_writers, "writers") ] > File > > "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", > line 400, in from_sizes > pool.add_workgroup(size, name = name) > File > > "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", > line 368, in add_workgroup > group = self.comm.comm.Get_group().Incl(ranks) > AttributeError: 'NoneType' object has no attribute > 'Get_group' > > > On 03/13/2018 02:01 PM, Nathan Goldbaum wrote: > > > > On Tue, Mar 13, 2018 at 1:52 PM, Nick Gnedin > <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>>> wrote: > > > Britton, > > I am trying to run Rockstar, but it does not > seem to be > packaged > with yt by default, import fails: > > Traceback (most recent call last): > File "hfc.py", line 3, in <module> > from > yt.analysis_modules.halo_finding.rockstar.api import > RockstarHaloFinder > File > > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/api.py", > line 16, in <module> > from .rockstar import RockstarHaloFinder > File > > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 28, in <module> > from . import rockstar_interface > ImportError: cannot import name > 'rockstar_interface' > > > I think I am using the latest versions of > python and yt: > > > yt module located at: > > /home/gnedin/anaconda3/lib/python3.6/site-packages > > The current version of yt is: > > --- > Version = 3.4.1 > --- > > > It is not packaged with yt out of the box. The > easiest way to > get a copy of yt built with the rockstar bindings > is to install > yt with the install script. You'll need to modify > the script so > that INST_ROCKSTAR=1 once you've downloaded it. > > Altenatively you can manually build yt and rockstar > following > the instructions in the docs here: > > http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r… > <http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r…> > > <http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r… > <http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r…>> > > The rockstar halo finder is licensed under GPLv3 so > unfortunately we can't distribute it with the yt > binaries on > pypi or conda-forge without changing their license > as well. > > > n > > > On 03/13/2018 12:44 PM, Britton Smith wrote: > > Hi Nick, > > Thanks for your report. Your timing data > confirms my > suspicion > about which part of the code isn't > scaling. The > rejoining of > the halo list after the halo finder is run > makes heavy > use of > MPI broadcast calls. Reworking this > shouldn't be too > difficult, > just a question of someone finding the > time. If anyone is > interested in trying to fix this, I can > direct them to the > places that need the attention. > > Nick, in the mean time, you might try the > Rockstar halo > finder > (either the one built-in to yt or the > standalone), > which scales > quite well. The output from both Rockstar > versions is > loadable > with yt. > > Britton > > On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin > <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>> > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>> > > <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>>>> > > wrote: > > > This is just a notice to the developer. > > I have run a HOP halo finder for a > large ART > simulation (1024^3 > particles) on BlueWaters with the > following code: > > import yt > from > yt.analysis_modules.halo_analysis.api import > HaloCatalog > yt.enable_parallelism() > path = > "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D" > aexps = [ "0.1280", "0.1203", > "0.1115", "0.1002", > "0.0907" ] > for aexp in aexps: > d = yt.load(path+"/rei40_a"+aexp > +"/rei40_a"+aexp+".art") > hc = > > HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"}) > hc.create() > > Because of memory constraints, I have > to run it on > at least > 4 MPI > ranks, and I noticed that yt > implementation of HOP > does not > scale - > a 4-rank job takes 14.5 hours and an > 8-rank one > takes 15.75 > hours. > Surely, halo finding for billion > particles should > scale > better than > that. > > Here is some timing info (I can > provide a full log > if you care) > > 4-rank run: > P000 yt : [INFO ] 2018-03-11 > 17:31:25,531 > Parameters: ... > P000 yt : [INFO ] 2018-03-11 > 18:36:49,781 > Initializing > HOP [1h] > P002 yt : [INFO ] 2018-03-11 > 22:14:15,486 Parsing > outputs [3.5h] > P000 yt : [INFO ] 2018-03-12 > 08:06:38,231 > Saving halo > ... [10h] > > 8-rank run: > P000 yt : [INFO ] 2018-03-10 > 21:03:27,226 > Parameters: ... > P000 yt : [INFO ] 2018-03-10 > 21:43:52,543 > Initializing > HOP [0.75h] > P005 yt : [INFO ] 2018-03-10 > 23:52:10,389 Parsing > outputs [2h] > P000 yt : [INFO ] 2018-03-11 > 12:43:46,645 > Saving halo > ... [12.5h *] > > * - does not scale at all. > > n > > > _______________________________________________ > yt-users mailing list -- > yt-users(a)python.org <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>> > <mailto:yt-users@python.org > <mailto:yt-users@python.org> <mailto:yt-users@python.org > <mailto:yt-users@python.org>>> > <mailto:yt-users@python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>> <mailto:yt-users@python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>>>> > To unsubscribe send an email to > yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>>>> > > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org > <mailto:yt-users@python.org>> > <mailto:yt-users@python.org > <mailto:yt-users@python.org> <mailto:yt-users@python.org > <mailto:yt-users@python.org>>> > To unsubscribe send an email to > yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>>> > > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org <mailto:yt-users@python.org>> > To unsubscribe send an email to > yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > > > >

2 1

Re: MPI scaling of HOP
by Nick Gnedin March 13, 2018

March 13, 2018

Britton, That helps, and Rockstar starts and loads data, but now it gives an internal error: P001 yt : [INFO ] 2018-03-13 16:03:11,005 Created 2048 chunks for ARTIO P001 yt : [WARNING ] 2018-03-13 16:05:09,506 Total Particle Count: 1.342e+08 [Error] Couldn't open # dsname index :strict! (Err: Servname not supported for ai_socktype) That error comes from the function default_addrinfo(...) in rockstar source code, file inet/socket.c. I don't know how well you guys know rockstar - let me know if you think this is the dead end. n On 03/13/2018 03:18 PM, Britton Smith wrote: > Hi Nick, > > You might need to add the "num_readers" and "num_writers" keywords to > this line: > rhf = RockstarHaloFinder(d, num_readers=8, num_writers=8) > The readers are the i/o nodes and the writers are the actual halo > finding instances. 8 for each is a good place to start. When you run, > you'll need to have (num_readers + num_writers + 1) MPI processes, where > the extra one is for the server. > > Hopefully, that takes care of it. > > Britton > > > On Tue, Mar 13, 2018 at 12:43 PM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com>> wrote: > > > Nathan, > > Thank you. I followed the online instructions and now Rockstar > starts, but gives me an error, irrespective of whether I call it > directly or via a HaloCatalog object: > > ---------- code --------- > d = yt.load(root+"/rei20_a"+aexp+"/rei20_a"+aexp+".art") > rhf = RockstarHaloFinder(d) > rhf.run() > ------------------------- > > Traceback (most recent call last): > File "hfc.py", line 15, in <module> > rhf = RockstarHaloFinder(d) > File > "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 230, in __init__ > self.pool, self.workgroup = self.runner.setup_pool() > File > "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 105, in setup_pool > (self.num_writers, "writers") ] > File > "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", > line 400, in from_sizes > pool.add_workgroup(size, name = name) > File > "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", > line 368, in add_workgroup > group = self.comm.comm.Get_group().Incl(ranks) > AttributeError: 'NoneType' object has no attribute 'Get_group' > > > On 03/13/2018 02:01 PM, Nathan Goldbaum wrote: > > > > On Tue, Mar 13, 2018 at 1:52 PM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>> wrote: > > > Britton, > > I am trying to run Rockstar, but it does not seem to be > packaged > with yt by default, import fails: > > Traceback (most recent call last): > File "hfc.py", line 3, in <module> > from yt.analysis_modules.halo_finding.rockstar.api import > RockstarHaloFinder > File > > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/api.py", > line 16, in <module> > from .rockstar import RockstarHaloFinder > File > > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 28, in <module> > from . import rockstar_interface > ImportError: cannot import name 'rockstar_interface' > > > I think I am using the latest versions of python and yt: > > > yt module located at: > /home/gnedin/anaconda3/lib/python3.6/site-packages > > The current version of yt is: > > --- > Version = 3.4.1 > --- > > > It is not packaged with yt out of the box. The easiest way to > get a copy of yt built with the rockstar bindings is to install > yt with the install script. You'll need to modify the script so > that INST_ROCKSTAR=1 once you've downloaded it. > > Altenatively you can manually build yt and rockstar following > the instructions in the docs here: > > http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r… > <http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r…> > > The rockstar halo finder is licensed under GPLv3 so > unfortunately we can't distribute it with the yt binaries on > pypi or conda-forge without changing their license as well. > > > n > > > On 03/13/2018 12:44 PM, Britton Smith wrote: > > Hi Nick, > > Thanks for your report. Your timing data confirms my > suspicion > about which part of the code isn't scaling. The > rejoining of > the halo list after the halo finder is run makes heavy > use of > MPI broadcast calls. Reworking this shouldn't be too > difficult, > just a question of someone finding the time. If anyone is > interested in trying to fix this, I can direct them to the > places that need the attention. > > Nick, in the mean time, you might try the Rockstar halo > finder > (either the one built-in to yt or the standalone), > which scales > quite well. The output from both Rockstar versions is > loadable > with yt. > > Britton > > On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin > <ngnedin(a)gmail.com <mailto:ngnedin@gmail.com> > <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>> > <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com> > > <mailto:ngnedin@gmail.com <mailto:ngnedin@gmail.com>>>> > wrote: > > > This is just a notice to the developer. > > I have run a HOP halo finder for a large ART > simulation (1024^3 > particles) on BlueWaters with the following code: > > import yt > from yt.analysis_modules.halo_analysis.api import > HaloCatalog > yt.enable_parallelism() > path = "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D" > aexps = [ "0.1280", "0.1203", "0.1115", "0.1002", > "0.0907" ] > for aexp in aexps: > d = yt.load(path+"/rei40_a"+aexp > +"/rei40_a"+aexp+".art") > hc = > > HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"}) > hc.create() > > Because of memory constraints, I have to run it on > at least > 4 MPI > ranks, and I noticed that yt implementation of HOP > does not > scale - > a 4-rank job takes 14.5 hours and an 8-rank one > takes 15.75 > hours. > Surely, halo finding for billion particles should > scale > better than > that. > > Here is some timing info (I can provide a full log > if you care) > > 4-rank run: > P000 yt : [INFO ] 2018-03-11 17:31:25,531 > Parameters: ... > P000 yt : [INFO ] 2018-03-11 18:36:49,781 > Initializing > HOP [1h] > P002 yt : [INFO ] 2018-03-11 22:14:15,486 Parsing > outputs [3.5h] > P000 yt : [INFO ] 2018-03-12 08:06:38,231 > Saving halo > ... [10h] > > 8-rank run: > P000 yt : [INFO ] 2018-03-10 21:03:27,226 > Parameters: ... > P000 yt : [INFO ] 2018-03-10 21:43:52,543 > Initializing > HOP [0.75h] > P005 yt : [INFO ] 2018-03-10 23:52:10,389 Parsing > outputs [2h] > P000 yt : [INFO ] 2018-03-11 12:43:46,645 > Saving halo > ... [12.5h *] > > * - does not scale at all. > > n > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org <mailto:yt-users@python.org>> > <mailto:yt-users@python.org > <mailto:yt-users@python.org> <mailto:yt-users@python.org > <mailto:yt-users@python.org>>> > To unsubscribe send an email to > yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>>> > > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org <mailto:yt-users@python.org>> > To unsubscribe send an email to yt-users-leave(a)python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > To unsubscribe send an email to yt-users-leave(a)python.org > <mailto:yt-users-leave@python.org> > >

2 2

Error in Setting Particle Filter in EAGLE simulation
by Stephanie Ho March 13, 2018

March 13, 2018

Hi, I tried to set a particle filter with EAGLE and got an error. Here is a simple example, which uses the EAGLE dataset from yt sample data. import yt from yt.data_objects.particle_filters import add_particle_filter eagle_fpath = './data/snapshot_028_z000p000/snap_028_z000p000.0.hdf5' unit_base = {'UnitLength_in_cm' : 3.085678e+24, 'UnitMass_in_g' : 1.989e+43, 'UnitVelocity_in_cm_per_s' : 100000.0} ds = yt.load(eagle_fpath,unit_base=unit_base) ad = ds.all_data() def cold_gas(pfilter, data): temperature = data[pfilter.filtered_type, "Temperature"] filter = (temperature.in_units('K') <= 1e5) return filter add_particle_filter("gas_cold", function=cold_gas, filtered_type='PartType0', requires=["Temperature"]) ds.add_particle_filter('gas_cold') Below shows the error. UnboundLocalError Traceback (most recent call last) /Users/stephaniehho/EAGLE_sim/sho/test_with_yt/test_z000p000/particle_filter_simple.py in <module>() 20 21 add_particle_filter("gas_cold", function=cold_gas, filtered_type='PartType0', requires=["Temperature"]) ---> 22 ds.add_particle_filter('gas_cold') /Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in add_particle_filter(self, filter) 673 if f is None: 674 return False --> 675 used = self._setup_filtered_type(f) 676 if used: 677 filter = f /Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in _setup_filtered_type(self, filter) 714 if filter.name not in self.filtered_particle_types: 715 self.filtered_particle_types.append(filter.name) --> 716 new_fields = self._setup_particle_types([filter.name]) 717 deps, _ = self.field_info.check_derived_fields(new_fields) 718 self.field_dependencies.update(deps) /Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in _setup_particle_types(self, ptypes) 723 if ptypes is None: ptypes = self.ds.particle_types_raw 724 for ptype in set(ptypes): --> 725 df += self._setup_particle_type(ptype) 726 return df 727 /Users/stephaniehho/yt-git/yt/data_objects/static_output.pyc in _setup_particle_type(self, ptype) 889 def _setup_particle_type(self, ptype): 890 orig = set(self.field_info.items()) --> 891 self.field_info.setup_particle_fields(ptype) 892 return [n for n, v in set(self.field_info.items()).difference(orig)] 893 /Users/stephaniehho/yt-git/yt/frontends/owls/fields.pyc in setup_particle_fields(self, ptype) 111 112 super(OWLSFieldInfo,self).setup_particle_fields( --> 113 ptype, num_neighbors=self._num_neighbors, ftype=ftype) 114 115 UnboundLocalError: local variable 'ftype' referenced before assignment What I find is that, this problem seems to be temporarily solved if I add en else case at line 111 of the code: https://github.com/yt-project/yt/blob/master/yt/frontends/owls/fields.py, else: ftype=ptype But this will raise the questions of, what are ftype and ptype designed for at the first place, or will this temporary fix cause other errors that I have not yet encountered. Thanks, Stephanie

2 1

Re: MPI scaling of HOP
by Nick Gnedin March 13, 2018

March 13, 2018

Nathan, Thank you. I followed the online instructions and now Rockstar starts, but gives me an error, irrespective of whether I call it directly or via a HaloCatalog object: ---------- code --------- d = yt.load(root+"/rei20_a"+aexp+"/rei20_a"+aexp+".art") rhf = RockstarHaloFinder(d) rhf.run() ------------------------- Traceback (most recent call last): File "hfc.py", line 15, in <module> rhf = RockstarHaloFinder(d) File "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", line 230, in __init__ self.pool, self.workgroup = self.runner.setup_pool() File "/scratch/midway2/gnedin/TMP/yt/yt/analysis_modules/halo_finding/rockstar/rockstar.py", line 105, in setup_pool (self.num_writers, "writers") ] File "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 400, in from_sizes pool.add_workgroup(size, name = name) File "/scratch/midway2/gnedin/TMP/yt/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 368, in add_workgroup group = self.comm.comm.Get_group().Incl(ranks) AttributeError: 'NoneType' object has no attribute 'Get_group' On 03/13/2018 02:01 PM, Nathan Goldbaum wrote: > > > On Tue, Mar 13, 2018 at 1:52 PM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com>> wrote: > > > Britton, > > I am trying to run Rockstar, but it does not seem to be packaged > with yt by default, import fails: > > Traceback (most recent call last): > File "hfc.py", line 3, in <module> > from yt.analysis_modules.halo_finding.rockstar.api import > RockstarHaloFinder > File > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/api.py", > line 16, in <module> > from .rockstar import RockstarHaloFinder > File > "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/rockstar.py", > line 28, in <module> > from . import rockstar_interface > ImportError: cannot import name 'rockstar_interface' > > > I think I am using the latest versions of python and yt: > > > yt module located at: > /home/gnedin/anaconda3/lib/python3.6/site-packages > > The current version of yt is: > > --- > Version = 3.4.1 > --- > > > It is not packaged with yt out of the box. The easiest way to get a copy > of yt built with the rockstar bindings is to install yt with the install > script. You'll need to modify the script so that INST_ROCKSTAR=1 once > you've downloaded it. > > Altenatively you can manually build yt and rockstar following the > instructions in the docs here: > > http://yt-project.org/docs/dev/installing.html#installing-support-for-the-r… > > The rockstar halo finder is licensed under GPLv3 so unfortunately we > can't distribute it with the yt binaries on pypi or conda-forge without > changing their license as well. > > > n > > > On 03/13/2018 12:44 PM, Britton Smith wrote: > > Hi Nick, > > Thanks for your report. Your timing data confirms my suspicion > about which part of the code isn't scaling. The rejoining of > the halo list after the halo finder is run makes heavy use of > MPI broadcast calls. Reworking this shouldn't be too difficult, > just a question of someone finding the time. If anyone is > interested in trying to fix this, I can direct them to the > places that need the attention. > > Nick, in the mean time, you might try the Rockstar halo finder > (either the one built-in to yt or the standalone), which scales > quite well. The output from both Rockstar versions is loadable > with yt. > > Britton > > On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com> <mailto:ngnedin@gmail.com > <mailto:ngnedin@gmail.com>>> wrote: > > > This is just a notice to the developer. > > I have run a HOP halo finder for a large ART simulation (1024^3 > particles) on BlueWaters with the following code: > > import yt > from yt.analysis_modules.halo_analysis.api import HaloCatalog > yt.enable_parallelism() > path = "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D" > aexps = [ "0.1280", "0.1203", "0.1115", "0.1002", "0.0907" ] > for aexp in aexps: > d = yt.load(path+"/rei40_a"+aexp +"/rei40_a"+aexp+".art") > hc = > > HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"}) > hc.create() > > Because of memory constraints, I have to run it on at least > 4 MPI > ranks, and I noticed that yt implementation of HOP does not > scale - > a 4-rank job takes 14.5 hours and an 8-rank one takes 15.75 > hours. > Surely, halo finding for billion particles should scale > better than > that. > > Here is some timing info (I can provide a full log if you care) > > 4-rank run: > P000 yt : [INFO ] 2018-03-11 17:31:25,531 Parameters: ... > P000 yt : [INFO ] 2018-03-11 18:36:49,781 Initializing > HOP [1h] > P002 yt : [INFO ] 2018-03-11 22:14:15,486 Parsing > outputs [3.5h] > P000 yt : [INFO ] 2018-03-12 08:06:38,231 Saving halo > ... [10h] > > 8-rank run: > P000 yt : [INFO ] 2018-03-10 21:03:27,226 Parameters: ... > P000 yt : [INFO ] 2018-03-10 21:43:52,543 Initializing > HOP [0.75h] > P005 yt : [INFO ] 2018-03-10 23:52:10,389 Parsing > outputs [2h] > P000 yt : [INFO ] 2018-03-11 12:43:46,645 Saving halo > ... [12.5h *] > > * - does not scale at all. > > n > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > <mailto:yt-users@python.org <mailto:yt-users@python.org>> > To unsubscribe send an email to yt-users-leave(a)python.org > <mailto:yt-users-leave@python.org> > <mailto:yt-users-leave@python.org > <mailto:yt-users-leave@python.org>> > > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > To unsubscribe send an email to yt-users-leave(a)python.org > <mailto:yt-users-leave@python.org> > >

2 1

Re: MPI scaling of HOP
by Nick Gnedin March 13, 2018

March 13, 2018

Britton, I am trying to run Rockstar, but it does not seem to be packaged with yt by default, import fails: Traceback (most recent call last): File "hfc.py", line 3, in <module> from yt.analysis_modules.halo_finding.rockstar.api import RockstarHaloFinder File "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/api.py", line 16, in <module> from .rockstar import RockstarHaloFinder File "/home/gnedin/anaconda3/lib/python3.6/site-packages/yt/analysis_modules/halo_finding/rockstar/rockstar.py", line 28, in <module> from . import rockstar_interface ImportError: cannot import name 'rockstar_interface' I think I am using the latest versions of python and yt: yt module located at: /home/gnedin/anaconda3/lib/python3.6/site-packages The current version of yt is: --- Version = 3.4.1 --- n On 03/13/2018 12:44 PM, Britton Smith wrote: > Hi Nick, > > Thanks for your report. Your timing data confirms my suspicion about > which part of the code isn't scaling. The rejoining of the halo list > after the halo finder is run makes heavy use of MPI broadcast calls. > Reworking this shouldn't be too difficult, just a question of someone > finding the time. If anyone is interested in trying to fix this, I can > direct them to the places that need the attention. > > Nick, in the mean time, you might try the Rockstar halo finder (either > the one built-in to yt or the standalone), which scales quite well. The > output from both Rockstar versions is loadable with yt. > > Britton > > On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com>> wrote: > > > This is just a notice to the developer. > > I have run a HOP halo finder for a large ART simulation (1024^3 > particles) on BlueWaters with the following code: > > import yt > from yt.analysis_modules.halo_analysis.api import HaloCatalog > yt.enable_parallelism() > path = "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D" > aexps = [ "0.1280", "0.1203", "0.1115", "0.1002", "0.0907" ] > for aexp in aexps: > d = yt.load(path+"/rei40_a"+aexp +"/rei40_a"+aexp+".art") > hc = > HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"}) > hc.create() > > Because of memory constraints, I have to run it on at least 4 MPI > ranks, and I noticed that yt implementation of HOP does not scale - > a 4-rank job takes 14.5 hours and an 8-rank one takes 15.75 hours. > Surely, halo finding for billion particles should scale better than > that. > > Here is some timing info (I can provide a full log if you care) > > 4-rank run: > P000 yt : [INFO ] 2018-03-11 17:31:25,531 Parameters: ... > P000 yt : [INFO ] 2018-03-11 18:36:49,781 Initializing HOP [1h] > P002 yt : [INFO ] 2018-03-11 22:14:15,486 Parsing outputs [3.5h] > P000 yt : [INFO ] 2018-03-12 08:06:38,231 Saving halo ... [10h] > > 8-rank run: > P000 yt : [INFO ] 2018-03-10 21:03:27,226 Parameters: ... > P000 yt : [INFO ] 2018-03-10 21:43:52,543 Initializing HOP [0.75h] > P005 yt : [INFO ] 2018-03-10 23:52:10,389 Parsing outputs [2h] > P000 yt : [INFO ] 2018-03-11 12:43:46,645 Saving halo ... [12.5h *] > > * - does not scale at all. > > n > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > To unsubscribe send an email to yt-users-leave(a)python.org > <mailto:yt-users-leave@python.org> > >

2 1

Re: MPI scaling of HOP
by Nick Gnedin March 13, 2018

March 13, 2018

Britton, Thank you, I will try RockStar. I also do not promise anything, but am willing to take a look at those MPI calls. n On 03/13/2018 12:44 PM, Britton Smith wrote: > Hi Nick, > > Thanks for your report. Your timing data confirms my suspicion about > which part of the code isn't scaling. The rejoining of the halo list > after the halo finder is run makes heavy use of MPI broadcast calls. > Reworking this shouldn't be too difficult, just a question of someone > finding the time. If anyone is interested in trying to fix this, I can > direct them to the places that need the attention. > > Nick, in the mean time, you might try the Rockstar halo finder (either > the one built-in to yt or the standalone), which scales quite well. The > output from both Rockstar versions is loadable with yt. > > Britton > > On Tue, Mar 13, 2018 at 8:46 AM, Nick Gnedin <ngnedin(a)gmail.com > <mailto:ngnedin@gmail.com>> wrote: > > > This is just a notice to the developer. > > I have run a HOP halo finder for a large ART simulation (1024^3 > particles) on BlueWaters with the following code: > > import yt > from yt.analysis_modules.halo_analysis.api import HaloCatalog > yt.enable_parallelism() > path = "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D" > aexps = [ "0.1280", "0.1203", "0.1115", "0.1002", "0.0907" ] > for aexp in aexps: > d = yt.load(path+"/rei40_a"+aexp +"/rei40_a"+aexp+".art") > hc = > HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"}) > hc.create() > > Because of memory constraints, I have to run it on at least 4 MPI > ranks, and I noticed that yt implementation of HOP does not scale - > a 4-rank job takes 14.5 hours and an 8-rank one takes 15.75 hours. > Surely, halo finding for billion particles should scale better than > that. > > Here is some timing info (I can provide a full log if you care) > > 4-rank run: > P000 yt : [INFO ] 2018-03-11 17:31:25,531 Parameters: ... > P000 yt : [INFO ] 2018-03-11 18:36:49,781 Initializing HOP [1h] > P002 yt : [INFO ] 2018-03-11 22:14:15,486 Parsing outputs [3.5h] > P000 yt : [INFO ] 2018-03-12 08:06:38,231 Saving halo ... [10h] > > 8-rank run: > P000 yt : [INFO ] 2018-03-10 21:03:27,226 Parameters: ... > P000 yt : [INFO ] 2018-03-10 21:43:52,543 Initializing HOP [0.75h] > P005 yt : [INFO ] 2018-03-10 23:52:10,389 Parsing outputs [2h] > P000 yt : [INFO ] 2018-03-11 12:43:46,645 Saving halo ... [12.5h *] > > * - does not scale at all. > > n > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > To unsubscribe send an email to yt-users-leave(a)python.org > <mailto:yt-users-leave@python.org> > >

1 0

MPI scaling of HOP
by Nick Gnedin March 13, 2018

March 13, 2018

This is just a notice to the developer. I have run a HOP halo finder for a large ART simulation (1024^3 particles) on BlueWaters with the following code: import yt from yt.analysis_modules.halo_analysis.api import HaloCatalog yt.enable_parallelism() path = "/mnt/c/scratch/sciteam/ngnedin/PERM/B40/D" aexps = [ "0.1280", "0.1203", "0.1115", "0.1002", "0.0907" ] for aexp in aexps: d = yt.load(path+"/rei40_a"+aexp +"/rei40_a"+aexp+".art") hc = HaloCatalog(data_ds=d,finder_method='hop',output_dir=path+"/a="+aexp+"/hop",finder_kwargs={"dm_only":False,"ptype":"N-BODY"}) hc.create() Because of memory constraints, I have to run it on at least 4 MPI ranks, and I noticed that yt implementation of HOP does not scale - a 4-rank job takes 14.5 hours and an 8-rank one takes 15.75 hours. Surely, halo finding for billion particles should scale better than that. Here is some timing info (I can provide a full log if you care) 4-rank run: P000 yt : [INFO ] 2018-03-11 17:31:25,531 Parameters: ... P000 yt : [INFO ] 2018-03-11 18:36:49,781 Initializing HOP [1h] P002 yt : [INFO ] 2018-03-11 22:14:15,486 Parsing outputs [3.5h] P000 yt : [INFO ] 2018-03-12 08:06:38,231 Saving halo ... [10h] 8-rank run: P000 yt : [INFO ] 2018-03-10 21:03:27,226 Parameters: ... P000 yt : [INFO ] 2018-03-10 21:43:52,543 Initializing HOP [0.75h] P005 yt : [INFO ] 2018-03-10 23:52:10,389 Parsing outputs [2h] P000 yt : [INFO ] 2018-03-11 12:43:46,645 Saving halo ... [12.5h *] * - does not scale at all. n

2 1

Re: PARTICLE_AGE NOT DEFINED FOR RAMSES DATA
by Corentin CADIOU March 13, 2018

March 13, 2018

Hi, YT reads the particle fields in the order they are found in the output. So according to your information, you should have three (I don’t know why you have five actually) |particle_extra_field|, representing in order the potential, birth_epoch and metallicity. If you want to load the data correctly, use: |ds = yt.load(my_output, extra_particle_fields=[ ('particle_potential', 'd'), ('particle_birth_time', 'd'), ('particle_metallicity', 'd') ]) | Please also note the by default, yt doesn’t know the unit of the potential so you’ll probably get something unit-less which you have to convert yourself into the right unit using a derived field (see http://yt-project.org/doc/developing/creating_derived_fields.html). However, be careful that in |output_part.f90|, that part that outputs the potential may be between |#ifdef OUTPUT_PARTICLE_POTENTIAL […]#endif|. So if you compiled RAMSES without the flag |OUTPUT_PARTICLE_POTENTIAL|, the potential won’t be printed out and you should skip the line loading the |particle_potential| in the |yt.load|-lines above. Cheers, Corentin ------------------------------------------------------------------------ There are two other solutions to your problem: 1. Use a more recent version of |output_part.f90| (eg. https://bitbucket.org/rteyssie/ramses/src/6ba4dbb10b47f305917b7a31488f5c267…) and eventually |output_hydro.f90|. You’ll also have to add the file |io/dump_utils.f90| and adapt the |Makefile| accordingly. You can probably use the latest version of the |Makefile|. If you do so, RAMSES will dump some metadata that yt can read to help it figure out which field is which. This is what I recommend. If you already ran some simulations, use the second solution. 2. Add a file named |part_file_descriptor.txt| and |hydro_file_descriptor.txt| in your output directories (|output_00001|, …, |output_xxxxx|) containing respectively |# version: 1 # ivar, variable_name, variable_type 1, position_x, d 2, position_y, d 3, position_z, d 4, velocity_x, d 5, velocity_y, d 6, velocity_z, d 7, mass, d 8, identity, i 9, levelp, i 10, potential, d 11, birth_time, d 12, metallicity, d | for the |part_file_descriptor.txt|. Do /not/ add a newline at the end of the file as it would make yt crash. You can add in the order outputted by |output_part.f90| extra fields that I may have missed. |hydro_file_descriptor.txt| should contain |# version: 1 # ivar, variable_name, variable_type 1, density, d 2, velocity_x, d 3, velocity_y, d 4, velocity_z, d 5, pressure, d 6, metallicity, d | Once again, adapt it if you have more that 6 variables. Here’s also a one-liner that copies |part_file_descriptor.txt| into the directories. You should run it from the folder containing your outputs |find . -name 'output_?????' -type d | xargs -I{} cp /path/to/part_file_descriptor.txt {} | On 2018-03-12 22:56, Vadlamani Samhitha wrote: > Thanks for the response. But, how do I figure out which of the extra > fields correspond to particle_age? The output order in output_part.f90 > in the ramses patch is as follows: position, velocity, mass, > identifier, refinement level, potential, birth epoch and metallicity. > But the fields being printed seem to be in alphabetic order. How to > find out /verify the order in which YT reads these fields from my output? > > On Mon, Mar 12, 2018 at 11:03 AM, Corentin CADIOU > <corentin.cadiou(a)iap.fr <mailto:corentin.cadiou@iap.fr>> wrote: > > Hi, > > Until the RAMSES version of ~December 2017, no information > whatsoever was left for post-processing tools to figure out what > fields are stored in the particle files. This means that yt can’t > figure out whether a field in the file is actually the age of > particle or, e.g., its metallicity. As a result, by default, yt > detects these extra unknown field and serves it in the > |extra_particle_field_xx| with no unit. In your case, one of these > field is the particle age, in unit_time, directly as read from the > output. > > If you want to convert it in the right unit, you can either add a > new field using a derived field where you convert from code unit > to physical unit (be careful in cosmological simulation that the > time is the conformal time). The other option is to teach yt which > extra_fields to expect. Assuming the birth time is stored before > the metallicity, you can use: > > |ds = yt.load('output_xxxxx/info_xxxxx.txt', > extra_particle_fields=[('particle_birth_time', 'd'), > ('particle_metallicity', 'd')]) ('io', ''particle_birth_time in > ds.derived_field_list # is True ds.r['io', 'particle_age'] # is > now in seconds. Check twice it has been converted correctly. | > > FYI, yt knows that |particle_age| and |particle_birth_time| are > expressed in time unit, so adding them in the > |extra_particle_fields| is enough for them to be defined with a > time unit. Any other name would become a dimensionless number > (like metallicity). > > I hope this helps and don’t hesitate to ask any question / send > suggestion of how to improve. I know the particle age is a little > messy right now. > > Cheers, > > Corentin > > On 2018-03-11 00:10, Vadlamani Samhitha wrote: > >> Thank you for your efforts. I looked at those "extra" fields and >> none of them corresponds to time: they're all dimensionless. >> >> On Sat, Mar 10, 2018 at 10:32 PM, Nathan Goldbaum >> <nathan12343(a)gmail.com <mailto:nathan12343@gmail.com>> wrote: >> >> I’m sorry but I’m not familiar enough with the RAMSES output >> format to be able to confidently answer your questions. >> >> I’ve pinged Corentin Cadiou, who has been doing a lot of work >> recently to improve the RAMSES frontend in yt, hopefully >> he’ll have a better idea. >> >> The changeset hash is a way of identifying precisely which >> version of yt you have installed from the git repository. If >> you don’t know what that is that’s ok, I was just curious if >> you happened to have an old version installed. It’s possible >> installing the latest development version of yt might help >> things if you originally installed a version from last Fall, >> although if you installed yt recently you probably have a >> version reflecting the latest state of the RAMSES frontend. >> >> >> >> On Sat, Mar 10, 2018 at 2:44 PM Vadlamani Samhitha >> <vadlamani.samhitha(a)gmail.com >> <mailto:vadlamani.samhitha@gmail.com>> wrote: >> >> I see. One possible check I could do is to find out the >> units for these fields and hope for only one of them to >> have time units. Is there anything else I could do to >> confirm the field? I’m not really sure of what a change >> set hash is or if I could find out at this point? Would >> updating yt help in any way? >> >> On Sat, 10 Mar 2018 at 9:10 PM, Nathan Goldbaum >> <nathan12343(a)gmail.com <mailto:nathan12343@gmail.com>> wrote: >> >> On Sat, Mar 10, 2018 at 1:44 PM Vadlamani Samhitha >> <vadlamani.samhitha(a)gmail.com >> <mailto:vadlamani.samhitha@gmail.com>> wrote: >> >> ds.field_list: >> >> ('all', 'particle_extra_field_1') >> >> ('all', 'particle_extra_field_2') >> >> ('all', 'particle_extra_field_3') >> >> ('all', 'particle_extra_field_4') >> >> ('all', 'particle_extra_field_5') >> >> >> If I had to guess, particle_age is one of these. >> >> ('all', 'particle_identifier') >> >> ('all', 'particle_mass') >> >> ('all', 'particle_position_x') >> >> ('all', 'particle_position_y') >> >> ('all', 'particle_position_z') >> >> ('all', 'particle_refinement_level') >> >> ('all', 'particle_velocity_x') >> >> ('all', 'particle_velocity_y') >> >> ('all', 'particle_velocity_z') >> >> ('io', 'particle_extra_field_1') >> >> ('io', 'particle_extra_field_2') >> >> ('io', 'particle_extra_field_3') >> >> ('io', 'particle_extra_field_4') >> >> ('io', 'particle_extra_field_5') >> >> ('io', 'particle_identifier') >> >> ('io', 'particle_mass') >> >> ('io', 'particle_position_x') >> >> ('io', 'particle_position_y') >> >> ('io', 'particle_position_z') >> >> ('io', 'particle_refinement_level') >> >> ('io', 'particle_velocity_x') >> >> ('io', 'particle_velocity_y') >> >> ('io', 'particle_velocity_z') >> >> ('ramses', 'Density') >> >> ('ramses', 'Metallicity') >> >> ('ramses', 'Pressure') >> >> ('ramses', 'var6') >> >> ('ramses', 'x-velocity') >> >> ('ramses', 'y-velocity') >> >> ('ramses', 'z-velocity') >> >> >> version: >> >> Version = 3.5.dev0 >> >> >> This means you built yt from source. Do you know what >> changeset hash you used? >> >> >> On Sat, Mar 10, 2018 at 3:50 PM, Nathan Goldbaum >> <nathan12343(a)gmail.com >> <mailto:nathan12343@gmail.com>> wrote: >> >> Hi, >> >> Just a head's up, the gmail account you used >> to contact the mailing list isn't subscribed >> so I had to manually let it through. >> >> It may be the case that your dataset doesn't >> have the particle_age field defined, I think >> only a subset of RAMSES datasets have it defined. >> >> Can you tell us what ds.field_list is? Can >> you also tell us what yt version you are using? >> >> -Nathan >> >> On Sat, Mar 10, 2018 at 5:16 AM, Samhitha >> Vadlamani <vadlamani.bits(a)gmail.com >> <mailto:vadlamani.bits@gmail.com>> wrote: >> >> Hi, >> >> I've recently started working with yt and >> am trying to extract "particle_age" for >> newly formed stars in my simulation: >> >> def formed_star(pfilter, data): >> >> filter = data["all", >> "particle_age"] < 0 >> >> return filter >> >> >> add_particle_filter("formed_star", >> function=formed_star, >> filtered_type='all', >> >> >> requires=["particle_age"]) >> >> However, I encounter an error: >> >> the fields >> >> ('all', 'particle_age'), >> >> required by the "formed_star" >> particle filter, are not defined for >> this dataset. >> >> >> Could anyone please recommend a possible >> solution for why "particle_age" is not >> being detected for my dataset? Thanks. >> >> >> _______________________________________________ >> yt-users mailing list -- >> yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to >> yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> _______________________________________________ >> yt-users mailing list -- >> yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to >> yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> >> >> _______________________________________________ >> yt-users mailing list -- yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to >> yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> _______________________________________________ >> yt-users mailing list -- yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to >> yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> >> _______________________________________________ >> yt-users mailing list -- yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to >> yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> _______________________________________________ >> yt-users mailing list -- yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to >> yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> _______________________________________________ >> yt-users mailing list -- yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> >> _______________________________________________ >> yt-users mailing list -- yt-users(a)python.org >> <mailto:yt-users@python.org> >> To unsubscribe send an email to yt-users-leave(a)python.org >> <mailto:yt-users-leave@python.org> >> >> >> >> >> _______________________________________________ >> yt-users mailing list -- yt-users(a)python.org <mailto:yt-users@python.org> >> To unsubscribe send an email to yt-users-leave(a)python.org <mailto:yt-users-leave@python.org> > > > -- > Corentin Cadiou > PhD student > Institut d'Astrophysique de Paris (IAP), desk 142b > 98 bis boulevard Arago > 75014 Paris, France > > phone: +33.6.43.18.66.83 <tel:+33%206%2043%2018%2066%2083> > > -- Corentin Cadiou PhD student Institut d'Astrophysique de Paris (IAP), desk 142b 98 bis boulevard Arago 75014 Paris, France phone: +33.6.43.18.66.83

1 0

PARTICLE_AGE NOT DEFINED FOR RAMSES DATA
by Samhitha Vadlamani March 10, 2018

March 10, 2018

Hi, I've recently started working with yt and am trying to extract "particle_age" for newly formed stars in my simulation: > def formed_star(pfilter, data): > > filter = data["all", "particle_age"] < 0 > > return filter > > >> add_particle_filter("formed_star", function=formed_star, >> filtered_type='all', > > requires=["particle_age"]) > > However, I encounter an error: > the fields > > ('all', 'particle_age'), > > required by the "formed_star" particle filter, are not defined for this > dataset. > Could anyone please recommend a possible solution for why "particle_age" is not being detected for my dataset? Thanks.

3 6