Mailman 3 June 2014 - yt-dev

Issue #850: ParticleTrajectories fails with weakly-referenced object no longer exists (yt_analysis/yt)
by Kacper Kowalik June 19, 2014

June 19, 2014

New issue 850: ParticleTrajectories fails with weakly-referenced object no longer exists https://bitbucket.org/yt_analysis/yt/issue/850/particletrajectories-fails-w… Kacper Kowalik: Following script: ``` #!python from yt.analysis_modules.particle_trajectories.api import \ ParticleTrajectories my_fns = ["enzo_tiny_cosmology/DD%4.4i/DD%4.4i.hierarchy" % (i, i) for i in range(40)] trajs = ParticleTrajectories(my_fns, [596]) ``` fails with: ``` #!text Traceback (most recent … [View More]

1 0

Weird nose test failure
by John ZuHone June 18, 2014

June 18, 2014

Hi all, Have any of you seen this failure before? http://paste.yt-project.org/show/4801/ I can't seem to run any nosetests on the answer tests because of it. Best, John

2 3

Issue #849: Answer tests for fits frontend are not working (yt_analysis/yt)
by Kacper Kowalik June 18, 2014

June 18, 2014

New issue 849: Answer tests for fits frontend are not working https://bitbucket.org/yt_analysis/yt/issue/849/answer-tests-for-fits-fronte… Kacper Kowalik: Currently `test_outputs.py` for *fits* frontend yields: ``` #!python File "/tmp/yt/yt/frontends/fits/tests/test_outputs.py", line 39, in test_grs pf = data_dir_load(grs, nan_mask=0.0) TypeError: data_dir_load() got an unexpected keyword argument 'nan_mask' ``` I'm using `4e6992f76dfc+ (yt-3.0) tip` Responsible: jzuhone

1 0

Leave of presence
by Matthew Turk June 16, 2014

June 16, 2014

Hey all, Just wanted to drop a line to let you know that the next couple weeks, particularly the next 10 days or so, are going to be pretty busy for me, so I might have much higher latency replying to emails and pull requests. Additionally, I think we should aim for a 3.0 final by the end of June at the *latest* so we can have something of a shakeout period before SciPy. If you're working on 3.0, please try to keep the trello boards updated: https://trello.com/yt_analysis The specific … [View More]

2 2

Regression for particle deposition
by Matthew Turk June 14, 2014

June 14, 2014

Hi all, As a quick note, on my laptop I found a regression in particle deposition. The snapshot_033 ("deposit","all_density") is messed up. It was my fault, as it came from changeset 2dc75b72d271 and is a result of F/C ordering (and I can fix as soon as I am back at my desk in an hour or so). Should only affect octree particle deposition, went in very briefly, and I've CC'd everyone who might have been affected in the last two days. But, to the point, I'm not sure why the answer testing … [View More]

4 5

Re: [yt-dev] Zombie jobs on eudora?
by Nathan Goldbaum June 12, 2014

June 12, 2014

Hey all, I'm looking at a memory leak that Philip (cc'd) is seeing when iterating over a long list of FLASH datasets. Just as an example of the type of behavior he is seeing - today he left his script running and ended up consuming 300 GB of RAM on a viz node. FWIW, the dataset is not particularly large - ~300 outputs and ~100 MB per output. These are also FLASH cylindrical coordinate simulations - so perhaps this behavior will only occur in curvilinear geometries? I've been playing with … [View More]objgraph to try to understand what's happening. Here's the script I've been using: http://paste.yt-project.org/show/4762/ Here's the output after one iteration of the for loop: http://paste.yt-project.org/show/4761/ It seems that for some reason a lot of data is not being garbage collected. Could there be a reference counting bug somewhere down in a cython routine? Objgraph is unable to find backreferences to root grid tiles in the flash dataset, and all the other yt objects that I've looked at seem to have backreference graphs that terminate at a FLASHGrid object that represents a root grid tile in one of the datasets. That's the best guess I have - but definitely nothing conclusive. I'd appreciate any other ideas anyone else has to help debug this. Thanks for your help in debugging this! -Nathan On Mon, Jun 9, 2014 at 7:36 PM, Phillip Macias <pmacias(a)ucsc.edu> wrote: > Done. Also this sim is just a test haha so don't judge the crappy quality > :) > > > On Mon, Jun 9, 2014 at 7:28 PM, Nathan Goldbaum <nathan12343(a)gmail.com> > wrote: > >> Any chance you could make the folder that the dataset lives in visible to >> me? >> >> chmod -R +r /pfs/pmacias/runs/hoseon/2pi/diff_angle/coriolis/ >> >> should do the trick. >> >> >> On Mon, Jun 9, 2014 at 7:25 PM, Nathan Goldbaum <nathan12343(a)gmail.com> >> wrote: >> >>> This looks like a memory leak in yt. Let me see if can figure out >>> what's going wrong. >>> >>> >>> On Mon, Jun 9, 2014 at 7:21 PM, Phillip Macias <pmacias(a)ucsc.edu> wrote: >>> >>>> No problem, sorry to be a pain! >>>> >>>> I've attached the script. To run I'm just sourcing the yt installation >>>> on hyades and doing >>>> >>>> python dens_plot.py >>>> >>>> Maybe I need to be clearing some things as I go through the loop? Idk >>>> I'm sure it's something really silly :/ >>>> >>>> Thanks again! >>>> -Phil >>>> >>>> >>>> On Mon, Jun 9, 2014 at 6:56 PM, Nathan Goldbaum <nathan12343(a)gmail.com> >>>> wrote: >>>> >>>>> Thanks for killing them. I'd be happy to take a look at your script. >>>>> >>>>> >>>>> On Mon, Jun 9, 2014 at 6:46 PM, Phillip Macias <pmacias(a)ucsc.edu> >>>>> wrote: >>>>> >>>>>> Hey Nathan, >>>>>> >>>>>> Sorry about that!! I was actually running yt scripts, I'm not sure >>>>>> how they could use so much ram :(. When I get home can I send you my script >>>>>> and you can tell me how I'm being dumb? >>>>>> >>>>>> -Phil >>>>>> >>>>>> >>>>>> >>>>>> On Jun 9, 2014, at 6:43 PM, Nathan Goldbaum <nathan12343(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>> > Hey Philip, >>>>>> > >>>>>> > It looks like you have two scripts that have been running for 7 >>>>>> hours - using ~300 GB of ram. Are these zombie processes? >>>>>> > >>>>>> > -Nathan >>>>>> >>>>> >>>>> >>>> >>> >> > [View Less]

4 36

3.0 rundown
by Matthew Turk June 11, 2014

June 11, 2014

Hi all, Last I checked, the main blockers to 3.0 were: * Enabling answer testing (done) * Enabling doc building (done) * Redoing all relevant docs (in progress?) * Cookbook (in progress?) * Finishing up any outstanding frontend issues * Break the volume rendering interface, and rebuild it (sort of in progress) Does anybody know where we're at on these things? Looking over the board: https://trello.com/b/Y5XV4Hod/yt-3-0 we're dangerously close. I think the final one, VR, might have … [View More]

4 4

Re: [yt-dev] About Covering Problem while plotting 3D surface and sketchfab pictures
by 吳佳鴻 June 11, 2014

June 11, 2014

Hi Matt, I ran the script accordingly and the results came out: Not smoothed 143496 0 Smoothed 143496 0 and "143496" is exact the total number of my grids. Is there any clue to this problem? Many thanks, Jia -Hung -------------Previous Email----------------- Hi Jia-Hung, Sorry for the delay! The key error you're seeing is because yt was unable to fill in all of the cells needed for the smoothed covering grid. Can you try this, and see what happens? ngood = 0 nbad = 0 for g in pf.h.… [View More]grids: try: g.retrieve_ghost_zones(1, ["Ones"], smoothed=False) except KeyError: nbad += 1 cnotinue ngood += 1 print "Not smoothed", ngood, nbad ngood = 0 nbad = 0 for g in pf.h.grids: try: g.retrieve_ghost_zones(1, ["Ones"], smoothed=True) except KeyError: nbad += 1 cnotinue ngood += 1 print "Smoothed", ngood, nbad and let us know the results? On Fri, Jun 6, 2014 at 10:52 AM, ??? <r00222055(a)ntu.edu.tw> wrote: > Dear all, > > I want to say thank you to all of you for making how powerful and > convenient yt is. I really appreciate everything you have done. > I?m now using yt 2.6 version and trying to construct our AMR code > named ?GAMER? frontend to yt. > > So far I have several preliminary results that I could plot picture > with projection /slice/ contour /radial profile successfully, and the > outcome pictures all seem fine. > However when I tried to plot ?3D Surface and Sketchfab?, I failed. > > It popped out some error messages. There exist some covering problems > in my data structure. Actually this question also happened when I > tried to plot ?VorticityMagnitude?. > I couldn?t figure it out, and every data value and positions seem fine > when I plot pictures in projection and slice. > > Therefore, I believe there is some fundamental problem that I am too > careless to pay attention to. Could anyone help me with this problem? > Thank you in advance. > > > > Jia-Hung Wu > > > > > > The following is the error message. > > ======================== Error Message > =========================================== > > ? > > In [75]: surface["Density"] > > yt : [ERROR ] 2014-06-06 22:49:12,262 Covering problem: 163 cells are > uncovered > > ---------------------------------------------------------------------- > ----- > > KeyError Traceback (most recent call last) > > <ipython-input-75-c8eab0fb4ee7> in <module>() > > ----> 1 surface["Density"] > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in __getitem__(self, key) > > 316 if key not in self.fields: > > 317 self.fields.append(key) > > --> 318 self.get_data(key) > > 319 return self.field_data[key] > > 320 > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc in get_data(self, fields, sample_type) > > 4233 my_verts = self._extract_isocontours_from_grid( > > 4234 g, self.surface_field, self.field_value, > > -> 4235 fields, sample_type) > > 4236 if fields is not None: > > 4237 my_verts, svals = my_verts > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc in save_state(self, grid, field, *args, **kwargs) > > 79 old_keys = grid.field_data.keys() > > 80 grid.field_parameters = self.field_parameters > > ---> 81 tr = func(self, grid, field, *args, **kwargs) > > 82 grid.field_parameters = old_params > > 83 grid.field_data = YTFieldData( [(k, grid.field_data[k]) for > k in old_keys] ) > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc in _extract_isocontours_from_grid(self, grid, field, > value, sample_values, > sample_type) > > 4257 sample_type = "face"): > > 4258 mask = self.data_source._get_cut_mask(grid) * > grid.child_mask > > -> 4259 vals = grid.get_vertex_centered_data(field, no_ghost = > False) > > 4260 if sample_values is not None: > > 4261 svals = grid.get_vertex_centered_data(sample_values) > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/grid_pat > ch.pyc in get_vertex_centered_data(self, field, smoothed, no_ghost) > > 498 np.power(10.0, new_field, new_field) > > 499 else: > > --> 500 cg = self.retrieve_ghost_zones(1, field, > smoothed=smoothed) > > 501 np.add(new_field, cg[field][1: ,1: ,1: ], new_field) > > 502 np.add(new_field, cg[field][:-1,1: ,1: ], new_field) > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/grid_pat > ch.pyc in retrieve_ghost_zones(self, n_zones, fields, all_levels, > smoothed) > > 465 if smoothed: > > 466 cube = self.hierarchy.smoothed_covering_grid( > > --> 467 level, new_left_edge, **kwargs) > > 468 else: > > 469 cube = self.hierarchy.covering_grid(level, > new_left_edge, **kwargs) > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in __init__(self, *args, **kwargs) > > 3869 (self.pf.domain_right_edge - self.pf.domain_left_edge) > / > > 3870 self.pf.domain_dimensions.astype("float64")) > > -> 3871 AMRCoveringGridBase.__init__(self, *args, **kwargs) > > 3872 self._final_start_index = self.global_startindex > > 3873 > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc in __init__(self, level, left_edge, dims, fields, pf, > num_ghost_zones, use_pbar, **kwargs) > > 3713 self.domain_width = np.rint((self.pf.domain_right_edge - > > 3714 > self.pf.domain_left_edge)/self.dds).astype('int64') > > -> 3715 self._refresh_data() > > 3716 > > 3717 def _get_list_of_grids(self, buffer = 0.0): > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in _refresh_data(self) > > 3730 > > 3731 def _refresh_data(self): > > -> 3732 AMR3DData._refresh_data(self) > > 3733 self['dx'] = self.dds[0] * np.ones(self.ActiveDimensions, > dtype='float64') > > 3734 self['dy'] = self.dds[1] * np.ones(self.ActiveDimensions, > dtype='float64') > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in _refresh_data(self) > > 304 """ > > 305 self.clear_data() > > --> 306 self.get_data() > > 307 > > 308 def keys(self): > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in get_data(self, field) > > 3893 try: > > 3894 #print "Generating", field > > -> 3895 self._generate_field(field) > > 3896 continue > > 3897 except NeedsOriginalGrid, ngt_exception: > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in _generate_field(self, field) > > 3780 # First we check the validator; this might even raise! > > 3781 self.pf.field_info[field].check_available(self) > > -> 3782 self[field] = self.pf.field_info[field](self) > > 3783 else: # Can't find the field, try as it might > > 3784 raise KeyError(field) > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/field_in > fo_container.pyc > in __call__(self, data) > > 383 ii = self.check_available(data) > > 384 original_fields = data.keys() # Copy > > --> 385 dd = self._function(self, data) > > 386 dd *= self._convert_function(data) > > 387 for field_name in data.keys(): > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/field_in > fo_container.pyc > in _TranslationFunc(field, data) > > 139 def TranslationFunc(field_name): > > 140 def _TranslationFunc(field, data): > > --> 141 return data[field_name] > > 142 return _TranslationFunc > > 143 > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in __getitem__(self, key) > > 316 if key not in self.fields: > > 317 self.fields.append(key) > > --> 318 self.get_data(key) > > 319 return self.field_data[key] > > 320 > > > > /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_con > tainers.pyc > in get_data(self, field) > > 3935 n_bad = (self[field]==-999).sum() > > 3936 mylog.error("Covering problem: %s cells are > uncovered", n_bad) > > -> 3937 raise KeyError(n_bad) > > 3938 if self._use_pbar: pbar.finish() > > 3939 > > > > KeyError: 163 > > ====================================================================== > =================== > > > _______________________________________________ > yt-dev mailing list > yt-dev(a)lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > ------------------------------ Subject: Digest Footer _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org ------------------------------ End of yt-dev Digest, Vol 66, Issue 6 ************************************* [View Less]

2 1

About Covering Problem while plotting 3D surface and sketchfab pictures
by 吳佳鴻 June 9, 2014

June 9, 2014

Dear all, I want to say thank you to all of you for making how powerful and convenient yt is. I really appreciate everything you have done. I’m now using yt 2.6 version and trying to construct our AMR code named “GAMER” frontend to yt. So far I have several preliminary results that I could plot picture with projection /slice/ contour /radial profile successfully, and the outcome pictures all seem fine. However when I tried to plot “3D Surface and Sketchfab”, I failed. It popped out some error … [View More]messages. There exist some covering problems in my data structure. Actually this question also happened when I tried to plot “VorticityMagnitude”. I couldn’t figure it out, and every data value and positions seem fine when I plot pictures in projection and slice. Therefore, I believe there is some fundamental problem that I am too careless to pay attention to. Could anyone help me with this problem? Thank you in advance. Jia-Hung Wu The following is the error message. ======================== Error Message =========================================== “ In [75]: surface["Density"] yt : [ERROR ] 2014-06-06 22:49:12,262 Covering problem: 163 cells are uncovered --------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-75-c8eab0fb4ee7> in <module>() ----> 1 surface["Density"] /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 316 if key not in self.fields: 317 self.fields.append(key) --> 318 self.get_data(key) 319 return self.field_data[key] 320 /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields, sample_type) 4233 my_verts = self._extract_isocontours_from_grid( 4234 g, self.surface_field, self.field_value, -> 4235 fields, sample_type) 4236 if fields is not None: 4237 my_verts, svals = my_verts /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in save_state(self, grid, field, *args, **kwargs) 79 old_keys = grid.field_data.keys() 80 grid.field_parameters = self.field_parameters ---> 81 tr = func(self, grid, field, *args, **kwargs) 82 grid.field_parameters = old_params 83 grid.field_data = YTFieldData( [(k, grid.field_data[k]) for k in old_keys] ) /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _extract_isocontours_from_grid(self, grid, field, value, sample_values, sample_type) 4257 sample_type = "face"): 4258 mask = self.data_source._get_cut_mask(grid) * grid.child_mask -> 4259 vals = grid.get_vertex_centered_data(field, no_ghost = False) 4260 if sample_values is not None: 4261 svals = grid.get_vertex_centered_data(sample_values) /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.pyc in get_vertex_centered_data(self, field, smoothed, no_ghost) 498 np.power(10.0, new_field, new_field) 499 else: --> 500 cg = self.retrieve_ghost_zones(1, field, smoothed=smoothed) 501 np.add(new_field, cg[field][1: ,1: ,1: ], new_field) 502 np.add(new_field, cg[field][:-1,1: ,1: ], new_field) /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.pyc in retrieve_ghost_zones(self, n_zones, fields, all_levels, smoothed) 465 if smoothed: 466 cube = self.hierarchy.smoothed_covering_grid( --> 467 level, new_left_edge, **kwargs) 468 else: 469 cube = self.hierarchy.covering_grid(level, new_left_edge, **kwargs) /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __init__(self, *args, **kwargs) 3869 (self.pf.domain_right_edge - self.pf.domain_left_edge) / 3870 self.pf.domain_dimensions.astype("float64")) -> 3871 AMRCoveringGridBase.__init__(self, *args, **kwargs) 3872 self._final_start_index = self.global_startindex 3873 /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __init__(self, level, left_edge, dims, fields, pf, num_ghost_zones, use_pbar, **kwargs) 3713 self.domain_width = np.rint((self.pf.domain_right_edge - 3714 self.pf.domain_left_edge)/self.dds).astype('int64') -> 3715 self._refresh_data() 3716 3717 def _get_list_of_grids(self, buffer = 0.0): /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _refresh_data(self) 3730 3731 def _refresh_data(self): -> 3732 AMR3DData._refresh_data(self) 3733 self['dx'] = self.dds[0] * np.ones(self.ActiveDimensions, dtype='float64') 3734 self['dy'] = self.dds[1] * np.ones(self.ActiveDimensions, dtype='float64') /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _refresh_data(self) 304 """ 305 self.clear_data() --> 306 self.get_data() 307 308 def keys(self): /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, field) 3893 try: 3894 #print "Generating", field -> 3895 self._generate_field(field) 3896 continue 3897 except NeedsOriginalGrid, ngt_exception: /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 3780 # First we check the validator; this might even raise! 3781 self.pf.field_info[field].check_available(self) -> 3782 self[field] = self.pf.field_info[field](self) 3783 else: # Can't find the field, try as it might 3784 raise KeyError(field) /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.pyc in __call__(self, data) 383 ii = self.check_available(data) 384 original_fields = data.keys() # Copy --> 385 dd = self._function(self, data) 386 dd *= self._convert_function(data) 387 for field_name in data.keys(): /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.pyc in _TranslationFunc(field, data) 139 def TranslationFunc(field_name): 140 def _TranslationFunc(field, data): --> 141 return data[field_name] 142 return _TranslationFunc 143 /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 316 if key not in self.fields: 317 self.fields.append(key) --> 318 self.get_data(key) 319 return self.field_data[key] 320 /work1/ftd/yt_2.6_Version/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, field) 3935 n_bad = (self[field]==-999).sum() 3936 mylog.error("Covering problem: %s cells are uncovered", n_bad) -> 3937 raise KeyError(n_bad) 3938 if self._use_pbar: pbar.finish() 3939 KeyError: 163 ========================================================================================= [View Less]

2 1

Hangout next week
by Matthew Turk June 9, 2014

June 9, 2014

Hi all, (Hope everybody's enjoying June! :) I'd like to have a brief hangout next week on Tuesday, just to have a quick catchup and see where we're at on a few things. * yt 3.0 capability * yt 3.0 docs * Possible workshop this fall * SciPy I've set up a doodle poll here: http://doodle.com/w447is5n6mnycpm8 If there's a strong preference to another day, let's shift it around, but I probably can't do it any day from Wed-Fri, so we might want to bump to the following week. -Matt

1 1