Mailman 3 February 2014 - yt-dev

Issue #775: Cannot generate isosurfaces of fields that depend on field parameters (yt_analysis/yt)
by Nathan Goldbaum 06 Feb '14

06 Feb '14

New issue 775: Cannot generate isosurfaces of fields that depend on field parameters https://bitbucket.org/yt_analysis/yt/issue/775/cannot-generate-isosurfaces-… Nathan Goldbaum: Currently in 3.0 isosurfaces that depend on field parameters cannot be generated. As an example: ``` #!python from yt.mods import * pf = load('HiResIsolatedgalaxy/DD0044/DD0044') dd = pf.h.disk('c', [0,0,1], (20, 'kpc'), (3, 'kpc')) radii = (np.arange(100)+1)/100.*20 fluxes = [] for r in radii: iso = pf.h.surface(dd, 'radius', r) print iso['density'] flux = iso.calculate_flux('x-velocity', 'y-velocity', 'z-velocity', 'Density') fluxes.append(flux/1.9891e33*3.15569e13) # solar masses per megayear print r, fluxes[-1] ``` On both unitrefactor and the current 3.0 tip, this fails with the following traceback: ``` #!python Traceback (most recent call last): File "radial_mass_flux.py", line 21, in <module> print iso['density'] File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/data_containers.py", line 205, in __getitem__ self.get_data(f) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/construction_data_containers.py", line 826, in get_data mask, fields, sample_type) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/construction_data_containers.py", line 846, in _extract_isocontours_from_grid vals = grid.get_vertex_centered_data(field, no_ghost = False) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/grid_patch.py", line 293, in get_vertex_centered_data np.add(new_field, cg[field][1: ,1: ,1: ], new_field) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/data_containers.py", line 205, in __getitem__ self.get_data(f) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/construction_data_containers.py", line 491, in get_data if len(gen) > 0: self._generate_fields(gen) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/data_containers.py", line 626, in _generate_fields fd = self._generate_field(field) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/data_containers.py", line 242, in _generate_field tr = self._generate_fluid_field(field) File "/Users/goldbaum/Documents/yt-hg/yt/data_objects/data_containers.py", line 262, in _generate_fluid_field rv = finfo(gen_obj) File "/Users/goldbaum/Documents/yt-hg/yt/fields/derived_field.py", line 181, in __call__ dd = self._function(self, data) File "/Users/goldbaum/Documents/yt-hg/yt/fields/geometric_fields.py", line 49, in _radius return get_radius(data, "") File "/Users/goldbaum/Documents/yt-hg/yt/fields/field_functions.py", line 19, in get_radius center = data.get_field_parameter("center").in_units("cm") AttributeError: 'NoneType' object has no attribute 'in_units' ``` I ran this script in unitrefactor, but the same issue is present in the current 3.0 tip. In this context, 'data' is a covering grid that gets generated by `grid_patch` objects, which have no field parameters. A quick fix might be to somehow pass the field parameters associated with the disk object to the grid patches. When I chatted with Matt a little while ago about this issue, he mentioned that we would need to use the chunking system to generate the covering grid, rather than the old grid-based method. I'm unsure how to do that, so I'm reporting the issue here in hopes of getting some advice or guidance on how to fix this issue.

1 0

Merging the open unitrefactor PR
by Nathan Goldbaum 06 Feb '14

06 Feb '14

Hi all, Over the past month or so development has been proceeding apace on the unit refactor in Matt's fork of yt. We did this because Matt initially opened the unitrefactor PR and at that time it was not ready for merging. However, thanks to everyone's hard work, unitrefactor is getting more and more stable and I think it's time to think about merging it into the main repo, even if there are open issues or some remaining bits of functionality that haven't been updated yet. With tons of development going on in Matt's fork, I think we're possibly leaving out people who aren't watching his repo. Additionally, since Matt is the only one who can merge pull requests into his repo, we need to use a lot more of his attention to keep work moving forward. It's true that there is some functionality that still needs to be ported and bugs that need to be fixed. Matt's trello board summarizes most of the issues (BTW, I see a couple missing issues, would you be open to adding more users to it?): https://trello.com/b/yv7o0dTp/unit-refactor One option would be to open a new named branch in the main repo, while still keeping the current 3.0 tip in a 'stable' state. I'm less inclined to go this route because I think unitrefactor is such a big improvement over the current codebase, since many new features have snuck in besides just adding units. Another concern is that there aren't any docs yet. That's definitely true, but there aren't any docs for the current 3.0 tip either. In fact, now that there are a set of docs bundled in the repo in the unitrefactor bookmark, merging should improve the documentation effort for 3.0 going forward by making it more straightforward to enforce a rule that things need to be documented before they get merged. What do you all think? -Nathan

6 13

RFC: Remove DD0010, DD0000 and tests/
by Matthew Turk 06 Feb '14

06 Feb '14

Hi all, My understanding is that we don't use the tests/ directory anymore, ever. It serves the purpose of having an example of writing your own answer test runner, but we don't use it for that, either. Additionally, we have the DD0010 and DD0000 datasets in there. These aren't *big*, but they are kind of pointless. Proposals: 1) Remove all *.py files from this directory, as we do not use them and our tests are elsewhere 2) Remove DD0010 since we have lots of examples on yt-project.org/data/ . -Matt

6 5

Issue #774: We shouldn't need to extend the transfer function (yt_analysis/yt)
by Matthew Turk 06 Feb '14

06 Feb '14

New issue 774: We shouldn't need to extend the transfer function https://bitbucket.org/yt_analysis/yt/issue/774/we-shouldnt-need-to-extend-t… Matthew Turk: I don't remember why we do this anymore -- NaNs? running of the edges? segfaults? -- but we typically extend the TF, and we shouldn't need to do this. Responsible: samskillman

1 0

Docs have been imported
by Matthew Turk 06 Feb '14

06 Feb '14

Hi all, As discussed, I imported the docs into the 3.0 branch in my MatthewTurk/yt repository. I did this as an anonymous user, not myself. Bookmark is 30docs. The will be updated during subsequent PRs and development. -Matt

2 3

Unit refactor, Trello, SPH, etc
by Matthew Turk 05 Feb '14

05 Feb '14

Hi all, Lately, a lot of the yt-3.0 dev has been going on in the unitrefactor domain. Britton, John, Nathan and I (in alphabetical order :) have been pushing pretty hard on getting things in order. I've been trying to keep things organized a bit by using Trello, which Sam showed me and was pretty cool -- I'm not sure I can give out access to writing to the boards to anybody, but I think the yt boards are all "public" readable, and you can request write privs: https://trello.com/b/yv7o0dTp/unit-refactor Anyway, as an update, here's where we stand on unit refactor. Big blockers: * MOAB, GDF, Pluto, Chombo frontends * A few remaining (minor) cosmology issues that Britton's just about got finished and for which I owe some unit tests * Some discussion about metallicity units: https://bitbucket.org/MatthewTurk/yt/pull-request/38/fixing-cgs-unit-conver… * Docs haven't been copied in yet, but John Z has updated the bootcamp to 3.0 (units) style. I think that's ... just about it. At some point, maybe not *immediately*, we also want to get non-CGS units much, much easier to set as the default. So, the other thing -- we *just* landed (in unit refactor) SPH support with smoothing fields. More on this is forthcoming, but I'm pretty pleased with where it is now, and best of all there's room for improvement and optimization. I'm going to be traveling a good amount the next couple weeks, but I hope to be able to address the remaining blockers, and then we can land the unit refactor in mainline yt-3.0 at some point after the documentation is included. We'll let that sit for a while, and then at some point the "rebranding" PR (which will be more invasive technically, but *less* invasive API-wise) can be issued, tested, and landed. I actually think it might be reasonable to get 3.0 out ... kind of soon-ish. Anyway, if you want to test it out, grab the unitrefactor bookmark from MatthewTurk/yt ; I'll update the PR into the main yt repository as soon as these next couple PRs pass into my repo. -Matt

4 6

Reducing memory usage in time series
by John Wise 04 Feb '14

04 Feb '14

Hi all, I've been trying to run rockstar in a sizable Enzo simulation (150k grids) with ~100 outputs, where it's running out of memory just loading the hierarchies. One hierarchy instance consumes almost 1GB! I've found this to be problem not specific to rockstar but time series objects. My solution is to explicitly delete the hierarchy's metadata and grids. Since I haven't contributed to yt-3.0 yet, I wanted to run this by everyone before submitting a PR. My question is about coding style, in that I see very few __del__() functions now. In my working version, I've defined a __del__ function for the grid_geometry_handler as def __del__(self): del self.grid_dimensions del self.grid_left_edge del self.grid_right_edge del self.grid_levels del self.grid_particle_count del self.grids When I delete pf._instantiated_hierarchy after each loop of a time series iterator, I don't see any excessive memory usage anymore. It just reuses the allocated memory from the previous iteration, which is totally fine by me. However, when I include this in a __del__ function for a static_output, I still see excessive memory usage, which is bizarre to me. Should I define a new routine in the grid_geometry_handler, something like clear_hierarchy(), or keep the __del__ function? I ask because I want to keep in line with the overall structure of yt-3.0. This could also be included in the clear_all_data() call. What do people think the best approach would be? Thanks, John -- John Wise Assistant Professor of Physics Center for Relativistic Astrophysics, Georgia Tech http://cosmo.gatech.edu

2 2

Issue #773: Unitrefactor: cannot process units for magnetic fields (yt_analysis/yt)
by John ZuHone 04 Feb '14

04 Feb '14

New issue 773: Unitrefactor: cannot process units for magnetic fields https://bitbucket.org/yt_analysis/yt/issue/773/unitrefactor-cannot-process-… John ZuHone: This script: ``` #!python pf = load("GasSloshing/sloshing_nomag2_hdf5_plt_cnt_0100") v, c = pf.h.find_max("density") sl = pf.h.slice(0, c[0]) print sl["magnetic_field_z"].shape ``` results in this: ``` --------------------------------------------------------------------------- UnitParseError Traceback (most recent call last) <ipython-input-3-9aa7caa31868> in <module>() 5 print sl["z"] 6 print sl["pdx"] ----> 7 print sl["magnetic_field_z"].shape /Users/jzuhone/Source/yt-units-notbroke/yt/data_objects/data_containers.pyc in __getitem__(self, key) 203 return self.field_data[f] 204 else: --> 205 self.get_data(f) 206 # fi.units is the unit expression string. We depend on the registry 207 # hanging off the dataset to define this unit object. /Users/jzuhone/Source/yt-units-notbroke/yt/data_objects/data_containers.pyc in get_data(self, fields) 607 608 fields_to_generate += gen_fluids + gen_particles --> 609 self._generate_fields(fields_to_generate) 610 611 def _generate_fields(self, fields_to_generate): /Users/jzuhone/Source/yt-units-notbroke/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 630 raise RuntimeError 631 self.field_data[field] = fd --> 632 fd.convert_to_units(fi.units) 633 except GenerationInProgress as gip: 634 for f in gip.fields: /Users/jzuhone/Source/yt-units-notbroke/yt/units/yt_array.pyc in convert_to_units(self, units) 330 331 """ --> 332 new_units = self._unit_repr_check_same(units) 333 conversion_factor = self.units.get_conversion_factor(new_units) 334 /Users/jzuhone/Source/yt-units-notbroke/yt/units/yt_array.pyc in _unit_repr_check_same(self, units) 312 # let Unit() handle units arg if it's not already a Unit obj. 313 if not isinstance(units, Unit): --> 314 units = Unit(units, registry = self.units.registry) 315 316 if not self.units.same_dimensions_as(units): /Users/jzuhone/Source/yt-units-notbroke/yt/units/unit_object.pyc in __new__(cls, unit_expr, cgs_value, dimensions, registry, **assumptions) 133 else: 134 # lookup the unit symbols --> 135 cgs_value, dimensions = _get_unit_data_from_expr(unit_expr, registry.lut) 136 137 # Sympy trick to get dimensions powers as Rationals /Users/jzuhone/Source/yt-units-notbroke/yt/units/unit_object.pyc in _get_unit_data_from_expr(unit_expr, unit_symbol_lut) 407 dimensions = 1 408 for expr in unit_expr.args: --> 409 unit_data = _get_unit_data_from_expr(expr, unit_symbol_lut) 410 cgs_value *= unit_data[0] 411 dimensions *= unit_data[1] /Users/jzuhone/Source/yt-units-notbroke/yt/units/unit_object.pyc in _get_unit_data_from_expr(unit_expr, unit_symbol_lut) 397 398 if isinstance(unit_expr, Pow): --> 399 unit_data = _get_unit_data_from_expr(unit_expr.args[0], unit_symbol_lut) 400 power = unit_expr.args[1] 401 conv = float(unit_data[0]**power) /Users/jzuhone/Source/yt-units-notbroke/yt/units/unit_object.pyc in _get_unit_data_from_expr(unit_expr, unit_symbol_lut) 415 raise UnitParseError("Cannot parse for unit data from '%s'. Please supply" \ 416 " an expression of only Unit, Symbol, Pow, and Mul" \ --> 417 "objects." % str(unit_expr)) 418 419 UnitParseError: Cannot parse for unit data from 'sqrt(cm)'. Please supply an expression of only Unit, Symbol, Pow, and Mulobjects. : UnitParseError: Cannot parse for unit data from 'sqrt(cm)'. Please supply an expression of only Unit, Symbol, Pow, and Mulobjects. [yt.units.unit_object] ``` Responsible: MatthewTurk

1 0

Issue #772: Unitrefactor: Off-axis slice plots don't look right when the width is explicitly set (yt_analysis/yt)
by John ZuHone 04 Feb '14

04 Feb '14

New issue 772: Unitrefactor: Off-axis slice plots don't look right when the width is explicitly set https://bitbucket.org/yt_analysis/yt/issue/772/unitrefactor-off-axis-slice-… John ZuHone: This script: ``` #!python pf = load("GasSloshing/sloshing_nomag2_hdf5_plt_cnt_0100") slc = SlicePlot(pf, [0.1, -0.1, 0.2], ["temperature"]) slc.show() ``` produces this plot: ![good image](http://i.imgur.com/BacoRSg.png) but if we try to set the width: ``` #!python slc.set_width(500.,"kpc") ``` produces this one: ![bad_image](http://i.imgur.com/cV3mBc4.png) Responsible: ngoldbaum

1 0

ProfileND: soliciting opinions
by John ZuHone 04 Feb '14

04 Feb '14

Hi all, So Matt and I are using the pressure of the yt / Athena workshop coming up in a few days to get some things ironed out with unitrefactor, and I've come up on an issue that I'd like to get feedback on with regard to the setup of ProfileND objects. Let's take Profile1D as an example: prof = Profile1D(sp, "density", 32, rho_min, rho_max, True, weight_field="density") prof.add_fields(["temperature","velocity_magnitude"]) You can access the dependent fields dict-like, as you should: prof["temperature"] which gives you the average value of that field in the bin. However, we have no "easy" way to get the values of the independent field (in this case, "density") at the bin centers. There are a couple of attributes for the values of this field, but oddly they give the same thing: prof.x_bins == values of the field at bin edges prof.x == values of the field at bin edges Is there a reason for this? My proposal is to make prof.x (and the corresponding prof.y and prof.z for Profile2D and Profile3D) the values at the bin centers. Best, John Z

6 11