Mailman 3 September 2016 - yt-dev

Issue #1280: ds.box() overrides units of YTArray inputs (yt_analysis/yt)
by scott_feister Sept. 20, 2016

Sept. 20, 2016

New issue 1280: ds.box() overrides units of YTArray inputs https://bitbucket.org/yt_analysis/yt/issues/1280/dsbox-overrides-units-of-y… scott_feister: BUG SUMMARY: ds.box(left_edge, right_edge) ignores the YT Units of left_edge, right_edge EXAMPLE OF BUG: ``` #!python import yt from yt import YTArray ds = yt.load("Enzo_64/DD0043/data0043") print("Domain info:") print(ds.domain_width.in_units('Mpc')) print(ds.domain_center.in_units('Mpc')) print ('Defining two data objects that should be … [View More]

1 0

Registration open for the Learn yt Workshop: Oct 10-12, 2016
by Nathan Goldbaum Sept. 20, 2016

Sept. 20, 2016

Hi all, Registration is now open for the Learn yt Workshop at the University of Illinois Urbana-Champaign, October 10-12, 2016. This workshop will introduce attendees to the yt project (yt-project.org), a python toolkit for the analysis and visualization of volumetric data. We have create a website for the workshop with more information: http://yt-project.org/workshop2016/ In addition, if you wold like to register for the workshop, please fill out this online form: https://goo.gl/forms/… [View More]

2 1

Issue #1279: Non-intuitive, undocumented methods for 3D volume rendering (yt_analysis/yt)
by scott_feister Sept. 16, 2016

Sept. 16, 2016

New issue 1279: Non-intuitive, undocumented methods for 3D volume rendering https://bitbucket.org/yt_analysis/yt/issues/1279/non-intuitive-undocumented… scott_feister: Scene's functionality on 3D volume rendering and camera rendering in 'perspective' lens mode is not well-documented for users. ## Two bugs: ## 1. In perspective lens mode, the typical use is moving the camera position by cam.set_camera_position([x,y,z]) and pointing the camera by cam.set_camera_focus([x,y,z]). Calling these … [View More]functions automatically adjusts the lens viewpoint appropriately. However, functions like cam.roll(), cam.rotate() etc. currently change the viewpoint of the lens independently of camera position. How confusing! Also, cam.zoom() doesn't "zoom" the image as you'd expect. 2. When following the Volume Rendering example, it seems like modifying the colormap is performed by "source.tfh.set_bounds((3e-35, 5e-27))" (where source is a reference to the scene's main source) should update the scene rendering, but it isn't. On all but the first render, many more steps are involved (see below!) ## Action items: ## 1. (Easy) The volume rendering examples posted prominently on the yt website should include examples of how to update a scene, including changing colormap, camera angle, etc. for a perspective lens. Nathan has already written up an example such as this in response to my question to the yt-users list. (http://paste.yt-project.org/show/6823/) 2. (Harder) The scene.render() and scene.save() should know not to render the scene without updating the transfer function. Why should it take four steps? ``` #!python # This method works only for the first render, and is given in the Volume Renderer example source.tfh.set_bounds((3e-35, 5e-27)) ... sc.render() ``` ``` #!python # This method is what actually works after the first render source.tfh.set_bounds((3e-35, 5e-27)) source.tfh.build_transfer_function() source.tfh.setup_default() source.transfer_function = source.tfh.tf ... sc.render() ``` Intuitively, it should take one or two steps: ``` #!python # Something like this is what I'm suggesting source.tfh.set_bounds((3e-35, 5e-27)) source.update() # for example, does all three steps above (and possibly more) ... sc.render() ``` 3. (Harder) cam.roll(), cam.rotate(), etc. should change the camera position when the camera is in perspective mode, I believe, rather than the viewpoint. cam.zoom() should change the position, as well. 4. (Harder) The Camera and Lens documentation can be expanded to reflect the complexity of the problem, and clearly address which settings are relevant under which lenses. (e.g. the fact that set_width() doesn't do what you might think in perspective mode). The main user documentation page especially on Camera and Lenses (http://yt-project.org/doc/visualizing/volume_rendering.html#camera-lenses) is where I'd expect to find a clear explanation of the various types of lenses with good usage examples on how to move around the camera, aim it places, zoom in, etc. in each case. The rest of this bug report contains the contextual description of problem from emails on yt-users that led to this bug report: While working through examples, I hit a roadblock, and I've had some difficulty finding good documentation on the what the "Scene.save()" function actually does; specifically, why it does different things on the first and second call. The first example in the user tutorial page for 3D volume rendering (http://yt-project.org/doc/visualizing/volume_rendering.html) looks something like this: ``` #!python ... sc = yt.create_scene(ds, lens_type='perspective') ... source = sc[0] source.tfh.set_bounds((3e-31, 5e-27)) ... sc.save('rendering.png', sigma_clip=6.0) ``` And, voila, volume rendering saved to png. However, if I naively continue the script to re-render with new settings: ``` #!python source.tfh.set_bounds((3e-35, 5e-27)) sc.camera.zoom(2.0) sc.save('rendering2.png', sigma_clip=6.0) ``` I find that none of my new settings are reflected in "rendering2.png" -- it's just a duplicate of "rendering.png"! But if I start again from scratch with a new scene, the settings take hold. This leaves me (a new user) scratching my head. So I posed the question to yt-users: Once you've created and saved a scene once, how do you change scene settings like colormap and camera angle? Nathan Golbaum kindly responded with the following: Hi Scott, So this is really two issues. The first, is that the camera.zoom() function doesn't really zoom in on the image, instead it decreases the "width" the volume rendering region by the zoom factor. This makes a lot of sense for plane-parallel volume renderings (the default), but not so much for perspective lenses as you saw. For a perspective lens you should instead reposition the camera to be closer to the focus to get the same effect. Next, you also saw that manipulating the TransferFunctionHelper object (source.tfh) after you've already done a volume rendering doesn't update the transfer function. That's because the TransferFunctionHelper object is only used to generate the transfer function (source.transfer_function) if it isn't set yet. If it's already set, it reuses it. So in your example the first time you called save(), the VolumeSource saw that no one had manually created a transfer function, and used the TransferFunctionHelper object to build one. Then, when you asked it for the second rendering, it just reused the same one because right now the VolumeSource doesn't track if the TransferFunctionHelper has been updated. To do what you mean, you need to manually set the transfer function to be the one generated by the TransferFunctionHelper: source.tfh.build_transfer_function() source.tfh.setup_default() source.transfer_function = source.tfh.tf After manipulating source.tfh. Using as an example the script from the docs, here's how to make the second image come out as you would expect for the perspective lens: http://paste.yt-project.org/show/6823/ Which makes these two images: http://i.imgur.com/gwUTWz1.png (rendering.png) http://i.imgur.com/M1lSz0N.png (rendering2.png) I think we could probably do a better job of detecting that the TransferFunctionHelper has been manipulated and avoid this confusion, if you'd like I invite you to open an issue about this on our issue tracker. One might also argue that the zoom function should adjust the camera position for perspective lenses. Sorry the volume renderer isn't totally intuitive for this use case. I did some work before the yt 3.3.1 release to improve things, but it's still definitely not perfect. I think there's a lot of power there but it also really needs some love from someone who is willing to think about corner cases and interactive workflows. -Nathan [View Less]

1 0

Issue #1278: Unable to filter a dataset using a derived field (yt_analysis/yt)
by Danielle Sponseller Sept. 16, 2016

Sept. 16, 2016

New issue 1278: Unable to filter a dataset using a derived field https://bitbucket.org/yt_analysis/yt/issues/1278/unable-to-filter-a-dataset… Danielle Sponseller: When defining a particle filter, a derived field cannot be used to determine which particles to filter out. The example below shows a field defined to be equivalent to the "particle_position_x" field. When using "particle_position_x' in the filter definition, the filter works as expected, however when the derived field is used an … [View More]

1 0

A particle approach to model Ray and AbsorptionSpectrum
by Bili Dong - Gmail Sept. 15, 2016

Sept. 15, 2016

Hi all, I came up with an idea to bring a particle approach to model Ray and AbsorptionSpectrum. But before I move on to implement it in yt, I'd like to let you know what it is and get feedbacks about it. The situation is that I have an SPH simulation, and I want to model the Ray (in order to get the AbsorptionSpectrum) as accurately as possible. Currently when we create a Ray object, it's always created from the deposited grid. Although it is a good approximation to the true particle … [View More]representation, it is still not the most accurate way. I'd like to be able to do it in the particle way (like in SPLASH). In the long term, I know that Matt and Meagan is working on a new system for particle dataset. The work I'm going to propose could be thought as lying on top of that, in that the method could be made faster utilizing Matt and Meagon's work, but the main infrastructure would stay the same. To introduce what I plan to do, let's have a look at the first figure here <http://yt-project.org/docs/dev/analyzing/analysis_modules/light_ray_generat…>. The core concept of a Ray object is the *path length*, *`dl`*. Basically, if we combine the normal fields with the `dl` field, we get a Ray object. Now imagine instead of a ray intersecting a lot of grid cells, we have a ray intersecting a lot of SPH particles. How do we define *`dl`* then? We could define it as the *integral of the SPH kernel along the intersection*! And that's the whole trick. From this we could define a particle Ray that just looks the 'same' as the original grid Ray. Then any analysis built on top of the Ray object, AbsorptionSpectrum for example, don't need to change a lot. They will work in different ways simply when provided with different different kinds of Ray object. The main difficulty in the implementation is the construction of the particle `dl` field. Currently I'm doing it brutal-forcedly by computing `dl` for all the particles and mask out those with zero values. Matt and Meagan's work will accelerate this by providing the neighboring information, so I could do the computation on a small set of particles then. The brutal-force method is not unbearably slow though. And the computation acceleration could be saved for future work. I have an external implementation of the particle approach, and have used it in my current research. I have compared results using the particle method and those from Trident and they agree statistically as we expected (thanks Cameron for the help). Now that it looks mature, I'd like to implement it in yt. If anyone has any comments, opinions and suggestions, I'd like to hear them. Thanks for reading, Bili [View Less]

4 3

Issue #1277: Test suite failures under python3.6b1 (yt_analysis/yt)
by Nathan Goldbaum Sept. 15, 2016

Sept. 15, 2016

New issue 1277: Test suite failures under python3.6b1 https://bitbucket.org/yt_analysis/yt/issues/1277/test-suite-failures-under-… Nathan Goldbaum: Here's the output from running the unit tests on my mac: ``` ====================================================================== ERROR: yt.analysis_modules.level_sets.tests.test_clump_finding.test_clump_tree_save ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/… [View More]python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/Users/goldbaum/Documents/yt-hg-python3/yt/analysis_modules/level_sets/tests/test_clump_finding.py", line 104, in test_clump_tree_save find_clumps(master_clump, c_min, c_max, step) File "/Users/goldbaum/Documents/yt-hg-python3/yt/analysis_modules/level_sets/clump_handling.py", line 433, in find_clumps clump.find_children(min_val) File "/Users/goldbaum/Documents/yt-hg-python3/yt/analysis_modules/level_sets/clump_handling.py", line 162, in find_children nj, cids = identify_contours(self.data, self.field, min_val, max_val) File "/Users/goldbaum/Documents/yt-hg-python3/yt/analysis_modules/level_sets/contour_finder.py", line 36, in identify_contours for (g, node, (sl, dims, gi)) in data_source.tiles.slice_traverse(): File "/Users/goldbaum/Documents/yt-hg-python3/yt/data_objects/data_containers.py", line 1126, in tiles self._tree = AMRKDTree(self.ds, data_source=self) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 181, in __init__ data_source=data_source) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 83, in __init__ self.build() File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 99, in build self.add_grids(grids) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 90, in add_grids self.comm_rank, self.comm_size) File "yt/utilities/lib/amr_kdtools.pyx", line 207, in yt.utilities.lib.amr_kdtools.Node.add_grids (yt/utilities/lib/amr_kdtools.c:5903) File "yt/utilities/lib/amr_kdtools.pyx", line 244, in yt.utilities.lib.amr_kdtools.Node.add_grids (yt/utilities/lib/amr_kdtools.c:5443) ValueError: Unexpected format string character: 'q' ====================================================================== ERROR: Failure: RuntimeError (This appears to be a FITS file, but AstroPy is not installed.) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/nose/failure.py", line 39, in runTest raise self.exc_val.with_traceback(self.tb) File "/usr/local/lib/python3.6/site-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/local/lib/python3.6/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/lib/python3.6/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python3/3.6.0b1_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 234, in load_module return load_source(name, filename, file) File "/usr/local/Cellar/python3/3.6.0b1_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 172, in load_source module = _load(spec) File "<frozen importlib._bootstrap>", line 675, in _load File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 676, in exec_module File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed File "/Users/goldbaum/Documents/yt-hg-python3/yt/frontends/fits/tests/test_outputs.py", line 29, in <module> @requires_ds(grs) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/answer_testing/framework.py", line 939, in requires_ds elif not can_run_ds(ds_fn, file_check): File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/answer_testing/framework.py", line 271, in can_run_ds load(ds_fn) File "/Users/goldbaum/Documents/yt-hg-python3/yt/convenience.py", line 79, in load if c._is_valid(*args, **kwargs): candidates.append(n) File "/Users/goldbaum/Documents/yt-hg-python3/yt/frontends/fits/data_structures.py", line 645, in _is_valid raise RuntimeError("This appears to be a FITS file, but AstroPy is not installed.") RuntimeError: This appears to be a FITS file, but AstroPy is not installed. ====================================================================== ERROR: Failure: ValueError (Unexpected format string character: 'q') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/nose/failure.py", line 39, in runTest raise self.exc_val.with_traceback(self.tb) File "/usr/local/lib/python3.6/site-packages/nose/loader.py", line 252, in generate for test in g(): File "/Users/goldbaum/Documents/yt-hg-python3/yt/geometry/tests/test_grid_container.py", line 123, in test_grid_arrays_view grid_arr = tree.grid_arrays File "yt/geometry/grid_container.pyx", line 130, in yt.geometry.grid_container.GridTree.grid_arrays.__get__ (yt/geometry/grid_container.c:4659) ValueError: Unexpected format string character: 'q' ====================================================================== ERROR: yt.utilities.lib.tests.test_bounding_volume_hierarchy.test_bounding_volume_hierarchy ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/lib/tests/test_bounding_volume_hierarchy.py", line 30, in test_bounding_volume_hierarchy ds = yt.load(fn) File "/Users/goldbaum/Documents/yt-hg-python3/yt/convenience.py", line 98, in load raise YTOutputNotIdentified(args, kwargs) yt.utilities.exceptions.YTOutputNotIdentified: Supplied ('/Users/goldbaum/Documents/test/MOOSE_sample_data/out.e-s010',) {}, but could not load! ====================================================================== ERROR: yt.utilities.tests.test_amr_kdtree.test_amr_kdtree_set_fields ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/tests/test_amr_kdtree.py", line 69, in test_amr_kdtree_set_fields dd.tiles.set_fields(fields, [True, True], False) File "/Users/goldbaum/Documents/yt-hg-python3/yt/data_objects/data_containers.py", line 1126, in tiles self._tree = AMRKDTree(self.ds, data_source=self) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 181, in __init__ data_source=data_source) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 83, in __init__ self.build() File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 99, in build self.add_grids(grids) File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/amr_kdtree/amr_kdtree.py", line 90, in add_grids self.comm_rank, self.comm_size) File "yt/utilities/lib/amr_kdtools.pyx", line 207, in yt.utilities.lib.amr_kdtools.Node.add_grids (yt/utilities/lib/amr_kdtools.c:5903) File "yt/utilities/lib/amr_kdtools.pyx", line 259, in yt.utilities.lib.amr_kdtools.Node.add_grids (yt/utilities/lib/amr_kdtools.c:5651) ValueError: Unexpected format string character: 'q' ====================================================================== ERROR: Failure: ModuleNotFoundError (No module named 'mock') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/nose/failure.py", line 39, in runTest raise self.exc_val.with_traceback(self.tb) File "/usr/local/lib/python3.6/site-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/local/lib/python3.6/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/lib/python3.6/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python3/3.6.0b1_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 234, in load_module return load_source(name, filename, file) File "/usr/local/Cellar/python3/3.6.0b1_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 172, in load_source module = _load(spec) File "<frozen importlib._bootstrap>", line 675, in _load File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 676, in exec_module File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed File "/Users/goldbaum/Documents/yt-hg-python3/yt/utilities/tests/test_config.py", line 11, in <module> import mock ModuleNotFoundError: No module named 'mock' ====================================================================== FAIL: yt.analysis_modules.level_sets.tests.test_clump_finding.test_clump_finding ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/Users/goldbaum/Documents/yt-hg-python3/yt/analysis_modules/level_sets/tests/test_clump_finding.py", line 69, in test_clump_finding assert_equal(len(master_clump.children), 2) File "/usr/local/lib/python3.6/site-packages/numpy/testing/utils.py", line 379, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 0 DESIRED: 2 >> raise AssertionError('\nItems are not equal:\n ACTUAL: 0\n DESIRED: 2') ---------------------------------------------------------------------- Ran 8511 tests in 12666.888s FAILED (errors=6, failures=1) ``` Most of these have to do with the 'q' format code (corresponding to the struct format 'q', or long long, see https://docs.python.org/2/library/struct.html#format-characters. The other one is due to an issue with the mock module that is fixed in PR 2377. [View Less]

1 0

Spack package for yt
by Erik Schnetter Sept. 14, 2016

Sept. 14, 2016

I have created a Spack <https://github.com/LLNL/spack> package for yt: < https://github.com/LLNL/spack/pull/1751>. Spack is a user-mode package manager for HPC systems. Once this package is accepted into Spack, people can install yt by typing "spack install yt". This will then install all necessary prerequisites (Python, HDF5, maybe MPI) as well. -erik -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

4 3

Backporting bugfixes for yt 3.3.2
by Nathan Goldbaum Sept. 13, 2016

Sept. 13, 2016

Hi all, Today I'm going to spend some time backporting bugfixes to the stable branch so we can cut yt 3.3.2. I expect it will be merged in next week at the PR review hangout (which I will send out an invite for this weekend). Please let me know if there are any bugfixes you'd particularly like this release to include or if you'd like me to wait on something you have in the works. -Nathan

3 5

Issue #1276: Run subset of unittests on Travis or Drone.io (yt_analysis/yt)
by Kacper Kowalik Sept. 12, 2016

Sept. 12, 2016

New issue 1276: Run subset of unittests on Travis or Drone.io https://bitbucket.org/yt_analysis/yt/issues/1276/run-subset-of-unittests-on… Kacper Kowalik: * Identify the minimum set of tests that would fit the free tier * Setup webhooks Responsible: xarthisius

1 0

Coordinate transformations and periodic boundaries
by John ZuHone Sept. 9, 2016

Sept. 9, 2016

Hi all, I’m working on some updates to my yt-based pyXSIM package and I’m stuck and I thought I would email this list for some guidance. I’m thinking of a situation where we have a data object (sphere, box, whatever) and it straddles a periodic boundary. I want to convert the coordinates such that the coordinates are translated with respect to some arbitrary origin (say, the center of a sphere but in theory it could be anywhere), but are also continuous, i.e, they do not wrap at the boundary.… [View More]

4 3