Mailman 3 November 2016 - yt-dev

the future of analysis_modules
by Britton Smith 05 Dec '16

05 Dec '16

Greetings, I would like to open a discussion on the idea of moving most of yt’s analysis modules into an external yt extensions package. For ease of reading, I will separate this email into what this would mean for the code, what I see are the pros, cons, logistics, and open questions. I would very much appreciate comment on this. What this means If we did this, most of the contents of yt/analysis_modules would be moved into a repository named something like yt_astro_analysis. Installing this would be an option in the install script and would likely also be pip installable. Imports would change from from yt.analysis_modules.halo_analysis.api import HaloCatalog to from yt.extensions.astro_analysis.halo_analysis.api import HaloCatalog After creating yt_astro_analysis, there would be a period where the old analysis_modules would still exist but be deprecated before being removed at some point down the road. Pros - Almost all of the current analysis modules are specific to astrophysics. As we continue to make the core functionality of yt less astro specific, it’s not clear how to make room for non-astro analysis modules. Putting everything together under analysis_modules will make navigation and documentation messy and confusing. This would also significantly slim down the yt codebase. - Many of the tools in analysis_modules are very old and are in need of API-breaking refactor. Some of these, like two_point_functions, did not make the jump from yt-2 to yt-3 and are no longer usable. Many tools no longer have a champion, someone interested in using, maintaining, and updating them as yt’s core functionality develops and changes. Moving analysis_modules from yt decouples them from yt’s release cycle, allowing interested parties to make updates and releases on a separate, likely shorter timescale. Some analysis_modules may even be better suited to be moved into other extensions that are actively developed, such as the case of the AbsorptionSpectrum with the Trident project. - Similar to the point above, yt releases would not be slowed by the need to update all of the championless modules. Individual analysis modules can be tied to specific stable releases of yt and so assured to work there. Cons - This will take a non-zero amount of work. See below for a summary of the primary tasks. - There are some outstanding logistical questions. See below. - Not having yt and analysis modules explicitly tied to the same codebase/releases could result in analysis tools falling behind and out of date. - The disruption and need to alter scripts could irritate and alienate users. Proposed progression This is roughly how this would happen. Here is a table with all existing analysis modules, their status, and potential future: https://goo.gl/HZykQA 1. Create yt_astro_analysis repo with all analysis modules that are to be moved. Add an entry to the extensions page on yt-project.org. Make it installed by default in the install script, at least at first. 2. Deprecate all moved modules in yt. 3. After some time, remove deprecated modules from yt. Open issues Here are some logistics and questions that still need to be worked out. - Can we set up separate testing for yt_astro_analysis? Would maintaining this be a pain? - How/where would the documentation be hosted? - How would we move the analysis modules source code and maintain its revision history? Questions to yt-dev - Are you +/-1 on this? Any other comments? - Changes to the analysis_modules spreadsheet (https://goo.gl/HZykQA)? - Interested in helping out with this? If this happens, I propose anyone interested meets for a hangout to discuss how to proceed. Thanks for reading! Britton

6 9

Issue #1300: yt.create_profile fails to find the innermost cells (yt_analysis/yt)
by Hsi-Yu Schive 29 Nov '16

29 Nov '16

New issue 1300: yt.create_profile fails to find the innermost cells https://bitbucket.org/yt_analysis/yt/issues/1300/ytcreate_profile-fails-to-… Hsi-Yu Schive: The following script and dataset fail to find the innermost cells and return zeros in all profile bins. Data are spherical symmetric with a center at (7.13 Mpc, 7.13 Mpc, 7.13 Mpc). The innermost 8 cells have the same peak density about 1.23651e-25 g/cm^3. [Dataset URL](http://use.yt/upload/ca211222) Script: ``` #!python ds = yt.load('./Data_000000') sp = ds.sphere( 'c', (15, "kpc") ) print 'peak density = ', sp['density'].max() profile = yt.create_profile( sp, 'radius', 'density', n_bins=5 ) print 'radial bins = ', profile.x.value print 'density profile = ', profile['density'].value ```

1 0

Issue #1299: order of parameters set in VR matters (yt_analysis/yt)
by Michael Zingale 22 Nov '16

22 Nov '16

New issue 1299: order of parameters set in VR matters https://bitbucket.org/yt_analysis/yt/issues/1299/order-of-parameters-set-in… Michael Zingale: consider the following script: ``` import yt import numpy as np from yt.visualization.volume_rendering.api import \ Scene, \ VolumeSource # fake test dataset ds = yt.testing.fake_vr_orientation_test_ds(128) # no log please ds._get_field_info("density").take_log = False sc = Scene() vol = VolumeSource(ds, field=('gas', 'density')) sc.add_source(vol) cam = sc.add_camera(ds, lens_type="perspective") cam.resolution = (1280, 720) cam.north_vector = np.array([0., 0., 1.]) cam.focus = ds.arr(np.array([1., 0., 0.]), 'code_length') cam.position = ds.arr(np.array([0., 0., 0.]), 'code_length') #sc.annotate_domain(ds) sc.render() sc.save("test_{:03}.png".format(0), sigma_clip=6.0) ``` this runs as-is. If you change the order of the properties to: ``` cam.north_vector = np.array([0., 0., 1.]) cam.position = ds.arr(np.array([0., 0., 0.]), 'code_length') cam.focus = ds.arr(np.array([1., 0., 0.]), 'code_length') ``` it aborts with ``` RuntimeError: Cannot set the camera focus and position to the same value ``` it you change them to ``` cam.focus = ds.arr(np.array([1., 0., 0.]), 'code_length') cam.position = ds.arr(np.array([0., 0., 0.]), 'code_length') cam.north_vector = np.array([0., 0., 1.]) ``` it aborts with: ``` yt.utilities.exceptions.YTException: normal_vector and north_vector cannot be aligned. ``` these checks should wait until the user is done setting all the properties and the order of setting them should not matter.

1 0

Issue #1298: set_*label - command for ProfilePlot (yt_analysis/yt)
by Katharina Wollenberg 19 Nov '16

19 Nov '16

New issue 1298: set_*label - command for ProfilePlot https://bitbucket.org/yt_analysis/yt/issues/1298/set_-label-command-for-pro… Katharina Wollenberg: Hi together, so far in yt, ProfilePlot does not have an "easy" way to set the plot axes labels individually. It would be nice to have in the future a command similar to the set_x/y/zlabel as is given for e.g. SlicePlot. Kind regards, Katharina

1 0

Issue #1297: ds.print_stats() fails (yt_analysis/yt)
by yurivict 17 Nov '16

17 Nov '16

New issue 1297: ds.print_stats() fails https://bitbucket.org/yt_analysis/yt/issues/1297/dsprint_stats-fails yurivict: The example from http://yt-project.org/doc/quickstart/data_inspection.html fails: ``` import yt ds = yt.load("IsolatedGalaxy/galaxy0030/galaxy0030") ds.print_stats() ``` ``` $ python2 chapter-inspect.py yt : [INFO ] 2016-11-17 00:12:48,682 Parameters: current_time = 0.00600002000283 yt : [INFO ] 2016-11-17 00:12:48,682 Parameters: domain_dimensions = [32 32 32] yt : [INFO ] 2016-11-17 00:12:48,683 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2016-11-17 00:12:48,683 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2016-11-17 00:12:48,683 Parameters: cosmological_simulation = 0.0 Traceback (most recent call last): File "chapter-inspect.py", line 6, in <module> ds.print_stats() File "/usr/local/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 320, in root_only return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 457, in print_stats self.index.print_stats() File "/usr/local/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 419, in index self, dataset_type=self.dataset_type) File "/usr/local/lib/python2.7/site-packages/yt/frontends/enzo/data_structures.py", line 212, in __init__ GridIndex.__init__(self, ds, dataset_type) File "/usr/local/lib/python2.7/site-packages/yt/geometry/geometry_handler.py", line 50, in __init__ self._setup_geometry() File "/usr/local/lib/python2.7/site-packages/yt/geometry/grid_geometry_handler.py", line 51, in _setup_geometry self._parse_index() File "/usr/local/lib/python2.7/site-packages/yt/frontends/enzo/data_structures.py", line 272, in _parse_index pbar = get_pbar("Parsing Hierarchy ", self.num_grids) File "/usr/local/lib/python2.7/site-packages/yt/funcs.py", line 414, in get_pbar pbar = TqdmProgressBar(title,maxval) File "/usr/local/lib/python2.7/site-packages/yt/funcs.py", line 346, in __init__ self._pbar = tqdm(leave=True, total=maxval, desc=title) File "/usr/local/lib/python2.7/site-packages/yt/extern/tqdm/_tqdm.py", line 297, in __init__ ncols = _environ_cols_wrapper()(file) TypeError: 'NoneType' object is not callable ```

1 0

Issue #1296: Python setuputils fail to record __hg_version__.py* files as installed (yt_analysis/yt)
by yurivict 16 Nov '16

16 Nov '16

New issue 1296: Python setuputils fail to record __hg_version__.py* files as installed https://bitbucket.org/yt_analysis/yt/issues/1296/python-setuputils-fail-to-… yurivict: setup.py writes these files separately, and it looks like this causes the problem. The install command ``` python setup.py install --record {file-list} ``` fails to record these files.

1 0

to the community
by Britton Smith 11 Nov '16

11 Nov '16

Hi everyone, In light of the recent election and some of the negative events that have happened, we want to reassure you, and at the same time make it clear to all, that we expect the yt community to continue to be a place of tolerance, kindness, and respect for everyone. We will certainly not judge or discriminate against people based on their values, beliefs, political affiliation or lack thereof, lifestyles, life experiences, race, color, religion, sex, national origin, ancestry, age, order of protection status, genetic information, marital status, disability, sexual orientation including gender identity, unfavorable discharge from the military or status as a protected veteran. We have a code of conduct, which can be reviewed at http://yt-project.org/community.html , which covers interactions on the mailing list, at in person workshops, on Slack or IRC, and commit messages. Importantly, we want to make sure you are aware that this tolerance of viewpoints does *not* extend to tolerance of harassing, rude, or intolerant statements or behavior. If any of you experience or witness something along these lines, please let members of the steering committee know. If you have something you wish to report that you would like to remain confidential, please email confidential(a)yt-project.org , and we will work to address this issue. For more information, this blog post on the Committee for the Status of Women in Astronomy has suggested actions for when you find yourself to be a bystander: http://womeninastronomy.blogspot.com/2016/11/reflection-moment-importance-o… , including links to other resources. Based on our experiences with the yt community as a group and individually, none of us anticipate this to be a concern. However, we cannot condemn strongly enough hateful language or actions, and want to take this opportunity to reassure this community that we will not tolerate it in our midst. Thank you, yt steering committee

1 0

Issue #1295: [GDF] writing and loading dataset yields different result (yt_analysis/yt)
by Kacper Kowalik 04 Nov '16

04 Nov '16

New issue 1295: [GDF] writing and loading dataset yields different result https://bitbucket.org/yt_analysis/yt/issues/1295/gdf-writing-and-loading-da… Kacper Kowalik: As reported by Andrea Negri on the [yt-user ml](http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2016-Novemb… following script: ``` #!/usr/bin/env python # -*- coding: utf-8 -*- import numpy as np import yt def main(): dims = np.array([2, 2, 2]) grid_data = [ dict(left_edge = [0.0, 0.0, 0.0], right_edge = [1.0, 1.0, 1.0], level = 0, dimensions = dims), dict(left_edge = [0.25, 0.25, 0.25], right_edge = [0.75, 0.75, 0.75], level = 1, dimensions = dims), ] bbox = np.array([[0, 1], [0, 1], [0, 1]]) ds = yt.load_amr_grids(grid_data, dims, bbox=bbox, periodicity=(0,0,0), \ length_unit=1., refine_by=2) #make a nice slice slc = yt.SlicePlot(ds, "z", "grid_level") slc.set_log("grid_level", False) slc.save('original_slice.png') yt.utilities.grid_data_format.writer.write_to_gdf(ds, 'test.h5', clobber=True) da = yt.load('test.h5') slc = yt.SlicePlot(da, "z", "grid_level") slc.set_log("grid_level", False) slc.save('test.png') if __name__ == '__main__': main() ``` yields different results ![original_slice.png] (https://bitbucket.org/repo/BXbAb/images/355574044-original_slice.png) ![test.png](https://bitbucket.org/repo/BXbAb/images/3456721044-test.png) for the same dataset that was exported and read as a GDF file. Responsible: xarthisius

1 0