yt-dev December 2014

yt-dev@python.org

11 participants
38 discussions

by Cameron Hummels

Fellow yt users: A few weeks ago, we asked yt users and developers to fill out a brief survey (http://goo.gl/forms/hRNryOWTPO) to provide us with feedback on how well yt is meeting the needs of the community and how we can improve. Thank you to all 39 people who responded, as it has given us a great deal to consider as we move forward with the code. We summarize the results of the survey below, but I start with the basic takeaway from the survey: Overall Survey Takeaway: The survey respondents are generally pleased with yt. It meets their needs, has a wonderful community, is relatively easy to install, and has fair documentation. Major short-term requests were for improvements in documentation, particularly in API docs and source code commenting, as well as more cross-linking in the existing documentation and making sure docs were up to date. Furthermore, people wanted more attention to making sure existing code in 3.0 works and for resurrecting all 2.x functionality in 3.0. The single biggest takeaway from the survey is that the transition to yt 3.0 has been fraught with difficulties. Many submitters expressed satisfaction with the new functionality in 3.0, but the overall process of transition through documentation, analysis modules and community response has been found to be lacking. Background: There were 39 unique responses to our survey. 75% of the respondents were grads and postdocs with a smattering of faculty, undergrads, and researchers. Nearly everyone is at 4-year universities. 50% of the respondents consider themselves intermediate users, 20% novice, 20% advanced, and 10% gurus. Installation: 90% of the respondents use the standalone install script, with a several users employing other methods (potentially in addition to the standalone script). 95% of the respondents rated installation as a 3 or better (out of 5) with most people settling on a 4 out of 5. Installation comments were aimed at having better means of installing on remote supercomputing systems and/or making pip installs work more frequently. Community Responsiveness: 72% of respondents gave yt 5 out of 5 and 97% were 3 or greater for community responsiveness. Clearly this is our strong point. There was a very wide distribution of ways in which people contacted the community for help with the most popular means being the mailing lists, the irc channel, mailing developers directly, and searching google. Comments in this section were mostly positive, but one user wished for more concrete action to be taken after bugs were reported. Documentation: 77% of respondents gave 4 or 5 out of 5 for the overall rating of the documentation. Individual docs components were more of a mix. Cookbooks were ranked very highly, and quickstart notebooks and narrative docs were generally ranked well. The two documentation components that seemed be ranked lower (although still fair) were API docs and comments in the source code with 15% of respondents noting that they were “mostly not useful” (ie 2 / 5). There were a lot of comments regarding ways to improve the docs, which I bullet point here: - Organization of docs is difficult to parse; Hard to find what you’re looking for. - Hard to know what to search for, so make command list (ie API docs) more prominent - Docs not always up to date (even between 3.0 and dev) - Discrepancies between API docs and narrative docs - Examples are either too simple or too advanced--need more intermediate examples - Units docs need more explanation - Not enough source code commenting or API docs - Not enough cross-linking between docs - More FAQ / Gotchas for common mistakes - API docs should include more examples and also note how to use all of the options, not just the most common. Functionality: 88% of respondents found yt to meet their research needs (4 or 5 out of 5). Respondents are generally using yt on a variety of datasets including grid data, octree data, particle data, and MHD with only a handful of users dealing with spherical or cylindrical data at present. Nearly all of the frontends are being used by respondents, with a few exceptions: Chombo, Moab, Nyx, Pluto, and non-astro data. Visualization remains the main use of yt with 97% of respondents, but simple analysis received 82% and advanced analysis received 62%. Interestingly, 31% of respondents use halo analysis tools, with only 15% using synthetic observation analysis. Big Picture: 51% of respondents gave yt 5 out of 5 for general satisfaction, with 28% 4 out of 5 and 15% 3 out of 5. Overall, this is pretty good but probably biased by the fact that people filled out this survey. Comments on the greatest strengths of yt include: - visualization capabilities - community support - flexibility Comments on the biggest shortcomings of yt include: - documentation (see above) - learning to “think in yt” - making new functionality when there is existing broken functionality (or missing documentation) - making sure 3.0 matches all functionality from 2.x - keeping the documentation up to date - making the transition from 2.x to 3.0 easier (how to update scripts) Things to focus on in the next year: - documentation (almost unanimously) - making sure 3.0 can do all functionality from 2.x Thank you for all of the valuable feedback. We sincerely appreciate the constructive criticism in making for a better code and community! We will put together a blueprint of how to address these shortcomings soon. Look for it after the holiday break. Have a wonderful holiday! On behalf of the yt development team, Cameron -- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #963: RAMSES particle depositition onto mesh does not work with filtered datasets (yt_analysis/yt)

by Benjamin Thompson

New issue 963: RAMSES particle depositition onto mesh does not work with filtered datasets https://bitbucket.org/yt_analysis/yt/issue/963/ramses-particle-depositition… Benjamin Thompson: Hello. With a RAMSES dataset. The deposition of particles onto a mesh works fine for unfilltered cases. For example, should I call ``` #!python print data["deposit,"io_density"] ``` I get answers which are sensible. However, I wish to filter my particle data for stars/dark matter and treat the particles seperatly. If for example I do this, ``` #!python def star_filter(pfilter,data): filter = np.logical_and(data.particles.source["particle_age"] != 0, data.particles.source["particle_age"] != None) return filter add_particle_filter("stars", function=star_filter, filtered_type="all", requires=["particle_age"]) data.ds.add_particle_filter("stars") ``` Which filters the particles as expected, which also creates the fields ('deposit', 'io_cic') ('deposit', 'io_cic_velocity_x') ('deposit', 'io_cic_velocity_y') ('deposit', 'io_cic_velocity_z') ('deposit', 'io_count') ('deposit', 'io_density') ('deposit', 'io_mass') ('deposit', 'io_nn_velocity_x') ('deposit', 'io_nn_velocity_y') ('deposit', 'io_nn_velocity_z') ('deposit', 'stars_cic') ('deposit', 'stars_cic_velocity_x') ('deposit', 'stars_cic_velocity_y') ('deposit', 'stars_cic_velocity_z') ('deposit', 'stars_count') ('deposit', 'stars_density') ('deposit', 'stars_mass') ('deposit', 'stars_nn_velocity_x') ('deposit', 'stars_nn_velocity_y') ('deposit', 'stars_nn_velocity_z') As you would expect However, upon trying to call print data["deposit,"stars_density"] I get the error below ``` #!shell Traceback (most recent call last): File "selene_gradients.py", line 46, in <module> print cylinder[("deposit", "stars_density")] File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 240, in __getitem__ self.get_data(f) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 673, in get_data self._generate_fields(fields_to_generate) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 693, in _generate_fields fd = self._generate_field(field) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 277, in _generate_field tr = self._generate_fluid_field(field) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 295, in _generate_fluid_field rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 311, in _generate_spatial_fluid ind += o.select(self.selector, self[field], rv, ind) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 240, in __getitem__ self.get_data(f) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 673, in get_data self._generate_fields(fields_to_generate) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 693, in _generate_fields fd = self._generate_field(field) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 277, in _generate_field tr = self._generate_fluid_field(field) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 297, in _generate_fluid_field rv = finfo(gen_obj) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/fields/derived_field.py", line 184, in __call__ dd = self._function(self, data) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/fields/particle_fields.py", line 106, in particle_density pos = data[ptype, coord_name] File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.py", line 76, in __getitem__ tr = super(OctreeSubset, self).__getitem__(key) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 240, in __getitem__ self.get_data(f) File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 615, in get_data with f.apply(self): File "/home/bthompson1/python/yt/yt-x86_64/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/gpfs/home/bthompson1/python/yt/yt-x86_64/src/yt-hg/yt/data_objects/particle_filters.py", line 46, in apply filter = self.function(self, dobj) File "/gpfs/home/bthompson1/ramses_pp/ramses_pp/ben/real_selene_analysis/ramses_pp/modules/yt/YT.py", line 59, in star_filter filter = np.logical_and(data.particles.source["particle_age"] != 0, data.particles.source["particle_age"] != None) AttributeError: 'RAMSESDomainSubset' object has no attribute 'particles' ``` _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #962: Unit warnings and errors should reference locations higher on the stack (yt_analysis/yt)

by Nathan Goldbaum

New issue 962: Unit warnings and errors should reference locations higher on the stack https://bitbucket.org/yt_analysis/yt/issue/962/unit-warnings-and-errors-sho… Nathan Goldbaum: Warnings like these aren't useful: ![Untitled.png](https://bitbucket.org/repo/BXbAb/images/3161691190-Untitled.png) These warnings are happening when we invoke numpy on the array data - normally these warnings would be issued wherever we are manipulating numpy arrays elsewhere, but since we access numpy indireactly via YTArray, the warnings show up as function calls in YTArray. We should be doing these superclass calls in try/except blocks to catch the warnings and if they happen manipulate them so the warning corresponds to a useful place in the call stack - probably a call or two higher. Responsible: ngoldbaum _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #961: total mass quantity called on a clump breaks subsequent data selection (yt_analysis/yt)

by Nathan Goldbaum

New issue 961: total mass quantity called on a clump breaks subsequent data selection https://bitbucket.org/yt_analysis/yt/issue/961/total-mass-quantity-called-o… Nathan Goldbaum: The following script: ``` #!python import numpy as np import yt from yt.analysis_modules.level_sets.api import * ds = yt.load("IsolatedGalaxy/galaxy0030/galaxy0030") data_source = ds.disk([0.5, 0.5, 0.5], [0., 0., 1.], (8, 'kpc'), (1, 'kpc')) field = ("gas", "density") step = 2.0 c_min = 10**np.floor(np.log10(data_source[field]).min() ) c_max = 10**np.floor(np.log10(data_source[field]).max()+1) master_clump = Clump(data_source, field) master_clump.add_validator("min_cells", 20) find_clumps(master_clump, c_min, c_max, step) leaf_clumps = get_lowest_clumps(master_clump) clump = leaf_clumps[3] print clump['density'] try: clump.quantities.total_mass() except Exception as e: print e print clump['density'] ``` will print out ``` #!python [ 7.07273796e-24 6.71093874e-24 6.95491269e-24 6.82937561e-24 6.80951625e-24 6.57924229e-24 7.29300493e-24 7.07554671e-24 7.30696998e-24 7.55819187e-24 7.65949253e-24 6.95166906e-24 7.52178995e-24 7.07804069e-24 6.73055650e-24 7.13322789e-24 7.65117396e-24 7.31980022e-24 7.29300148e-24 7.06273581e-24 7.00048981e-24 7.11120382e-24 6.79811352e-24 6.67183093e-24 7.16742091e-24 7.36740952e-24 7.14456988e-24 7.34582239e-24 6.71862846e-24 7.32380039e-24 7.49710554e-24 6.95885833e-24 6.82522495e-24 6.91253709e-24 7.12972196e-24 6.76674443e-24 6.97601938e-24 7.56780126e-24 7.45219661e-24 7.33906110e-24 7.06027220e-24 6.85427194e-24 6.74850585e-24 7.30664210e-24 6.73289034e-24 7.46033156e-24 7.15491925e-24 7.32940616e-24 7.13456220e-24 6.72901856e-24] g/cm**3 Can't mix particle/discrete and fluid/mesh conditions or quantities. Field: ('all', 'particle_mass') and Conditions specified: obj['contours_3ed9f4d5c4c8464d85b336044c891112'] == 18 [] g/cm**3 ``` h/t to @dcollins4096 for reporting this on the mailing list. _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #960: Bad sub-volume plots for BoxLib/Maestro spherical data (yt_analysis/yt)

by Adam Jacobs

New issue 960: Bad sub-volume plots for BoxLib/Maestro spherical data https://bitbucket.org/yt_analysis/yt/issue/960/bad-sub-volume-plots-for-box… Adam Jacobs: When I try to plot an off-axis slice of a subset of BoxLib/Maestro spherical data (yt test dataset maestro_subCh_plt00248) I find that my data gets junked if the plot includes any regions outside of the subset. My understanding is that in yt the regions outside of the sub-volume should be white and the other data should be plotted as it is when the full dataset is plotted. This issue feels similar to a previously resolved issue dealing with off-axis slices of spherical Maestro data (https://bitbucket.org/yt_analysis/yt/issue/955/off-axis-slices-returning-in…). I've attached code that produces the error as well as images of the output. Responsible: ngoldbaum _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #959: Ray provides non-contiguous cell information (yt_analysis/yt)

by Nicholas Earl

New issue 959: Ray provides non-contiguous cell information https://bitbucket.org/yt_analysis/yt/issue/959/ray-provides-non-contiguous-… Nicholas Earl: Using the [Ray object](http://yt-project.org/doc/reference/api/generated/yt.data_objects.s… directly with enzo simulation data that uses AMR seems to give non-contiguous cell information. Mainly, it looks like the higher-resolution cells are appended to the end of the array. For a simple example of a high density cloud in an ambient medium, you can see a slice of the hydrogen number density in the simulation [here](https://www.dropbox.com/s/xhjjqntc64h0lfk/DD0001_Slice_z_H_number_den… with refinement overlaid. You can see in the [Ray's hydrogen number density](https://www.dropbox.com/s/a3dp3ip7wjmnbh7/h_num_density_ray.png?dl… that the sharp peak around 80 corresponds with the density gradient at the edges of the cloud while the higher resolution information of the cloud interior has been tacked on to the end (i.e. the "top-hat" portion). Printing of the [Ray's fcoords](http://paste.yt-project.org/show/5269/) also supports this. You can see that the x-coordinates of the cells between lines 167 and 200 are out of place. _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #958: Document detailed cuts on particle chunks (yt_analysis/yt)

by Cameron Hummels

New issue 958: Document detailed cuts on particle chunks https://bitbucket.org/yt_analysis/yt/issue/958/document-detailed-cuts-on-pa… Cameron Hummels: This is a followup on a conversation from the yt-users email list: --- *Hi, Everybody-* *I have some old code in which I pulled particle ids from grid objects, and I'm porting it to yt3.0.* *My old code used a "cut mask" defined like the following:* ``` #!python for grid in data._grids: cut_mask= data._get_cut_mask(grid) ``` *I see that grids is now handled through chunks, is there a modern equivalent of the cut_mask? Or, is there a new-and-improved method to get particle ids from complex objects?* --- *The yt-3.0 equivalent using the chunking system looks like this:* http://nbviewer.ipython.org/gist/ngoldbaum/700d2235d24a5faf9d58 --- Perhaps a note about this functionality should go in the "Particle Filters" section of the docs? http://yt-project.org/docs/dev/analyzing/filtering.html#filtering-particle-… _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

3.1?

by John ZuHone

Today was the day we were nominally supposed to release 3.1. I know we have Cameron's docs PR. Anything else? John ZuHone Kavli Center for Astrophysics and Space Research Massachusetts Institute of Technology 77 Massachusetts Ave., 37-582G Cambridge, MA 02139 (w) 617-253-2354 (m) 781-708-5004 jzuhone(a)space.mit.edu jzuhone(a)gmail.com http://www.jzuhone.com _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #957: Ragged array tests fail on numpy 1.9 (yt_analysis/yt)

by Nathan Goldbaum

New issue 957: Ragged array tests fail on numpy 1.9 https://bitbucket.org/yt_analysis/yt/issue/957/ragged-array-tests-fail-on-n… Nathan Goldbaum: Running the following command with numpy 1.9 installed generates the following: ``` #!base $nosetests utilities.lib.tests.test_ragged_arrays FFFFFF..FFF.................................FFFF.F...F.......................................................................................................................... ``` See http://paste.yt-project.org/show/5263/ for the tracebacks returned by nose. _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

6 years, 4 months

Issue #956: enable_parallelism dies due to missing rootLogger (yt_analysis/yt)

by Nathan Goldbaum

New issue 956: enable_parallelism dies due to missing rootLogger https://bitbucket.org/yt_analysis/yt/issue/956/enable_parallelism-dies-due-… Nathan Goldbaum: This script: ``` #!python import yt yt.enable_parallelism() ``` Will fail with the following (somewhat garbled) traceback when run in parallel: ``` #!python Traceback (most recent call last): File "test.py", line 2, in <module> yt.enable_parallelism() File "/Users/goldbaum/Documents/yt-hg/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 116, in enable_parallelism Traceback (most recent call last): File "test.py", line 2, in <module> if len(yt.utilities.logger.rootLogger.handlers) > 0: yt.enable_parallelism() AttributeError: 'module' object has no attribute 'rootLogger' File "/Users/goldbaum/Documents/yt-hg/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 116, in enable_parallelism if len(yt.utilities.logger.rootLogger.handlers) > 0: AttributeError: 'module' object has no attribute 'rootLogger' Traceback (most recent call last): File "test.py", line 2, in <module> yt.enable_parallelism() File "/Users/goldbaum/Documents/yt-hg/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 116, in enable_parallelism if len(yt.utilities.logger.rootLogger.handlers) > 0: AttributeError: 'module' object has no attribute 'rootLogger' yt : [INFO ] 2014-12-13 12:04:00,092 Global parallel computation enabled: 0 / 4 Traceback (most recent call last): File "test.py", line 2, in <module> yt.enable_parallelism() File "/Users/goldbaum/Documents/yt-hg/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 116, in enable_parallelism if len(yt.utilities.logger.rootLogger.handlers) > 0: AttributeError: 'module' object has no attribute 'rootLogger' ``` This was caused by PR 1355. I've provisionally assigned this issue to John ZuHone because he's been thinking about loggers a lot lately. I'm also not sure what the parallel analysis interface is doing with the loggers here, so I can't fix this myself immediately. If anyone else knows how to handle this, feel free to jump in with a patch :) Responsible: jzuhone

6 years, 4 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

yt-dev December 2014