yt-dev August 2013

yt-dev@python.org

15 participants
47 discussions

by Stuart Mumford

Hello, First an introduction, I am Stuart Mumford I am doing a solar physics PhD at Sheffield University in the UK. I work primarily on MHD wave simulations of the solar atmosphere studying energy propagation and wave excitation. The code SAC (http://www.aanda.org/10.1051/0004-6361:200809800) solves for perturbations ontop of a fixed background to deal with the very wide range of the parameters e.g. density and temperature in the solar atmosphere. I also attended SciPy this year, where I met some of you and discovered yt, mainly as a consequence of my contributions to SunPy (www.sunpy.org) which is a solar physics data analysis library. So that's that over with, now having been playing with yt on and off since SciPy I started using it properly this week, as the need arose. I am very impressed with yt's capabilities and I want to be able to make better use of it. To that end I am trying to write a frontend for SAC, as I have been speaking to some of you about in IRC (I am Cadair in IRC and on GH). I have some questions regarding this, as far as I can ascertain a frontend provides a mapper between the stored data and a abstracted format that yt understands? I have started looking in the code for FLASH and Stream to try and get an idea where to start and not got very far. So some questions to start: Where should the file handling be done? Where is the file opened? What things should the StaticOutput subclass define? Thanks once again for your help on IRC over the last few days and I hope you can point me in the right direction. Stuart

7 years, 6 months

Issue #638: Document IPython integration (yt_analysis/yt)

by Nathan Goldbaum

New issue 638: Document IPython integration https://bitbucket.org/yt_analysis/yt/issue/638/document-ipython-integration Nathan Goldbaum: There should be a section in the narrative docs that goes over yt's IPython integration. Specifically, this should cover `imods`, `yt notebook`, and IPython pretty display for plots and volume renderings.

7 years, 6 months

Issue #637: Matplotlib 1.3.0 doesn't seem compatible with yt.pmods (yt_analysis/yt)

by Matthew Turk

New issue 637: Matplotlib 1.3.0 doesn't seem compatible with yt.pmods https://bitbucket.org/yt_analysis/yt/issue/637/matplotlib-130-doesnt-seem-c… Matthew Turk: Running "from yt.pmods import *" even from within a non-MPI python results in: ``` #!python File "pm.py", line 1, in <module> from yt.pmods import * File "/home/mturk/yt/yt/yt/pmods.py", line 364, in <module> from yt.mods import * File "/home/mturk/yt/yt/yt/pmods.py", line 235, in __import_hook__ m = __load_tail__(q, tail) File "/home/mturk/yt/yt/yt/pmods.py", line 339, in __load_tail__ m = __import_module__(head, mname, m) File "/home/mturk/yt/yt/yt/pmods.py", line 279, in __import_module__ m = imp.load_module(fqname, fp, pathname, stuff) File "/home/mturk/yt/yt/yt/mods.py", line 129, in <module> from yt.visualization.api import \ File "/home/mturk/yt/yt/yt/pmods.py", line 235, in __import_hook__ m = __load_tail__(q, tail) File "/home/mturk/yt/yt/yt/pmods.py", line 339, in __load_tail__ m = __import_module__(head, mname, m) File "/home/mturk/yt/yt/yt/pmods.py", line 279, in __import_module__ m = imp.load_module(fqname, fp, pathname, stuff) File "/home/mturk/yt/yt/yt/visualization/api.py", line 31, in <module> from color_maps import \ File "/home/mturk/yt/yt/yt/pmods.py", line 234, in __import_hook__ q, tail = __find_head_package__(parent, name) File "/home/mturk/yt/yt/yt/pmods.py", line 323, in __find_head_package__ q = __import_module__(head, qname, parent) File "/home/mturk/yt/yt/yt/pmods.py", line 279, in __import_module__ m = imp.load_module(fqname, fp, pathname, stuff) File "/home/mturk/yt/yt/yt/visualization/color_maps.py", line 26, in <module> import matplotlib File "/home/mturk/yt/yt/yt/pmods.py", line 234, in __import_hook__ q, tail = __find_head_package__(parent, name) File "/home/mturk/yt/yt/yt/pmods.py", line 328, in __find_head_package__ q = __import_module__(head, qname, parent) File "/home/mturk/yt/yt/yt/pmods.py", line 279, in __import_module__ m = imp.load_module(fqname, fp, pathname, stuff) File "/home/mturk/yt-x86_64/lib/python2.7/site-packages/matplotlib-1.3.0-py2.7-linux-x86_64.egg/matplotlib/__init__.py", line 130, in <module> from matplotlib.compat import subprocess File "/home/mturk/yt/yt/yt/pmods.py", line 239, in __import_hook__ __ensure_fromlist__(m, fromlist) File "/home/mturk/yt/yt/yt/pmods.py", line 357, in __ensure_fromlist__ submod = __import_module__(sub, subname, m) File "/home/mturk/yt/yt/yt/pmods.py", line 279, in __import_module__ m = imp.load_module(fqname, fp, pathname, stuff) File "/home/mturk/yt-x86_64/lib/python2.7/site-packages/matplotlib-1.3.0-py2.7-linux-x86_64.egg/matplotlib/compat/subprocess.py", line 79, in <module> CalledProcessError = subprocess.CalledProcessError AttributeError: 'module' object has no attribute 'CalledProcessError' ```

7 years, 6 months

Announcing: yt 2.5.5

by Matthew Turk

Hi all, We're proud to release yt version 2.5.5. This is an unscheduled point release that includes all bug fixes identified and fixed since the release of 2.5.4 on July 2nd. We missed our scheduled release date a bit, and have continued to push off a 2.6 as development on the 2.x branch winds down. Additions, changes and bug fixes include: * New absorption spectrum analysis module with documentation * Fix for volume rendering on the command line * map_to_colormap will no longer return out-of-bounds errors * Fixes for dds in covering grid calculations * Library searching for build process is now more reliable * Unit fix for "VorticityGrowthTimescale" field * Pyflakes stylistic fixes * Adding ability to draw lines with Grey Opacity in volume rendering * Updated physical constants to reflect 2010 CODATA data * Number density added to FLASH * Dependency updates (including IPython 1.0) * Many fixes for Athena frontend * Radius and ParticleRadius now work for reduced-dimensionality datasets * Source distributions now work again! * Better notebook support for yt plots * Athena data now 64 bits everywhere * Grids displays on plots are now shaded to reflect the level of refinement * show_colormaps() is a new function for displaying all known colormaps * PhasePlotter by default now adds a colormap. If you are using the stable branch of yt from an installation script, you can upgrade using "yt update" or "yt update --all" to upgrade your full dependency stack. If you are using the development branch, you may already have these fixes. A tarball of this release has been uploaded to the Python Package Index (PyPI). yt releases often feature contributions from many individuals; this release includes first time contributions from John Forbes, Noel Scudder and Hilary Egan. Documentation for this release can be found at: http://yt-project.org/docs/dev/ Thanks very much, Matt, on behalf of the yt development team

7 years, 6 months

Dealing with overlapping grids in Athena data

by John Zuhone

Hi all, I wanted to call attention to an oddity I discovered yesterday with regard to some Athena data I'm working with. For background, Athena data consists of "domains" which may or may not exist on different levels of refinement. They adhere to some fairly strict rules, which may be found here: https://trac.princeton.edu/Athena/wiki/AthenaDocsSMRGeom Domains are further subdivided into "grids". If the run was done on a single processor, then each domain just has one grid, and the two are identical. If it was a parallel job, then a domain will usually have multiple grids. The important point here is that there is no requirement that a grid on a given refinement level needs to be fully self-contained within a parent grid on the level just below it (though Athena does adhere to refine_by = 2). To further complicate things, for parallel runs the grids are split into different vtk files in different directories for each processor and also different directories for each refinement level. We've constructed the Athena frontend so that this data may be read as well as data that was produced by a serial run. There is a tool in the Athena source, join_vtk, which allows one to take the data from a parallel run and merge it into single vtk files, which has the consequence of reducing the number of grids to one per domain. This brings me to the dataset I mentioned in the beginning. It consists of a simple set of 5 3D cubical nested domains, all centered on the domain center, which all of them except the highest-level domain exactly half the size on a side of the previous one. The last one is 3/4 the size on a side of the previous one. The run was done on 512 processors, and each domain is split up into 512 grids. The result looks like this: https://hub.yt-project.org/nb/bfd6uv As you can see, the issue is that the grids on the highest refinement level really don't have a well-defined notion of parent grids, because these grids are overlapping multiple grids on the parent level. However, at the cell level there are no overlaps of this type--every cell on a given level is fully contained within a cell on the level below it. Here is the same data, except that it has been merged with join_vtk and now there are only five grids: https://hub.yt-project.org/nb/7qxvas In this case, every grid has a well-defined parent, so everything looks correct. In fact, I realized today that there is no provision in the Athena frontend for establishing parent-child relationships amongst grids--so apparently this is being determined somewhere in the code as a fallback when the frontend doesn't provide this information? I'd just like to open this up for discussion as to how we might make it possible to at least work with this data in a limited way, since it seems like it's a situation that might come up often enough. I've created an issue: https://bitbucket.org/yt_analysis/yt/issue/636/overlapping-grids-in-athena-… Best, John Z -- John ZuHone Postdoctoral Researcher NASA/Goddard Space Flight Center jzuhone(a)gmail.com john.zuhone(a)nasa.g

7 years, 6 months

Announcing: yt 3.0 alpha 3

by Matthew Turk

Hi everyone, We're proud to announce the third ALPHA release of yt 3.0. yt has recently transitioned to a time-based release plan ( https://ytep.readthedocs.org/en/latest/YTEPs/YTEP-0008.html ) and this is the third scheduled alpha of 3.0, although we've grossly missed the estimated date of July 15. No date for a "final" release has yet been set, but there will likely be several more alphas before that time. The yt 2.5 codebase, and further updates in the 2.x series, will be supported for a considerable amount of time and you do not need to upgrade. = yt 3.0?! = yt 3.0 represents a new direction forward for yt: getting rid of all the underlying assumptions that data needs to be sectioned off into nice little grid patches. This includes supporting Octree codes natively (NMSU-ART and RAMSES), eventual support for SPH codes, and even opaque data structures where the data is extremely large (ARTIO). We're even planning support for natively handling cylindrical and spherical coordinates. More: http://blog.yt-project.org/post/WhatsUpWith30.html However, this *is* an alpha release. Not all of the existing codes have been ported to 3.0. Additionally, this release benefits from the technical and non-technical contributions from many new people. yt is developed in the context of a community of contributors, and with the push toward a new architecture, we aim to expand that community considerably. In particular, this release has considerably benefited from contributions from many new individuals. = Getting It! = To try out yt 3.0, you can now pull from the main yt repository, update to the yt-3.0 branch, and rebuild your extensions. Or, if you would like to create a new, safely sectioned off environment, simply re-run the normal "development" install script after changing the variable BRANCH to "yt-3.0". If you would like to try out yt 3.0 and are having trouble, please write to the yt-users mailing list for assistance. The yt 3.0 install script may also work, which can be obtained by executing these commands: wget http://hg.yt-project.org/yt/raw/yt-3.0/doc/install_script.sh bash install_script.sh = What's New? = A demo notebook demonstrating much new functionality, and including the full release notes, can be found here: http://hub.yt-project.org/nb/88u14f The bullet-pointed release notes can be found at the end of this email, as well. The main improvements include *considerable* memory usage reduction, ARTIO spatial data indexing support, better particle support, an overhaul of the selection methods for data objects, and a mechanism for on-the-fly definitions of particle types based on boolean filters. Additionally, this release was used in the production of the AGORA project flagship paper. The analysis script used there can be seen at this short URL: http://goo.gl/SwY9Uv and the paper can be found here: http://arxiv.org/abs/1308.2669 = Reporting Problems = If you test out yt 3.0 we want to hear if it DID or DID NOT work! Feedback is crucial at this time. yt-users and yt-dev are both good forums for discussion, asking questions, and reporting problems. Lots of things have changed on the backend, but we have attempted to minimize the user-facing changes. To report a bug please go here: https://bitbucket.org/yt_analysis/yt/issues/new Note that you will not receive updates if you are not logged in when you create the bug. = What's Next? = The next alpha release (3.0a4) will be released sometime this fall, but development can be monitored either at http://bitbucket.org/yt_analysis/yt-3.0 or in the main yt repository under the named branch "yt-3.0". We hope to focus on generalizing Octree support further, adding better non-Cartesian support, and supporting generic smoothing kernel definitions. If you'd like to participate in yt development, please stop by #yt on irc.freenode.net ( http://yt-project.org/irc.html ) or yt-dev ( http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org ), or submit a pull request on BitBucket. Thanks very much, Matt, on behalf of the yt development team Changelog: (Including 434 changesets from 7 contributors since 3.0a2) * Fixes for volume rendering and porting of the Cython kD-tree. VR now works for octree codes, albeit slowly. * Major reorganization and simplification of selector code, including generic periodicity * ARTIO now has spatial data support * Major bug fix for FLASH IO. FLASH works again, following a regression in 3.0a2. * Merged with mainline yt development, bringing developments from yt 2.5.3 and 2.5.4 into 3.0. * OWLS (Gadget-HDF5) data now updated to match functionality of other Gadget datasets. * Creationg of "arbitrary_grid" data selector for flexible particle deposition operations. This enables creation of a grid of arbitrary dimensions that can have particles smoothed or deposited within them. * Conversion of all CIC operations to particle deposition operations. This results in much more flexible particle type selection and speed improvements. * Refactoring of particle fields to enable simpler creation and collection of particle fields. Particle fields defined for Eulerian codes and Lagrangian codes are now interchangeable. * New, currently unused parallel ring iteration. To be explored in the future for smoothing kernels. * Many improvements for NMSU-ART * Fixed a crazy QuadTree deposition bug for projections that showed up when projecting octree particle deposition results through a restricted data object. * Fixed a crazy off-by-one bug for binary Gadget IO. * Enormous Octree diet. * Oct leaf nodes now cost 256 bits each, which may eventually be reduced to 192 bits. Previous versions were considerably larger. * Octree traversal is strictly recursive, enabling distributed-memory octrees to be implemented. * RAMSES octrees are created by-domain instead of globally. This will enable better parallelism in the future and faster ghost zone generation when that is implemented. * Particle octree are now constructed via Z-curve generation. This is considerably faster (10-20x) as well as much more memory conservative. This was designed to utilize parallel octree construction in the future. * Particle octrees are now indexed by coarse base-level indexing of domain regions. This enables unsorted particle files to be indexed for IO as well as spatial selection without mandating that leaf nodes are fully-contained within a single file. Reduction in total oct leaf count of ~8x for multi-file particle datasets. * YTDataContainer no longer requires .shape and .size * Only requested fields are plotted, not translated fields * Reduce memory usage of spatially-chunked fields for patch datasets * Added arbitrary particle loader, load_particles. * Units fix for RAMSES boxlen!=1.0 datasets * Particle filtering for dynamically-created particle types, similar to derived fields * Improve type consistency in TotalQuantity and other derived quantities. * Fixed major error with ghost zone generation and uninitialized values * Some resiliency for particle datasets that do not specify a domain * Bugfix for 1- and 2-D Enzo datasets * Improvements to the Tipsy frontend, including auto-detecting parameter files * Enable specification of n_ref when creating particle dataset * Enable particle deposition to back-act on particle fields * Major fix for octree particle counting, resulting in smaller meshes * Enable "source" specification for volume rendering * Enable initial volume rendering of octree datasets * Fixed error with RAMSES C/F ordering of data * Many fixes related to field dependency calculation * FLASH cylindrical & polar pixelization fix

7 years, 6 months

Issue #636: Overlapping grids in Athena SMR data (yt_analysis/yt)

by John ZuHone

New issue 636: Overlapping grids in Athena SMR data https://bitbucket.org/yt_analysis/yt/issue/636/overlapping-grids-in-athena-… John ZuHone: Athena grids on a given refinement level are not required to be fully self-contained within grids on the level just below them, so unique parent-child relationships are not established in some cases. A notebook showing this issue: https://hub.yt-project.org/nb/bfd6uv If the data from the different grids is merged together (outside of yt) so that there is one grid per domain, the data looks correct: https://hub.yt-project.org/nb/7qxvas Responsible: jzuhone

7 years, 6 months

Issue #635: find_max() is broken for particle codes: (yt_analysis/yt)

by Nathan Goldbaum

New issue 635: find_max() is broken for particle codes: https://bitbucket.org/yt_analysis/yt/issue/635/find_max-is-broken-for-parti… Nathan Goldbaum: ``` pf.h.find_max(('deposit', 'Gas_density')) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-5-d81b7d6ae9ea> in <module>() ----> 1 pf.h.find_max(('deposit', 'Gas_density')) AttributeError: 'ParticleGeometryHandler' object has no attribute 'find_max' ``` I think this operation will fail for RAMSES as well.

7 years, 6 months

Issue #634: Set up derived quantities for individual particle types (yt_analysis/yt)

by Matthew Turk

New issue 634: Set up derived quantities for individual particle types https://bitbucket.org/yt_analysis/yt/issue/634/set-up-derived-quantities-fo… Matthew Turk: Much like how we have the `particle_fields.py` file for creating specific types of particle fields, we should have this for derived quantities as well. Responsible: MatthewTurk

7 years, 6 months

Documentation repo not updating yt.readthedocs.org in over a week

by Cameron Hummels

Hello everyone (and Britton and Kacper), I had a PR accepted last week to create an additional cookbook entry on overplotting grid edges on a PlotWindow instance. I've looked, and the change is in the repository: yt-doc/source/cookbook/complex_plots.rst ( the header: "Plotting Grid Edges Over Fluids" right after "Plotting Particles Over Fluids"). Unfortunately, it does not appear on either the yt.readthedocs.org page, nor does it appear yt-project.org docs page. I thought that the readthedocs page got rebuilt immediately after every PR, or is this not the case? Does anyone know what might be going on here? I wonder how long this may have been happening and if our docs are lagged behind the changes that have occurred to them recently? Britton, Matt suggested I talk to you since you own the docs repo on BB. Kacper, I figured I'd include you here since you're in charge of the yt-project.org/docs page. Cameron Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org

7 years, 6 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

yt-dev August 2013