Mailman 3 August 2014 - yt-dev

GDF PR--call for some feedback
by John Zuhone 22 Sep '14

22 Sep '14

Hi all, I have a work-in-progress PR in the hopper regarding GDF reading and writing: https://bitbucket.org/yt_analysis/yt/pull-request/1001/wip-some-gdf-writing… What it does is implement direct writing of covering grids to GDF files (something I've been using in my research, to work with the resulting files directly in yt) and clobbering existing files with the same path if the user allows for it. However, when fiddling around with this I determined that it was necessary to make GDF more unit-aware. Ultimately, I determined that changes were needed that would break the standard that we had determined for the files (http://yt-project.org/gdf.txt). The two main changes are: 1) Remove "field_to_cgs", and allow the fields to be in whatever units we wish them to be in the file (which are specified as HDF5 attributes). 2) Add a new top-level group containing the information for "length","mass", and "time" units. These will be used when the file is opened up in yt to determine the units. Since (for now) we do automatic conversion to cgs, that is something that is still left unimplemented, but otherwise I think this is all to do for now. I'm writing the email in case anyone has any suggestions or objections to the format changes--particularly Jeff or Kacper. We'll obviously need to document them if they are accepted. Best, John -- John ZuHone Postdoctoral Researcher NASA/Goddard Space Flight Center jzuhone(a)gmail.com john.zuhone(a)nasa.gov

8 9

let's talk about Governance
by Britton Smith 01 Sep '14

01 Sep '14

Greeting yt developers, First, I want to congratulate everyone here on the successful release of yt-3.0. This was a massive effort on the part of so many and a true testament to the strength of this team. At the time of writing this, there are 78 members of the yt-dev mailing list. As someone who does most of their work in very small collaborations, this amazes me and make me very proud. In case you're wondering, the yt-users list has 268 members. As a project, yt has a significant amount of infrastructure: code review with pull requests, issue tracking, automated testing, emails lists, an IRC channel, enhancement proposals, workshops. All of this is evidence of our legitimacy as a Real Thing. However, one big missing piece is a system of governance. I don't know exactly what this means, but I have some ideas, which I will share below. What I want to do right now is to start a discussion that will, hopefully, include as many people as possible on this list. For me, governance means (roughly) the following: - a set of procedures in writing for how various things are to be done, such as acceptance of pull requests, releases, designating developers as core contributors, etc. - a governing body to make decisions and help guide the project. This accomplishes a number of things, which as a project I think we need, such as: - overall stability of the project. - providing a system for conflict resolution. - maintaining the spirit of yt as a team effort. - providing a way for active contributors to get credit for their contribution in the form of official recognition. So, these are my initial thoughts, but I really think this deserves a thorough discussion with as many people participating as possible. Please, think about what governance means to you, whether we need it, what it should be, and what we might get out of it, and share your thoughts over the next few days. I look forward to this discussion. Britton

12 30

YT field detection
by Douglas Harvey Rudd 30 Aug '14

30 Aug '14

Hi all I have a set of fixes for Issue #887 (field detection). Most apply without breaking any tests, but to address all of the problems in that issue I need to add a check to YTDataContainer._determine_fields that doesn't allow a field to be added if it doesn't exist in field_list or derived_field_list. This fixes my bug, but breaks a test based on code in yt/visualization/volume_rendering/camera.py which adds a field "temp_weightfield" (not a tuple!). When this field is passed to _determine_fields, _get_field_info first guesses at ("gas","temp_weightfield"), but since that doesn't exist in field_info, it replaces it with ("index","temp_weightfield"), which I don't see as being any better. There are several places where the syntax _get_field_info(*field) is used, which will break when field is a string, not a tuple. Should I fix camera.py to use an explicit fluid type? If so, what? I've tried to use the fluid type of the input field, but that apparently is also not always a tuple. e.g. fields = [field] if self.weight is not None: self.weightfield = (field[0], "temp_weightfield") # This is a temporary field, which we will remove at the end. def _make_wf(f, w): def temp_weightfield(a, b): tr = b[f].astype("float64") * b[w] return b.apply_units(tr, a.units) return tr return temp_weightfield ds.field_info.add_field(self.weightfield, function=_make_wf(self.field, self.weight)) # Now we have to tell the dataset to add it and to calculate # its dependencies.. deps, _ = ds.field_info.check_derived_fields([self.weightfield]) ds.field_dependencies.update(deps) fields = [self.weightfield, self.weight] Douglas Rudd Scientific Computing Consultant Research Computing Center drudd(a)uchicago.edu

2 2

Issue #889: yt update --all fails when updating from numpy 1.7 to 1.8 on some systems (yt_analysis/yt)
by Cameron Hummels 28 Aug '14

28 Aug '14

New issue 889: yt update --all fails when updating from numpy 1.7 to 1.8 on some systems https://bitbucket.org/yt_analysis/yt/issue/889/yt-update-all-fails-when-upd… Cameron Hummels: Since the addition of Pull Request #1168, which updates the version of numpy from 1.7 to 1.8, some attempts to update yt's dependencies in place fail with this error: ``` #!python ImportError: cannot import name multiarray ``` See here for full traceback: http://paste.yt-project.org/show/5077/ This appears to be a problem on mac systems and TACC Stampede, and perhaps others. One solution to this is to wipe yt and reinstall from the install script, but a different solution is desired.

1 0

Particle-Only Plots
by Andrew Myers 28 Aug '14

28 Aug '14

Hi folks, A while back, Jeremy Ritter asked the yt-users list if yt could produce a particle plot like that obtained through annotate_particles(), but without loading any mesh data. The answer was "no", but that it isn't too hard to use matplotlib directly. Personally, this is a visualization task that I do all the time, and I think it would be useful to make this type of plot a one-liner in yt - provided it handles units and labels and such automatically like any other yt plot. I've added this feature in my fork of yt. The interface is the same as for any other yt plots, and the standard plot modification mechanisms should work. Examples are here <http://nbviewer.ipython.org/gist/atmyers/f8616c9ed5a9d2b027e8> and here <http://nbviewer.ipython.org/gist/atmyers/8d979d990268f48c9688>. If there is interest in having this in mainline yt I'll write some docs and issue a pull request. -Andrew

3 5

Issue #888: find_max() returning odd values (yt_analysis/yt)
by Brian O'Shea 28 Aug '14

28 Aug '14

New issue 888: find_max() returning odd values https://bitbucket.org/yt_analysis/yt/issue/888/find_max-returning-odd-values Brian O'Shea: I am using yt-3, and am running find_max() on a series of Enzo cosmological datasets (see link to z=0 dataset below). Using the script linked to below, The field matter_overdensity produces unreasonably large values (~1e+40 - 1e+302, unitless) for later data outputs (see output linked to below), when the script is run in parallel using the command "mpirun -np 16 python dmdensity.py --parallel". The field "density" produces sensible results in parallel, and both matter_overdensity and density produce sensible results when run in serial. script: http://paste.yt-project.org/show/5073/ output: http://paste.yt-project.org/show/5074/ z=0 dataset: http://galactica.pa.msu.edu/~bwoshea/data/datasets/RD0006.tar

1 0

Issue #887: Asking for naked ARTIO particle field type corrupts field detection (yt_analysis/yt)
by Douglas Rudd 27 Aug '14

27 Aug '14

New issue 887: Asking for naked ARTIO particle field type corrupts field detection https://bitbucket.org/yt_analysis/yt/issue/887/asking-for-naked-artio-parti… Douglas Rudd: Field detection shows strange behavior when asking for a field without a particle type when that field doesn't exist for all types. - Example script: http://paste.yt-project.org/show/5071/ - Example output: http://paste.yt-project.org/show/5072/ Explanation: I load a dataset with STAR type particles which has the unique field "BIRTH_TIME" - ("STAR","BIRTH_TIME") in ds.field_list == True - ("all","BIRTH_TIME") in ds.field_list == False If I request ["all","BIRTH_TIME"], I correctly receive the exception ``` #!python Could not find field '('all', 'BIRTH_TIME')' in AGORA_LOW_1.art. ``` If I request ["BIRTH_TIME"], it promotes this to ("all","BIRTH_TIME"), but then proceeds to chunk and attempt to load this non-existent field, leading to the exception (after I/O!): ``` #!python Something has gone terribly wrong, _function is NullFunc for ('STAR', 'BIRTH_TIME') ``` Subsequent requests for ("all","BIRTH_TIME") then produce this latest exception, showing that field detection is caching some bad information somewhere. I've tracked through the entire line of execution (I originally included it but it's so crazy and convoluted I think it's only useful for me). I've found several things that look like bugs, and I'll submit a PR with some "fixes." Since I don't know this part of the code it will need some review.

1 0

Deposited particle velocities
by Nathan Goldbaum 27 Aug '14

27 Aug '14

Hi all, I'd like to setup deposition fields for particle velocities. This means doing a mass-weighted nearest neighbor or cloud-in-cell deposition of the particle velocities in a grid dataset. I'm using this to analyze the stellar velocity dispersions in a simulation, so it's much more convenient to have the particle velocities interpolated onto a grid to calculate the local velocity dispersion using a 3D convolution. Does anyone have an example of setting up a particle deposition field? If not, where should I look in the source? Thanks for your help! -Nathan

2 4

Proto-announcement for dev workshop
by Matthew Turk 26 Aug '14

26 Aug '14

Hi all, I wanted to write to the list to let you all know that I'm looking at having a dev workshop at NCSA on October 16 and 17, with overflow onto the 18th for whoever wants to have a weekend here. This will follow on the heels of a Blue Waters workshop on the 13-15. I'll be following up with some more information when I get it, including a "proper" announcement, but I thought I'd send out something of a save-the-date now. If there's some reason that this definitely works, or definitely *doesn't* work for you and you really want to go, please let me know off-list and we'll see what we can work out. Also, there will be a barbecue at my place. So please, someone remind me to buy a barbecue. And to learn how to use it. (Maybe I should file a ticket on bitbucket.) -Matt

1 0

field list
by Michael Zingale 26 Aug '14

26 Aug '14

Where is the script that generates the field list for the docs: http://yt-project.org/docs/3.0/reference/field_list.html The boxlib field list is incomplete, and I'd like to take a hack at adding the code-specific fields. Mike -- Michael Zingale Associate Professor Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 *phone*: 631-632-8225 *e-mail*: Michael.Zingale(a)stonybrook.edu *web*: http://www.astro.sunysb.edu/mzingale

2 2