Mailman 3 July 2012 - yt-dev

Why do halo objects need their particles?
by Matthew Turk July 12, 2012

July 12, 2012

Hi all, This week at the HIPACC Summer School, Chris Moody and I have been looking at finishing up Rockstar (rockstar.googlecode.com) integration. The current status is that Rockstar *works*, and Chris has added on TimeSeries support for it, but it alas only works for single-mass simulations. (The author of Rockstar has provided some suggestions about how to add multi-mass support.) Rockstar does have an interesting method of calculating halo properties. Specifically, you add in variables and whatnot to a struct and then insert your own routines into the C code, recompile, rerun, and get additional results. The way we've set up Rockstar integration, we could actually supply a generic Python function, and have it fill in a set of values that are then calculated on a halo. I think this is where we want to go with the Rockstar stuff -- you specify a set of functions you want calculated on a halo, these get calculated, you get back the halo object and the particles are flushed from memory. What this brings up in my mind is, why do we hold on to the particles at all? When you run the halo finders in yt, the returned halo objects keep all the particles that belong to each halo. This lets the user specify, later on, which analysis to run. But, it also leads to a *lot* of memory management, parallel-distribution of objects (in a way that's different than we do now), and really I think doesn't add a *huge* amount of value. Nearly all the remaining analysis that gets performed either requires you to dump the particle IDs to disk anyway (merger trees) or could be performed for very little cost inline. So if we think about this the way Christine, Stephen and everyone was talking about the other day, imagine you had a halo finder that was run like this: catalog = HaloFinder(ts, analysis = ["CenterOfMass", "MaximumRadii", "EllipsoidalParameters"]) The halo finder would go out, perform the analysis on each dataset in the time series, and then return to you the full tree. But where this would be different than before is that the halos would have the properties calculated ahead of time, and the particles themselves would be gone. They'd be far more manageable, and we could discard nearly *all* of the complexity in the existing halo objects. You could write your halo analysis routines *nearly identically* to the way you'd write your field definitions: def MaximumRadii(halo, pf): dx = halo["x"] - halo.properties["CenterOfMass"] ... return max_r The downside would be that to add on any analysis that requires *raw particle data*, you would either have to write out the particle IDs or information to disk, or re-run the halo finder. But honestly, I think this would probably be just fine to do. And the vast improvements to the infrstructure would probably be worth it. For instance, we would probably no longer need to subclass the halo objects, and we would be able to get rid of "claiming" on a processor-by-processor basis. Thoughts? Stephen, what do you think? -Matt

3 2

Issue with PlotWindow and Callbacks
by John Zuhone July 12, 2012

July 12, 2012

Hi all, I've found an issue with PlotWindow and callbacks. The example script I have included adds velocity vectors on top of a slice: http://paste.yt-project.org/show/2544/ Later on in the script, I try to resize the plot, but I get this error: http://paste.yt-project.org/show/2545/ This is because the field list now includes x-velocity and y-velocity, but there are not any plots corresponding to these fields in the list. I figure that the answer to this is to retain some sort of field list that corresponds to the fields that are actually being plotted in the colorscale, but I figured before I just went in and dropped it in somewhere I'd ask for opinions as to where it should go. Best, John Z -- John ZuHone Postdoctoral Researcher NASA/Goddard Space Flight Center jzuhone(a)gmail.com john.zuhone(a)nasa.gov

4 7

6000 commits
by Matthew Turk July 12, 2012

July 12, 2012

Hi all, I just noticed in IRC that we crested 6000 commits to the yt_analysis/yt repository. Congrats everyone! Ohloh also updated our project today: https://www.ohloh.net/p/yt_amr -Matt

3 2

cannot do from yt.mods import *
by Geoffrey So July 11, 2012

July 11, 2012

Hi, I just installed the latest dev as of 30 minutes ago or so, but I get the following when I do >from yt.mods import * in python. Traceback (most recent call last): File "proj.py", line 1, in <module> from yt.mods import * File "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/mods.py", line 98, in <module> from yt.frontends.art.api import \ File "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/frontends/art/api.py", line 31, in <module> from .data_structures import \ File "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/frontends/art/data_structures.py", line 42, in <module> from .fields import \ File "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/frontends/art/fields.py", line 39, in <module> import yt.utilities.amr_utils as amr_utils ImportError: No module named amr_utils Is my installation broken or is there something wrong with the latest dev yt? From G.S.

5 8

Fwd: [yt_analysis/yt] Load Text Halos (pull request #192)
by Nathan Goldbaum July 10, 2012

July 10, 2012

This is in response to the discussion on the list about halos. -------- Original Message -------- Subject: [yt_analysis/yt] Load Text Halos (pull request #192) Date: Tue, 10 Jul 2012 22:25:16 -0000 From: Stephen Skory <pullrequests-noreply(a)bitbucket.org> To: goldbaum(a)ucolick.org A new pull request has been opened by Stephen Skory. sskory/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/192/load-text-halos Title: Load Text Halos This adds the ability to read in simple text files listing the locations and radii of halos, with optional supplementary data fields. For now the supplementary data fields are assumed to be floats, but eventually (hopefully) this will be changed so that string/integer/other fields can be read in. Perhaps with a re-write/upgrade of halo machinery. The supplementary data is accessed off a halo object using the .supp dict {{{ #!python from yt.analysis_modules.halo_finding.api import * h2 = LoadTextHaloes(pf, "Halos.out", {'x':7,'y':8,'z':9,'r':13, 'm':1}) h2[0].supp['m'] # 1357151571000000.0 h2[0].total_mass() # 1706602465579130.2 }}} The first mass is read from the file and is what the halo finder finds; the second is the total DM mass contained in the sphere, which is why the latter is bigger. The x, y, z, and r columns are required, of course. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

1 0

Docs Proposal
by Sam Skillman July 10, 2012

July 10, 2012

Hi all, We had a fairly good discussion in the docs hangout about how the structure of the docs might be improved. What we've come up with is a reorganization of the front page of http://yt-project.org/doc/index.html to have the Overview paragraph, followed by a table of contents entitled "User Guide.” This is what the structure would look like: - User Guide (See full Table of Contents → link to full, or toggle full/collapsed to 2? levels) - “Getting Started” - Installing and updating yt - Cookbook - Field List - FAQ - Analyzing Data - Loading Data - Enzo Data - Using and Manipulating Objects and Fields - What Types of Fields are there? - - etc - Visualizing Data - etc - Advanced - Getting Involved - Other Versions: - Latest Docs Build - Current Release (2.4) - yt-project.org/docs/2.1 - yt-project.org/docs/2.2 - yt-project.org/docs/2.3 - yt-project.org/docs/2.4 Here's a GDoc with this layout, as well as some responsibilities in pushing out 2.4: https://docs.google.com/document/d/1gwqrzDbsqKz-V1J0HMcaptPcy0ETsmeVkWKzWII… Sam, Britton, John Z., Nathan

2 2

text halo supplementary data
by Stephen Skory July 10, 2012

July 10, 2012

Hi all, I didn't read my email until this morning, when I saw the text halos thread had exploded. I had no idea how strongly people felt about halos! Here is a proposal for a solution that I think reduces confusion and also adds capability. Using Christine's request as an example, what I would do is add to the LoadTextHaloes a supplementary data column dict: pf = load("data") halos = LoadTextHaloes(pf, "file.txt", columns = {'x':0, 'y':1, 'z':2, 'r':3}, supp = {'m':4}) And for each halo object there would be a supp dict hanging off of it: h0 = halos[0] h0.supp['m'] This means that h0.total_mass() will correspond to the sphere, while the 'm' above to whatever's in the text file. This supp dict could be used to store temporary data for each halo object outside the LoadTextHaloes function, and if I hear that it might be useful, it could be included in a halo.dump() save. One issue is what happens to the supp columns? Do I float() them, or do I keep them strings, which allows users to convert them if needed? If I keep them strings, 'type' string columns would be useful. Or do I have yet another dict that specifies type in the call to LoadTextHaloes? -- Stephen Skory s(a)skory.us http://stephenskory.com/ 510.621.3687 (google voice)

4 5

Prep for 2.4
by Matthew Turk July 10, 2012

July 10, 2012

Hi all, A few of us have been working on documentation in preparation for releasing yt 2.4. I believe docs are the major change at this point, and once we have those in line we can pull the trigger. The items below still need to be handled, and we'd really appreciate anybody that wants to pitch in. The doc repository is at https://bitbucket.org/yt_analysis/yt-doc . Please feel free to add items, edit existing docs, remove cruft, and whatever else seems appropriate. 1) Every single function or class imported in yt.mods needs to have at least a simple docstring that summarizes what it does. I'd prefer these all be of the numpy docstring standard (see docs/example_docstring.rst) Can a few volunteers examine the stuff in yt/mods.py and see if they can help out adding docstrings? 2) What used to be "cookbook examples" in the docs are being transformed into "simple scripts," and cookbook scripts are being pulled out and put in a different place (you can see this in the yt_analysis/cookbook repo, which will have something like this: http://yt-project.org/cookbook/ ) . The simple scripts are being converted to use the more complex datasets from the workshop datasets. Each will then be included manually. Nathan and I are handling these. If anyone wants to take a crack at adding more *simple* scripts to both source/cookbook/ as .py files, that would be helpful. We could use things like "getting enclosed mass" and "how to save out profiles to disk" and "setting colorbars" and whatnot, but nothing too complex. 3) The workshop page still has to be added -- this is basically a version of this section: http://yt-project.org/workshop2012/#workshop where we have a list 4) The narrative docs should be converted to reference PlotWindow, rather than PlotCollection (wherever possible, although for now callbacks do not work with PlotWindow). I think Nathan might tackle this. 5) The Volume Rendering docs need to be updated to be, well, in line with current best practices. 6) Low-level data inspection docs should be added. 7) ...anything else? = Here are the remaining tickets: https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&mileston… and the current draft of the changelog: http://paste.yt-project.org/show/MfdV3E8s2H7pc1pC3mVp/ . If you have additions, please feel free to reply with them! I'm going to prepare some (html) slides that lay out new features, describe changes in behavior, and have examples. Does anyone have any comments or suggestions? -Matt

4 11

Callbacks for the Plot Window
by Nathan Goldbaum July 9, 2012

July 9, 2012

Hi all, Matt and I have been thinking about how to port the already existing callbacks to the new Plot Window plotting interface. We've come down to two choices that we'd like to the put to the list to vote on. The first choice is to leave the interface exactly the same and use dict-like access through a .modify member that hangs off the PlotWindow object. So, for example, if you wanted to add grids to a plot, it would look something like this: >> slc = SlicePlot(pf, 0, 'Density') >> slc.modify['grids']() Alternatively, the callbacks could hang off of the plot window object as attributes: >> slc = SlicePlot(pf, 0, 'Density') >> slc.annotate_grids() Personally, I prefer the second choice since it leaves the list of annotations available at runtime. In ipython, for example, I could say slc.annotate<tab> and then get a list of all of the available plot modifications for this object. Both choices are about equally complicated syntactically since the attributes can be dynamically generated based on a list of available callbacks in plot_modifications.py. If you have an opinion, please reply with a vote for your preferred option. Cheers, Nathan Goldbaum

4 4

Fwd: [yt_analysis/yt] Updating the command line tool to use yt plot. (pull request #187)
by Matthew Turk July 6, 2012

July 6, 2012

Nathan notes in here that moving to PlotWindow means we can't use the callbacks (grid, time) from the command line for a bit. I will attempt to address this, but even if we can't, I think it's okay for now. If anyone thinks we should reconsider, please be heard! ---------- Forwarded message ---------- From: Nathan Goldbaum <pullrequests-noreply(a)bitbucket.org> Date: Fri, Jul 6, 2012 at 7:16 PM Subject: [yt_analysis/yt] Updating the command line tool to use yt plot. (pull request #187) To: matthewturk(a)gmail.com A new pull request has been opened by Nathan Goldbaum. ngoldbaum/yt-ngoldbaum has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/187/updating-the-command-… Title: Updating the command line tool to use yt plot. I had to temporarily disable the callbacks from the command line. @MatthewTurk agrees that this is ok as a temporary measure until we can port the callbacks over to the plotwindow. If anyone is against this change I can hold off until after yt 2.4. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

1 0