Mailman 3 January 2013 - yt-dev

RFC: SPH plan in yt 3.0
by Matthew Turk Jan. 17, 2013

Jan. 17, 2013

Hi all, I am writing today to request comments and suggestions for supporting SPH data in yt. This is part of a broader effort -- which includes the native Octree support -- in yt 3.0 to correctly support non-gridpatch data. This email contains a lot of information about where we are, and I was hoping to get some feedback from people who have more experience with SPH data, Gadget, AREPO, etc. In particular, (if they're around and have the time) it'd be great to hear from Marcel Haas, … [View More]Michael J. Roberts, and Chris Moody. Additionally, I am very interested in approaching this as an iterative, incremental process. Minimum Viable Product (MVP) first, then improving as we learn and go along. = Background = As seen in this pull request: https://bitbucket.org/yt_analysis/yt-3.0/pull-request/11/initial-n-body-dat… I've implemented a first pass at N-body data support. This proceeds through the following steps: * Read header * Create octree * For each file in output: * Add particles to octree * When a cell crests a certain number of particles (or when particles from more than one file are co-located) refine * Directly query particles to get data. As it stands, for N-body data this is very poor at memory conservation. In fact, it's extremely poor. I have however written it so that in the future, we can move to a distributed memory octree, which should alleviate difficulties. For quantitative analysis of quantities that are *in the output*, this already works. It does not yet provide any kind of kernel or density estimator. By mandating refinement based on file that the particles are in, load on demand operations can (more) efficiently read data. Many of the related items are outlined in the corresponding YTEP: https://yt.readthedocs.org/projects/ytep/en/latest/YTEPs/YTEP-0005.html Currently, it works for OWLS data. However, since there has been something of a Gadget fragmentation, there is considerable difficulty in "supporting Gadget" when that includes supporting the binary format. I've instead taken the approach of attempting to isolate nearly all the functionality into classes so that, at worst, someone could implement a couple functions, supply them to the Gadget interface, and it will use those to read data from disk. Some integration work on this is still to come, but on some level it will be a 'build your own' system for some particle codes. OWLS data (which was run with Gadget) on the other hand is in HDF5, is self-describing, has a considerable number of metadata items affiliated with it, and even outputs the Density estimate for the hydro particles. Where I'm now stuck is how to proceed with handling SPH data. By extension, this also applies somewhat to handling tesselation codes like AREPO and PHURBAS. = Challenges = Analyzing quantitative data, where fields can be generated from fully-local information, is not difficult. We can in fact largely consider this to be done. This includes anything that does not require applying a smoothing kernel. The hard part comes in in two areas: * Processing SPH particles as fluid quantities * Visualizing results Here are a few of our tools: * Data selection and reading: we can get rectangular prisms and read them from disk relatively quickly. Same for spheres. We can also get boolean regions of these and read them from disk. * Neighbor search in octrees. Given Oct A at Level N, we can get all M (<=26) Octs that neighbor A at Levels <= N. * Voronoi tesselations: we have a wrapper for Voro++ which is relatively performant, but not amazing. The DTFE method provides a way to use this to our advantage, although this requires non-local information. (The naive approach, of getting density by dividing the particle mass by the volume of the voronoi cell, does not give good agreement at least for the OWLS data.) Here's what we want to support: * Quantitative analysis (profiles, etc) * Visualization (Projections, slices, probably not VR yet) * Speed == Gridding Data == In the past, one approach we have explored has been to simply grid all of the values. Assuming you have a well-behaved kernel and a large number of particles, you can calculate at every cell-center the value of the necessary fields. If this were done, we could reduce the complexity of the problem *after* gridding the data, but it could potentially increase the complexity of the problem *before* gridding the data. For instance, presupposing we took our Octree and created blocks of 8^3 (like FLASH), we could then analyze the resultant eulerian grid identically to other codes -- projections, slices, and so on would be done exactly as elsewhere. This does bring with it the challenge of conducting the smoothing kernel. Not only are there a handful of free parameters in the smoothing kernel, but it also requires handling edge effects. Individual octs are confined to a single domain; neighbors, however, are not. So we'd potentially be in a situation where to calculate the values inside a single Oct, we would have to read from many different files. From the standpoint of implementing in yt, though, we could very simply implement this using the new IO functions in 3.0. We already have a _read_particles function, so the _read_fluid function would identify the necessary region, read the particles, and then proceed to smooth them. We would need to ensure that we were not subject to edge effects (something Daniel Price goes into some detail about in the SPLASH paper) which may prove tricky. It's been impressed upon me numerous times, however, that this is a sub-optimal solution. So perhaps we should avoid it. == Not Gridding Data == I spent some time thinking about this over the last little while, and it's no longer obvious to me that we need to grid the data at all even to live within the yt infrastructure. What gridded data buys us is a clear "extent" of fluid influence (i.e., a particle's influence may extend beyond its center) and an easy way of transforming a fluid into an image. The former is hard. But I think the latter is easier than I realized before. Recent, still percolating changes in how yt handles pixelization and ray traversal should allow a simple mechanism for swapping out pixelization functions based on characteristics of the dataset. A pixelization function takes a slice or projection data object (which is, in fact, an *actual* data object and not an image) and then converts them to an image. For SPH datasets, we could mandate that the pixelization object be swapped out for something that can apply an SPH-appropriate method of pixelization (see Daniel Price's SPLASH paper, or the GigaPan paper, for a few examples). We would likely also want to change out the Projection object, as well, and we may need to eliminate the ability to avoid IO during the pixelization procedure (i.e., the project-once-pixelize-many advantage of grid data). But I think it would be do-able. So we could do this. I think this is the harder road, but I think it is also possible. We would *still* need to come up with a mechanism for applying the smoothing kernel. = Conclusions and Contributions = So I think we have two main paths forward -- gridding, and not gridding. I would very, very much appreciate comments or feedback on either of these approaches. I believe that, regardless of what we choose to do, we'll need to figure out how to best apply a smoothing kernel to data; I think this is probably going to be difficult. But I am optimistic we are nearing the point of usability. For pure N-body data, we are already there -- the trick is now to look at the data as fluids, not just particles. The best way to contribute help right now is to brainstorm and give feedback. I'd love to hear ideas, criticisms, flaws in my logic, ways to improve things, and so on and so forth. I think once we have an MVP, we will be better set up to move forward. If you'd like to add a reader for your own flavor of particle data, we should do that as well -- especially if you are keen to start exploring and experimenting. The code so far can be found in the repository linked above. I'd also appreciate comments (or acceptance) of the pull request. I am also happy to hold a Hangout to discuss this, go over things, help set up readers or whatever. I consider this a relatively high-priority project. It will be a key feature of yt 3.0. (Speaking of which, today I've allotted time to focus on getting the last major patch-refinement feature done -- covering grids. Hopefully patch-refinement people can test it out after that's done!) Thanks for your patience with such a long email! :) -Matt [View Less]

5 10

data on different AMR levels
by Elizabeth Tasker Jan. 17, 2013

Jan. 17, 2013

Hi, I'm working with Nick Gnedin on looking at the density properties of my galaxy on different AMR levels. For this, we'd like to grab all the cells on one AMR level, divide them so that they have a fixed cell size and output. Is there a quick way of doing this in yt? I *think* the answer will be no, since (for example) covering_grid will grab the best data available to make the mesh, not just the data from a single level. Is that right? We specifically need to avoid interpolation. If so,… [View More]

3 8

3.0 version labeled as '2.5-dev'
by Anthony Scopatz Jan. 16, 2013

Jan. 16, 2013

Hello All, Is there some special reason that the __version__ in yt-3.0 is listed as '2.5-dev'? Am I missing something subtle about the release schedule or is this safe to change? Be Well Anthony

4 8

[yt_analysis/yt] save figure with a given suffix (issue #487)
by Suoqing Ji Jan. 16, 2013

Jan. 16, 2013

--- you can reply above this line --- New issue 487: save figure with a given suffix https://bitbucket.org/yt_analysis/yt/issue/487/save-figure-with-a-given-suf… Suoqing Ji: Warning will appear if, say, SlicePlot.save('1.pdf') yt : [WARNING ] 2013-01-16 04:12:20,035 Unknown suffix pdf, defaulting to Agg One way to solve is change the last line: def get_image_suffix(name): suffix = os.path.splitext(name)[1].lstrip('.') return suffix if suffix in ['png', 'eps', 'ps', 'pdf'] else ''… [View More]

1 0

Announcing the 2013 yt developer conference - March 6-8, Santa Cruz California
by Nathan Goldbaum Jan. 15, 2013

Jan. 15, 2013

Dear Colleagues, We are proud to officially announce the upcoming 2013 yt developer workshop. This three day event, to be held on the campus of the University of California, Santa Cruz on March 6th through 8th, will bring together a diverse group of students, researchers, and developers. yt is a community-developed analysis and visualization toolkit, primarily directed at astrophysical hydrodynamics simulations, offering full support for output from the Enzo, FLASH, Orion, and Nyx codes, … [View More]with preliminary support for several others. It provides access to simulation data using an intuitive python interface, can quickly and easily perform many common visualization and analysis tasks, and offers a framework for conducting data reductions and analysis of simulation data. More details about yt, including download and installation instructions as well as full documentation are available on our website: http://yt-project.org. Up to this point, yt development has proceeded largely over the internet, leveraging e-mail lists, our online code repository (http://bitbucket.org/yt_analysis), and IRC to track, discuss, and evaluate changes to the code. Unfortunately, this model makes it difficult to jump in to development as a newcomer. This workshop will not only allow experienced developers to collaborate in person on new features, but will also be a means for new developers to learn what yt is about and begin contributing. The workshop will include several training sessions, constituting a primer in distributed version control, test-based development, and best practices for scientific programming. The workshop will also serve as a venue to plan and execute some of the major new features we are planning for the upcoming yt 3.0 release, including full support for outputs of Lagrangian codes like Gadget, Gasoline and AREPO, better support for oct-based codes like ART and RAMSES, initial conditions generation, the new Grid Data Format, a new way of handling units and unit conversion, non-cartesian geometries, and advanced graphical browser widgets inside the iPython notebook. More information and a registration form for the workshop are available on the workshop website: http://yt-project.org/workshop2013/. We have limited funding support for hotels and airfare. Funds will be preferentially distributed to students and the level of individual support will depend on demand. If you have questions or concerns about the workshop, please feel free to contact the organizers at workshop2013(a)yt-project.org. On behalf of the organizing committee, Nathan Goldbaum [View Less]

1 0

kwargs for SlicePlot and ProjectionPlot?
by Andrew Myers Jan. 14, 2013

Jan. 14, 2013

Hi Folks, I was wondering if there are any objections (particularly from Nathan) to adding an optional **kwargs argument to SlicePlot and ProjectionPlot, so that we can pass in field parameters like this: pw = SlicePlot(pf, 2, 'GravitationalForceMag', center=c, width=(1e4, 'au'), particle_data=pd) instead of the (very slighty) more clunky: slice = pf.h.slice(2, c[2], ['GravitationalForceMag'], particle_data=pd) pw = slice.to_pw('GravitationalForceMag', center=c, width=(1e4, 'au')) I think … [View More]

3 3

[yt_analysis/yt] Callbacks and OffAxisProjectionPlot (issue #486)
by Nathan Goldbaum Jan. 12, 2013

Jan. 12, 2013

--- you can reply above this line --- New issue 486: Callbacks and OffAxisProjectionPlot https://bitbucket.org/yt_analysis/yt/issue/486/callbacks-and-offaxisproject… Nathan Goldbaum: A number of callbacks don't work using OffAxisProjectionPlot. This is mostly because OffAxisProjectionPlot is a bit of a hack, as it wraps the volume render and camera by creating a dummy data object that doesn't inherit from AMRData. In principle we should be able to get the callbacks working either by … [View More]

1 0

Module search path in iyt
by Andrew Myers Jan. 12, 2013

Jan. 12, 2013

I recently updated my yt installation, and I noticed that the current working directory no longer gets added to the module search path when running yt in interactive shell mode. So, if "StarParticles.py" is a file in my current working directory, I can't just type import StarParticles to import that code as a module. I fixed this in my installation by adding: import sys sys.path.insert(0, ' ') to the top of the bin/iyt script, but not sure what the right way to do this more generally is, so … [View More]

2 1

pmods
by Stephen Skory Jan. 12, 2013

Jan. 12, 2013

Hi all, has anyone tried "from yt.pmods import *" lately? I'm getting this error: http://paste.yt-project.org/show/3053/ Interestingly, if I "import h5py" prior to "from yt.pmods import *" things appear to go fine. I get this error with both h5py 2.1.0 (which is what's in the install script) and the latest 2.1.1. -- Stephen Skory s(a)skory.us http://stephenskory.com/ 510.621.3687 (google voice)

2 2

Fwd: [yt_analysis/ytep] YTEP-0006: Periodicity (pull request #6)
by Matthew Turk Jan. 10, 2013

Jan. 10, 2013

Before the new year there was some discussion of periodicity. I've outlined a first pass at a YTEP for improving periodicity. Any long-form discussion can occur here, but please feel free to leave comments in the YTEP itself. ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Thu, Jan 10, 2013 at 2:52 PM Subject: [yt_analysis/ytep] YTEP-0006: Periodicity (pull request #6) To: yt(a)enzotools.org A new pull request has been opened … [View More]

2 1