Mailman 3 March 2010 - yt-dev

Multivariate Volume Rendering
by Matthew Turk 16 Apr '10

16 Apr '10

Hi all, Over the last little while, the volume renderer has really taken off in popularity! I'm a bit conflicted about this, since I'd rather people were super duper into things like phase plots, radial profiles and clump finding, but it's also good to know people are using yt for volume visualization as well as other types of analysis. I used it myself to what I think was great effect at a conference last week, and I hope that other people are getting similar use out of it. Anyway, a couple months ago John wrote to yt-users and suggested that we try doing multivariate volume rendering, similar to what's in Kaehler, Wise, & Abel. For a while mostly nothing happened, but over the Friday->Sunday, Jeff and I hammered it out and it's now working. Jeff took care of the emission spectra (with some assistance from John's RGB code), I took a whack at adding multiple variables to the partitioned grids and the volume rendering itself, and we can now do multivariate rendering -- although absorption still needs some work, which I'll discuss below. The new mechanism for grid partitioning only returns the indices into the vertex-centered data. The actual processing of the vertex-centered data. This means the actual locating of sweeps is much faster -- right now we don't take advantage of that, but I can see it being useful in the future. A few weeks ago I wrote a distributed object collection so that we could do 3D domain decomp to get the bricks, then pass them around between processes for the actual volume rendering, which was parallelized via 2D image plane decomp into decomposed inclined boxes. Now the 3D decomp will only pass around single arrays that describe the indices into the bricks, and then each image-decomp processor will generate and slice up the data as necessary -- this isn't yet working, but it's a clear process and I'll try to take care of it soon. Transfer functions now have several more parameters, and I'm working to convert the old-style transfer functions to this. The idea is that now you add field interpolation tables which correspond to a specific field, link them to channels (R, Ralpha, G, Galpha, B, Balpha), and optionally have their values weighted by other fields. So for an example, here's how the Planck function sets up its various channels: http://hg.enzotools.org/yt/file/ede5d76cddd4/yt/extensions/volume_rendering… (there's a bit of cruft where I tested adding lines for density absorption.) It creates a TransferFunction, links it to a field (0) and a weighting field (2), then links it to channels. There's a lot of complexity in this, because it's a complex thing and we want to enable very complex behavior, but for the most part it will be hidden in the ColorTransferFunction, which will abstract this all away. Absorption works slightly differently now. Because we do back-to-front integration, I believe we do not have to store the accumulated opacity -- so while right now the volume renderer has 6xNxM arrays, I think it should just be 3. Additionally, we're presented with the radiative transfer equation: dI/ds = -alpha * I_0 + j Here j would be our emission at a given sample, I_0 is the current value of the image plane, and ds is the distance between samples. Since we're doing back-to-front, I believe we are directly solving this, and so our alpha fed in directly corresponds to the absorption. However, because ds is so low, careful calibration between alpha and emissivity is necessary. Alternately, converting everything back to cgs is probably not difficult... (approximate) Scattering should be easy to put in, it just hasn't been yet. However, that's neither here nor there. I believe that some exciting things can be done with this (Britton and Jeff and I have chatted about line transfer as well as LyA maps) and I think it's rapidly converging on a workable implementation. I made up some images with it. The first is a sample dataset of John's with an HII region: http://yt.enzotools.org/attachment/wiki/Screenshots/hiiregion_rgb.png The second is one of my protostars from my thesis (I had to aggressively clip the image here) just at around 1.5e4K. This image is 3AU on a side: http://yt.enzotools.org/attachment/wiki/Screenshots/protostar_rgb.png And, just for fun, I got the transfer function widget to steal the camera position from the VTK interface, so I made up an image of the two playing nicely together: http://yt.enzotools.org/attachment/wiki/Screenshots/vtk_tf_vr.png Anyway, that's my status update. All of the MultiVariate stuff went in the "vr-multivariate" branch, so feel free to take a look. (You'll probably have to re-cython.) An example script is here: http://paste.enzotools.org/show/367/ I'll shoot another, much shorter, email to the list when it's working and merged back into the main hg branch. We'll then have a parallel software volume renderer that is fully multi-channel, multi-variate, and comes with black body transfer functions. All of this is only achievable thanks to the development community we have, and our team efforts. So, thank you all for your hard work and your enthusiasm. -Matt

2 3

Call for assistance: Ghost Zones
by Matthew Turk 31 Mar '10

31 Mar '10

Hi all, I need some help with the ghost zone generation. Right now it's too slow, because it performs a cascading interpolation from the lowest levels to the highest levels, filling in available cells, then interpolating to the next level and repeating the process. Looking over the Enzo source code, I think we can do better: I think we should be able to replicate the Enzo logic much more closely, which only implicitly requires cascading interpolation, by checking parents and sibling grids. The issue of Subling grids I think we will also need to address. The current code is in AMRIntSmoothedCoveringGridBase. It's not that pretty. Does anybody have any ideas on whether or not we can do this better than it currently is done? Getting vertex centered data requires the ghost zones, and right now getting ghost zones is *easily* the most time consuming step of the volume rendering for many datasets. Dave, John, I know you've both dealt with this issue but if anybody has any ideas, they'd be greatly appreciated! -Matt

3 3

Cool Merger Tree PDF
by Stephen Skory 31 Mar '10

31 Mar '10

Hi all, I've attached a cool merger tree I just made up while doing some testing of the Merger Tree code. This is the tree for the most massive halo at z=0 for a recent simulation. Time goes forward from top to bottom, and you can see the redshift in the small black boxes on the right of each row. The colors of the boxes are from green to red, smallest to largest at each snapshot (so color doesn't always mean the same mass). The arrows show the parent-child relationship, and the percentages next to the arrow is how much mass of the parent went to the child. The halo info is inside each box, with the particle mass (Msol) as calculated by the halo finder and the CoM position. If you look carefully, you can see cases where a halo took more than one pass to fully integrate into the most massive halo. _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

3 4

Release 1.7
by Matthew Turk 29 Mar '10

29 Mar '10

Hi all, I've uploaded a dev build of the docs for 1.7 here: http://yt.enzotools.org/doc/dev_build/ The changelog is here: http://yt.enzotools.org/doc/dev_build/changelog.html#version-1-7 I think this will be the last in the 1.x series. I've made good progress on re-organizing (can almost import yt!) in the yt-reorg hg repository, which I encourage you to check out and browse, but I warn it's not yet usable. I'll be adding in some compatibility layers to catch some common usages of lagos, raven, etc. Once the unit tests pass I'm going to merge it into the main hg branch, at which point I'll need some help trying to break it. That will then become the SVN trunk and then 2.0, which will be mostly free of clever names and be structured more logically and clearly. Stephen has again stepped up and done a lot of the features going into 1.7, and I am happy that we're going to close out 1.x on a good note. I think we should aim for 1.7 in a couple weeks, and then give maybe a week buffer time before moving the reorganized source into svn trunk. I think Stephen's working on getting a better (or at least different) kD-tree implementation into the source tree and testing his structure function code before this release goes out. -Matt

2 1

possible bug in mpi get_data
by John Wise 29 Mar '10

29 Mar '10

Hi, I think I found a little bug when calling get_data() in parallel jobs. I was trying to make basic slices of Density, Temperature, and Electron_Fraction. When it gets to the Electron_Fraction slice, it already had the data in memory, so temp_data == {}. I'm not sure why it was already in memory, I hadn't accessed Electron_Density yet. Perhaps it was loaded from the .yt file? Then the routine tries to accumulate temp_data with _mpi_catdict(), and this is where things go wrong because temp_data is empty! An if-statement before this call avoided this bug, diff -r c248cd4313a8 yt/lagos/BaseDataTypes.py --- a/yt/lagos/BaseDataTypes.py Sun Mar 28 23:31:52 2010 -0700 +++ b/yt/lagos/BaseDataTypes.py Mon Mar 29 12:47:20 2010 -0400 @@ -593,7 +593,8 @@ # Now the next field can use this field self[field] = temp_data[field] # We finalize - temp_data = self._mpi_catdict(temp_data) + if temp_data != {}: + temp_data = self._mpi_catdict(temp_data) # And set, for the next group for field in temp_data.keys(): self[field] = temp_data[field] I just wanted to make sure that this is the right thing to do before committing. John

2 1

Enzo Users' Workshop - June 28-30, 2010 (San Diego, CA)
by Matthew Turk 22 Mar '10

22 Mar '10

Dear Colleague, Please pass this email along to any researchers that you feel may be interested, and apologies if you receive multiple copies of this message. The Enzo development consortium is pleased to announce the first Enzo Users Workshop, which will take place at the San Diego Supercomputer Center on June 28-30, 2010. This workshop will be take place shortly after the release of Enzo 1.6, which will have several new features (including MHD and radiation transport), and will be targeted at both new and experienced Enzo users. We will present tutorials on both beginning and advanced Enzo subjects, including running existing simulation types, developing new problem initialization routines, and data analysis and visualization using the 'yt' analysis package. Several experienced Enzo users and developers will be available to help users with both beginning and advanced questions. The website for the workshop is http://lca.ucsd.edu/workshops/enzo2010 , which contains a mailing list sign up, draft schedule, and accommodation details. We will be able to partially defray accommodation costs for attending students. We hope to see you there! Sincerely, Matthew Turk, Michael Norman, and Brian O'Shea (on behalf of the Enzo development team)

1 0

projection plotting tools
by Britton Smith 19 Mar '10

19 Mar '10

Hey guys, Again stealing Sam Skillman's sense of style, I wrote up some single and multiple panel image functions for making nice looking images from fixed resolution projection data saved to hdf5 files. Color bar and tick label locations are easily settable (top, bottom, left, right). I put a few example images here for you to check out: http://casa.colorado.edu/~brittons/sample_images/ The plotting functions are here: http://paste.enzotools.org/show/374/ The script I used to make the sample images is here: http://paste.enzotools.org/show/375/ Let me know if you have any comments. Is it worth it to have something like this in yt itself? If so, we might also want to add a simple hdf5 writer for fixed resolution projections. Britton

3 2

proposition to close svn-trunk to active development
by Britton Smith 17 Mar '10

17 Mar '10

Hi guys, It seems that most of yt development has migrated away from the svn-trunk to the various mercurial branches. Like most of you guys, I imagine, I am now using the main hg branch of yt almost exclusively, since that's where everything new that I need gets put right away. It's rare, but every now and then something goes straight into the svn trunk without passing through hg. Just to make it official, I would like to propose that the svn-trunk not be used for any active development, but only a staging area for updates coming from the hg branch and destined for a release version. This way, we don't run into the issue of having to merge changes in two directions between svn and hg, which I think is a bit of a pain. I get the impression that almost all development is taking place in hg already, but I think it may be beneficial to have an official policy on how this sort of thing should be done. Thoughts, anybody? Britton

4 6

Interesting paper on volume remappings
by Matthew Turk 17 Mar '10

17 Mar '10

Hi guys, There was a paper on the arxiv today, http://arxiv.org/abs/1003.3178 about remapping cubical, periodic volumes into non-cubical volumes -- suitable for things like light cones and whatnot. The group provides code as well as further description here: http://mwhite.berkeley.edu/BoxRemap/ I wonder if this could be used as an enhancement in the light cone extension? Britton, Eric, what are your thoughts? -Matt

3 3

Structure Function Generator
by Stephen Skory 13 Mar '10

13 Mar '10

Hi all, I have just added to the hg-devel branch a parallel (in memory and workload) structure function generator. It's very alpha in the sense that I haven't even really done any comparisons to other generators, so it may be completely wrong. It is not be trusted right now. The results I have got in testing are reasonable, so if it is wrong, it's not in a big way. I've written some documentation for it, and there is a simple example in here: http://hg.enzotools.org/yt-doc/file/c1f1442dc71e/source/extensions/struct_f… There can be multiple structure functions operating at once, and the structure functions are completely flexible. You can use defined fields in yt, and output multiple outputs, meaning you can make multidimensional structure functions. You can define density cutoffs for the structure function points as well by putting that into your function. The analysis region can be the entire volume, or a subregion you define. It uses very little blocking global MPI, so it should be fairly scalable, but I haven't done any big tests, so I really don't know. Even though I just wrote software to do this, I still find the whole structure function stuff confusing. So if the documentation isn't clear enough, can you help me out and say what could improve it? And if you get the chance, or you need it, please try running it, and let me know how it goes. _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

2 2