Mailman 3 June 2012 - yt-dev

Photon fields
by John ZuHone 15 Jun '12

15 Jun '12

Hi all, Just wanted to sort of poll all of you on an idea I've been kicking around. I don't know how useful it will be for non X-ray people, but I thought I'd give it a shot. There are a number of tools out there designed to create synthetic X-ray observations from simulations, both AMR and SPH. Of the tools adapted for AMR codes, they tend to be a little clunky and most of them use IDL to extract data. Almost always, gas grid data is regridded to a uniform grid before doing anything with it, making working on large volumes with small resolution prohibitive. PHOX is a tool written by Veronica Biffi and Klaus Dolag to generate X-ray observations from (primarily) Gadget simulations. You can find out about it here: http://www.mpa-garching.mpg.de/~kdolag/Phox/ The pipeline for most virtual X-ray observatories goes like this: 1) Using the gas properties (density, temperature, metallicity, whatever), generate X-ray emissivity fields. 2) Project these fields along the chosen line of sight to give a 2D map of surface brightness. 3) Using this map as a DF for the photons, generate simulated photons which then may be processed later. PHOX's (in my estimation, better) approach is: 1) Using the gas properties (density, temperature, metallicity, whatever), generate X-ray emissivity fields. 2) Generate simulated photons from the emissivity field, assuming a fiducial instrument area and exposure time. 3) Project these photons along a given line of sight, which may be processed later. The nice thing about doing things this way is that you can save the photons to disk to be re-used for a variety of different projections and X-ray instruments. I am writing a tool that uses yt to get data from AMR simulations to generate photon files readable by PHOX. My intention is that this be a standalone tool that I will make publically available, but it got me thinking about the possible usefulness of "photon fields" in yt, akin to the 3D fields of photons that are written to PHOX photon files. Basically, these are akin to particle fields, with the (x,y,z) position of the photon, its (unshifted) energy, and the (vx,vy,vz) velocity of the emitting gas for Doppler shifting. I don't know if the more general concept of these photon fields would be a useful construct to include in yt at some level or not for constructing simulated observations. The main difference between these fields and others in yt is that these entities are slightly more abstract than physical: real photons are of course emitted in a particular direction and received by the observer. But I would guess that this subtle point would not be much of an issue since really the only thing you would be doing with them after generating them would be projecting them along a line of sight. So I guess my idea would be to build in the framework for photon fields, along with capabilities for projecting them and doppler shifting them, while letting the details of the generation of their positions and energies up to the user. Would any of the rest of you or your colleagues find such a thing useful? Thoughts? Best, John Z

4 6

light cone problems
by Stephen Skory 15 Jun '12

15 Jun '12

Hi, I have been messing around with the example light cone generator example for halos: http://yt-project.org/doc/cookbook/index.html#cookbook-light-cone-halo-mask and I have come across a few issues. My guess is that the various things the light cone uses (the halo profiler, FRB) have changed underneath the light cone. I think I've got it working with some changes, but at least one is not a good permanent solution, and I'd like Britton eyes on this since I don't really understand all the details of the LC and I'm not sure that these changes are good. 1. In the example above, the "add_redshift_label" keyword to "project_light_cone()" doesn't appear to exist any more. 2. I've made these changes to the yt source to get things to work: http://paste.yt-project.org/show/2466/ a. The changes to halo_mask.py are hacks. It appears that the halo profiler now adds the overdensity factor to "RadiusMpc" and "TotalMassMsun" (e.g. "RadiusMpc_200"). It seems to me that the correct way to fix this is to pass the halo profiler kwargs down to the _make_slice_halo_list() function where these lines appear so the overdensity factor can be included in these strings. What do you think? b. In light_cone.py I've changed the default of set_parameters to an empty dict, which dict.update() accepts to perform no action, when None makes it croak. c. frb.field_data[] doesn't appear to work. Thanks! -- Stephen Skory s(a)skory.us http://stephenskory.com/ 510.621.3687 (google voice)

2 2

Fwd: [yt_analysis/yt] Volume Rendering Refactor (pull request #168)
by Matthew Turk 15 Jun '12

15 Jun '12

Sam and I chatted, and we'd like to request three people test this and if they all agree, we can accept it. Can three people give it a go? ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Wed, Jun 13, 2012 at 11:12 AM Subject: [yt_analysis/yt] Volume Rendering Refactor (pull request #168) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt-refactor has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/168/volume-rendering-refa… Title: Volume Rendering Refactor This includes all of the changes to the volume rendering system that were discussed here: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/002039.… The main user-facing changes that could cause problems: 1) Cython 0.16 is now required. 2) Off-axis projections no longer interpolate. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

7 17

Transfer Function Plots
by Sam Skillman 10 Jun '12

10 Jun '12

Hi all, I've been experimenting with different ways to plot the transfer functions, and in particular in the context of the InteractiveCamera. What are people's thoughts about this behavior: http://i.imgur.com/CGZ7Z.png http://i.imgur.com/OCsVt.jpg http://i.imgur.com/VNEse.png btw, you can get this behavior by changing your script to call pf.h.interactive_camera, and then running in ipython with a working matplotlib backend. I'll let Nathan bring up his concerns rather than me try to convey them. Sam

8 14

contours on column densiy
by david collins 08 Jun '12

08 Jun '12

Hi, all- What would it take to get the contour finder in yt to work on a projection? d. -- Sent from my computer.

2 1

Rendering & Off-Axis Projection Changes: OpenMP, Opaque Renders, Lighting
by Sam Skillman 08 Jun '12

08 Jun '12

Hi all, Matt and I have been working a bunch in the volume_refactor bookmark in our yt-refactor forks and have added a bunch of functionality and performance improvements. We are now entering a point where we will be issuing a PR soon. At that point it should be tested by as many people as possible to ensure coverage. I’m going to break this down into things that will change behavior and new things. The underlying code in the volume renderer has been refactored, simplified and streamlined. Adding new types of volume rendering should now be much easier, since the act of walking the cells and the accumulation of values in the plane are now completely separated. --Things that will change in some way: *Off-axis Projections: We have implemented a new way of calculating off-axis projections which no-longer requires ghost zones and interpolation. Instead, as the rays traverse the cells, they take the value directly from the value for that cell. This leads to several changes. First, it is way faster! Second, because we aren’t interpolating anymore, that means you won’t get quite as smooth of an image as you may have before. Below are before and after images. Note that the transpose of the image is due to changes in the default behavior of setting up the looking vectors and the writing of the png in write_image. left: old right: new ** The performance gains for a sample (largish) dataset: Old: 186 seconds, including IO New (single-threaded): 20 seconds, including IO If you use 8 OpenMP threads (see later), the new drops down to 10 seconds, with an improvement from 18 to 8 in the section that is threaded. Not perfect scaling, but there’s not a lot of work per cell at that point. *Isocontour/Other Renderings: These may change in their appearance slightly, mostly because of the changes made to allow for opaque and semi-opaque renderings. If you were using transfer functions that have values close to 1.0 for the alpha, then things may start getting opaque. More on this later. -- Things that are new: *OpenMP Threading: We’ve added support for OpenMP in some of the cython files such as grid_traversal.pyx. This means that by default your renderings will run with OpenMP threads. By default it will use the number of cores on whatever computer/node you are on. You can set this by either adjusting the environment variable OMP_NUM_THREADS or using the num_threads keyword in camera.snapshot(). Just be aware of it, and take a gander at top to see yt running at >100%! This will hopefully allow users to run without mpi for rendering on their desktops/laptops. Here’s a snapshot from John running on (presumably) 24 cores: * * ** * * A side effect is that Cython 0.16 is now *required*. This is included in the install_script.sh, which can be re-run at any time over top of an existing installation to update dependencies. *Opaque Rendering: Support for opaque rendering is now possible. This is mostly enabled by a few changes to the allowed values in the transfer function. The opacity is dictated by the values in the rgb of the transfer function. This is a continuous behavior, so if you slowly ramp up the rgb values from 0.1 to 10, you’ll see it get more and more opaque. In general it works out that rgb values around 1.0 start making things opaque, but you may need to experiment to get the desired behavior. By the time 2.4 is released we plan on having a tutorial on how to make more and more complex renderings, starting from a simple rendering and moving to a more complicated example. Currently the behavior is set so that all colors are opaque to all other colors. The integration step (in yt/utilities/lib/field_interpolation_tables.pxd) is: ttot = trgba[0] + trgba[1] + trgba[2] ta = fmax(1.0 - dt*ttot, 0.0) for i in range(3): rgba[i] = (1.0-ta)*trgba[i] + ta*rgba[i] trgba is the transfer function rgba rgba is the image values *Lighting: We also now enable adding a light source to the rendering. The behavior is enabled by use_light=True in the camera() call. It is then controlled by cam.light_dir and cam.light_rgba. The light direction is then dotted with the gradient of the field being rendered to either add or subtract color from that region. Here the integration is: ta = fmax(1.0-dt*(trgba[0] + trgba[1] + trgba[2]), 0.0) for i in range(3): rgba[i] = (1.-ta)*trgba[i]*(1. + dot_prod*l_rgba[i]) + ta * rgba[i] where dot_prod is the (gradient of field) dot (light_direction), and l_rgba is the light color. What this does is amplify the emission when the dot product is positive, and decrease it when it is negative. So, what would be awesome is if you could try out your old scripts, and then try out some of the new features. Remember, you may need to run the install script again to get an up-to-date cython. There are two simple options to try this out. The first is to pull the volume_refactor bookmark into your local repository. hg pull -B volume_refactor https://bitbucket.org/MatthewTurk/yt-refactor hg up volume_refactor This will create a head, which you shouldn’t worry about merging with any other head, with the bookmark volume_refactor. Then a python setup.py develop or whatever you like should do it. If you don’t want to use bookmarks (though you really should because they are awesome!), then maybe just cloning the whole repo is for you. Once we let this simmer on yt-dev for a while, we will put up a PR to get this merged into the yt tip, and send out a compact version of this email to the yt-users list. And yes, the PR will have lot of updated docs to go with it. We are also working on a narrative example of how to go from a simple rendering like in the cookbook to something much more complex, and maybe even a screencast of one of us doing that in real time. Cheers, Sam & Matt

3 2

Fwd: [IPython-dev] Data viz in the notebook with JS
by Matthew Turk 04 Jun '12

04 Jun '12

Hi all (including Fernando, who I forgot was subscribed when I forwarded that message a few days ago!), This may also be of interest here. It's more from IPython mailing lists about implementing widgets like what we have in Reason, using their notebook engine. As my work on Reason lately has been refactoring it for client-side maintainability and to work with ExtJS4, I haven't thought about the python-side of it lately, and this would have to touch that a bit. If you want to play with the most recently released IPython notebook, it's available when yt is installed with the INST_0MQ option set to 1. The command "ipython notebook" will get you started. Any enterprising devs that want to work on integration should definitely speak up; my time is otherwise mostly occupied for the near/mid-future, but I'd be happy to lend a hand where I can. I'd want this to be part of the ExtJS4 port, as I don't see any point in having two major, independent sets of changes to Reason. Advantages to using IPython-notebook: * Using websockets will reduce latency between actions and responses by a considerable amount * Outsource all of the kernel launching (MPI/Queue/local), which would probably ease the development of interactive in-browser parallel VR * Outsource dealing with process handling * Payloads sent from yt widget code would not *necessarily* need to be received within Reason, but could possibly be displayed in a vanilla Notebook Disadvantages: * Several additional external dependencies (zeromq, pyzeromq, jinja2, tornado). Three of these are currently optional dependencies in the install script. * Payload system will have to be rewritten * Unclear amount of our REPL/Notebook code will have to be rewritten * The IPython notebook's storage format (SQLite) may not play nicely with all Lustre systems * Possible impedance mismatch between JS libraries used in Notebook and ExtJS4 * My understanding of how WebSockets work doesn't cover how they work when SSH tunneling to bind to a socket, so I am putting potential glitches from this approach under "disadvantage" For those interested, my ExtJS4 work is in my fork at https://bitbucket.org/MatthewTurk/yt-extjs4 . The code is considerably cleaned up from the first generation of Reason, and I think more inviting for development. As an example, compare the code for these sets of files between the old Reason and the new: New: yt/gui/reason/html/app/controller/widgets/PlotWindow.js yt/gui/reason/html/app/view/widgets/PlotWindow.js Old: yt/gui/reason/html/js/widget_plotwindow.js View & handler code has been completely separated, and I believe there's just enough magic to make developing new widgets easy and straightforward. I'm still porting existing widgets and bringing basic functionality up to spec, but the architecture is complete. -Matt ---------- Forwarded message ---------- From: Fernando Perez <fperez.net(a)gmail.com> Date: Sat, Jun 2, 2012 at 5:09 PM Subject: [IPython-dev] Data viz in the notebook with JS To: IPython Development list <ipython-dev(a)scipy.org>, IPython User list <ipython-user(a)scipy.org> Hi folks Here's a quick preview of what we can start having now: http://imgur.com/vk2Q6 courtesy of Cameron Bates: https://github.com/crbates/ipython-flot/blob/master/IPython/frontend/html/n… I suggest simply to grab the single flot.py file and put it in your working dir for testing. We still have a ways to go with this kind of setup, because the user experience upon saving is a bit awkward: these javascript objects are only rendered when their code executes, so the saved page shows empty placeholders. We'll have to think more about all this, but it's great to have a concrete point to start from. Now that all the JavaScript refactoring has been merged, writing dynamic code with JS in the notebook will be much cleaner. If anyone is looking for toys of this nature to play with, here's a great summary page of viz JS libraries that could all be very elegantly plugged into the notebook: http://selection.datavisualization.ch Let us know what you come up with! Cheers, f _______________________________________________ IPython-dev mailing list IPython-dev(a)scipy.org http://mail.scipy.org/mailman/listinfo/ipython-dev

2 2

Downgrading Grid Resolution in yt
by Cameron Hummels 01 Jun '12

01 Jun '12

Hello everyone, Does anyone know of a way to universally downgrade the grid refinement by 1 level in yt (for an enzo output)? I know that one can resample an output to any arbitrary maximum refinement level using a covering grid (or a smoothed covering grid), but the result only "downgrades" data at the highest levels of refinement. What I'm instead looking for is a method to "downgrade" the resolution of the data at every grid element regardless of its refinement. For example, if I have a two adjacent grids, one with 10 levels of refinement and the other with 3, I would like them resampled at 9 and 2, respectively, as opposed to 9 and 3, which is what happens in the case of the covering grid. Additionally, it might be nice to retain the AMR structure of the original output, so that the memory consumption is relatively low (much lower than an equivalent covering grid). This technique would be very useful for resolution studies to determine the effects of poorer resolution on post-processing analysis. Anyway, has anyone played around with this? If not, I'll look into writing something up in the next few weeks, but I figured I'd ask. Cameron -- Cameron Hummels PhD Candidate, Astronomy Department of Columbia University Public Outreach Director, Astronomy Department of Columbia University NASA IYA New York State Student Ambassador http://outreach.astro.columbia.edu PGP: 0x06F886E3

2 6

Fwd: [IPython-dev] Another monster landed: refactored JavaScript, new tooltips and interactive widgets in the notebook
by Matthew Turk 01 Jun '12

01 Jun '12

This might of interest to people here. In theory we could refactor Reason to use the IPython kernel, with these developments. My attempts to do so earlier in the year were blocked on exactly this set of changes going in. ---------- Forwarded message ---------- From: Fernando Perez <fperez.net(a)gmail.com> Date: Thu, May 31, 2012 at 8:30 PM Subject: [IPython-dev] Another monster landed: refactored JavaScript, new tooltips and interactive widgets in the notebook To: IPython Development list <ipython-dev(a)scipy.org>, IPython User list <ipython-user(a)scipy.org> Hi folks, After the merge of the previous big two branches (the merging of the interactive and parallel kernels into one object and the magics refactoring/cell support), the third monster we had in flight has now landed: https://github.com/ipython/ipython/pull/1711/ For those of you not following the github hooks obsessively, here's the high-level view of the goodies: - New graphical tooltip with keyboard control: successive presses of the TAB key will expand it, pin it for 10 s (a clock icon is shown), and send its content to the bottom help pager (which is now resizable). - Completer has been refactored into a new class and now can analyze the current cell to support completions on objects that don't exist in the kernel yet (because the cell hasn't been executed). - All the client-side JavaScript has been decoupled to ease reuse of parts of the machinery without having to build a full-blown notebook. This will make it much easier to communicate with an IPython kernel from existing web pages and to integrate single cells into other sites, without loading the full notebook document-like UI. - This refactoring also enables the possibility of writing dynamic javascript widgets that are returned from Python code and that present an interactive view to the user, with callbacks in Javascript executing calls to the Kernel. This will enable many interactive elements to be added by users in notebooks. An example of this capability has been provided as a proof of concept in `docs/examples/widgets` that lets you directly communicate with one or more parallel engines, acting as a mini-console for parallel debugging and introspection. Kudos to Brian and Matthias for the excellent work they put into this, it was over 84 commits and involved the merging of several intermediate pull requests. But the goodies shipped therein should be *very* useful. We've seen recently attempts at interactive plots with d3/flot and other similar tools; now the machinery is ready to do that cleanly, so by all means have fun! Cheers, f _______________________________________________ IPython-dev mailing list IPython-dev(a)scipy.org http://mail.scipy.org/mailman/listinfo/ipython-dev

2 1