Mailman 3 July 2013 - yt-dev

yt Strategy Meeting Monday the 22nd
by Matthew Turk 04 Sep '13

04 Sep '13

Hi all, I'd like to call a brief, approximately one hour, strategy meeting on Monday, April 22nd at 3PM Eastern. This will cover where yt is at, where we can go, and how things are progressing. Some of this will be dedicated to progress updates (from me and others) as well as soliciting feedback on a number of things. Many other projects -- notably Cactus -- do these on a fixed schedule, so based on the success/failure of this one we can consider having them semi-regularly. The agenda I would like to cover will include: * yt 3.0 status, including near/long term plans for development and support for non-grid codes * Rundown of all YTEPs accepted and outstanding * yt's "presence" on the web * 3.0 paper I will also be soliciting feedback in roundtable style on: * How are we doing? * How is the procedure for code contributions working? * How is the yt "infrastructure" meeting people's needs? (Mailing lists, Bitbucket, etc) * Where are energies being spent that could be better spent elsewhere? If you have either agenda items or discussion items, please reply to this email and suggest them. I think having a large presence at this meeting would be very useful, so if you are able to make it, please try to do so. It'll be held as a google hangout, and I'll send details here when it starts up. If there's a pressing need to reschedule, please write in reply to this email. Thanks, and looking forward to seeing you there, Matt

2 3

Fwd: [IPython-dev] interactive widgets
by Matthew Turk 30 Jul '13

30 Jul '13

Hi all, The IPython devs have been sketching out the protocol for their javascript widgets. They've also been setting a good timetable for when external devs can start working on it. Since this is -- in my opinion -- one of the most promising directions for Reason and other yt GUIs/explorers, it's definitely interesting to see it take shape. (Frankly, I think it's the most promising direction for *all* scientific python data exploration, but maybe that's just me ...) The message spec Matthias links to does a pretty interesting job of laying out how Python would communicate with the GUI and how it would look to the slightly-higher-level infrastructure developer. Definitely something to keep an eye on, and also seems like porting existing Reason code (which is very similar in feel, although not as well thought-out or implemented) would be possible. -Matt ---------- Forwarded message ---------- From: Matthias BUSSONNIER <bussonniermatthias(a)gmail.com> Date: Tue, Jul 30, 2013 at 10:36 AM Subject: Re: [IPython-dev] interactive widgets To: IPython developers list <ipython-dev(a)scipy.org> Hi Jason. You can have allow here https://hackpad.com/IPython-Interactive-JavaScript-Widget-Design-jRD6dHFt5I8 We have only define the rough edges of what we will do. - create widget step (from kernel to javascript IIRC) - bidirectionnal update_state js<-> kernel - possibly a destroy widget message Data in json form. No binary websocket for now. -- Matthias Le 30 juil. 2013 à 18:11, Jason Grout a écrit : > Could someone summarize some of the discussion that happened last week > on interactive widgets (like manipulate/interact controls, etc.)? I'm > working more on extending Sage's implementation this week, and I'm > curious what happened last week in the IPython camp. > > Thanks, > > Jason > _______________________________________________ > IPython-dev mailing list > IPython-dev(a)scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev _______________________________________________ IPython-dev mailing list IPython-dev(a)scipy.org http://mail.scipy.org/mailman/listinfo/ipython-dev

1 0

Turn data serialization off by default
by Nathan Goldbaum 30 Jul '13

30 Jul '13

Hi all, I've just issued a PR that will hopefully fix a whole class of buggy behavior that both new and experienced yt users commonly run into. Specifically, I'd like it if we could turn off data serialization by default. This changes a long-lived default value in yt's configuration, so I wanted to bring this change to the attention of both the yt user and developer community. What is data serialization? Currently, yt will save the result of certain expensive calculations, including projections, the structure of the grid hierarchy, and the list of fields present in the data. While this does have the beneficial effect of saving time when a user needs to repetitively calculate these quantities on the same dataset, it has a number of features which lead to buggy, annoying behavior. Specifically, If I am developing my simulation code or repeatedly restarting my code, searching for a way to grind past a code crash, I will quite often regenerate the same simulation output file over and over, changing a line of code or switching out the value of a parameter each time. If yt's data serialization is turned on, it's likely that yt's visualizations will correspond to old versions of the data file. Since only certain operations are serialized, it's also possible for yt to get into an inconsistent state - one operation will show the current data file, while another operation will show an old version. It's possible to fix a bug in your code, but because yt is still loading the old data, you won't be able to tell that your bug is fixed until you realize that you have .yt and .harrays files littering your filesystem. I've personally wasted a lot of time due to yt's serialization 'feature' and denizens of our IRC channel and mailing list can attest to how often new users run into this behavior as well. My pull request only turns off serlialization by default, it doesn't disable the capability completely. Once the pull request is merged in, you can turn on serialization either by adding an entry to your config file: $ cat ~/.yt/config [yt] serialize = True Or on a per-script basis: from yt.config import ytcfg ytcfg['yt', 'serialize'] = 'True' from yt.mods import * The pull request is here: https://bitbucket.org/yt_analysis/yt/pull-request/558 I know several of you are big fans of this feature, so if you object to this change please leave a comment on the pull request so we can figure out a way forward. -Nathan

7 11

Issue #613: MultiVariateTransferFunction does not support grey_opacity (yt_analysis/yt)
by Noel Scudder 30 Jul '13

30 Jul '13

New issue 613: MultiVariateTransferFunction does not support grey_opacity https://bitbucket.org/yt_analysis/yt/issue/613/multivariatetransferfunction… Noel Scudder: File "/home/noel/shocks/yt-x86_64/src/yt-hg/yt/visualization/volume_rendering/camera.py", line 611, in finalize_image if self.transfer_function.grey_opacity is False: AttributeError: 'MultiVariateTransferFunction' object has no attribute 'grey_opacity' It seems the MultiVariate (as well as Planck, see issue #605) transfer function was not updated after opaque volume rendering was added.

1 0

Issue #612: Reloading an athena dataset leads to corruption. (yt_analysis/yt)
by Nathan Goldbaum 29 Jul '13

29 Jul '13

New issue 612: Reloading an athena dataset leads to corruption. https://bitbucket.org/yt_analysis/yt/issue/612/reloading-an-athena-dataset-… Nathan Goldbaum: I'm using a 2D orszag tang dataset to test this. The failing script is here: ``` #!python from yt.mods import * fns = sorted(glob.glob('OrszagTang.????.vtk')) fn = fns[0] #fn = '/home/goldbaum/tar_files/IsolatedGalaxy/galaxy0030/galaxy0030' pf = load(fn) slc = SlicePlot(pf, 2, 'Density') slc.save() pf = load(fn) slc = SlicePlot(pf, 2, 'Density') slc.save() ``` This produces the following traceback: ``` #!python Traceback (most recent call last): File "test.py", line 15, in <module> slc.save() File "/home/goldbaum/yt-x86_64/src/yt-hg/yt/visualization/plot_window.py", line 1091, in save names.append(v.save(n,mpl_kwargs)) File "/home/goldbaum/yt-x86_64/src/yt-hg/yt/visualization/base_plot_types.py", line 70, in save canvas.print_figure(name, **mpl_kwargs) File "/home/goldbaum/yt-x86_64/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 2096, in print_figure **kwargs) File "/home/goldbaum/yt-x86_64/lib/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 492, in print_png FigureCanvasAgg.draw(self) File "/home/goldbaum/yt-x86_64/lib/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 435, in draw self.renderer = self.get_renderer() File "/home/goldbaum/yt-x86_64/lib/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 454, in get_renderer self.renderer = RendererAgg(w, h, self.figure.dpi) File "/home/goldbaum/yt-x86_64/lib/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 85, in __init__ self._renderer = _RendererAgg(int(width), int(height), dpi, debug=False) ValueError: cannot convert float NaN to integer ``` When I switch out the athena dataset for the enzo IsolatedGalaxy dataset, I get no crash, so I think this has to be an issue with the athena frontend. I will try to dive in and fix this tonight, but if @samskillman, @jzuhone, or @MatthewTurk have any idea where the error might be, I'd love to hear your thoughts.

1 0

3.0a3 slightly delayed
by Matthew Turk 26 Jul '13

26 Jul '13

Hi all, I'm starting the release process of yt 3.0a3 today, but I won't finish it for a few days. Basically, I'm writing up a bunch of notebooks demoing new features and showing speed/memory improvements. So the release will likely be delayed until later this week. Any outstanding pull requests you'd like to have go in should be issued today or tomorrow morning, and I'll try to review and have them go in. If there are major outstanding bugs, or improvements that aren't ready to land yet, we can hold off a few more days but absolutely no later than Thursday of next week. -Matt PS As a sidenote, I will be out of email contact for some time not-yet-determined starting between July 30 and through at least August 12.

2 2

YTSliceBase.reslice broken
by Douglas Harvey Rudd 23 Jul '13

23 Jul '13

Hi all, In writing test cases for the slice selector I discovered that the reslice function of YTSliceBase appears to be broken due to caching at several levels. 1) the selector object stores the slice coordinate and was never updated 2) the grid selection is cached 3) the cell mask (created by fill_mask) is only updated when the selector id changes I fixed 1) and 2) by overloading the clear_data function in YTSelectionDataContainer: def clear_data(self): """ Clears out all data from the YTSelectionDataContainer instance, freeing memory. """ self._selector = None self._current_chunk = None self.field_data.clear() and replaced the call to self.field_data.clear() in YTSliceBase.reslice with self.clear_data() This forces the selector to be recreated when it is next requested. Unfortunately the problem persisted, inconsistently, and I tracked it down to 3). The id of the newly created selector was sometimes identical to the one prior to the reslice command. In the grid_patch hierarchy, the cell mask is only recreated if the id of the selector object has changed, and so in this case an old mask persisted. I see several options for solving the problem: 1) replace using the python id function with a random identifier or hash of the actual selector data, to ensure it detects when the parameters have changed 2) reach into the hierarchy from the clear_data function and erase the cached cell mask 3) add a check in the hierarchy to detect if the chunking has changed along with the selector (not sure how this could be done, but probably a good idea anyway, for consistency) 4) remove the reslice function and prevent selection data containers from changing after creation Thoughts? Douglas Rudd Scientific Computing Consultant Research Computing Center drudd(a)uchicago.edu

3 2

Issue #611: Incorrect docstrings in plot_window.py (yt_analysis/yt)
by Andrew Myers 22 Jul '13

22 Jul '13

New issue 611: Incorrect docstrings in plot_window.py https://bitbucket.org/yt_analysis/yt/issue/611/incorrect-docstrings-in-plot… Andrew Myers: The docstrings for SlicePlot, ProjectionPlot, OffAxisSlicePlot and OffAxisProjection say that the center defaults to the maximum density cell, but in fact it looks like they default to the center of the domain. I'm filing an issue for this rather than just fixing it myself because I'm not sure what people behavior is "right." Responsible: atmyers

1 0

Proposed: Remove RAMSES and ART from 2.x and force migration to 3.0
by Matthew Turk 22 Jul '13

22 Jul '13

Hi all, I'd like to propose we actually remove the RAMSES and ART (NMSU) frontends from yt 2.x, since they probably don't work, and certainly not to the level of support they have in 3.0. In fact, I'd go so far as to say that NMSU ART and RAMSES are production quality in 3.0, unlike in 2.x where they don't quite work properly. I think we need confirmation from Chris Moody before this can be done, but does anyone have an objection to this? I will simply replace them with stubs that say, please use yt 3.0. -Matt

3 2

Issue #610: Trouble plotting a simple ramses dataset (yt_analysis/yt)
by Nathan Goldbaum 22 Jul '13

22 Jul '13

New issue 610: Trouble plotting a simple ramses dataset https://bitbucket.org/yt_analysis/yt/issue/610/trouble-plotting-a-simple-ra… Nathan Goldbaum: Using the following script: ``` #!python from yt.mods import * pf = load('/pfs/hennebel/KH/output_00037/info_00037.txt') slc = SlicePlot(pf, 'z', 'Density') ``` I get the following traceback: ``` #!python traceback Traceback (most recent call last): File "test.py", line 3, in <module> slc = SlicePlot(pf, 'z', 'Density') File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/visualization/plot_window.py", line 1254, in __init__ slc = pf.h.slice(axis, center[axis], File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/data_objects/static_output.py", line 218, in hierarchy self, data_style=self.data_style) File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/frontends/ramses/data_structures.py", line 303, in __init__ super(RAMSESGeometryHandler, self).__init__(pf, data_style) File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/geometry/geometry_handler.py", line 68, in __init__ self._setup_geometry() File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/geometry/oct_geometry_handler.py", line 50, in _setup_geometry self._initialize_oct_handler() File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/frontends/ramses/data_structures.py", line 308, in _initialize_oct_handler for i in range(self.parameter_file['ncpu'])] File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/frontends/ramses/data_structures.py", line 72, in __init__ self._read_amr() File "/home/goldbaum/yt-x86_64/src/yt-3.0/yt/frontends/ramses/data_structures.py", line 250, in _read_amr assert(n == ng) AssertionError ``` The dataset is a simple 3D kelvin helmholtz test. I've uploaded a copy to my website (warning, 5GB): http://ucolick.org/~goldbaum/files/kh.tar I don't know much about this dataset as I got it precanned from Patrick Hennebelle so sorry if I can't provide much detail. FWIW, this is on the current tip of 3.0 (250ee66bf017).

3 2