Hi Matt, I'm springing off this email, but it's a separate point, so I gave it a new heading. Regarding breaking of backwards compatibility, I'm definitely +1 for avoiding it. That said, I also appreciate that changes are necessary to make yt more flexible to its ever expanding user base. A few of my scripts do use some of the lower functionality of yt (e.g. accessing an the parents of an individual cell etc) that I could see might change with an overhaul of structure, invisible to most cookbook-usage. I actually wouldn't issue more than a passing grumble, providing it was super clear how to fix my code for the new set-up. (So easy that I don't need to re-understand my long and (undoubtedly) poorly annotated script to do it). How about a spin-off page from the FAQ for "I updated and my code no longer works, WTH?". This could show an example of new call *and* the old call (so it's easy to see how to update your script). An example of what might be in there would be the call to EnzoSimulation which has undergone a number of revisions over the years. I don't think this would address Cameron's issue, which I think it about the change in the result rather than the usage, but it might be good for the above related problem. Elizabeth On Jun 14, 2012, at 12:40 PM, Matthew Turk wrote:
Hi Sam,
Thanks for doing this.
I'll update my PR shortly; this needs to be a part of it before it's accepted.
This has also highlighted a part of the process that I think we're all trying to do -- peer review not just of the quantitative changes to the code (i.e., breaking tests!) but also qualitative. In this case, Cameron saw something that would change functionality, and he stood up for keeping the ability to retain backwards compatibility. I think this is a valid point and a valuable position; in the case of the off-axis projection, when I initially made this change it was because I felt this was truer to the underlying data. (There are extremely few places in the yt codebase where it interpolates.)
However, retaining a solid foundation for users is also extremely important. The growth of yt over the last couple years has put us in a sticky situation; we've developed a community, but it's a community that relies on functionality not changing without good reason, and more importantly that relies on keeping a solid basis for repeatable analysis and consistent results. This change broke that -- even though it's arguably an improvement of the solution -- and the old method was no longer available.
I'm really happy that this system has worked; while I was initially -1 on restoring interpolation, Cameron swayed me in IRC by bringing up this remarkably important point. I think turning the default to non-interpolated but retaining the interpolated mode are both the right decisions, and I'm +1. And more than that, I'm glad this all happened before the PR was accepted.
So thank you, Sam, and Cameron, for restoring the old functionality. I'm excited about moving forward. The results in this thread have really gotten me excited about the success of this new set of changes.
-Matt
On Jun 13, 2012 11:26 PM, "Sam Skillman" <samskillman@gmail.com> wrote: Hi all,
Earlier today Cameron brought up some concerns about removing the behavior of the off-axis-projection using interpolated data from vertex-centered data. If you recall from earlier discussions on the impact of the new threading + opaque renderings + refactoring, one of the things we've done is switch from the interpolation to using the raw values within each cell a given ray traverses for use in the off-axis-projection. This is mathematically a more accurate representation of the data, but can sometimes lead to blocky looking images.
What I've done is go through and add an optional interpolated=(False/True) argument to the off_axis_projection call, where the default is False. Additionally, I've set this up so that it now wraps the ProjectionCamera class, which is derived from the Camera class, so it has things like .rotate, etc.
Just for reference, I'm including below both timings, results, and example images, made using this script: http://paste.yt-project.org/show/2462/
Here is the output: Comparison of Raw vs Interpolated Density Raw -- Time: 1.173431e+00 seconds, min: 8.046089e-06, max: 1.177795e-02 Int -- Time: 6.393837e+00 seconds, min: 8.322453e-06, max: 1.113421e-02
Density-weighted Temperature Raw -- Time: 8.513930e-01 seconds, min: 4.681834e+00, max: 1.237399e+07 Int -- Time: 8.450805e+00 seconds, min: 1.105447e+01, max: 1.213922e+07
The main difference that you see here is that the interpolated doesn't capture the minima and maxima as well, which makes sense since it is interpolating the data and smoothing out the peaks. Note that the Interpolated use the no_ghost=False option, which brings it much closer to the Raw values. This highlights the improvement off the off_axis_projection for quantitative results, as it is both more accurate and faster. This is also a fairly small dataset. Improvements for larger simulations are even more impressive.
You can see the diff between this changeset and Matt's changeset that is currently the PR changeset here: https://bitbucket.org/samskillman/yt-refactor/compare/..MatthewTurk/yt-refac...
Please take a look. I'm hopeful that with these changes pulled into Matt's PR, we should be closer to this being accepted.
Along with these changes come some updated docs, as well as getting rid of those pesky print statements that Britton saw.
Cheers, Sam
On Wed, Jun 13, 2012 at 8:36 PM, Sam Skillman <samskillman@gmail.com> wrote: Hi Chris,
Hmm, maybe Pleiades sets OMP_NUM_THREADS=1 by default. Can you check: echo $OMP_NUM_THREADS
If it is either unset or set to 1, could you give it a shot with export OMP_NUM_THREADS=12? The HT shouldn't help too much.
Best, Sam
On Wed, Jun 13, 2012 at 8:14 PM, Christopher Moody <cemoody@ucsc.edu> wrote: Hi Matt, Yep, this is an ART dataset. I had a node running with 12 (24 with HT) cores at NASA Pleiades. It didn't go above 4% (=1/24) usage, which indicates means it likely did not use more than one core (but I didn't tell it to). If there's a parallel version of the script I can try it out tonight, and see the scaling.
chris
On Wed, Jun 13, 2012 at 1:01 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Chris, is this an ART dataset rendering? I love it. Great work. And, did you note on your machine what the load went up to? Did it scale with the numebr of cores okay?
On Wed, Jun 13, 2012 at 3:47 PM, Christopher Moody <cemoody@ucsc.edu> wrote:
Works for me!
On one core, no_ghost=True, kd-Tree took 5 seconds, ray casting 56 seconds. Impressive.
chris
On Wed, Jun 13, 2012 at 10:45 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Also, for what it's worth, John, Sam and I have been working on creating a movie sequence with this in HiDef for a display project. Once that's done we're going to be posting the movie, the original scripts, and potentially even a screencast of how one goes about doing a complex volume rendering.
On Wed, Jun 13, 2012 at 12:34 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi all,
Here's a sample script that exposes a few of the new features such as opacity and the map_to_colormap function. http://paste.yt-project.org/show/2458/
I'll see if I can get some more scripts written in a portable way.
Cheers, Sam
On Wed, Jun 13, 2012 at 10:22 AM, John ZuHone <jzuhone@gmail.com> wrote:
Matt,
I'll give it a whirl... it may take a day or two.
John Z
On Jun 13, 2012, at 11:12 AM, Matthew Turk wrote:
> Sam and I chatted, and we'd like to request three people test this and > if they all agree, we can accept it. > > Can three people give it a go? > > > ---------- Forwarded message ---------- > From: Matthew Turk <pullrequests-noreply@bitbucket.org> > Date: Wed, Jun 13, 2012 at 11:12 AM > Subject: [yt_analysis/yt] Volume Rendering Refactor (pull request #168) > To: yt@enzotools.org > > > A new pull request has been opened by Matthew Turk. > > MatthewTurk/yt-refactor has changes to be pulled into yt_analysis/yt. > > > https://bitbucket.org/yt_analysis/yt/pull-request/168/volume-rendering-refac... > > Title: Volume Rendering Refactor > > This includes all of the changes to the volume rendering system that > were discussed here: > > > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/002039.h... > > The main user-facing changes that could cause problems: > > 1) Cython 0.16 is now required. > 2) Off-axis projections no longer interpolate. > > Changes to be pulled: > > > > -- > This is an issue notification from bitbucket.org. > You are receiving this either because you are the participating > in a pull request, or you are following it. > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org