Mailman 3 June 2012 - yt-dev

Near- and mid-term roadmap: 2.4, 2.x, 3.0
by Matthew Turk Aug. 7, 2012

Aug. 7, 2012

Hi all, Sam and I are working on issuing a PR about the volume rendering refactor. This is the last major code change to go in place for 2.4, pending the quad_proj changes being accepted or rejected. I'm still thinking about this refine_by issue, and that might also make it in. After the volume_refactor, it's just testing and documentation left: https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&mileston… After 2.4 is released, I am going to shift my new development exclusively to the yt-3.0 branch, which is currently hosted here: http://bitbucket.org/yt_analysis/yt-3.0 This will include geometry refactoring and many remaining de-Enzoifications of the code base. 3.0 is nearing a usable state, but will not be ready for production for a while. However, I think that allowing the codebases to diverge too rapidly will lead to problems: a fracturing. So once the 3.0 codebase can muster feature parity with 2.x, I am going to ask that all development on the 2.x branch cease, and all pull requests for features and new functionality be applied only to the 3.0 branch. In advance of this, I would like to solicit help: we've identified a number of relatively straightforward changes to this code base that can and should be made. For instance, the changing of field names. I'd also like to consider moving to a more spread-out source code layout, like one class per file. Changes like these make it difficult to do cross-branch merges and transplants, which is why I would like to save them for after feature parity has been reached and development on 2.x has halted. Previous threads: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001941… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001949… If anyone has any strong objections to this strategy, please reply. Estimated Timeline: * June 26, 2.4 release * July 30, 3.0 reaches rough feature parity * Fall sometime: 3.0 merged into main 'yt' branch * Some time after that: 3.0 released Criteria for Feature Parity: * All patch-based AMR codes support data region selection * Parallelism is functional for all parallel-capable routines * All patch-based AMR codes support projections, slices, cutting planes * All patch-based AMR codes support volume rendering * All patch-based AMR codes support halo finding * Octree-based AMR codes (RAMSES and ART) support data region selection * Octree-based AMR codes (RAMSES and ART) support slices, projections and cutting planes If any of you are interested in helping with this reengineering effort, I would be extremely interested in collaborating. My goal for this effort is to focus on making yt a future-compatible code, and changing it from an organically grown system into one that we design based on the years of experience we all have now had with it. -Matt

4 6

Re: [yt-dev] off_axis_projection
by Cameron Hummels July 3, 2012

July 3, 2012

Hey guys, I guess I don't entirely understand why these two functionalities are so different, but I'll chat with you, Matt, a bit more about this offline when I'm back at CU. Cameron > Okay. > > Cameron, I know you rely upon the off_axis_projection -- would it be > okay, in your opinion, if we deprecated the interpolated = True option > in some 2.X release and then in 3.0, if you want interpolation, you > need to set up a projection camera yourself? I ask because the setup > for these two is very different, and (in the spirit of the plot > window!) I'd like to try to reduce the complexity. If you're -1 on > this, then we can keep it the way it is. > > On Thu, Jun 28, 2012 at 11:57 AM, Sam Skillman <samskillman(a)gmail.com> > wrote: >> That capability doesn't exist yet for interpolated=True. When that is >> possible, I would be +1 with moving from volume to data_source. >> >> >> On Thu, Jun 28, 2012 at 9:53 AM, Matthew Turk <matthewturk(a)gmail.com> >> wrote: >>> >>> On Thu, Jun 28, 2012 at 11:51 AM, Sam Skillman <samskillman(a)gmail.com> >>> wrote: >>> > Hi all, >>> > >>> > I think the only kwarg I'd be really +1 with getting rid of would be >>> > no_ghost. interpolated=True should just trigger no_ghost=False, and >>> > interpolated=False doesn't use the ghost zones no matter what. I >>> think >>> > volume should stay, especially since I think it will soon be possible >>> to >>> > do >>> > an off_axis_projection of a data object which would probably be fed >>> in >>> > through the volume. >>> >>> Why not get rid of volume and instead supply data_source, like >>> projections? >>> >>> > >>> > Sam >>> > >>> > >>> > On Thu, Jun 28, 2012 at 9:39 AM, Matthew Turk <matthewturk(a)gmail.com> >>> > wrote: >>> >> >>> >> Hi Elizabeth, >>> >> >>> >> I agree, this makes sense. >>> >> >>> >> Last night I tried to take a crack at this, but I got very confused >>> >> with the various definitions of volume, interpolated, etc etc, in >>> the >>> >> routine off_axis_projection. Cameron and Sam, do you think it would >>> >> be okay if we sim

4 9

Odd compile, OSX10.6, OMP, x86 vs. i386
by david collins July 3, 2012

July 3, 2012

Hi, all-- Sam and I tracked down an odd glitch. It started with trying to run the test runner, and I got an error on the "from yt.utilities.answer_testing.api import" (full trace below) For some reason the build of yt/utilities/lib/grid_traversal.pyx was looking for $DEST_DIR/lib64 and not finding it, trying to use /usr/local/lib/libgomp.dylib which, for some reason is built for i386 rather than x86_64. Putting /usr/local/lib/x86_64 in LD_LIBRARY_PATH didn't seem to work, but soft linking /usr/local/lib/x86_64 to $DEST_DIR/lib64 worked. More elegant solutions are welcome, but this worked for me. d. /run>./test_runner.py --quicksuite=True --output-dir=/Users/dccollins/ThisScratch/TestRunnerFiducial Traceback (most recent call last): File "./test_runner.py", line 29, in <module> from yt.utilities.answer_testing.api import \ File "/Users/dccollins/local2/src/yt-hg/yt/utilities/answer_testing/__init__.py", line 26, in <module> import runner File "/Users/dccollins/local2/src/yt-hg/yt/utilities/answer_testing/runner.py", line 26, in <module> import matplotlib File "/Users/dccollins/local2/lib/python2.7/site-packages/matplotlib/__init__.py", line 133, in <module> from matplotlib.rcsetup import (defaultParams, File "/Users/dccollins/local2/lib/python2.7/site-packages/matplotlib/rcsetup.py", line 19, in <module> from matplotlib.colors import is_color_like File "/Users/dccollins/local2/lib/python2.7/site-packages/matplotlib/colors.py", line 52, in <module> import numpy as np File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/__init__.py", line 137, in <module> import add_newdocs File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/core/__init__.py", line 5, in <module> import multiarray ImportError: dlopen(/Users/dccollins/local2/lib/python2.7/site-packages/numpy/core/multiarray.so, 2): no suitable image found. Did find: /Users/dccollins/local2/lib/python2.7/site-packages/numpy/core/multiarray.so: mach-o, but wrong architecture -- Sent from my computer.

2 2

possible Forthon 0.8.8 and 0.8.9 issues?
by Geoffrey So July 1, 2012

July 1, 2012

Has anyone installed yt-dev from scratch recently and used the newer than 0.8.7 Forthon(s)? I was able to use make to compile the 8.8 and 8.9 after removing the --with-numpy argument from the Makefile inside the yt/utilities/kdtree, however, I get fKD' is not defined when I use functions that requires the kd-tree. I was able to bypass this problem by reinstalling YT and then using Forthon 0.8.7 (removing Forthon 8.8/8.9 then installing 8.7 then reinstallling YT did not work). I was wondering if anyone else ran into the same issue? Not urgent because 8.7 works. From G.S.

2 1

Fwd: [IPython-dev] [ANN] IPython 0.13 is officially out!
by Matthew Turk June 30, 2012

June 30, 2012

---------- Forwarded message ---------- From: "Fernando Perez" <fperez.net(a)gmail.com> Date: Jun 30, 2012 4:56 AM Subject: [IPython-dev] [ANN] IPython 0.13 is officially out! To: "IPython Development list" <ipython-dev(a)scipy.org>, "IPython User list" <ipython-user(a)scipy.org>, "Discussion of Numerical Python" < numpy-discussion(a)scipy.org>, "Matplotlib Users" < matplotlib-users(a)lists.sourceforge.net>, "SciPy Users List" < scipy-user(a)scipy.net> Hi all, on behalf of the IPython development team, and just in time for the imminent Debian freeze and SciPy 2012, I'm thrilled to announce, after an intense 6 months of work, the official release of IPython 0.13. This version contains several major new features, as well as a large amount of bug and regression fixes. The previous version (0.12) was released on December 19 2011, so in this development cycle we had: - ~6 months of work. - 373 pull requests merged. - 742 issues closed (non-pull requests). - contributions from 62 authors. - 1760 commits. - a diff of 114226 lines. This means that we closed a total of 1115 issues over 6 months, for a rate of almost 200 issues closed per month and almost 300 commits per month. We are very grateful to all of you who have contributed so enthusiastically to the project and have had the patience of pushing your contributions through our often lengthy review process. We've also welcomed several new members to the core IPython development group: Jörgen Stenarson (@jstenar - this really was an omission as Jörgen has been our Windows expert for a long time) and Matthias Bussonier (@Carreau), who has been very active on all fronts of the project. *Highlights* There is too much new work to write up here, so we refer you to our full What's New document (http://ipython.org/ipython-doc/rel-0.13/whatsnew/version0.13.html) for the full details. But the main highlights of this release are: * Brand new UI for the notebook, with major usability improvements (real menus, toolbar, and much more) * Manage all your parallel cluster configurations from the notebook with push-button simplicity (cluster start/stop with one button). * Cell magics: commands prefixed with %% apply to an entire cell. We ship with many cell magics by default, including timing, profiling, running cells under bash, Perl and Ruby as well as magics to interface seamlessly with Cython, R and Octave. * The IPython.parallel tools have received many fixes, optimizations, and a number of API improvements to make writing, profiling and debugging parallel codes with IPython much easier. * We have unified our interactive kernels (the basic ipython object you know and love) with the engines running in parallel, so that you can now use all IPython special tricks in parallel too. And you can connect a console or qtconsole to any parallel engine for direct, interactive execution, plotting and debugging in a cluster. *Downloads* Download links and instructions are at: http://ipython.org/download.html And IPython is also on PyPI: http://pypi.python.org/pypi/ipython Those contain a built version of the HTML docs; if you want pure source downloads with no docs, those are available on github: Tarball: https://github.com/ipython/ipython/tarball/rel-0.13 Zipball: https://github.com/ipython/ipython/zipball/rel-0.13 Please see our release notes for the full details on everything about this release: http://ipython.org/ipython-doc/rel-0.13/whatsnew/version0.13.html As usual, if you find any other problem, please file a ticket --or even better, a pull request fixing it-- on our github issues site (https://github.com/ipython/ipython/issues). Many thanks to all who contributed! Fernando, on behalf of the IPython development team. http://ipython.org _______________________________________________ IPython-dev mailing list IPython-dev(a)scipy.org http://mail.scipy.org/mailman/listinfo/ipython-dev

1 0

off_axis_projection
by Elizabeth Tasker June 28, 2012

June 28, 2012

Hi, "off_axis_projection" doesn't seem to accept field parameters. This isn't a big problem and I can definitely get around it by using global variables, but should it have a field_parameter option for consistency with add_projection? Elizabeth

3 5

broken ellipsoid
by Geoffrey So June 28, 2012

June 28, 2012

I was trying to use my ellipsoid in the dev-yt installation to check things, and apparently some of the changes of defining RX,Y,Z relative to get_rotation_matrix in math_utils.py didn't get pulled in or was left out and replaced with calls to get_rotation_matrix, but one of the lines in halo_objects.py has a typo: r1 = (e0_vector * rz).sum(axis = 1) should be replaced with r1 = (e0_vector * RZ).sum(axis = 1) where RZ are both capitalized. Fixing that makes the ellipsoid work, can someone check it in please? Thanks in advance. From G.S.

2 1

Call for testing: Projection performance
by Matthew Turk June 27, 2012

June 27, 2012

Hi all, I implemented this "quadtree extension" that duplicates the quadtree on all processors, which may make it nicer to scale projections. Previously the procedure was: 1) Locally project 2) Merge across procs: 2a) Serialize quadtree 2b) Point-to-point communciate 2c) Deserialize 2d) Merge local and remote 2d) Repeat up to 2a 3) Finish I've added a step 0) which is "initialize entire quadtree", which means all of step 2 becomes "perform sum of big array on all procs." This has good and bad elements: we're still doing a lot of heavy communication across processors, but it will be managed by the MPI implementation instead of by yt. Also, we avoid all of the costly serialize/deserialize procedures. So for a given dataset, step 0 will be fixed in cost, but step 1 will be reduced as the number of processors goes up. Step 2, which now is a single (or two) communication steps, will increase in cost with increasing number of processors. So, it's not clear that this will *actually* be helpful or not. It needs testing, and I've pushed it here: bb://MatthewTurk/yt/ hash 3f39eb7bf468 If anybody out there could test it, I'd be might glad. This is the script I've been using: http://paste.yt-project.org/show/2343/ I'd *greatly* appreciate testing results -- particularly for proc combos like 1, 2, 4, 8, 16, 32, 64, ... . On my machine, the results are somewhat inconclusive. Keep in mind you'll have to run with the option: --config serialize=False to get real results. Here's the shell command I used: ( for i in 1 2 3 4 5 6 7 8 9 10 ; do mpirun -np ${i} python2.7 proj.py --parallel --config serialize=False ; done ) 2>&1 | tee proj_new.log Comparison against results from the old method would also be super helpful. The alternate idea that I'd had was a bit different, but harder to implement, and also with a glaring problem. The idea would be to serialize arrays, do the butterfly reduction, but instead of converting into data objects simply progressively walk hilbert indices. Unfortunately this only works for up to 2^32 effective size, which is not going to work in a lot of cases. Anyway, if this doesn't work, I'd be eager to hear if anybody has any ideas. :) -Matt

4 7

Summary of a yt sprint
by Matthew Turk June 27, 2012

June 27, 2012

Hi all, This last weekend, John, Nathan, Britton, Sam and I were fortunate enough to be in the same place for a couple days and we had a sprint on some yt functionality. Almost all of the changes have gone into the code base already, but we wanted to share them with you all. Feel free to chime in with questions, comments, or more information. = Parallelism Improvements = Britton implemented a pure-MPI task queuing system that works with the existing parallel_objects machinery. It is currently not threaded (and opinions vary as to whether that would be productive or not) and so one MPI task is dedicated as the controller node that distributes objects. This can either be manually used or you can simply supply "dynamic=True" to the parallel_objects machinery. I have to say, the changes to yt's parallelism in the last year or so have been pretty much my favorite improvements, and Britton really knocked this one out of the park, in my opinion. As I noted in my other email, with the pull request for Reason, Britton also added an MPI-parallel backend for Reason. I won't repeat my email here, but if you want to test it out you'll need Pyro4 and the launcher script in scripts/ . This will be documented in more detail when I finish up the remaining task, which is to deliver intermediate payloads from the MPI controller back to the web browser. = Plot Window = Perhaps Nathan would be better at describing this, but he's now implemented a mechanism (built on Jeff's initial design) for generating nice plots easily and complex plots straightforwardly. This was discussed here: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/002111.… Nathan, anything you wanted to add? = Reason = This was described in my other email as well: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/002117.… and this PR: https://bitbucket.org/yt_analysis/yt/pull-request/179/reason-gui-refactor-e… If anybody can test this out, I'd appreciate it. I did note this weekend that when running simulations, I spied a few of the other sprinters using it. I think seeing developers using something means it has "arrived." (And I know Sam has a volume rendering widget in mind ...) = Ghost Zones = John spent a good amount of time implementing a fast sibling finder, as well as a ghost zone filling mechanism. This is designed to replace the existing mechanism of a smoothed covering grid, which requires interpolation from the coarsest level up to the requested level. This should lead to a considerable performance increase when doing anything having ghost zones. I think this is going to be PR'd sometime in the next while. -- Anyway, I think I speak for everybody when I say it was a pretty successful sprint, and we'd love to hear your feedback! -Matt

2 2

Setup script uses the wrong libpng headers
by Nathan Goldbaum June 27, 2012

June 27, 2012

Hi all, I've just run into an odd issue. I'm using Matt's yt-extJS4 fork to make 2D plots. In doing so, I discovered that png_writer.so is being compiled with the wrong libpng.h header. This produces errors that look like: libpng warning: Application was compiled with png.h from libpng-1.5.4 libpng warning: Application is running with png.c from libpng-1.2.43 libpng error: Incompatible libpng version in application and library Segmentation fault: 11 Does anyone know why the setup script isn't finding the libpng headers that come with yt? Thanks for your help, Nathan

1 1