Near- and mid-term roadmap: 2.4, 2.x, 3.0
by Matthew Turk
Hi all,
Sam and I are working on issuing a PR about the volume rendering
refactor. This is the last major code change to go in place for 2.4,
pending the quad_proj changes being accepted or rejected. I'm still
thinking about this refine_by issue, and that might also make it in.
After the volume_refactor, it's just testing and documentation left:
https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&milest...
After 2.4 is released, I am going to shift my new development
exclusively to the yt-3.0 branch, which is currently hosted here:
http://bitbucket.org/yt_analysis/yt-3.0
This will include geometry refactoring and many remaining
de-Enzoifications of the code base. 3.0 is nearing a usable state,
but will not be ready for production for a while. However, I think
that allowing the codebases to diverge too rapidly will lead to
problems: a fracturing. So once the 3.0 codebase can muster feature
parity with 2.x, I am going to ask that all development on the 2.x
branch cease, and all pull requests for features and new functionality
be applied only to the 3.0 branch. In advance of this, I would like
to solicit help: we've identified a number of relatively
straightforward changes to this code base that can and should be made.
For instance, the changing of field names. I'd also like to consider
moving to a more spread-out source code layout, like one class per
file. Changes like these make it difficult to do cross-branch merges
and transplants, which is why I would like to save them for after
feature parity has been reached and development on 2.x has halted.
Previous threads:
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0...
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0...
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0...
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/0019...
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/0019...
If anyone has any strong objections to this strategy, please reply.
Estimated Timeline:
* June 26, 2.4 release
* July 30, 3.0 reaches rough feature parity
* Fall sometime: 3.0 merged into main 'yt' branch
* Some time after that: 3.0 released
Criteria for Feature Parity:
* All patch-based AMR codes support data region selection
* Parallelism is functional for all parallel-capable routines
* All patch-based AMR codes support projections, slices, cutting planes
* All patch-based AMR codes support volume rendering
* All patch-based AMR codes support halo finding
* Octree-based AMR codes (RAMSES and ART) support data region selection
* Octree-based AMR codes (RAMSES and ART) support slices, projections
and cutting planes
If any of you are interested in helping with this reengineering
effort, I would be extremely interested in collaborating. My goal for
this effort is to focus on making yt a future-compatible code, and
changing it from an organically grown system into one that we design
based on the years of experience we all have now had with it.
-Matt
6 years, 7 months
Re: [yt-dev] off_axis_projection
by Cameron Hummels
Hey guys,
I guess I don't entirely understand why these two functionalities are so
different, but I'll chat with you, Matt, a bit more about this offline
when I'm back at CU.
Cameron
> Okay.
>
> Cameron, I know you rely upon the off_axis_projection -- would it be
> okay, in your opinion, if we deprecated the interpolated = True option
> in some 2.X release and then in 3.0, if you want interpolation, you
> need to set up a projection camera yourself? I ask because the setup
> for these two is very different, and (in the spirit of the plot
> window!) I'd like to try to reduce the complexity. If you're -1 on
> this, then we can keep it the way it is.
>
> On Thu, Jun 28, 2012 at 11:57 AM, Sam Skillman <samskillman(a)gmail.com>
> wrote:
>> That capability doesn't exist yet for interpolated=True. When that is
>> possible, I would be +1 with moving from volume to data_source.
>>
>>
>> On Thu, Jun 28, 2012 at 9:53 AM, Matthew Turk <matthewturk(a)gmail.com>
>> wrote:
>>>
>>> On Thu, Jun 28, 2012 at 11:51 AM, Sam Skillman <samskillman(a)gmail.com>
>>> wrote:
>>> > Hi all,
>>> >
>>> > I think the only kwarg I'd be really +1 with getting rid of would be
>>> > no_ghost. interpolated=True should just trigger no_ghost=False, and
>>> > interpolated=False doesn't use the ghost zones no matter what. I
>>> think
>>> > volume should stay, especially since I think it will soon be possible
>>> to
>>> > do
>>> > an off_axis_projection of a data object which would probably be fed
>>> in
>>> > through the volume.
>>>
>>> Why not get rid of volume and instead supply data_source, like
>>> projections?
>>>
>>> >
>>> > Sam
>>> >
>>> >
>>> > On Thu, Jun 28, 2012 at 9:39 AM, Matthew Turk <matthewturk(a)gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi Elizabeth,
>>> >>
>>> >> I agree, this makes sense.
>>> >>
>>> >> Last night I tried to take a crack at this, but I got very confused
>>> >> with the various definitions of volume, interpolated, etc etc, in
>>> the
>>> >> routine off_axis_projection. Cameron and Sam, do you think it would
>>> >> be okay if we sim
6 years, 8 months
Odd compile, OSX10.6, OMP, x86 vs. i386
by david collins
Hi, all--
Sam and I tracked down an odd glitch. It started with trying to run
the test runner, and I got an error on the "from
yt.utilities.answer_testing.api import" (full trace below)
For some reason the build of yt/utilities/lib/grid_traversal.pyx was
looking for $DEST_DIR/lib64 and not finding it, trying to use
/usr/local/lib/libgomp.dylib which, for some reason is built for i386
rather than x86_64. Putting /usr/local/lib/x86_64 in LD_LIBRARY_PATH
didn't seem to work, but soft linking /usr/local/lib/x86_64 to
$DEST_DIR/lib64 worked.
More elegant solutions are welcome, but this worked for me.
d.
/run>./test_runner.py --quicksuite=True
--output-dir=/Users/dccollins/ThisScratch/TestRunnerFiducial
Traceback (most recent call last):
File "./test_runner.py", line 29, in <module>
from yt.utilities.answer_testing.api import \
File "/Users/dccollins/local2/src/yt-hg/yt/utilities/answer_testing/__init__.py",
line 26, in <module>
import runner
File "/Users/dccollins/local2/src/yt-hg/yt/utilities/answer_testing/runner.py",
line 26, in <module>
import matplotlib
File "/Users/dccollins/local2/lib/python2.7/site-packages/matplotlib/__init__.py",
line 133, in <module>
from matplotlib.rcsetup import (defaultParams,
File "/Users/dccollins/local2/lib/python2.7/site-packages/matplotlib/rcsetup.py",
line 19, in <module>
from matplotlib.colors import is_color_like
File "/Users/dccollins/local2/lib/python2.7/site-packages/matplotlib/colors.py",
line 52, in <module>
import numpy as np
File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/__init__.py",
line 137, in <module>
import add_newdocs
File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/add_newdocs.py",
line 9, in <module>
from numpy.lib import add_newdoc
File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/lib/__init__.py",
line 4, in <module>
from type_check import *
File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/lib/type_check.py",
line 8, in <module>
import numpy.core.numeric as _nx
File "/Users/dccollins/local2/lib/python2.7/site-packages/numpy/core/__init__.py",
line 5, in <module>
import multiarray
ImportError: dlopen(/Users/dccollins/local2/lib/python2.7/site-packages/numpy/core/multiarray.so,
2): no suitable image found. Did find:
/Users/dccollins/local2/lib/python2.7/site-packages/numpy/core/multiarray.so:
mach-o, but wrong architecture
--
Sent from my computer.
6 years, 8 months
possible Forthon 0.8.8 and 0.8.9 issues?
by Geoffrey So
Has anyone installed yt-dev from scratch recently and used the newer than
0.8.7 Forthon(s)? I was able to use make to compile the 8.8 and 8.9 after
removing the --with-numpy argument from the Makefile inside the
yt/utilities/kdtree, however, I get
fKD' is not defined
when I use functions that requires the kd-tree. I was able to bypass this
problem by reinstalling YT and then using Forthon 0.8.7 (removing Forthon
8.8/8.9 then installing 8.7 then reinstallling YT did not work). I was
wondering if anyone else ran into the same issue? Not urgent because 8.7
works.
From
G.S.
6 years, 8 months
Fwd: [IPython-dev] [ANN] IPython 0.13 is officially out!
by Matthew Turk
---------- Forwarded message ----------
From: "Fernando Perez" <fperez.net(a)gmail.com>
Date: Jun 30, 2012 4:56 AM
Subject: [IPython-dev] [ANN] IPython 0.13 is officially out!
To: "IPython Development list" <ipython-dev(a)scipy.org>, "IPython User list"
<ipython-user(a)scipy.org>, "Discussion of Numerical Python" <
numpy-discussion(a)scipy.org>, "Matplotlib Users" <
matplotlib-users(a)lists.sourceforge.net>, "SciPy Users List" <
scipy-user(a)scipy.net>
Hi all,
on behalf of the IPython development team, and just in time for the
imminent Debian freeze and SciPy 2012, I'm thrilled to announce, after
an intense 6 months of work, the official release of IPython 0.13.
This version contains several major new features, as well as a large
amount of bug and regression fixes. The previous version (0.12) was
released on December 19 2011, so in this development cycle we had:
- ~6 months of work.
- 373 pull requests merged.
- 742 issues closed (non-pull requests).
- contributions from 62 authors.
- 1760 commits.
- a diff of 114226 lines.
This means that we closed a total of 1115 issues over 6 months, for a
rate of almost 200 issues closed per month and almost 300 commits per
month. We are very grateful to all of you who have contributed so
enthusiastically to the project and have had the patience of pushing
your contributions through our often lengthy review process.
We've also welcomed several new members to the core IPython
development group: Jörgen Stenarson (@jstenar - this really was an
omission as Jörgen has been our Windows expert for a long time) and
Matthias Bussonier (@Carreau), who has been very active on all fronts
of the project.
*Highlights*
There is too much new work to write up here, so we refer you to our
full What's New document
(http://ipython.org/ipython-doc/rel-0.13/whatsnew/version0.13.html)
for the full details. But the main highlights of this release are:
* Brand new UI for the notebook, with major usability improvements
(real menus, toolbar, and much more)
* Manage all your parallel cluster configurations from the notebook
with push-button simplicity (cluster start/stop with one button).
* Cell magics: commands prefixed with %% apply to an entire cell. We
ship with many cell magics by default, including timing, profiling,
running cells under bash, Perl and Ruby as well as magics to interface
seamlessly with Cython, R and Octave.
* The IPython.parallel tools have received many fixes, optimizations,
and a number of API improvements to make writing, profiling and
debugging parallel codes with IPython much easier.
* We have unified our interactive kernels (the basic ipython object
you know and love) with the engines running in parallel, so that you
can now use all IPython special tricks in parallel too. And you can
connect a console or qtconsole to any parallel engine for direct,
interactive execution, plotting and debugging in a cluster.
*Downloads*
Download links and instructions are at: http://ipython.org/download.html
And IPython is also on PyPI: http://pypi.python.org/pypi/ipython
Those contain a built version of the HTML docs; if you want pure
source downloads with no docs, those are available on github:
Tarball: https://github.com/ipython/ipython/tarball/rel-0.13
Zipball: https://github.com/ipython/ipython/zipball/rel-0.13
Please see our release notes for the full details on everything about
this release:
http://ipython.org/ipython-doc/rel-0.13/whatsnew/version0.13.html
As usual, if you find any other problem, please file a ticket --or
even better, a pull request fixing it-- on our github issues site
(https://github.com/ipython/ipython/issues).
Many thanks to all who contributed!
Fernando, on behalf of the IPython development team.
http://ipython.org
_______________________________________________
IPython-dev mailing list
IPython-dev(a)scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev
6 years, 8 months
off_axis_projection
by Elizabeth Tasker
Hi,
"off_axis_projection" doesn't seem to accept field parameters.
This isn't a big problem and I can definitely get around it by using global variables, but should it have a field_parameter option for consistency with add_projection?
Elizabeth
6 years, 9 months
broken ellipsoid
by Geoffrey So
I was trying to use my ellipsoid in the dev-yt installation to check
things, and apparently some of the changes of defining RX,Y,Z relative to
get_rotation_matrix in math_utils.py didn't get pulled in or was left out
and replaced with calls to get_rotation_matrix, but one of the lines in
halo_objects.py has a typo:
r1 = (e0_vector * rz).sum(axis = 1)
should be replaced with
r1 = (e0_vector * RZ).sum(axis = 1)
where RZ are both capitalized.
Fixing that makes the ellipsoid work, can someone check it in please?
Thanks in advance.
From
G.S.
6 years, 9 months
Call for testing: Projection performance
by Matthew Turk
Hi all,
I implemented this "quadtree extension" that duplicates the quadtree
on all processors, which may make it nicer to scale projections.
Previously the procedure was:
1) Locally project
2) Merge across procs:
2a) Serialize quadtree
2b) Point-to-point communciate
2c) Deserialize
2d) Merge local and remote
2d) Repeat up to 2a
3) Finish
I've added a step 0) which is "initialize entire quadtree", which
means all of step 2 becomes "perform sum of big array on all procs."
This has good and bad elements: we're still doing a lot of heavy
communication across processors, but it will be managed by the MPI
implementation instead of by yt. Also, we avoid all of the costly
serialize/deserialize procedures. So for a given dataset, step 0 will
be fixed in cost, but step 1 will be reduced as the number of
processors goes up. Step 2, which now is a single (or two)
communication steps, will increase in cost with increasing number of
processors.
So, it's not clear that this will *actually* be helpful or not. It
needs testing, and I've pushed it here:
bb://MatthewTurk/yt/
hash 3f39eb7bf468
If anybody out there could test it, I'd be might glad. This is the
script I've been using:
http://paste.yt-project.org/show/2343/
I'd *greatly* appreciate testing results -- particularly for proc
combos like 1, 2, 4, 8, 16, 32, 64, ... . On my machine, the results
are somewhat inconclusive. Keep in mind you'll have to run with the
option:
--config serialize=False
to get real results. Here's the shell command I used:
( for i in 1 2 3 4 5 6 7 8 9 10 ; do mpirun -np ${i} python2.7 proj.py
--parallel --config serialize=False ; done ) 2>&1 | tee proj_new.log
Comparison against results from the old method would also be super helpful.
The alternate idea that I'd had was a bit different, but harder to
implement, and also with a glaring problem. The idea would be to
serialize arrays, do the butterfly reduction, but instead of
converting into data objects simply progressively walk hilbert
indices. Unfortunately this only works for up to 2^32 effective size,
which is not going to work in a lot of cases.
Anyway, if this doesn't work, I'd be eager to hear if anybody has any ideas. :)
-Matt
6 years, 9 months
Summary of a yt sprint
by Matthew Turk
Hi all,
This last weekend, John, Nathan, Britton, Sam and I were fortunate
enough to be in the same place for a couple days and we had a sprint
on some yt functionality. Almost all of the changes have gone into
the code base already, but we wanted to share them with you all. Feel
free to chime in with questions, comments, or more information.
= Parallelism Improvements =
Britton implemented a pure-MPI task queuing system that works with the
existing parallel_objects machinery. It is currently not threaded
(and opinions vary as to whether that would be productive or not) and
so one MPI task is dedicated as the controller node that distributes
objects. This can either be manually used or you can simply supply
"dynamic=True" to the parallel_objects machinery. I have to say, the
changes to yt's parallelism in the last year or so have been pretty
much my favorite improvements, and Britton really knocked this one out
of the park, in my opinion. As I noted in my other email, with the
pull request for Reason, Britton also added an MPI-parallel backend
for Reason. I won't repeat my email here, but if you want to test it
out you'll need Pyro4 and the launcher script in scripts/ . This will
be documented in more detail when I finish up the remaining task,
which is to deliver intermediate payloads from the MPI controller back
to the web browser.
= Plot Window =
Perhaps Nathan would be better at describing this, but he's now
implemented a mechanism (built on Jeff's initial design) for
generating nice plots easily and complex plots straightforwardly.
This was discussed here:
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/00211...
Nathan, anything you wanted to add?
= Reason =
This was described in my other email as well:
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/00211...
and this PR:
https://bitbucket.org/yt_analysis/yt/pull-request/179/reason-gui-refactor...
If anybody can test this out, I'd appreciate it. I did note this
weekend that when running simulations, I spied a few of the other
sprinters using it. I think seeing developers using something means
it has "arrived." (And I know Sam has a volume rendering widget in
mind ...)
= Ghost Zones =
John spent a good amount of time implementing a fast sibling finder,
as well as a ghost zone filling mechanism. This is designed to
replace the existing mechanism of a smoothed covering grid, which
requires interpolation from the coarsest level up to the requested
level. This should lead to a considerable performance increase when
doing anything having ghost zones. I think this is going to be PR'd
sometime in the next while.
--
Anyway, I think I speak for everybody when I say it was a pretty
successful sprint, and we'd love to hear your feedback!
-Matt
6 years, 9 months
Setup script uses the wrong libpng headers
by Nathan Goldbaum
Hi all,
I've just run into an odd issue. I'm using Matt's yt-extJS4 fork to make 2D plots. In doing so, I discovered that png_writer.so is being compiled with the wrong libpng.h header.
This produces errors that look like:
libpng warning: Application was compiled with png.h from libpng-1.5.4
libpng warning: Application is running with png.c from libpng-1.2.43
libpng error: Incompatible libpng version in application and library
Segmentation fault: 11
Does anyone know why the setup script isn't finding the libpng headers that come with yt?
Thanks for your help,
Nathan
6 years, 9 months
