Hi all,
I implemented this "quadtree extension" that duplicates the quadtree
on all processors, which may make it nicer to scale projections.
Previously the procedure was:
1) Locally project
2) Merge across procs:
2a) Serialize quadtree
2b) Point-to-point communciate
2c) Deserialize
2d) Merge local and remote
2d) Repeat up to 2a
3) Finish
I've added a step 0) which is "initialize entire quadtree", which
means all of step 2 becomes "perform sum of big array on all procs."
This has good and bad elements: we're still doing a lot of heavy
communication across processors, but it will be managed by the MPI
implementation instead of by yt. Also, we avoid all of the costly
serialize/deserialize procedures. So for a given dataset, step 0 will
be fixed in cost, but step 1 will be reduced as the number of
processors goes up. Step 2, which now is a single (or two)
communication steps, will increase in cost with increasing number of
processors.
So, it's not clear that this will *actually* be helpful or not. It
needs testing, and I've pushed it here:
bb://MatthewTurk/yt/
hash 3f39eb7bf468
If anybody out there could test it, I'd be might glad. This is the
script I've been using:
http://paste.yt-project.org/show/2343/
I'd *greatly* appreciate testing results -- particularly for proc
combos like 1, 2, 4, 8, 16, 32, 64, ... . On my machine, the results
are somewhat inconclusive. Keep in mind you'll have to run with the
option:
--config serialize=False
to get real results. Here's the shell command I used:
( for i in 1 2 3 4 5 6 7 8 9 10 ; do mpirun -np ${i} python2.7 proj.py
--parallel --config serialize=False ; done ) 2>&1 | tee proj_new.log
Comparison against results from the old method would also be super helpful.
The alternate idea that I'd had was a bit different, but harder to
implement, and also with a glaring problem. The idea would be to
serialize arrays, do the butterfly reduction, but instead of
converting into data objects simply progressively walk hilbert
indices. Unfortunately this only works for up to 2^32 effective size,
which is not going to work in a lot of cases.
Anyway, if this doesn't work, I'd be eager to hear if anybody has any ideas. :)
-Matt
Hi everyone,
I have a couple of comments related to the negative comment on the yt survey. In particular, there is one part I agree with and one I do not:
The comment about yt being heavily cosmology based I do not agree with. As a non-cosmo simulator --one who still intends to torch a fellow developer's house for declaring all my research 'test problems' at a user meeting-- I use yt for all my simulation analysis and use a wide range of its capabilities. In fact, the only part I can think that I do not use is the halo finder, but I do use connected sets which is a similar idea for gas.
I therefore feel this comment is not part of the general consensus and can be ignored.
However, the comment about the documentation being from the user's rather than the programmer's, point of view is interesting and think worth some thought. For a while, I skirted around yt, aware that it was powerful but baulking at the effort needed to use it comfortably. Last summer, I had to write an entirely new analysis program for my simulations and I bit the bullet and starting working extensively with yt and python. I've never picked up an IDL script since, but there was definitely an energy bump. I also think that I do find the documentation **significantly** clearer now I approach it from a programmer's perspective.
I have a couple of suggestions of what I think might help with this:
(1) The quickest option is to keep everything the same but add more actual examples. For instance, the routine call, e.g.
contour(self, field, ncont=5, factor=4, take_log=False, clim=None, plot_args=None):
is confusion for a non-programmer. When I first came to yt, I spent a while with thoughts like:
What is 'self'? Do I need to pass it like in C++ with Grid-> ?
Although it's slightly painful, I think it would be much improved if there was an actual call example for each one of the callbacks and other tools. So we'd have:
contour(self, field, ncont=5, factor=4, take_log=False, clim=None, plot_args=None):
(This is a proxy for ContourCallback.)
pc.modify["contours"]("Density", ncont=1, take_log=False)
Add contours in field to the plot. ncont governs the number of contours generated, factor governs the number of points used in the interpolation, take_log governs how it is contoured and clim gives the (upper, lower) limits for contouring.
(2) My second suggestion involves the gui and concerns the way yt is presented. Currently, we talk about scripts and then mention there is a gui. How about we do it the other way around? So a new user's first port of call should be the gui. Then, we get to the section "beyond the gui" and talk about scripting and cookbook-ing and then finally onto full simulation analysis, i.e. writing whatever analysis package you like in python, but making use of the routines yt has.
For instance, in the workshop we just had at Hokudai, we could have stood up and brought up the yt gui before anything else. *Then* gone onto scripts.
The docs could be rearranged in a similar fashion.
I think this might provide a much faster way for new users to start getting into yt and make the gui of much more interest. Currently, people come to it after they learn to script at which point, its value is much less because it's inevitably more limited.
Elizabeth
PS today's google-doodle (or possible tomorrow's for the USA) is awesome.
Hi all (especially Sam),
What is blocking the merge of volume_refactor into the main dev
branch? My recollection is that the only remaining issue was the
b2f/f2b ordering, which I thought Sam had taken a successful swing at.
-Matt
---------- Forwarded message ----------
From: Bitbucket <notifications-noreply(a)bitbucket.org>
Date: Fri, May 25, 2012 at 6:30 AM
Subject: [yt_analysis/yt] File open dialog in reason (pull request #161)
To: yt(a)enzotools.org
A new pull request has been opened by Matthew Turk.
MatthewTurk/yt has changes to be pulled into yt_analysis/yt.
https://bitbucket.org/yt_analysis/yt/pull-request/161/file-open-dialog-in...
Title: File open dialog in reason
This adds a first pass at a "File Open" menu item and dialog in
Reason. Comments welcome!
Changes to be pulled:
--
This is an issue notification from bitbucket.org.
You are receiving this either because you are the participating
in a pull request, or you are following it.
Hi everyone,
I've closed up the 2012 community survey. We had a fair number of
responses (24), which I think is a good cross-section of the
community. (There are 134 people on yt-users, so this is just under
20% of the list population, many of whom may not be actual or active
users.)
For the most part, it was very, very positive. We're definitely doing
some things very right. However, we had one *particularly* negative
response (explored in detail below), and a good amount of constructive
criticism scattered through the positive responses. I'll detail the
particularly negative response, and then discuss the rest in bulk. At
the end I've included the set of free-form comments left in the
survey. I think the big takeaways there are pretty clear, and mostly
are on the roadmap!
I'm happy to make available the original responses, but I've collated
most of them here. I have no doubt there is a *huge* selection
effect, and I'm *really* grateful that the strongly negative response
was left at all, so that we could get an insight into what went wrong!
Finally, I'd encourage everyone reading this to reply if you have
comments, suggestions, or any feedback that did not make it into the
survey. I don't want this to be the end of the conversation, but
rather the start of the feedback process.
= Negative =
The negative response was, indeed, quite negative. (And, it was
difficult to read, to be honest.) Here are the comments, unedited.
--
* What should be changed about the documentation?
"I want more narrative documentation, It should be written from the
user's rather than the programmer's point of view. Start with simple
tasks, and then build up to scripting and modification only slowly."
* Have you found anything in yt to be particularly annoying?
"Unreliability -- both times I've tried to use it, I had to invoke
developer support to even get basic work done. Terribly written
documentation -- virtually incomprehensible if you aren't both a
cosmologist and a programmer."
* If there's one message about yt you want to send, what is it?
"The concepts are powerful, but it should either be stated to be
specifically for cosmological visualization only, or its general
purpose use should be better supported."
* How do you find the yt community to be?
"Helpful and friendly, but rather too convinced of yt's usefulness, so
that people get sucked in for whom the tool is not at all ready yet."
* Anything to add about deficiencies in yt?
"It is far too focused on the cosmology use case to be useful as a
general purpose visualization tool. This impacts the documentation,
which only makes sense if you are in the mindset of halo-finding and
object property measurement, and the testing, which relies on users to
find such basic things as support for 2D model visualizations (the
most basic test problems) or non-square grids in nominally supported
codes."
--
Basically, my takeaways from this remarkably well-written, if blunt,
takedown were:
* The submitter thinks we're pretty self-satisfied, which gets in the
way of supporting new users.
* The submitter finds that we're cosmology-biased. I don't know that
this is necessarily the case, but it seems to come across in our
supporting materials.
* The submitter believes we have positioned ourselves as "general
purpose" but I'm not sure if this is in opposition to "astrophysically
focused" or in opposition to "cosmology focused"
* Our support for simplified dimensionality and test problems seems
lacking. I don't know how much to read in to the "nominally supported
codes" section. I assume this means FLASH, but it's not clear which
version of FLASH.
* We need much, much better and more descriptive error handling.
I am a bit sorry that this submitter did not include contact
information and instead chose to submit anonymously. A followup
discussion would have been very productive, particularly because I
think this response will spur a number of changes. I've shared this
response with a few other people on this list, and all of them have
been eager to make changes in response to the comments. It would have
been nice to be able to discuss them with this person, and to start an
ongoing discussion.
However, regardless, this should act as a wakeup call. I will
endeavor to ensure that we support reduced dimensionality test
problems, and during the next iteration of the documentation I hope we
can iterate on the tutorial, the narrative docs, and so on. I suppose
in the past we have neglected this population of computational
astrophysicists that are not as at home with programming, so perhaps
either adding better educational materials *or* identifying existing
resources that exist (and linking to them) would be useful.
One plan I'd had had been to add screencasts. I think organizing
these as how to install, then moving on to how to get going with
*each* simulation code type (and providing explicit sample scripts and
example data) independently would be a good strategy.
Does anyone have any additional thoughts on this? Furthermore, does
anyone else want to pile on?
= All Results =
We had 24 replies. Of these people:
--
20 (80%) said yt was easy to use
3 (12%) said yt was difficult to use
1 (4%) said yt was too hard to use
--
I think this is *great*. And, I think it's a huge compliment to the
work we've all put in to make the code easier to use.
--
16 (64%) said they use yt all the time (!)
7 (28%) use it once in a while
1 (4%) tried it, it broke, they stopped
--
Again, wow. While it's difficult to read too much into this because
of the selection bias, it's still great. And the final response is
valuable to put into context the overall survey results.
--
The GUI Reason:
2 (8%) Use it often
10 (40%) Use it once in a while
11 (44%) Don't use it
1 (4%) Never heard of it
--
This is interesting, but I'd put it out to the rest of you to
interpret. Sam, Jeff, thoughts?
--
The thing yt needs more than anything else:
2 (8%) More supported codes
3 (12%) Speed or parallelism improvements
6 (24%) More canned tasks for astrophysical analysis
1 (4%) Better/easier visualization not destined for publication
5 (20%) Better/easier visualization for publication
0 (0%) Improvements to the GUI
3 (12%) API documentation
3 (12%) Narrative documentation and tutorials
0 (0%) A more friendly development process or community
0 (0%) More reliability or error handling
0 (0%) Ability to conduct time series analysis
2 Other 2
--
It seems like the top two priorities are better/easier publication
quality viz, and canned astrophysical tasks. This is excellent, as
they are both well within our wheelhouse. One thing we should start
pushing harder on is encouraging users to submit analysis modules
upstream -- and to create uniform interfaces for analysis modules,
particularly ones that work on time series. Britton and I have talked
about smoothing up the various interfaces, making things more
"Dataflow" or "Pipeline" oriented, and this is a good point of focus
for that.
One sad bit is that I really want to improve the GUI (and I think John
and Sam have expressed a similar desire, and I know Jeff has) but it
looks like either due to indifference or its capabilities, this isn't
a priority for others.
--
What should be changed about the documentation?
0 (0%) There's too much -- slim it down!
7 (32%) I want more narrative documentation
17 (77%) I want more example scripts
6 (27%) The organization should be improved
2 (9%) I'd like more screencasts
6 (27%) Other
--
More examples! We have let the cookbook falter a bit. That should be
reorganized and made more extensive.
Of the 24 responses, 14 had watched one or more videos from the
workshop. This was the most surprising response.
--
Which of these pieces of functionality have you used:
10 (48%) yt pastebin (the yt pastebin)
6 (29%) Clump finder
11 (52%) Halo finder
17 (81%) Volume renderer
9 (43%) IRC for yt help
6 (29%) Time series analysis
--
I found this the most interesting. More people have used the volume
renderer than have used the pastebin! I guess it's a hit, and I would
guess it's also a big draw.
--
Volume rendering in yt is ...
15 (75%) Attractive
13 (65%) Hard to get right
2 (10%) Slow
2 (10%) Easy to set up
--
This completely jives with my experience. VR is super hard to make
look nice, easy to make look terrible, and gives great results one
you've set it up. I think coming up with new methods for rapid
iteration, pre-visualization, and so on are all going to be important
in the near future.
= Specific Comments =
Have you found anything in yt to be particularly annoying?
* It may be related with python. Particular version of yt need
particular version of python. It particularly makes an issue with mac
OS X.. (it is partially mac OSX fault). Now days, yt install its own
python in to mac but it makes more work... For example, if I want to
make my own routine with scipy and yt, it may cause issue.
* Deleting the *yt and *harrays for making some script work.
* "Unreliability -- both times I've tried to use it, I had to invoke
developer support to even get basic work done. Terribly written
documentation -- virtually incomprehensible if you aren't both a
cosmologist and a programmer."
* Most of the "cookbook" relies on plot collections which are clunky
in some situations. It would be nice to see more cookbook entries for
generating plots, figures, etc without relying on this data structure.
* I get lost in a hurry if I do something non-standard and need to
look under the hood. More under-the-hood docs would be cool.
* Yes, I found that simple funtions like figure control (i.e.,
changing axis, symbles, grid, changing font and size of text etc) are
quite annoying. I had difficulties in plotting multiple plots etc.
Size of simulation box is not displayed in YT and one has to manually
mention which could be useful.
If there's one message about yt you want to send, what is it?
* You guys are doing a great job - keep it up!
* Thanks!
* It is good tool! I hope it will continuously and actively improve.
* "Keep up the great job!
* Keep developing this great project! Congrats
* The concepts are powerful, but it should either be stated to be
specifically for cosmological visualization only, or its general
purpose use should be better supported.
* yt makes my life easier! Keep it up.
* Keep doing the good work.
* Good work thus far.
* yt is elegant.
How do you find the yt community to be?
* Very welcoming - no complaints!
* Great!
* friendly, helpful + dedicated
* It's a great place for receiving and giving help and suggestions
about real work.
* The friendliest community I know :)
* Great!
* Helpful and friendly, but rather too convinced of yt's usefulness,
so that people get sucked in for whom the tool is not at all ready
yet.
* I haven't been active on the list at all, but it seems like
response times are (extremely) short, and everyone is very helpful.
* Love it!
* Friendly and helpful.
* Invaluable! The IRC channel and email list are incredibly helpful
resources when something is missing from the docs or a bug creeps in
the code.
* Excellent. Super helpful.
* I found them very helpful.
* Very responsive and helpful.
Have you ever contributed code to yt? If so, what was good or bad
about that experience?
* Not yet.
* Not yet, but I look forward to it!
* Yes, hg was a bit rough experience at the beginning, but as soons
as I've achieved rookie level, I'm fine with it.
* No.
* Haven't made any changes.
* yes, a little bit. It was a very good experience.
* Yes. I felt it fixed a problem, and it was well-received.
* No
* Wish someone have short example of write code, add to doc, add test problem.
Anything to add about deficiencies in yt?
* Most of the deficiencies that ails me are already covered in yt-3.0
doc: multiple fluids, handling non-cartesian grids, handling vector
fields
* It is far too focused on the cosmology use case to be useful as a
general purpose visualization tool. This impacts the documentation,
which only makes sense if you are in the mindset of halo-finding and
object property measurement, and the testing, which relies on users to
find such basic things as support for 2D model visualizations (the
most basic test problems) or non-square grids in nominally supported
codes.
* It would be great to have more direct access to matplotlib axes
objects, say when you make a PlotCollection, to allow for easy
manipulation of tick marks, axes labels, etc.
* I would like volume rendering to be easier to use correctly (i.e.
interactive isocontour finding).
* I don't like the choices for the first question-- I think yt is
very easy for some things, and gets more difficult as you get off the
beaten path in a non-linear manner.
* Figure controls, 3-D visualization, Better GUI not just web based,
Importing data into YT with simple formats like ASCII or Binary etc,
Imroved halofinder like T_vir
* I would like to see it read Gadget snapshots.
* need transfer function alongside volume rendering in log space, or
make it customizable for the user. Right now it's printed in real
space so if you have multiple add gaussian, the smaller ones are
barely noticeable, and the background being white is not good for if
the volume rendering is done with a background. Would be nice to be
able to customize it. Maybe add transfer function and ability to add
strings of text to describe the volume rendering off to the sides or
top/bottom.
If you have trouble viewing or submitting this form, you can fill it out
online:
https://docs.google.com/spreadsheet/viewform?formkey=dEtHNXFXbXZJTXZUQ3dW...
yt users survey, Spring 2012
It's that time of year again! We'd like some feedback on your use and your
opinions about the present and future of yt. It's really short, and no
questions are mandatory, so feel free to fill in as many or as few as you'd
like.
Do you find that yt is:
Easy to use
Difficult to use
Too hard to use
How often do you use yt?
All the time
Once in a while
I tried, and it broke, so I stopped
Have you used the yt GUI, Reason?
Yes, often
Yes, once or twice
No
Never heard of it
The thing I think yt needs, more than anything else is ...
More supported codes
Speed or parallelism improvements
More canned tasks for astrophysical analysis
Better/easier visualization not destined for publication
Better/easier visualization for publication
Improvements to the GUI
API documentation
Narrative documentation and tutorials
A more friendly development process or community
More reliability or error handling
Ability to conduct time series analysis
Other:
Anything to add about deficiencies in yt?
What should be changed about the documentation?
There's too much -- slim it down!
I want more narrative documentation
I want more example scripts
The organization should be improved
I'd like more screencasts
Other:
Have you watched the yt workshop videos?
I was not there and did not watch any videos
I was not there and did watch one or more videos
I was there and did not watch any videos
I was there and did watch one or more videos
Which of these pieces of functionality have you used:
yt pastebin (the yt pastebin)
Clump finder
Halo finder
Volume renderer
IRC for yt help
Time series analysis
Volume rendering in yt is ...
Attractive
Hard to get right
Slow
Easy to set up
Have you found anything in yt to be particularly annoying?
How do you find the yt community to be?
Have you ever contributed code to yt? If so, what was good or bad about
that experience?
If there's one message about yt you want to send, what is it? If you want
to be contacted about anything you've written, put your contact info here.
Powered by Google Docs Report Abuse - Terms of Service - Additional Terms
After our discussion over the last week, I've issued a Pull Request.
I don't believe it's ready yet -- for instance, John reported a big
uptick in memory usage. However, I think we're nearing that point.
Please feel free to send comments to the other thread, rather than
leaving them on the PR itself.
-Matt
---------- Forwarded message ----------
From: Matthew Turk <pullrequests-noreply(a)bitbucket.org>
Date: Mon, May 7, 2012 at 10:29 PM
Subject: [yt_analysis/yt] Change the way quad tree projections are
reduced (pull request #156)
To: yt(a)enzotools.org
A new pull request has been opened by Matthew Turk.
MatthewTurk/yt has changes to be pulled into yt_analysis/yt.
https://bitbucket.org/yt_analysis/yt/pull-request/156/change-the-way-quad...
Title: Change the way quad tree projections are reduced
This implements the quad-tree extension code discussed in this thread:
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-May/001986...
Changes to be pulled:
--
This is an issue notification from bitbucket.org.
You are receiving this either because you are the participating
in a pull request, or you are following it.
Hi, I was installing YT using the default intel/mvapich2 modules loaded,
>module list
Currently Loaded Modulefiles:
1) binutils/2.22 2) intel/2011 3) mvapich2_ib/1.8a1p1
but when I tried to do >make in the kdtree folder after installing Forthon
I got:
Using Forthon ()
Forthon -F gfortran --compile_first fKD_source --no2underscores --fopt
"-O3" fKD fKD_source.f90
Building package fKD
Forthon fKD.v
F90Free fKD_p.F90
/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/f951: error while loading shared
libraries: libgmp.so.3: cannot open shared object file: No such file or
directory
make[1]: *** [fKD_p.o] Error 1
make: *** [fKD] Error 2
When I did a >locate libgmp.so.3 it only shows under gnu folders
/opt/gnu/gmp/4.3.2/lib/libgmp.so.3
/opt/gnu/gmp/4.3.2/lib/libgmp.so.3.5.2
Since it's looking for libgmp, does that mean I have to use the gnu
compilers?
From
G.S.
I was trying to reinstall dev YT and enabling parallel HOP, but ran into
the trouble that at the end of installing Forthon, I got the error pasted
at the end, has anyone seen this? I tried using
>sudo python setup.py install
but it gave me the same errors, so subsequently my kdtree make fails.
From
G.S.
Permissions on Notice and Src files needs to be set by hand
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/Forthon
copying Lib/__init__.py -> build/lib/Forthon
copying Lib/_Forthon.py -> build/lib/Forthon
copying Lib/cfinterface.py -> build/lib/Forthon
copying Lib/compilers.py -> build/lib/Forthon
copying Lib/Forthon_builder.py -> build/lib/Forthon
copying Lib/Forthon_options.py -> build/lib/Forthon
copying Lib/ForthonTimer.py -> build/lib/Forthon
copying Lib/fvars.py -> build/lib/Forthon
copying Lib/interfaceparser.py -> build/lib/Forthon
copying Lib/preprocess.py -> build/lib/Forthon
copying Lib/wrappergen_derivedtypes.py -> build/lib/Forthon
copying Lib/wrappergenerator.py -> build/lib/Forthon
running build_scripts
creating build/scripts-2.7
copying and adjusting Forthon -> build/scripts-2.7
changing mode of build/scripts-2.7/Forthon from 644 to 755
running install_lib
running install_scripts
copying build/scripts-2.7/Forthon -> /Users/gso/Documents/el-yt/bin
changing mode of /Users/gso/Documents/el-yt/bin/Forthon to 755
running install_data
error: can't copy 'Notice': doesn't exist or not a regular file
Hi all,
I've been trying to fix a bug in the opaque rendering. It comes down to
which way the rays get integrated (back-to-front/front-to-back) with
respect to the camera position. Right now I think the behavior is such
that the bricks are handed to the sampler from the back to front, but the
bricks themselves are rendered front to back. Anyways, the bug shows up
better in parallel, and I've been using
http://paste.yt-project.org/show/2372/
with
https://bitbucket.org/samskillman/yt-refactor/changeset/b4db9a5b5704
The behavior ends up being something like this (this particular image might
have the opposite L than the script):
http://imgur.com/Od3np.png
If you get something that looks okay, rotate it to a few different
positions and see if it still works.
If anyone wants to dig in, I would appreciate the help.
Cheers,
Sam
