yt-dev May 2012

yt-dev@python.org

11 participants
15 discussions

Call for testing: Projection performance

by Matthew Turk

Hi all, I implemented this "quadtree extension" that duplicates the quadtree on all processors, which may make it nicer to scale projections. Previously the procedure was: 1) Locally project 2) Merge across procs: 2a) Serialize quadtree 2b) Point-to-point communciate 2c) Deserialize 2d) Merge local and remote 2d) Repeat up to 2a 3) Finish I've added a step 0) which is "initialize entire quadtree", which means all of step 2 becomes "perform sum of big array on all procs." This has good and bad elements: we're still doing a lot of heavy communication across processors, but it will be managed by the MPI implementation instead of by yt. Also, we avoid all of the costly serialize/deserialize procedures. So for a given dataset, step 0 will be fixed in cost, but step 1 will be reduced as the number of processors goes up. Step 2, which now is a single (or two) communication steps, will increase in cost with increasing number of processors. So, it's not clear that this will *actually* be helpful or not. It needs testing, and I've pushed it here: bb://MatthewTurk/yt/ hash 3f39eb7bf468 If anybody out there could test it, I'd be might glad. This is the script I've been using: http://paste.yt-project.org/show/2343/ I'd *greatly* appreciate testing results -- particularly for proc combos like 1, 2, 4, 8, 16, 32, 64, ... . On my machine, the results are somewhat inconclusive. Keep in mind you'll have to run with the option: --config serialize=False to get real results. Here's the shell command I used: ( for i in 1 2 3 4 5 6 7 8 9 10 ; do mpirun -np ${i} python2.7 proj.py --parallel --config serialize=False ; done ) 2>&1 | tee proj_new.log Comparison against results from the old method would also be super helpful. The alternate idea that I'd had was a bit different, but harder to implement, and also with a glaring problem. The idea would be to serialize arrays, do the butterfly reduction, but instead of converting into data objects simply progressively walk hilbert indices. Unfortunately this only works for up to 2^32 effective size, which is not going to work in a lot of cases. Anyway, if this doesn't work, I'd be eager to hear if anybody has any ideas. :) -Matt

8 years, 1 month

Re: [yt-dev] 2012 Community Survey Summary

by Elizabeth Tasker

Hi everyone, I have a couple of comments related to the negative comment on the yt survey. In particular, there is one part I agree with and one I do not: The comment about yt being heavily cosmology based I do not agree with. As a non-cosmo simulator --one who still intends to torch a fellow developer's house for declaring all my research 'test problems' at a user meeting-- I use yt for all my simulation analysis and use a wide range of its capabilities. In fact, the only part I can think that I do not use is the halo finder, but I do use connected sets which is a similar idea for gas. I therefore feel this comment is not part of the general consensus and can be ignored. However, the comment about the documentation being from the user's rather than the programmer's, point of view is interesting and think worth some thought. For a while, I skirted around yt, aware that it was powerful but baulking at the effort needed to use it comfortably. Last summer, I had to write an entirely new analysis program for my simulations and I bit the bullet and starting working extensively with yt and python. I've never picked up an IDL script since, but there was definitely an energy bump. I also think that I do find the documentation **significantly** clearer now I approach it from a programmer's perspective. I have a couple of suggestions of what I think might help with this: (1) The quickest option is to keep everything the same but add more actual examples. For instance, the routine call, e.g. contour(self, field, ncont=5, factor=4, take_log=False, clim=None, plot_args=None): is confusion for a non-programmer. When I first came to yt, I spent a while with thoughts like: What is 'self'? Do I need to pass it like in C++ with Grid-> ? Although it's slightly painful, I think it would be much improved if there was an actual call example for each one of the callbacks and other tools. So we'd have: contour(self, field, ncont=5, factor=4, take_log=False, clim=None, plot_args=None): (This is a proxy for ContourCallback.) pc.modify["contours"]("Density", ncont=1, take_log=False) Add contours in field to the plot. ncont governs the number of contours generated, factor governs the number of points used in the interpolation, take_log governs how it is contoured and clim gives the (upper, lower) limits for contouring. (2) My second suggestion involves the gui and concerns the way yt is presented. Currently, we talk about scripts and then mention there is a gui. How about we do it the other way around? So a new user's first port of call should be the gui. Then, we get to the section "beyond the gui" and talk about scripting and cookbook-ing and then finally onto full simulation analysis, i.e. writing whatever analysis package you like in python, but making use of the routines yt has. For instance, in the workshop we just had at Hokudai, we could have stood up and brought up the yt gui before anything else. *Then* gone onto scripts. The docs could be rearranged in a similar fashion. I think this might provide a much faster way for new users to start getting into yt and make the gui of much more interest. Currently, people come to it after they learn to script at which point, its value is much less because it's inevitably more limited. Elizabeth PS today's google-doodle (or possible tomorrow's for the USA) is awesome.

8 years, 2 months

What is blocking volume_refactor merge?

by Matthew Turk

Hi all (especially Sam), What is blocking the merge of volume_refactor into the main dev branch? My recollection is that the only remaining issue was the b2f/f2b ordering, which I thought Sam had taken a successful swing at. -Matt

8 years, 2 months

Fwd: [yt_analysis/yt] File open dialog in reason (pull request #161)

by Matthew Turk

---------- Forwarded message ---------- From: Bitbucket <notifications-noreply(a)bitbucket.org> Date: Fri, May 25, 2012 at 6:30 AM Subject: [yt_analysis/yt] File open dialog in reason (pull request #161) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/161/file-open-dialog-in-r… Title: File open dialog in reason This adds a first pass at a "File Open" menu item and dialog in Reason. Comments welcome! Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

8 years, 2 months

2012 Community Survey Summary

by Matthew Turk

Hi everyone, I've closed up the 2012 community survey. We had a fair number of responses (24), which I think is a good cross-section of the community. (There are 134 people on yt-users, so this is just under 20% of the list population, many of whom may not be actual or active users.) For the most part, it was very, very positive. We're definitely doing some things very right. However, we had one *particularly* negative response (explored in detail below), and a good amount of constructive criticism scattered through the positive responses. I'll detail the particularly negative response, and then discuss the rest in bulk. At the end I've included the set of free-form comments left in the survey. I think the big takeaways there are pretty clear, and mostly are on the roadmap! I'm happy to make available the original responses, but I've collated most of them here. I have no doubt there is a *huge* selection effect, and I'm *really* grateful that the strongly negative response was left at all, so that we could get an insight into what went wrong! Finally, I'd encourage everyone reading this to reply if you have comments, suggestions, or any feedback that did not make it into the survey. I don't want this to be the end of the conversation, but rather the start of the feedback process. = Negative = The negative response was, indeed, quite negative. (And, it was difficult to read, to be honest.) Here are the comments, unedited. -- * What should be changed about the documentation? "I want more narrative documentation, It should be written from the user's rather than the programmer's point of view. Start with simple tasks, and then build up to scripting and modification only slowly." * Have you found anything in yt to be particularly annoying? "Unreliability -- both times I've tried to use it, I had to invoke developer support to even get basic work done. Terribly written documentation -- virtually incomprehensible if you aren't both a cosmologist and a programmer." * If there's one message about yt you want to send, what is it? "The concepts are powerful, but it should either be stated to be specifically for cosmological visualization only, or its general purpose use should be better supported." * How do you find the yt community to be? "Helpful and friendly, but rather too convinced of yt's usefulness, so that people get sucked in for whom the tool is not at all ready yet." * Anything to add about deficiencies in yt? "It is far too focused on the cosmology use case to be useful as a general purpose visualization tool. This impacts the documentation, which only makes sense if you are in the mindset of halo-finding and object property measurement, and the testing, which relies on users to find such basic things as support for 2D model visualizations (the most basic test problems) or non-square grids in nominally supported codes." -- Basically, my takeaways from this remarkably well-written, if blunt, takedown were: * The submitter thinks we're pretty self-satisfied, which gets in the way of supporting new users. * The submitter finds that we're cosmology-biased. I don't know that this is necessarily the case, but it seems to come across in our supporting materials. * The submitter believes we have positioned ourselves as "general purpose" but I'm not sure if this is in opposition to "astrophysically focused" or in opposition to "cosmology focused" * Our support for simplified dimensionality and test problems seems lacking. I don't know how much to read in to the "nominally supported codes" section. I assume this means FLASH, but it's not clear which version of FLASH. * We need much, much better and more descriptive error handling. I am a bit sorry that this submitter did not include contact information and instead chose to submit anonymously. A followup discussion would have been very productive, particularly because I think this response will spur a number of changes. I've shared this response with a few other people on this list, and all of them have been eager to make changes in response to the comments. It would have been nice to be able to discuss them with this person, and to start an ongoing discussion. However, regardless, this should act as a wakeup call. I will endeavor to ensure that we support reduced dimensionality test problems, and during the next iteration of the documentation I hope we can iterate on the tutorial, the narrative docs, and so on. I suppose in the past we have neglected this population of computational astrophysicists that are not as at home with programming, so perhaps either adding better educational materials *or* identifying existing resources that exist (and linking to them) would be useful. One plan I'd had had been to add screencasts. I think organizing these as how to install, then moving on to how to get going with *each* simulation code type (and providing explicit sample scripts and example data) independently would be a good strategy. Does anyone have any additional thoughts on this? Furthermore, does anyone else want to pile on? = All Results = We had 24 replies. Of these people: -- 20 (80%) said yt was easy to use 3 (12%) said yt was difficult to use 1 (4%) said yt was too hard to use -- I think this is *great*. And, I think it's a huge compliment to the work we've all put in to make the code easier to use. -- 16 (64%) said they use yt all the time (!) 7 (28%) use it once in a while 1 (4%) tried it, it broke, they stopped -- Again, wow. While it's difficult to read too much into this because of the selection bias, it's still great. And the final response is valuable to put into context the overall survey results. -- The GUI Reason: 2 (8%) Use it often 10 (40%) Use it once in a while 11 (44%) Don't use it 1 (4%) Never heard of it -- This is interesting, but I'd put it out to the rest of you to interpret. Sam, Jeff, thoughts? -- The thing yt needs more than anything else: 2 (8%) More supported codes 3 (12%) Speed or parallelism improvements 6 (24%) More canned tasks for astrophysical analysis 1 (4%) Better/easier visualization not destined for publication 5 (20%) Better/easier visualization for publication 0 (0%) Improvements to the GUI 3 (12%) API documentation 3 (12%) Narrative documentation and tutorials 0 (0%) A more friendly development process or community 0 (0%) More reliability or error handling 0 (0%) Ability to conduct time series analysis 2 Other 2 -- It seems like the top two priorities are better/easier publication quality viz, and canned astrophysical tasks. This is excellent, as they are both well within our wheelhouse. One thing we should start pushing harder on is encouraging users to submit analysis modules upstream -- and to create uniform interfaces for analysis modules, particularly ones that work on time series. Britton and I have talked about smoothing up the various interfaces, making things more "Dataflow" or "Pipeline" oriented, and this is a good point of focus for that. One sad bit is that I really want to improve the GUI (and I think John and Sam have expressed a similar desire, and I know Jeff has) but it looks like either due to indifference or its capabilities, this isn't a priority for others. -- What should be changed about the documentation? 0 (0%) There's too much -- slim it down! 7 (32%) I want more narrative documentation 17 (77%) I want more example scripts 6 (27%) The organization should be improved 2 (9%) I'd like more screencasts 6 (27%) Other -- More examples! We have let the cookbook falter a bit. That should be reorganized and made more extensive. Of the 24 responses, 14 had watched one or more videos from the workshop. This was the most surprising response. -- Which of these pieces of functionality have you used: 10 (48%) yt pastebin (the yt pastebin) 6 (29%) Clump finder 11 (52%) Halo finder 17 (81%) Volume renderer 9 (43%) IRC for yt help 6 (29%) Time series analysis -- I found this the most interesting. More people have used the volume renderer than have used the pastebin! I guess it's a hit, and I would guess it's also a big draw. -- Volume rendering in yt is ... 15 (75%) Attractive 13 (65%) Hard to get right 2 (10%) Slow 2 (10%) Easy to set up -- This completely jives with my experience. VR is super hard to make look nice, easy to make look terrible, and gives great results one you've set it up. I think coming up with new methods for rapid iteration, pre-visualization, and so on are all going to be important in the near future. = Specific Comments = Have you found anything in yt to be particularly annoying? * It may be related with python. Particular version of yt need particular version of python. It particularly makes an issue with mac OS X.. (it is partially mac OSX fault). Now days, yt install its own python in to mac but it makes more work... For example, if I want to make my own routine with scipy and yt, it may cause issue. * Deleting the *yt and *harrays for making some script work. * "Unreliability -- both times I've tried to use it, I had to invoke developer support to even get basic work done. Terribly written documentation -- virtually incomprehensible if you aren't both a cosmologist and a programmer." * Most of the "cookbook" relies on plot collections which are clunky in some situations. It would be nice to see more cookbook entries for generating plots, figures, etc without relying on this data structure. * I get lost in a hurry if I do something non-standard and need to look under the hood. More under-the-hood docs would be cool. * Yes, I found that simple funtions like figure control (i.e., changing axis, symbles, grid, changing font and size of text etc) are quite annoying. I had difficulties in plotting multiple plots etc. Size of simulation box is not displayed in YT and one has to manually mention which could be useful. If there's one message about yt you want to send, what is it? * You guys are doing a great job - keep it up! * Thanks! * It is good tool! I hope it will continuously and actively improve. * "Keep up the great job! * Keep developing this great project! Congrats * The concepts are powerful, but it should either be stated to be specifically for cosmological visualization only, or its general purpose use should be better supported. * yt makes my life easier! Keep it up. * Keep doing the good work. * Good work thus far. * yt is elegant. How do you find the yt community to be? * Very welcoming - no complaints! * Great! * friendly, helpful + dedicated * It's a great place for receiving and giving help and suggestions about real work. * The friendliest community I know :) * Great! * Helpful and friendly, but rather too convinced of yt's usefulness, so that people get sucked in for whom the tool is not at all ready yet. * I haven't been active on the list at all, but it seems like response times are (extremely) short, and everyone is very helpful. * Love it! * Friendly and helpful. * Invaluable! The IRC channel and email list are incredibly helpful resources when something is missing from the docs or a bug creeps in the code. * Excellent. Super helpful. * I found them very helpful. * Very responsive and helpful. Have you ever contributed code to yt? If so, what was good or bad about that experience? * Not yet. * Not yet, but I look forward to it! * Yes, hg was a bit rough experience at the beginning, but as soons as I've achieved rookie level, I'm fine with it. * No. * Haven't made any changes. * yes, a little bit. It was a very good experience. * Yes. I felt it fixed a problem, and it was well-received. * No * Wish someone have short example of write code, add to doc, add test problem. Anything to add about deficiencies in yt? * Most of the deficiencies that ails me are already covered in yt-3.0 doc: multiple fluids, handling non-cartesian grids, handling vector fields * It is far too focused on the cosmology use case to be useful as a general purpose visualization tool. This impacts the documentation, which only makes sense if you are in the mindset of halo-finding and object property measurement, and the testing, which relies on users to find such basic things as support for 2D model visualizations (the most basic test problems) or non-square grids in nominally supported codes. * It would be great to have more direct access to matplotlib axes objects, say when you make a PlotCollection, to allow for easy manipulation of tick marks, axes labels, etc. * I would like volume rendering to be easier to use correctly (i.e. interactive isocontour finding). * I don't like the choices for the first question-- I think yt is very easy for some things, and gets more difficult as you get off the beaten path in a non-linear manner. * Figure controls, 3-D visualization, Better GUI not just web based, Importing data into YT with simple formats like ASCII or Binary etc, Imroved halofinder like T_vir * I would like to see it read Gadget snapshots. * need transfer function alongside volume rendering in log space, or make it customizable for the user. Right now it's printed in real space so if you have multiple add gaussian, the smaller ones are barely noticeable, and the background being white is not good for if the volume rendering is done with a background. Would be nice to be able to customize it. Maybe add transfer function and ability to add strings of text to describe the volume rendering off to the sides or top/bottom.

8 years, 2 months

yt users survey, Spring 2012

by matthewturk＠gmail.com

If you have trouble viewing or submitting this form, you can fill it out online: https://docs.google.com/spreadsheet/viewform?formkey=dEtHNXFXbXZJTXZUQ3dWaU… yt users survey, Spring 2012 It's that time of year again! We'd like some feedback on your use and your opinions about the present and future of yt. It's really short, and no questions are mandatory, so feel free to fill in as many or as few as you'd like. Do you find that yt is: Easy to use Difficult to use Too hard to use How often do you use yt? All the time Once in a while I tried, and it broke, so I stopped Have you used the yt GUI, Reason? Yes, often Yes, once or twice No Never heard of it The thing I think yt needs, more than anything else is ... More supported codes Speed or parallelism improvements More canned tasks for astrophysical analysis Better/easier visualization not destined for publication Better/easier visualization for publication Improvements to the GUI API documentation Narrative documentation and tutorials A more friendly development process or community More reliability or error handling Ability to conduct time series analysis Other: Anything to add about deficiencies in yt? What should be changed about the documentation? There's too much -- slim it down! I want more narrative documentation I want more example scripts The organization should be improved I'd like more screencasts Other: Have you watched the yt workshop videos? I was not there and did not watch any videos I was not there and did watch one or more videos I was there and did not watch any videos I was there and did watch one or more videos Which of these pieces of functionality have you used: yt pastebin (the yt pastebin) Clump finder Halo finder Volume renderer IRC for yt help Time series analysis Volume rendering in yt is ... Attractive Hard to get right Slow Easy to set up Have you found anything in yt to be particularly annoying? How do you find the yt community to be? Have you ever contributed code to yt? If so, what was good or bad about that experience? If there's one message about yt you want to send, what is it? If you want to be contacted about anything you've written, put your contact info here. Powered by Google Docs Report Abuse - Terms of Service - Additional Terms

8 years, 2 months

Fwd: [yt_analysis/yt] Change the way quad tree projections are reduced (pull request #156)

by Matthew Turk

After our discussion over the last week, I've issued a Pull Request. I don't believe it's ready yet -- for instance, John reported a big uptick in memory usage. However, I think we're nearing that point. Please feel free to send comments to the other thread, rather than leaving them on the PR itself. -Matt ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Mon, May 7, 2012 at 10:29 PM Subject: [yt_analysis/yt] Change the way quad tree projections are reduced (pull request #156) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/156/change-the-way-quad-t… Title: Change the way quad tree projections are reduced This implements the quad-tree extension code discussed in this thread: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-May/001986.h… Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

8 years, 2 months

yt installation on Gordon

by Geoffrey So

Hi, I was installing YT using the default intel/mvapich2 modules loaded, >module list Currently Loaded Modulefiles: 1) binutils/2.22 2) intel/2011 3) mvapich2_ib/1.8a1p1 but when I tried to do >make in the kdtree folder after installing Forthon I got: Using Forthon () Forthon -F gfortran --compile_first fKD_source --no2underscores --fopt "-O3" fKD fKD_source.f90 Building package fKD Forthon fKD.v F90Free fKD_p.F90 /usr/libexec/gcc/x86_64-redhat-linux/4.1.2/f951: error while loading shared libraries: libgmp.so.3: cannot open shared object file: No such file or directory make[1]: *** [fKD_p.o] Error 1 make: *** [fKD] Error 2 When I did a >locate libgmp.so.3 it only shows under gnu folders /opt/gnu/gmp/4.3.2/lib/libgmp.so.3 /opt/gnu/gmp/4.3.2/lib/libgmp.so.3.5.2 Since it's looking for libgmp, does that mean I have to use the gnu compilers? From G.S.

8 years, 2 months

Forthon-0.8.8

by Geoffrey So

I was trying to reinstall dev YT and enabling parallel HOP, but ran into the trouble that at the end of installing Forthon, I got the error pasted at the end, has anyone seen this? I tried using >sudo python setup.py install but it gave me the same errors, so subsequently my kdtree make fails. From G.S. Permissions on Notice and Src files needs to be set by hand running install running build running build_py creating build creating build/lib creating build/lib/Forthon copying Lib/__init__.py -> build/lib/Forthon copying Lib/_Forthon.py -> build/lib/Forthon copying Lib/cfinterface.py -> build/lib/Forthon copying Lib/compilers.py -> build/lib/Forthon copying Lib/Forthon_builder.py -> build/lib/Forthon copying Lib/Forthon_options.py -> build/lib/Forthon copying Lib/ForthonTimer.py -> build/lib/Forthon copying Lib/fvars.py -> build/lib/Forthon copying Lib/interfaceparser.py -> build/lib/Forthon copying Lib/preprocess.py -> build/lib/Forthon copying Lib/wrappergen_derivedtypes.py -> build/lib/Forthon copying Lib/wrappergenerator.py -> build/lib/Forthon running build_scripts creating build/scripts-2.7 copying and adjusting Forthon -> build/scripts-2.7 changing mode of build/scripts-2.7/Forthon from 644 to 755 running install_lib running install_scripts copying build/scripts-2.7/Forthon -> /Users/gso/Documents/el-yt/bin changing mode of /Users/gso/Documents/el-yt/bin/Forthon to 755 running install_data error: can't copy 'Notice': doesn't exist or not a regular file

8 years, 2 months

rendering bug

by Sam Skillman

Hi all, I've been trying to fix a bug in the opaque rendering. It comes down to which way the rays get integrated (back-to-front/front-to-back) with respect to the camera position. Right now I think the behavior is such that the bricks are handed to the sampler from the back to front, but the bricks themselves are rendered front to back. Anyways, the bug shows up better in parallel, and I've been using http://paste.yt-project.org/show/2372/ with https://bitbucket.org/samskillman/yt-refactor/changeset/b4db9a5b5704 The behavior ends up being something like this (this particular image might have the opposite L than the script): http://imgur.com/Od3np.png If you get something that looks okay, rotate it to a few different positions and see if it still works. If anyone wants to dig in, I would appreciate the help. Cheers, Sam

8 years, 2 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

yt-dev May 2012