Mailman 3 September 2011 - yt-dev

Stumped on off-axis projections
by Matthew Turk 30 Sep '11

30 Sep '11

Hi all, I'm writing a helper function for off-axis projections. I'm stuck on deciding what to do for weighting. For a weighted projection, we integrate field * weight across the whole domain. For on-axis, this is well-defined. For off-axis, we have two choices of what to do, because we interpolate between vertices: * Construct a field that is everywhere field * weight, and interpolate inside that to get our v's * Interpolate field, interpolate weight, and multiply the two interpolated values to get our v's These are different operations, and can give very different results. (You can test this very easily in 1D; I was surprised at the magnitude of the differences.) I am not sure which one is more 'correct' for our particular goal. I can actually see arguments for both sides. The arguments in favor of #1 are that we are constructing a valid field everywhere in advance, and it will be more stable. The arguments for #2 are that it may more accurately reflect the local value you would get compared to, say, a column density taken off axis (which is what we ultimately divide by at the end of the calculation.) Does anybody here have an opinion? 1D example code: http://paste.yt-project.org/show/1837/ -Matt

3 3

Grid/yt data format meeting
by Matthew Turk 28 Sep '11

28 Sep '11

Hi all, Casey, Cameron, Sam and I are going to be meeting -- likely utilizing "Google Hangouts" -- to discuss the grid data format (Jeff has suggested we just call it the 'yt data format,' which I am also down for.) The meeting will be on Wednesday at 1:15PM Eastern. I'll announce it again on Wednesday (with URL), but only in IRC, not the mailing list. We'll be discussing what improvements could be made to the existing yt data format: http://yt-project.org/gdf.txt as well as Sam's (awesome) progress with importers/exporters, and integration with the IRATE-format. I think we're all hoping this could be a serialization format to take to the bank, so if you have thoughts on interop or ground-up design of a data format, we'd appreciate your participation. -Matt

2 1

Screencasts
by Cameron Hummels 28 Sep '11

28 Sep '11

Hey peeps, So it turns out that a bunch of people have viewed the "how to use reason" screencast that we put out last week. It seems like screencasts may be a good way of introducing certain concepts in the yt universe, because it really shows our audience exactly *how* to get certain functionality to work (along with a rewind button!). With that in mind, I wanted to see if anyone else wanted to create a screencast on some topic in yt. I tried to create a "How to" on making a yt screencast on the following page to help you out if you decide to make one. Furthermore, Matt and I outlined several possibilities for topics on the following page: https://bitbucket.org/yt_analysis/yt/wiki/Screencasts Topics include getting up and running, parallelization in yt, halo finding/usage, volume rendering, time-series/movie-making, defining your own field, etc. If you'd like to make one, feel free to volunteer and put your name next to one of the topics on the wiki (it should be editable by all), then just post one when you get a chance. We can put it in our blog and update the site with news of it. This may be a good way to supplement the documentation. Let me know if you have any comments! Cameron

6 7

Reason crash
by John Wise 27 Sep '11

27 Sep '11

Hi everyone, I'm trying to use Reason, and it's crashing when I try to create a projection or slice. Viewing the grid data works perfectly fine. I'm using a version that's installed without the install script. I'm using a similar set of dependencies as the install script (Python 2.7.1, h5py 1.3.1, the ext* and PhiloGL files from enzotools.org) Everything else has been working fine with yt. Here's the traceback: ----- Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/usr/local/lib/python2.7/dist-packages/yt-2.3dev-py2.7-linux-x86_64.egg/yt/gui/reason/extdirect_repl.py", line 123, in run task['name'], task['widget_data_name']) File "/usr/local/lib/python2.7/dist-packages/yt-2.3dev-py2.7-linux-x86_64.egg/yt/gui/reason/extdirect_repl.py", line 451, in _add_widget payload['data'] = self.locals[widget_data_name] KeyError: '_twidget_data' ----- When I inspect the task variable, it has these fields. (Pdb) p task {'widget_data_name': '_twidget_data', 'type': 'add_widget', 'name': '_tpw'} Here is my python session with me briefly looking around with pdb, but I'm not familiar enough with reason to properly debug this. I also tried reducing max_threads to 1, and it didn't have any effect. http://paste.enzotools.org/show/1828/ If you want to try to replicate this error, I've uploaded the dataset (50MB), but I have a feeling that it's not dataset's fault. http://www.physics.gatech.edu/~jw254/scpics/DD0036.tgz Has anyone had this error before? Thanks! John -- John Wise Assistant Professor of Physics Center for Relativistic Astrophysics, Georgia Tech

3 5

Re: [Yt-dev] Nautilus: ParallelHaloProfiler I/O issue
by Geoffrey So 23 Sep '11

23 Sep '11

Hi Stephen, I was running parallelHF on Nautilus when I got the following email from the sys admins, I'll try to answer the questions but let me know if I'm wrong or there's something else to be said. (1) How much data are read/written by program ? - After all the particles (3200^3 of them) are read in they are linked with a fortran KDtree if they satisfy some conditions. (2) How many parallel readers/writers are used by program ? - It is reading using 512 cores from my submission script. The amount of write to disk depends on the distribution of the particle haloes across processors, if they exist across processors then there will be more files written out by write_particle_lists. (3) Do you use MPI_IO ? or something else ? - Yes, the program uses mpi4py-1.2.2 installed in my home directory The details of the code can be found at: http://yt-project.org/doc/analysis_modules/running_halofinder.html#halo-fin… under the section "Parallel HOP" Currently I am using 512 cores with 4GB per core for a total of 2TB of ram for this 3200 cube unigrid simulation, should I decrease the amount of processors but keeping the same amount of ram? Or is there other ways to optimize as to not affect other users? Sorry for the inconvenience. From G.S. On Fri, Sep 23, 2011 at 9:12 AM, Patel, Pragneshkumar B <pragnesh(a)utk.edu>wrote: > Hello, > > We have noticed some I/O issue on Nautilus. We suspect that > "ParallelHaloProfiler" program is doing some very I/O-intensive operations > periodically (or may be checkpoint). We like to throttle these back a bit , > so that other users are not affected by it. > > I like to get some more information about your job #60891. E.g. > > (1) How much data are read/written by program ? > (2) How many parallel readers/writers are used by program ? > (3) Do you use MPI_IO ? or something else ? > > Please give me detail and we will work on your I/O issue. > > Thanks > Pragnesh > >

3 3

possible halo mass function bug
by Geoffrey So 22 Sep '11

22 Sep '11

I was running an old script to refresh my memory of how the halo mass function is used in YT, and it just siphoned all the memory out of my laptop on a tiny dataset. As I recalled I was able to test the halo mass function extensively on my laptop with this 64 cube unigrid run. Thinking memory is the issue (mine or my laptop's), I then tried to run the script on triton with the TCC node, it still hangs and runs out of memory, I then tried the PDAF node and got the same results even after updating to the latest dev (22ac529a1dab (yt) tip). link to the script: http://paste.enzotools.org/show/1815/ Or is there something obviously wrong with the script? From G.S.

2 2

Workshop IRC Meeting Summary
by Matthew Turk 19 Sep '11

19 Sep '11

Hi all, Today we had a meeting in IRC to discuss a potential 2012 yt workshop. I think it was very successful, and rather than having long, high-turnover discussions over email it seems like IRC would be a better venue. I've summarized the important points below, and the full logs can be found online at: http://paste.yt-project.org/show/1807/ The important points: 1) The FLASH center has *very* graciously offered to host us in Chicago, thanks to John ZuHone's efforts. 2) We are currently exploring the following dates, in order: Jan 23-26, Jan 31-Feb 3, or Feb 6-Feb 9. JohnZ will be looking into availability of hotel blocks. 3) A draft agenda has been posted here: https://docs.google.com/document/d/1gNGifqqLnu3x4DYVzM-EpYjriqIWraV-ZJHdwtb… which is available for commenting. Please leave comments if you have them, using the 'commenting' feature. 4) If anyone has any ideas about potential funding sources, please contact Stephen Skory or me. 5) We'll have presentations from some local (UC/FLASH/ANL) HPC experts as well. The current action items: 1) Figure out hotel situation with candidate dates (JZ) 2) Continue iterating on agenda (Britton, everyone else) 3) Investigate funding (SS, anyone else that has leads) Please feel encouraged to reply to this list with any thoughts you have. As new items develop we'll continue updating this list, and hopefully we will be able to begin advertising it and shoring up participants/support soon. Best wishes, and thanks to everyone who participated! This workshop should be fun, and I'm really excited about the energy I was getting in the chat room -- about topics, development, learning, sharing code, everything. -Matt

1 0

Meeting for yt Workshop Discussion
by Matthew Turk 19 Sep '11

19 Sep '11

Hi all, There are a number of exciting things to share and discuss about the possibility of having a yt workshop in early 2012. The discussion here and on yt-users has been fast and furious. To avoid that high-throughput for discussing several items that need to be discussed with some rapidity, Britton and I would like to call an IRC meeting. If you are interested in participating in organizing (and we hope that we can count on participation from everyone who has participated in the discussion thus far) please help us pick a time by filling out this doodle *by the end of the day Saturday*. http://www.doodle.com/frwnhzis5na8g5xi One of us will announce by Sunday morning through this medium when it will occur. The meeting will be held in #yt on irc.freenode.org. You can access this either through web-chat: http://yt-project.org/irc.html or through your IRC/IM client. We will keep a log of the discussion, which one of us will post to this list when concluded. The following items will be discussed: * Venue (and there is some exciting news on that front!) * Dates * Overall structure * Agenda (curriculum?) * Reaching out to other interested parties * Other items Thanks very much, Matt

1 2

ellipsoid part 2
by Geoffrey So 15 Sep '11

15 Sep '11

Hello Stephen, This is part 2 of the ellipsoid parameter finding. The method is outlined mainly in the Dubinski and Carlberg 1991 paper. But since we are doing this in YT and we already know the particle list to begin with, I put in the option that the user can specify if they want all the particles in the region to be taken into account or only the particles found by the halo finder algorithm. This is an option that I think should be amended to the first ellipsoid parameter finder. The script form is linked here: http://paste.enzotools.org/show/1782/ Once I have written out the methodology for myself, I'll work on the YT documentation, will probably need some help in this area when it comes to it. Doron Lemez has used this method in his paper and helped me tested the accuracy of this script. I was wondering do I need to ask him if he wants his name in the credits, or just mention his name as a special thanks? One thing he did in the paper that was especially useful is defined the starting sphere to have 2x the virial radius. Without that specification, I run into non-convergence sometimes, where after many iterations I end up with no particles. And I want to mention again that there's no guarantee that any or all the haloes will converge with this method. I tried to adhere to the PEP8 formatting, but old habits die hard so I'm sure I made mistakes in the formatting, let me know if you see them. From G.S.

3 7

AAS splinter meeting
by david collins 09 Sep '11

09 Sep '11

I emailed the contact person for setting up a splinter meeting. It sounds like this is an avenue open to us, and has some potential benefits. I kind of like the walk-in demo space idea, but that's probably outside of the scope we were looking for right now. They charge a a few hundred for the room, but that's likely going to be the same if we rent a conference room at a hotel. I'm definitely not married to the idea, so if y'all would rather do somewhere or somewhen else that's also grand. d. David, It sounds like a splinter meeting is exactly what you would need. Splinter meetings can be private (invitation only), or public, and public splinter meetings will be listed in the online meeting materials as well as in the printed meeting program. Anyone that is registered for the meeting may attend your splinter meeting. There is no set format for splinter meetings, it is kind of a "catch all" term we use for anything that is not specifically science session related. We can schedule invitation-only splinter meetings in any time slot you would like, but we try not to schedule public splinter meetings against our Plenary Invited talks (8:30am, 11:40am, 3:40pm, and 4:30pm each day). An exception to this would be if you wanted to have a "open-house" kind of set up all day, where people could come in and out at their leisure to test your software--we could schedule that for a long block of hours with no problem. It also sounds like you might need more than one format--maybe a walk-in space for demos, and a smaller meeting space for developers and private meetings. Either way, we can accommodate you. You can fill out the form available at http://aas.org/meetings/splinter_proposal_form.php, or you can send the details to me and we can figure out what will work best. The splinter form also has most of our AV pricing, for reference. I can take AV orders and catering orders (if applicable) as well. Thanks, Lisa Idem Meetings Manager American Astronomical Society 202-328-2010 ext. 126 On Thu, Sep 8, 2011 at 4:50 PM, David C. Collins <dccollins(a)lanl.gov> wrote: Hi there-- I have a question about Splinter Meetings, specifically for the January 2011 meeting in Austin. I'm trying to organize a meeting for a software package yt, which designed to analyze and visualize astrophysical simulation data. The meeting we're planning is a combination of new user instruction and developer meeting to discuss recent work that's been done from our various contributors, as well as future development ideas (with the goal of blurring the line between new users and developers). The AAS meeting is an ideal venue, as so many of us will be going already, and it's a good way to attract new users. The workshop application deadline has passed, but I was wondering if a splinter meeting might be an appropriate session type for us. What is the general setup for this kind of session, and do you think it would be adaptable for our needs? Thanks! David Collins. -- Sent from my computer.

1 0