Mailman 3 March 2009 - yt-dev

Hacking guidelines
by Matthew Turk March 25, 2009

March 25, 2009

Hi guys, I put up a wiki page today with some stuff I wrote up on the bus this morning: http://yt.enzotools.org/wiki/HackingGuidelines It's the first iteration, but I thought it would be a good way to start codifying what we want in terms of contributions. I have been following some discussions on other lists, where people are talking about contributions, and the relative impact of guidelines on the quality and quantity of contributions; with a release coming up (1.5, maybe May? maybe in conjunction with the Barn?) that I think will turn yt from a plotting tool into a full-fledged analysis and visualization solution, I thought we should start soliciting and encouraging contributions and bug fixes. These would also be great guidelines for us to follow as well; I am guilty of not realizing these ideals quite as well as I should, so now I can be publicly shamed. :) I think you all have wiki accounts, so if I've left something out you should feel free to fix it up. -Matt

1 0

parallel profiler problem
by Stephen Skory March 24, 2009

March 24, 2009

Hi Britton & Matt, first of all, thanks very much for working on the parallel halo profiler! I appreciate it very much. I am having some trouble running it on a L7 dataset. Specifically, I am getting the error below at all processor counts (this is a -np 1 run, rev 1226): http://paste.enzotools.org/show/83/ It's getting late so I'll take another whack at it tomorrow morning. I'm sending it out on the chance that one of you might have some insight. Thanks! _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

3 13

debugging decorator
by Matthew Turk March 23, 2009

March 23, 2009

Hi guys, I just added a new decorator in r1229. This is the 'run_pdb' decorator, which will drop you to an interactive pdb shell before calling the function that it is applied to. For instance, if I applied it to, say, _is_fully_enclosed inside BaseDataTypes, whenever that function is called I get a pdb prompt. ( http://docs.python.org/library/pdb.html ) This enables FAR easier debugging. (You can do the same thing by inserting pdb.set_trace() at any point in the code, but this way yt will handle the imports for you.) -Matt

1 0

yt-hop configuration improvements?
by Stephen Skory March 23, 2009

March 23, 2009

Hi guys, I had an email exchange with Brian a few days ago about trying to reproduce some data he made on L7. In particular, he gave me some settings for HOP hop.c and regroup.c that cannot be changed using the current setup. Happily, I was able to reproduce the list of haloes he found using his three step serial process with parallel yt-hop once I made some hard-coded changes to HOP. If we want to have as many options as the serial version Brian used, there would have to be about 10 optional inputs to yt-hop, which I know will kind of go against the aesthetic that Matt has for the current yt-hop. Unless there are any questions or protests, I'll try to add these options tomorrow. _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

3 9

minimum halo mass
by Matthew Turk March 20, 2009

March 20, 2009

Hi guys, We never came to an agreement. I think we need to decide on some padding-calculator, and then call that good and wrap SS_HopOutput.py back into the main source file. I really, really think somebody besides Matt should come up with this. (Since my halos are all 10^6 M_sun and I only care about the biggest one!) Can we come up with something? Until we do so, the parallel halo finder is just going to be sitting on the sidelines, and I'd really like it to be moved in as the primary mode of halo finding. -Matt

3 5

Auto-registration of callbacks and change in Callback API
by Matthew Turk March 19, 2009

March 19, 2009

Hi guys, I've been helping some people learn to use yt lately, and it is becoming apparent to me that the plotting API might need some work. I'm mostly okay with the way the PlotCollection stuff is handled (but see http://yt.enzotools.org/ticket/158 for more on that) but the callbacks specifically are not terribly intuitive. The current API is: p = pc.add_projection(...) p.add_callback(ContourCallback("Temperature")) Currently the Callback metaclass auto-registers each callback (http://yt.enzotools.org/browser/trunk/yt/raven/Callbacks.py#L38) so that they get imported automatically in yt.mods (http://yt.enzotools.org/browser/trunk/yt/mods.py#L55) but this can be done in a substantially more straightforward fashion. (However, the AMRData subclasses are also nearly auto-registering, which is something I intend to explore a bit more in the future.) I'd like to propose that each callback be given a static attribute _type_name -- i.e., "contour", "quiver" etc, and then this be used to access that particular callback type. I'm torn, however, on how this should be handled. There are three main ways I've considered for implementing this: == 1. Through a .modify(type_name, *args, **kwargs) command. You would then do: p.modify("contour", "Temperature") for instance, and it would add that callback. This has the downside that, while we could provide docstrings for all the possible callbacks, we would have to implement some introspection to do so. Not a huge deal, but somewhat annoying. 2. Through dictionary-style lookup, similar to DerivedQuantities: p.modify["contour"]("Temperature") this would allow us to have docstrings without any introspection. 3. Through the current style used for generating data objects from the hierarchy: p.contour("Temperature") This would also give us docstrings for free. == What do you think? The first appeals to me the strongest for aesthetic reasons, but the third appeals to me on the grounds of ease of use and understanding. Does anyone have any strong feelings? This would all be handled via an auto-registry situation, so the means of adding new callbacks would change, but only in that they would all require _type_name . Additionally, I see no reason to get rid of the current p.add_callback(...) method, which is more general that we're looking at here. -Matt

3 5

Ghetto Parallel Halo Profiler Report
by Stephen Skory March 18, 2009

March 18, 2009

Hi guys, You may find this interesting. Yesterday I did two runs of the halo profiler on L7. I took the full HopAnalysis.out file and split it up into 16 files in a round-robin fashion, and also in a spatial decomposition. I then ran with 16 separate parameter files for these 16 separate HOP files like this (for each setup): from mpi4py import MPI if MPI.COMM_WORLD.rank == 0: q = HP.HaloProfiler(blah, 'round-robin01.par') q.makeProfiles() if MPI.COMM_WORLD.rank == 1: ...'round-robin02.par' q.makeProfiles() ... ... MPI.COMM_WORLD.Barrier() The round-robin took a little under ten hours to finish, while the spatial decomposition took closer to eleven hours. This is because the numbers of haloes per task wasn't as balanced in the spatial case. The haloes per minute per process was more or less identical for both methods. _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

1 0

HOP on Kraken XT5
by Stephen Skory March 17, 2009

March 17, 2009

Hi guys, I had some trouble running HOP in a batch job on Kraken. Specifically, it was crashing at line 443 in hop_regroup.c where it's trying to open a temporary file (although looking at tempnam(NULL) at line 413 it should be in memory, but that wasn't working). I've made a fix for it that works for me, but I believe other people have run yt-HOP on Kraken (Britton?), and have you had this kind of trouble? I don't want to commit this fix if it's my problem only. Thanks! _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

3 10

smoothed plots
by david collins March 17, 2009

March 17, 2009

Hi, All-- Is it possible to smooth the image I get from plot_collection.add_projection? Thanks, d.

2 1

Matplot lib glitches: long underscore, linestyle
by david collins March 16, 2009

March 16, 2009

Hi-- I've been making a bunch of plots these days, some of them using matplotlib. I've noticed two strange glitches when making eps files with it. In both cases, the plots are made just fine, the eps files look good when you open them with gv/preview. But when you try to make them into a latex document, the document builds fine, but dvipdf hangs indefinitely. First, if you make a histogram with fill=False and linestyle='dashed' The second is if you try to include long text in a subscript in a caption. So I had plt.scatter(stuff, label='\rho_{truelove}') and it caused problems. I haven't sorted it out, but I figure most of y'all will be using matplot lib to put things into tex at some point. d.

2 2