Mailman 3 - yt-dev - python.org

parameter file store
by Matthew Turk 03 Mar '09

03 Mar '09

Hi guys, I rewrote -- again -- the parameter file store. This one I finished a couple weeks ago, and Dave, Britton and I have been testing it since then. (Implicitly, no less.) I'm committing it now, and it comes with new unit tests for the parameter file store. I think it's more robust than the previous one, and it works in parallel. Unfortunately, it might grind the disk slightly more, but I think it's on the scale of MUCH less than reading/writing the normal .yt file or parsing the hierarchy. Let me know if you have any problems; I committed in r1196. I updated the unit tests, but unfortunately I can't quite figure out how to run them without having instantiated the parameter file once yet. Specifically, Clearly this means I'm not an expert at writing unit tests or mock objects. Anyway, you can run them by first instantiating an EnzoStaticOutput of tests/DD0010/moving7_0010 and then doing tests/test_lagos.py. Sometime I'll see about writing some more of those, too. If there are any edge cases or problems, let me know. I wrote up (by copying a couple paragraphs from my thesis and extending them) a bit of why we need this, how it works, and what it does. This is the kind of tone I was thinking of; do y'all think it works, or should it be scrapped? http://blog.enzotools.org/ It will likely not get read -- that's pretty clear -- but I think it's a useful means of collecting information that can be processed back into documentation. -Matt

1 0

recipes
by Matthew Turk 03 Mar '09

03 Mar '09

Hey, I'm going to take a look at the Recipes sometime today or tomorrow. Could somebody else help me out, make sure there aren't any obvious problems? If you want to just reply to this message (or fix them) that'd be great. I'll also probably be adding a few more, but I don't know on *what* yet.

1 0

RMS mass overdensity method
by Stephen Skory 02 Mar '09

02 Mar '09

Hi all, I am currently attempting to write a (parallel) RMS mass overdensity method. This is the mass variance over the average density smoothed over a length scale. sigma_8 is this quantity smoothed over 8 Mpc at z=0 in a linearly collapsed universe, for example. I have a method that works (mostly), shown in the link below. It first finds the total mass in the box in order to calculate the average density field. It then picks a random center and gets the particle mass inside of a sphere at that center of the chosen length scale, normalizing by the average mass in that sphere. It does this many times, and then the RMS is calculated from the samples. http://paste.enzotools.org/show/60/ Some of you may notice a flaw in the code: if the random center is near to the box edge, part of the sphere will be outside the box, giving too low masses. I hope to address this, see below. If there are other flaws, let a brother know! Since I wish to use this on the L7 dataset, doing this in parallel is a must. And luckily, this proceedure requires very similar logic to parallel HOP. Spatial decomposition along with periodic oversampling (padding) will work, I think. The padding, which will be the radius of the smoothing spheres, will make sure that any center chosen in the 'real' box will have it's full compliment of particles inside of it, very similar to the parallel HOP concept. Below is a rough attempt at how I think a parallel implementation can work, but there are some issues I would like some advice from the yt cognesceti: http://paste.enzotools.org/show/61/ - I calculate the total mass on lines 61-65 by partitioning the box just like parallel HOP does, and there's no over counting. But when I get into the individual threads, on line 18 I'm trying to oversample for each sub-box. I've modified _partition_hierarchy_3d (see below) so that in a special case (like this) we can oversample the entire box in serial mode, although this doesn't work yet. What I'd like for serial mode, for example, are particle lists with oversampled reflected particles such that the range of points is [-padding, 1+padding]. But I'm not sure this is the right way to go, modifying _parititon_hierarchy_3d. http://paste.enzotools.org/show/62/ - in _get_over_density(), I am using the pf.h.sphere() method to cut out the particles inside the smoothing sphere, but I'm not sure I want to do this. I'm aware that as it is it won't work using the _partition_hierarchy_3d method. Can you give me a suggestion of the 'right' way to do this? I'd wonder if investing time in the KDtree scipy stuff Matt emailed about earlier is worth it, but we already have this domain decomposition plus oversampling machinery, and we definitely need the oversampling for this method. And domain decomposition makes data reading so much faster. Thanks for your comments! _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

2 11

Website Changes, dev blog?
by Matthew Turk 02 Mar '09

02 Mar '09

Hi guys, I'm rolling out some changes to the website. http://enzotools.org/ has already been updated, with a list of destinations. Right now these are yt, the microscope, the mercurial repositories and the pastebin. If you have any other projects you'd like to host, I would *LOVE* to put them up and add them as either a subdomain, svn repo or hg repo. (hg is the easiest to add.) The next change up is going to be to the front page of yt. Trac is not going away, but it kind of sucks as a public face. Here's my first draft: http://spacepope.org/temp/ for the front page. The list of links is not final. (Jeff is helping out with that.) It'll link into trac, the URLs for trac will all stay the same, but I'm trying to make it easier to handle. I'll also be reorganizing the wiki (without adding any content just now) to split the pages up to be smaller, unless they cover a single unified idea. (For instance, the Cray page -- very popular page these days, actually -- will stay one page. Installation might get split, and the front page will be split.) The trac page itself should be re-styled at some point; I've been playing with themes, but that's slightly longer term. The development principles are getting moved onto a dedicated, static page: http://spacepope.org/temp/principles.html but they'll still be on the wiki. All of the code for the yt-chrome is in the repository yt-chrome (although maybe it should be named differently) so you can play with it if you like. I've started fiddling with the colors on the pastebin. (This is all in the lodgeit-yt repository.) I'd like it to be more distinct from the original pastebin, so I have been moving it to a more reddish look. It's not done yet -- a few more elements to style up. I'll roll it out when I finish the remaining elements. Finally, and perhaps most importantly, I think it might be worthwhile to start recording a bit more about what we do. Now, of the five of us on this list, only three of us regularly commit -- and that's cool! -- but I think we should also keep a bit of a narrative for major new features, release concepts, and other things that might just not fit into the wiki. I try to do this, once in a while, on my own blog, but it'd be good if we could keep it in some semi-official place. Maybe even tips-n-tricks, etc. If anyone else would be willing to contribute, I'd be more than happy to set up a group blog. I just don't want to be the only one contributing. I think there's been a lot of stuff being put in, added, blah blah, and it would be excellent to keep a nice narrative description of the design decisions and new features, which we can then direct people to. What do y'all think? Of the blog, the website, etc? -Matt

3 9

shiny
by Matthew Turk 02 Mar '09

02 Mar '09

Hi guys, Can any of you send me some shiny happy plots? The cooler the better. Feel free to attach them here! -Matt

3 3

max on a region
by david collins 28 Feb '09

28 Feb '09

Hi-- Is there a way to find both the value and coordinate of the max values on an extracted region, like Reginald= pf.h.region(c,l,r), with a single command? d.

2 1

Implementation of a parallel cKDTree
by Matt 27 Feb '09

27 Feb '09

Just FYI, if you need fast kd-trees in C with a Python interface, scipy.spatial has them; additionally, scipy.spatial is largely separable from scipy as a whole, so any compilation problems with scipy will likely not apply to spatial. ( http://docs.scipy.org/doc/scipy/reference/spatial.html ) This could be extremely useful for a number of problems with generating subsets of data in yt, clustering algorithms, and ray tracing algorithms. And now it's multi-threaded. Sent to you by Matt via Google Reader: Implementation of a parallel cKDTree via gmane.comp.python.scientific.user by Sturla Molden on 2/27/09 I have fiddled a bit with scipy.spatial.cKDTree for better performance on multicore CPUs. I have used threading.Thread instead of OpenMP, so no special compilation or compiler is required. The number of threads defaults to the number of processors if it can be determined. The performance is not much different from what I get with OpenMP. It is faster than using cKDTree with multiprocessing and shared memory. Memory handling is also improved. There are checks for NULL pointers returned by malloc or realloc. setjmp/longjmp is used for error handling if malloc or realloc fail. A memory pool is used to make sure all complex data structures are cleaned up properly. I have assumed that crt functions malloc, realloc and free are thread safe. This is usually the case. If they are not, they must be wrapped with calls to PyGILState_Ensure and PyGILState_Release. I have not done this as it could impair scalability. Regards, Sturla Molden _______________________________________________ SciPy-user mailing list SciPy-user< at >scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user Things you can do from here: - Subscribe to gmane.comp.python.scientific.user using Google Reader - Get started using Google Reader to easily keep up with all your favorite sites

1 0

Parallel Hop
by Matthew Turk 25 Feb '09

25 Feb '09

Hi Stephen, Could you summarize quickly for us the following two points? I want -- very much -- to eliminate the SS_HopOutput.py file and place it in the old location, but we *cannot* do so until these are answered satisfactorily. You are the one in the best position to answer them. :) 1. What is the status of determining, automatically, the necessary padding? I have suggested that this could be some small multiple of the root grid dx, but I don't recall hearing back from you. (I believe you have been using on the order of 6 root grid cells as padding.) 2. Are there any outstanding bugs in SS_HopOutput that you have found prevent it from replacing the old one, and working in both serial and parallel? (I have found none.) Thanks very much! -Matt

3 13

OSX Installation Script
by Matthew Turk 24 Feb '09

24 Feb '09

Hi guys, I've written up a preliminary OSX installation script. I believe it only works with 10.5. If anybody feels like installing YT on a 10.5 machine, this will be the preferred method as soon as I am sure the kinks are worked out: http://bitbucket.org/MatthewTurk/yt/src/tip/doc/install_script_osx.sh (download link: http://bitbucket.org/MatthewTurk/yt/raw/f2899cb571fa/doc/install_script_osx… ) It's not in trunk yet because until somebody else tests it and it seems okay, I'm not fully trusting. So, if anybody feels like giving it a go ... let me know if it works!!! -Matt

1 0

open season
by Matthew Turk 24 Feb '09

24 Feb '09

Hi guys, I'm officially declaring open season on the yt wiki. Please feel free to make any changes or additions you think are appropriate -- sometimes I forget that things are out of date, or wrong, or whatever. So please feel free to modify or change things, make them simpler, de-Matt-ify them! (And go ahead and add anything you think needs to be added!) -Matt

1 0