Mailman 3 February 2009 - yt-dev

max on a region
by david collins 28 Feb '09

28 Feb '09

Hi-- Is there a way to find both the value and coordinate of the max values on an extracted region, like Reginald= pf.h.region(c,l,r), with a single command? d.

2 1

Implementation of a parallel cKDTree
by Matt 27 Feb '09

27 Feb '09

Just FYI, if you need fast kd-trees in C with a Python interface, scipy.spatial has them; additionally, scipy.spatial is largely separable from scipy as a whole, so any compilation problems with scipy will likely not apply to spatial. ( http://docs.scipy.org/doc/scipy/reference/spatial.html ) This could be extremely useful for a number of problems with generating subsets of data in yt, clustering algorithms, and ray tracing algorithms. And now it's multi-threaded. Sent to you by Matt via Google Reader: Implementation of a parallel cKDTree via gmane.comp.python.scientific.user by Sturla Molden on 2/27/09 I have fiddled a bit with scipy.spatial.cKDTree for better performance on multicore CPUs. I have used threading.Thread instead of OpenMP, so no special compilation or compiler is required. The number of threads defaults to the number of processors if it can be determined. The performance is not much different from what I get with OpenMP. It is faster than using cKDTree with multiprocessing and shared memory. Memory handling is also improved. There are checks for NULL pointers returned by malloc or realloc. setjmp/longjmp is used for error handling if malloc or realloc fail. A memory pool is used to make sure all complex data structures are cleaned up properly. I have assumed that crt functions malloc, realloc and free are thread safe. This is usually the case. If they are not, they must be wrapped with calls to PyGILState_Ensure and PyGILState_Release. I have not done this as it could impair scalability. Regards, Sturla Molden _______________________________________________ SciPy-user mailing list SciPy-user< at >scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user Things you can do from here: - Subscribe to gmane.comp.python.scientific.user using Google Reader - Get started using Google Reader to easily keep up with all your favorite sites

1 0

Parallel Hop
by Matthew Turk 25 Feb '09

25 Feb '09

Hi Stephen, Could you summarize quickly for us the following two points? I want -- very much -- to eliminate the SS_HopOutput.py file and place it in the old location, but we *cannot* do so until these are answered satisfactorily. You are the one in the best position to answer them. :) 1. What is the status of determining, automatically, the necessary padding? I have suggested that this could be some small multiple of the root grid dx, but I don't recall hearing back from you. (I believe you have been using on the order of 6 root grid cells as padding.) 2. Are there any outstanding bugs in SS_HopOutput that you have found prevent it from replacing the old one, and working in both serial and parallel? (I have found none.) Thanks very much! -Matt

3 13

OSX Installation Script
by Matthew Turk 24 Feb '09

24 Feb '09

Hi guys, I've written up a preliminary OSX installation script. I believe it only works with 10.5. If anybody feels like installing YT on a 10.5 machine, this will be the preferred method as soon as I am sure the kinks are worked out: http://bitbucket.org/MatthewTurk/yt/src/tip/doc/install_script_osx.sh (download link: http://bitbucket.org/MatthewTurk/yt/raw/f2899cb571fa/doc/install_script_osx… ) It's not in trunk yet because until somebody else tests it and it seems okay, I'm not fully trusting. So, if anybody feels like giving it a go ... let me know if it works!!! -Matt

1 0

open season
by Matthew Turk 24 Feb '09

24 Feb '09

Hi guys, I'm officially declaring open season on the yt wiki. Please feel free to make any changes or additions you think are appropriate -- sometimes I forget that things are out of date, or wrong, or whatever. So please feel free to modify or change things, make them simpler, de-Matt-ify them! (And go ahead and add anything you think needs to be added!) -Matt

1 0

Trac
by Matthew Turk 21 Feb '09

21 Feb '09

Hey guys, I think I have fixed the Trac problems. That's all. -Matt

3 2

Dumb colorbar question
by david collins 18 Feb '09

18 Feb '09

Hi-- I'm making a bunch of plots with a greyscale colormap. When I set the zlim, things get rescaled properly, but the colors on the colorbar are rgb, not grayscale. I noticed by accident that the following works: pf = lagos.TenPCStaticOutput('/lustre/scratch/collins/128_amr/OK1/DD2030/data2030') pc = raven.PlotCollection(pf) pc.add_projection('Density',0) pc.save('rescaled_no') pc.set_zlim(1e-3,5) pc.save('rescaled_yes') because of the first 'save' feature. Removing it makes a rgb colorbar. What does save do that forces the colorbar map? d.

2 1

Re: [Yt-dev] [yt-users] parallel data retrieval
by Matthew Turk 14 Feb '09

14 Feb '09

Hi guys, I'm moving this to -dev. This should be in the halos themselves; unfortunately we want to avoid double-communicating, both between procs and into the file system. I will work on that, using Stephen's script as a template. However, I've also set up a simple little way to add things to wrap, to get rid of the get_indices, get_velocities, etc functions. I think we can use this to prevent any getter/setter methods. http://paste.enzotools.org/show/46/ Stephen, this is just a simple little mod, but if you could test that this appropriately wraps the __getitem__ method, that'd be awesome. Then we can ditch the getters. -Matt On Fri, Feb 13, 2009 at 10:27 AM, Stephen Skory <stephenskory(a)yahoo.com> wrote: > Matt, > >> If you want to try it again, you might > >> consider manually setting (and then unsetting) the _processing >> attribute of your HopGroup to True; this should set it to turn off the >> locking/barrierizing. > > To follow up, your suggestion works, and I've pasted the full script below for everyone's edification. > > http://paste.enzotools.org/show/45/ > > Please note that I haven't fixed some things in the version I pasted, in particular "grouped_particles" will be incorrect. > > > Thanks! > > _______________________________________________________ > sskory(a)physics.ucsd.edu o__ Stephen Skory > http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student > ________________________________(_)_\(_)_______________ > _______________________________________________ > yt-users mailing list > yt-users(a)lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >

2 16

Mercurial Repository
by Matthew Turk 13 Feb '09

13 Feb '09

Hi everyone, Currently, I have yt checkouts on several different machines, and in these I have been having some issues with keeping things in sync. For a while I was using bzr with an svn plugin to manage this; but in the last couple days I've been having problems with this. Additionally, it has been unreliable lately. However, mercurial is substantially more reliable -- and furthermore, the existing (and easy_install-able) hgsvn is much less fancy in what it seeks to do. I have created a mercurial mirror on bitbucket.org. I'll be using this as my primary dumping point for experimental and possibly non-functional stuff. To merge changes back upstream, I will be using the workflow outlined here: http://www.selenic.com/mercurial/wiki/index.cgi/WorkingWithSubversion#head-… Basically you check out using the special hgsvn command, and then branch all you like, etc etc, and then update and then merge into a *separate* subversion repository using a patch command. This is ideal, I think, as it requires no additional metadata. However, the reason I'm bringing this up is that I'll be doing all of this on bitbucket.org, which is designed to foster and encourage social software development. It includes things like wikis in the checkout, issue tracking, and *extremely* simple forking and patching between users. It's designed for collaborations, as well as passing patches back and forth. http://bitbucket.org/help/collaborating/ This will NOT replace the wiki, source control, or issue tracking on yt.enzotools.org. Mercurial is FAR too niche and FAR too rapidly developed for that. (Although I would like to note that one of the coolest things about bitbucket is that you can get a tarball in the flavor of your choice right on the main project page.) But for experimental stuff, for feature hacking, for all of that, mercurial and bitbucket are useful tools. The things I hope to address here, in the BitBucket repo, would be things like: * int math covering grid (almost done!) * better vertex-centering of data (trickier) * GUI re-working and so on and so forth. Anyway, if this sort of thing appeals to you -- repositories of your changes that you can pass around more easily, scripts and whatnot -- I encourage you to go to BitBucket, sign up with an OpenID account (I used my GMail account) and then "fork" yt. (There's a button for that.) If you make changes you think I should have -- and vice versa! -- clicking on the "pull request" button helps communicate them back and forth. I think this will lead to more experimentation and possibly a better end product. http://bitbucket.org/MatthewTurk/yt/overview/ -Matt

1 0

covering grids
by Matthew Turk 12 Feb '09

12 Feb '09

Hey guys, I have implemented the integer-based covering grid. It's not really much faster; the interpolated one should be, but the primary one is not. (This was disappointing to me.) Anyway, this exposed a slight bug in the covering grid generation of ghost zones. I have attached an example plot. Upper left is the new, upper right is the old, lower left is the relative difference. abs(new-old)/(new+old). It looks to me like an off-by-one in the grid values. This should not change substantially any of the calculations, as it still gives *correct* values, so contouring and whatnot should still work exactly correctly. We just get a little bit of bonus data one one side and a little less on the other. It's *possible* that if your contour join was exclusively in the ghost zone of a single grid that you now have too many; this seems kind of unlikely to me, but it still should be addressed. I'm going to fix it, but probably by inserting the new routine rather than by fixing the old one. The only thing remaining (you can see for yourself at my bzr repo: http://bzr.enzotools.org/integer_cgrid/ ) thing to do is to do periodic domains and smoothing. For smoothing I will be implementing the (int math friendly) algorithm from Ralf Kaehler. -Matt

1 1

yt-dev February 2009