Mailman 3 September 2010 - yt-dev

Reading in Daniel Ceverino's ART output
by Christopher Moody 29 Sep '10

29 Sep '10

Howdy folks, I wanted to announce that I just committed to the hg repo a reader for Daniel Ceverino's particular version of the ART code. My research interest is to pipe his ART data to Sunrise via yt. As for the code, It's still a rudimentary reader, and I've included only a draft version of the frontend ('cevart'). I'll be using the RAMSES frontend as a starting point, fleshing it out to fit Daniel's data. Unfortunately, I believe Daniel's data is a bit post-processed, and so my work is not particularly generic. So if anyone has any information on how ART typically generates it's outputs, please contact me! Thanks, chris

3 2

Parallelism notes and call for patch testing
by Matthew Turk 25 Sep '10

25 Sep '10

Hi all, I prepared some strong scaling plots today (and I'm actually quite pleased with them -- I will share them next week) and discovered a couple items of interest. * Profile1D is substantially slower than Profile2D. I'm currently investigating why, but my current working belief is that the usage of pure-Python for conducting the histogram slows us down, versus the hand-coded C routine I wrote for 2D and 3D histograms. (Bin[23]DProfile in yt/utilities/data_point_utilities.c). I'll probably rewrite the 1D into C at some point. * Projections (old-style, not quadtree) scale much better than I think I realized, up to ~64 processors. At 128 on the dataset I tested on (512^3 L7) the algorithmic overhead and processor starvation combined to reduce scaling substantially. The good news is that it still only takes 40 seconds on 64 processors. This was on Triton, 8x8 node topology. I'm confident that on bigger datasets it would scale further. * With the coalescing of grid/cpu reading in the yt-Enzo code IO is not really an issue at the moment. * DerivedQuantities use _mpi_catlist, which I think I wrote something like two and a half years ago. At that time, I avoided any collective communications or non-blocking combines, so it proceeded by pickling lists that got passed over the wire (one by one) to the root processor, where they were joined and then broadcast back. This is extremely slow. The current mechanism that I think makes the most sense is to use the _recv_arrays routine that manages an alltoallv call. I've prepared two pastes. The first is a patch: http://paste.enzotools.org/show/1242/ that converts the Derived Quantity parallel join to _mpi_catarrays and converts _mpi_catarrays to use the alltoallv wrapper The second is the test script I used: http://paste.enzotools.org/show/1243/ which I was only able to test on my laptop this evening, as Triton's disk is down for a bit. It produced bitwise identical (once I handled the transposes correctly!) between parallel and serial. If anyone that uses parallelism in yt heavily has a chance I'd really appreciate it if you could run this script on a dataset you've got. If you've got other big parallel jobs that use the DerivedQuantities or Slice mechanisms (as those are the ones that have been touched) that would also be great to hear back about. Thanks! Matt

1 0

Call for testing: New install script
by Matthew Turk 19 Sep '10

19 Sep '10

Hi all, Jeff and I have re-written the install script to use mercurial. This involved adding in an optional bzip2 installation, as that was a sticking point for several machines, as well as re-organizing it to install bzip2 before Python and then hg before yt. If you get a chance, I'd really appreciate it if you could try running it: http://yt.enzotools.org/files/install_script.sh There's a "branch" name at the top that governs which yt branch gets cloned and updated to in the installation section. Any feedback would be constructive; we need to replace the current install script pretty soon, so let me know if this does or doesn't work! Thanks, Matt

4 7

Re: [Yt-dev] Call for testing: New install script (also some GUI things and OSX 10.6)
by Matthew Turk 15 Sep '10

15 Sep '10

Hi Rick, > Yes, I've used the 10.6 script in the past with no problem. Personally, all of the software I build on my Mac goes into /usr/local. This can be rationalized as a desire for consistency with other computes that I work with, but it's really just that I don't build enough OS X-specific software to learn about frameworks, etc. I've learned enough about Frameworks to mostly regret it. ;-) My recollection of the motivation for framework builds is twofold -- one for the access to the native widgets, and the other (possibly the same thing?) was the access to the ObjC frameworks. For instance, you can get bindings to do a lot of things. But with yt, specifically, we have no reason to use any of the ObjC frameworks. So I guess the question is, if the install_script.sh works, do we want to spend *any* effort on maintaining a second set of scripts for the framework builds? Or should we just say adios to them? I see that you've used the 10.6 script, but has the old install_script.sh (or even the current one with INST_BZLIB = 0) worked for you? > I think that not building GUIs by default is reasonable solution to providing a simple and reliable installation method. Web interfaces can limiting, but they're much more portable, and open up a lot of potential, especially given the number of remote systems we work on. Right now I have a couple things built on yt that might be cool, but might not be: * Fisheye: Tom Abel suggested I try to replicate the Jacques GUI based on YT. This uses Tkinter, is very simple, and basically recreates the ability to zoom, slice, project, change fields, recenter, and so on. It's a couple hundred lines of code. http://bitbucket.org/MatthewTurk/fisheye Requires only the Python stdlib. * Image Panner: This is a google maps style pan-n-scan with a cool colorbar attached. It uses the Chaco GUI, not the web browser, and all the pixelization is still handled by yt. Comes with yt, requires ETS and Qt. * Reason v2: This is essentially just a TVTK GUI, where you can make isocontours and cutting planes in a frame. *This* is kind of cool, but I think the challenge of writing our own marching cubes algorithm in raw GLSL is less than the challenge of getting people to install VTK. Requires basically the entire kitchen sink: Qt, ETS, VTK. * Simple OpenGL widgets: These are widgets for displaying images either stereoscopically or non-stereoscopically in OpenGL, a grid viewer, a grid slicer, and a volume renderer that runs on the hardware. (The volume renderer has some artifact I haven't figured out, but enough works to see the structure.) Requires only PyOpenGL. These are all the GUIs that I believe are currently built on yt that would function if you ran them right now. I only see a future for fisheye and the OpenGL widgets. (And there's a good chance the OpenGL widgets will go away due to a pretty cool possibility coming up in the not-too-distant future.) As for web stuff, there are a couple proof-of-concepts in the repo http://bitbucket.org/MatthewTurk/yt_ajax . These are a Mapquest (not google maps) style interface that pans around an image, and a simple python interpreter. (Neither are secure, so only run if you are sure you want to!) The python interpreter also comes with a convenience function to pass images back to the browser, so you end up with something that looks like: http://yt.enzotools.org/files/simple_repl.png (except I've cut off the web browser around the edges.) Neither of these require anything outside the standard Python library and yt, which I think is what makes them the most attractive to me: you don't *need* anything else installed, connections can be tunnelled over ssh *trivially*, and you can have images passed back and forth simply. Plus, passing HTML from the server to the client allows you to create widgets programmatically, so one could imagine a number of cool mechanisms for interaction. This fits extremely nicely with the "plot window" Jeff is working on. Anyway, this is where I see the future of GUIs in yt: either very raw OpenGL stuff, web stuff with few or (for preference) no external dependencies, or *possibly* Tkinter. What do you think of this possible approach, Rick and others? My personal feeling, and it's taken me a while to come to this, is that web stuff is the only way I see that we could go forward and not have any dependencies, whcih I think is the only way we'll be able to have a GUI that reaches anyone not reading (writing?) this email. Oh, so the upshot is that if we say, "No Native GUIs!" we can probably ditch the 105 and 106 install scripts. +1, -1 to ditching native GUIs, anybody? Best, Matt

2 3

Fwd: [mpi4py] [ANN] mpi4py release 1.2.2
by Matthew Turk 13 Sep '10

13 Sep '10

---------- Forwarded message ---------- From: Lisandro Dalcin <dalcinl(a)gmail.com> Date: Mon, Sep 13, 2010 at 2:46 PM Subject: [mpi4py] [ANN] mpi4py release 1.2.2 To: mpi4py <mpi4py(a)googlegroups.com> Release 1.2.2 [2010-09-13] ========================== * Add ``mpi4py.get_config()`` to retrieve information (compiler wrappers, includes, libraries, etc) about the MPI implementation employed to build mpi4py. * Workaround Python libraries with missing GILState-related API calls in case of non-threaded Python builds. * Windows: look for MPICH2, DeinoMPI, Microsoft HPC Pack at their default install locations under %ProgramFiles. * MPE: fix hacks related to old API's, these hacks are broken when MPE is built with a MPI implementations other than MPICH2. * HP-MPI: fix for missing Fortran datatypes, use dlopen() to load the MPI shared library before MPI_Init() * Many distutils-related fixes, cleanup, and enhancements, better logics to find MPI compiler wrappers. * Support for ``pip install mpi4py``. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 -- You received this message because you are subscribed to the Google Groups "mpi4py" group. To post to this group, send email to mpi4py(a)googlegroups.com. To unsubscribe from this group, send email to mpi4py+unsubscribe(a)googlegroups.com. For more options, visit this group at http://groups.google.com/group/mpi4py?hl=en.

1 1

Answer Testing
by Matthew Turk 08 Sep '10

08 Sep '10

Hi all, I've been working on a very simple answer testing module. It leverages what is currently being used by Enzo to do answer testing. The code is up here: http://bitbucket.org/MatthewTurk/yt_answer_testing There are three sets of tests right now: * Hierarchy consistency * Projections and profiles * Field values Jeff's coding up a mechanism for running this at SLAC on a cron job against datasets from Enzo, Orion, RAMSES and FLASH. If you have a dataset that you would like added to the mix, please email either of us. When it's up and running he'll email the list with the URL, but that might not be for a little while. If you have a test you would like to add, please fork the repo on bitbucket *or* clone it and patchbomb me with your patches. -Matt

1 0

Help with io.py and GDF
by Christopher Moody 03 Sep '10

03 Sep '10

Howdy all, I've been working on getting writing code (http://hg.enzotools.org/gadget_infrastructure/summary) that will transform gadget snapshots into the newly-minted gridded data format (GDF: http://yt.enzotools.org/wiki/GridDataFormat). The last few days I've been writing a very skeletal GDF reader in yt, and I've come across a few problems. I pushed changes to yt a few hours ago, and so should be up on the hg repository - what I'm talking about is referring to files in /frontends/gadget/* . 1. When reading, say, an array of particle data like position_x of some grid, (shape is just N, the number of particles in that grid) I get array shape errors. It tries to multiply the array by a weight_data but that's an array of the same shape as active dimensions for the grid (which in my case is [2,2,2] since I'm doing an octree refinement) - so it cries when it tries to multiply an N array with a (3,2)-shape array. To make that shape make sense, it seems like I want not the data for that particular grid, but all of it's children's data. Or something, I'm a bit confused. Check out io.py. 2. Is there going to be a problem with having parent grids with no particle data - that are almost empty? The position_x array for such a grid won't exist, so I'm not sure what to do in the _read_data_set() method. I get grids like this when I subdivide a grid into another 8 child grids and then all of the particles particles belonging to the parent grid are shuffled into the appropriate child bins. The parent is then left with none. 3. What does _read_data_slice() mean when a field like 'position' is already scalar-ified as position_x? I thought the slice 'axis' would've picked out the 'x' axis in position, but you start off already asking for position_x. I've included an example GDF data file here (http://dl.dropbox.com/u/206140/decay_100.gyt.hdf5 - not guaranteed to be alive forever) and a pastebin script (http://paste.enzotools.org/show/1145/) that highlights my first problem. chris

2 1

Request For Ideas: Particle Handling
by Matthew Turk 02 Sep '10

02 Sep '10

Hi everyone, It's becoming increasingly clear that particle handling in yt (and frankly in Enzo) needs to be improved. I'm at something of a loss for how to improve this, because of the issues with the following things: 1) How do different particle types get selected? Even in Enzo, this is variable, which drives me somewhat batty. The distinction via creation_time or particle_type is unfortunately quite vague. How can we do this without getting an enormous logic tree? On some level, because Enzo's handling of particles is relatively difficult to handle in this regard, I'm okay with pushing a change upstream to Enzo and then migrating toward that in yt over time. 2) Presupposing that this happens, how do we distinguish between fields for different particle types? What is the most pleasing API for selecting only the star particles in an AMRSphere? I see several possibilities: sphere["popiii_particle_velocity_x"] sphere["popiii", "particle_velocity_x"] sphere.particles["popiii"]["particle_velocity_x"] I can't decide which of these is the nicest. Or maybe there's even a better one I haven't thought of yet. The trick is figuring out the most simple way of addressing it, while still retaining the fundamental idea of data objects as conduits for data. The solution to this issue is not unlike any future solution to a multi-fluid problem, as well. So what does everyone think? I think we can agree, particularly as we move to more simulations of stellar clusters and whatnot, that a better mechanism for selecting and addressing particles needs to be figured out... ANY ideas would be greatly appreciated. Thanks, Matt

3 3

Kraken Dynamic Loader
by Matthew Turk 01 Sep '10

01 Sep '10

Hi all, I've been able to get yt to run on Kraken using only dynamic libraries. I think this means that -- while performance is probably slightly worse! -- it isn't necessary to maintain static libraries. The only trick is that the stack has to be completely located on the scratch/lustre disk. Here's the process that worked for me: I checked out the install script, replaced the svn checkout in the install script with an hg checkout, and ran it inside lustre with PrgEnv-gnu. I copied the following libraries from /lib64 or /usr/lib64, or even from the GNU installation location, into the yt-x86_64/lib directory: libm.so.6 libc.so.6 libutil.so.1 libdl.so.2 libpthread.so.0 libgcc_s.so.1 libstdc++.so.6 libfreetype.so.6 Inside the submission script I set LD_LIBRARY_PATH to point to the new directory on the scratch disk. Then, launching the scratch disk python2.6, it worked. Anyway, I'm going to write this up, but let me know if this works for anyone else. You can see which libraries a library requires in the lib/ directory by running ldd on it. -Matt

2 1

Quick Summary: Active Branches
by Matthew Turk 01 Sep '10

01 Sep '10

Hi everyone, Now that we've moved to mercurial for all yt development (which I'm pretty pleased about) I have placed a wiki page here: http://yt.enzotools.org/wiki/ExistingBranches with all the current active branches. But basically, it comes down to: "stable" => the previous yt head, which has the old names "yt" => the new head, reorganized and probably with some lingering import issues "kd-render" => Sam's branch for kD-tree volume rendering If you are able, I'd prefer if you could use and fix any issues that arise with imports directly in the "yt" branch, but if you'd rather continue using "stable" that's of course fine as well. Best, Matt PS Sorry about the confusion on the graph on the main web page. I think I could have mangled history enough so that branches would be closed AND have their last heads in the distant past, but it's too late now. Maybe some day we'll have the option of hiding topological heads on closed branches in the graph view, but that day isn't today.

1 0