Mailman 3 May 2009 - yt-dev

Installations
by Matthew Turk 25 May '09

25 May '09

Hi guys, Extremely quick question. Do any of you have installations on: Abe Cobalt BigBen Vanilla Debian/RedHat/Ubuntu If so, did you run the installation script? Did you have to do anything special? Also, do you have installations at any big supercomputing centers *other* than Kraken, Ranger, Verne? Anything special needed for them? Thanks! -Matt

4 4

Roadmap Change
by Matthew Turk 20 May '09

20 May '09

Hi guys, Sorry to send out another email about the YT roadmap. In the last 48 hours I've had a couple emails from people trying to install 1.0, and it made me realize we NEED to push out 1.5, even if it is without substantial documentation updates. To address this, I will be branching yt-1.5 this evening. I will not be tagging for release, but I will be announcing this on the yt-users and enzo-users-l mailing lists and I will be updating the Wiki to reflect this new stable branch. The trunk freeze, which in retrospect was over-engineering the problem, would be over. I'll also be emailing enzo-l asking to move over the svn:externals pointer to yt-1.5. Any thoughts or objections? -Matt

2 2

1.5 update, missed goal, feature free in trunk
by Matthew Turk 17 May '09

17 May '09

Hi everyone, This email has two major components. If you don't care about YT-1.5, skip to the bottom. A couple weeks or months ago I emailed out suggesting a May 15 deadling; at the time, I thought this was feasible. However, in the intervening time, other concerns have sprung to the forefront, displacing efforts to rewrite and shore up the documentation. The following tickets are still outstanding: http://yt.enzotools.org/report/12 I will summarize these here. #128: (Example Images) For documentation; I'm working on a sphinx extension that will do this, using hg repos of the sample data. Building the docs to include these will not be an easy task, but since we will ship pre-built docs, it's not a problem. #95: (Command-line script for radprofs) Britton's on this one, but I think actually right now the ball is in my court. So we're going to sync up in a week and change or so to finish it up. #214: (IPython demo scripts) This one might get canceled. I wrote up a little sphinx extension to run IPython.demo scripts and turn them into documentation, which could then be stepped through interactively (via IPython) and displayed inline in the docs. It was complicated. Interactive demos are super useful, and super cool, but I think this is over-engineered. #148: (set_zlim and set_cmap issue) This one I'm still confused on, and not sure how to fix. I can replicate it. I'll look into it. #132: (cross-reference) part of doc rewrite #144: (sphinx-extensions) might get canceled or rolled into #128 #152: (doc changes) specific suggestions that are being taken into account in the rewrite #154: (splash screen update for reason) I need to get on this, using the new logo #181: (docs about running in parallel) Britton and I are on this, and Stephen wrote some docs about HOP. So that's where we're at. Almost all documentation stuff. A new outline has been prepared, and there's a doc repo on hg.e.o. Anyway, I've decided we should aim for a June 15 release date. My thesis is being turned in on May 29. After that I'm goign to be meeting up with a couple of you and we can spend a bit of time talking about documentation and working. It's clear from recent discussions on the mailing list and offline that the docs need to be rethought. I take responsibility for this problem. We're going to work hard and fix it. Okay. That being said, and this is the important part, I am now officially *freezing* the yt trunk to *new features*. We will not have a new branch for new features, but instead new features can go into mercurial repositories and then after the release on June 15 we'll port them all back when we move trunk->tags/yt-1.5. (After porting anything new, we'll discuss a reorganization.) The mercurial repositories are open and available to everyone, so I believe this is a good compromise that doesn't prevent bleeding-edgers from grabbing stuff. This is not a new mode of development or a shift in character of our current development, but rather a temporary fix to make sure we do not increase our work before pushing out a stable release. Bug fixes can still go in. New features, not yet. hg clone a repo from hg.e.o (either yt-bitbucket or yt-trunk) and I'll give you a place to push it to on hg.e.o along with a username. Okay, thanks all. Thanks guys, and let me know if you have any problems with this, or if you can help with the documentation. The new proposed outline is on google docs, so if you'd like to contribute, let me know and I'll shoot you an invite. -Matt

1 0

HDF5 HaloFinder preloading
by matthewturk＠gmail.com 11 May '09

11 May '09

Looks mostly okay, but I think we need to rethink the initial mass calculation before it gets committed. http://codereview.appspot.com/59041/diff/1/2 File HaloFinding.py (right): http://codereview.appspot.com/59041/diff/1/2#newcode197 Line 197: Not sure we need this? http://codereview.appspot.com/59041/diff/1/2#newcode433 Line 433: padded, LE, RE, self._data_source = self._partition_hierarchy_3d(padding=self.padding) We should get rid of this initial step of partitioning the entire hierarchy, and instead move to using a DerivedQuantity. I'd say something like all_data = self.pf.h.all_data() all_data.quantities["TotalQuantity"]("ParticleMassMsun", lazy_reader=True) which will automatically parallelize. This would get rid of most of the following lines, and we could avoid the mpi_allsum, too. I'll need to add in the preloading to the DQ object, however. http://codereview.appspot.com/59041/diff/1/2#newcode450 Line 450: self._data_source.hierarchy.queue) Should be fine. http://codereview.appspot.com/59041

5 19

OpenMP HOP
by Stephen Skory 08 May '09

08 May '09

Hi y'all, I've spent the morning trying to merge the copy of OpenMP HOP I have with the version in trunk. Although HOP is not the bottleneck when running it, OpenMP HOP does have a concrete speedup, at 8 processors it's better than 5x faster. I wrapped everything in #ifdefs OPENMP and it runs identically as before if it's not turned on. With OpenMP turned on, it compiles, but it gives me this error when I run it on Ranger: from yt.mods import * File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/mods.py", line 32, in <module> import yt.lagos as lagos File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/__init__.py", line 105, in <module> from HaloFinding import * File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 29, in <module> from yt.lagos.hop.EnzoHop import RunHOP File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/hop/__init__.py", line 3, in <module> from EnzoHop import * ImportError: /share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/hop/EnzoHop.so: undefined symbol: omp_unset_lock I'm using gcc, and the relavent flags are "-fopenmp -DOPENMP". There are other omp calls that it isn't complaining about. I'm wondering if any of you have any thoughts. There's the very real chance this won't work, I can find nothing of substance online about doing this kind of thing. Thanks! _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

2 5

1024^3 HOP problems
by Stephen Skory 07 May '09

07 May '09

Hi all, I'm still trying to run HOP on the 1024^3 dataset on Ranger. It's been dying in two places inconsistently, meaning two consecutive runs will die either place. Line 196 in BaseDataTypes.py: self.fields.append(key) where it's complaining that self.fields doesn't have append. In HierarchyType.py at line 92: self.gridTree = [ [] for i in range(self.num_grids)] Sometimes it gives a memory error. self.num_grids is 440,000+, but in my testing it doesn't matter how lists it's trying to make, it that's where it wants to die it dies. I'm running on Kraken right now. We'll see how it goes. _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

2 15

moving yt from pytables to h5py
by Britton Smith 06 May '09

06 May '09

Hi everyone, Currently, the hdf5 operations in yt are handled by pytables. However, pytables is not the only python hdf5 module out there. For my own non-yt code, I have been using h5py (http://code.google.com/p/h5py/). For simple hdf5 i/o, I find h5py to be far more intuitive and flexible than pytables. I want to propose to people here that we switch the yt trunk over to h5py before the yt 1.5 release. Before I enter my sales pitch, I want to make it known that the work is already done. I converted a version of the yt-trunk to h5py this weekend. Matt and I have been testing it on various machines and it seems to be working. None of the yt function calls have changed. Only the internal calls to tables functionality have been replaced with h5py calls. Other than the ease of casual use, some things in h5py's favor include: - h5py is faster and relies on fewer python objects than tables - tables is not designed for our style of IO - better with reloading nodes than pytables (something very relevant to yt, esp. in parallel) - h5py installs easily: I did it from the source, which requires Cython, but it should be installable with easy_install with these environment variables: HDF5_DIR=path to hdf5 HDF5_API=16 I know that switching dependencies is something we want to do as little as possible, but I think it's worth it in this case. Matt will alter the install script so this change should be relatively seemless. If anyone has any major objections, please raise them. Matt, please chime in if I missed anything or presented anything incorrectly. Britton

3 6

HOP in-place patch
by Matthew Turk 06 May '09

06 May '09

Hi guys, I've applied (in r1293) the patch to HOP for in-place access rather than the copying of arrays that it currently does. We discussed this a bit ago and it went through with no objections, but I had been holding off on committing. Let me know if there are any problems. -Matt

1 0

preloading with HaloFinding
by Stephen Skory 05 May '09

05 May '09

Hi all, here is an attempt to make preloading work with HaloFinding. http://codereview.appspot.com/59041 Ignore hop_hop and hop_regroup, those changes aren't related to this issue and won't be added to trunk. I couldn't convince the upload script to ignore them. Looking at the stderr log, it appears that it is reading multiple grids. I'm not sure that it's not reading them again later. Can someone (Matt) take a quick look and see what you think? http://paste.enzotools.org/show/116/ Thanks! _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

1 1

naming conventions
by Stephen Skory 01 May '09

01 May '09

Matt, is there a reason why you used self.data_source in HaloFinding, but in other places you used self._data_source, like in Profiles? Is there a naming scheme you've got set up? _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

2 2