Hi guys,
Extremely quick question. Do any of you have installations on:
Abe
Cobalt
BigBen
Vanilla Debian/RedHat/Ubuntu
If so, did you run the installation script? Did you have to do
anything special?
Also, do you have installations at any big supercomputing centers
*other* than Kraken, Ranger, Verne? Anything special needed for them?
Thanks!
-Matt
Hi guys,
Sorry to send out another email about the YT roadmap. In the last 48
hours I've had a couple emails from people trying to install 1.0, and
it made me realize we NEED to push out 1.5, even if it is without
substantial documentation updates.
To address this, I will be branching yt-1.5 this evening. I will not
be tagging for release, but I will be announcing this on the yt-users
and enzo-users-l mailing lists and I will be updating the Wiki to
reflect this new stable branch. The trunk freeze, which in retrospect
was over-engineering the problem, would be over. I'll also be
emailing enzo-l asking to move over the svn:externals pointer to
yt-1.5.
Any thoughts or objections?
-Matt
Hi everyone,
This email has two major components. If you don't care about YT-1.5,
skip to the bottom.
A couple weeks or months ago I emailed out suggesting a May 15
deadling; at the time, I thought this was feasible. However, in the
intervening time, other concerns have sprung to the forefront,
displacing efforts to rewrite and shore up the documentation. The
following tickets are still outstanding:
http://yt.enzotools.org/report/12
I will summarize these here.
#128: (Example Images) For documentation; I'm working on a sphinx
extension that will do this, using hg repos of the sample data.
Building the docs to include these will not be an easy task, but since
we will ship pre-built docs, it's not a problem.
#95: (Command-line script for radprofs) Britton's on this one, but I
think actually right now the ball is in my court. So we're going to
sync up in a week and change or so to finish it up.
#214: (IPython demo scripts) This one might get canceled. I wrote up
a little sphinx extension to run IPython.demo scripts and turn them
into documentation, which could then be stepped through interactively
(via IPython) and displayed inline in the docs. It was complicated.
Interactive demos are super useful, and super cool, but I think this
is over-engineered.
#148: (set_zlim and set_cmap issue) This one I'm still confused on,
and not sure how to fix. I can replicate it. I'll look into it.
#132: (cross-reference) part of doc rewrite
#144: (sphinx-extensions) might get canceled or rolled into #128
#152: (doc changes) specific suggestions that are being taken into
account in the rewrite
#154: (splash screen update for reason) I need to get on this, using
the new logo
#181: (docs about running in parallel) Britton and I are on this, and
Stephen wrote some docs about HOP.
So that's where we're at. Almost all documentation stuff. A new
outline has been prepared, and there's a doc repo on hg.e.o.
Anyway, I've decided we should aim for a June 15 release date. My
thesis is being turned in on May 29. After that I'm goign to be
meeting up with a couple of you and we can spend a bit of time talking
about documentation and working. It's clear from recent discussions
on the mailing list and offline that the docs need to be rethought. I
take responsibility for this problem. We're going to work hard and
fix it.
Okay.
That being said, and this is the important part, I am now officially
*freezing* the yt trunk to *new features*. We will not have a new
branch for new features, but instead new features can go into
mercurial repositories and then after the release on June 15 we'll
port them all back when we move trunk->tags/yt-1.5. (After porting
anything new, we'll discuss a reorganization.)
The mercurial repositories are open and available to everyone, so I
believe this is a good compromise that doesn't prevent bleeding-edgers
from grabbing stuff. This is not a new mode of development or a shift
in character of our current development, but rather a temporary fix to
make sure we do not increase our work before pushing out a stable
release.
Bug fixes can still go in. New features, not yet. hg clone a repo
from hg.e.o (either yt-bitbucket or yt-trunk) and I'll give you a
place to push it to on hg.e.o along with a username.
Okay, thanks all. Thanks guys, and let me know if you have any
problems with this, or if you can help with the documentation. The
new proposed outline is on google docs, so if you'd like to
contribute, let me know and I'll shoot you an invite.
-Matt
Looks mostly okay, but I think we need to rethink the initial mass
calculation before it gets committed.
http://codereview.appspot.com/59041/diff/1/2
File HaloFinding.py (right):
http://codereview.appspot.com/59041/diff/1/2#newcode197
Line 197:
Not sure we need this?
http://codereview.appspot.com/59041/diff/1/2#newcode433
Line 433: padded, LE, RE, self._data_source =
self._partition_hierarchy_3d(padding=self.padding)
We should get rid of this initial step of partitioning the entire
hierarchy, and instead move to using a DerivedQuantity. I'd say
something like
all_data = self.pf.h.all_data()
all_data.quantities["TotalQuantity"]("ParticleMassMsun",
lazy_reader=True)
which will automatically parallelize. This would get rid of most of the
following lines, and we could avoid the mpi_allsum, too. I'll need to
add in the preloading to the DQ object, however.
http://codereview.appspot.com/59041/diff/1/2#newcode450
Line 450: self._data_source.hierarchy.queue)
Should be fine.
http://codereview.appspot.com/59041
Hi y'all,
I've spent the morning trying to merge the copy of OpenMP HOP I have with the version in trunk. Although HOP is not the bottleneck when running it, OpenMP HOP does have a concrete speedup, at 8 processors it's better than 5x faster. I wrapped everything in #ifdefs OPENMP and it runs identically as before if it's not turned on. With OpenMP turned on, it compiles, but it gives me this error when I run it on Ranger:
from yt.mods import *
File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/mods.py", line 32, in <module>
import yt.lagos as lagos
File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/__init__.py", line 105, in <module>
from HaloFinding import *
File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 29, in <module>
from yt.lagos.hop.EnzoHop import RunHOP
File "/share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/hop/__init__.py", line 3, in <module>
from EnzoHop import *
ImportError: /share/home/00649/tg457850/yt/lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/hop/EnzoHop.so: undefined symbol: omp_unset_lock
I'm using gcc, and the relavent flags are "-fopenmp -DOPENMP". There are other omp calls that it isn't complaining about. I'm wondering if any of you have any thoughts. There's the very real chance this won't work, I can find nothing of substance online about doing this kind of thing.
Thanks!
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
Hi all,
I'm still trying to run HOP on the 1024^3 dataset on Ranger. It's been dying in two places inconsistently, meaning two consecutive runs will die either place. Line 196 in BaseDataTypes.py:
self.fields.append(key)
where it's complaining that self.fields doesn't have append. In HierarchyType.py at line 92:
self.gridTree = [ [] for i in range(self.num_grids)]
Sometimes it gives a memory error. self.num_grids is 440,000+, but in
my testing it doesn't matter how lists it's trying to make, it that's where it wants to die it dies.
I'm running on Kraken right now. We'll see how it goes.
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
Hi everyone,
Currently, the hdf5 operations in yt are handled by pytables. However,
pytables is not the only python hdf5 module out there. For my own non-yt
code, I have been using h5py (http://code.google.com/p/h5py/). For simple
hdf5 i/o, I find h5py to be far more intuitive and flexible than pytables.
I want to propose to people here that we switch the yt trunk over to h5py
before the yt 1.5 release. Before I enter my sales pitch, I want to make it
known that the work is already done. I converted a version of the yt-trunk
to h5py this weekend. Matt and I have been testing it on various machines
and it seems to be working. None of the yt function calls have changed.
Only the internal calls to tables functionality have been replaced with h5py
calls.
Other than the ease of casual use, some things in h5py's favor include:
- h5py is faster and relies on fewer python objects than tables
- tables is not designed for our style of IO
- better with reloading nodes than pytables (something very relevant to yt,
esp. in parallel)
- h5py installs easily: I did it from the source, which requires Cython,
but it should be installable with easy_install with these environment
variables:
HDF5_DIR=path to hdf5
HDF5_API=16
I know that switching dependencies is something we want to do as little as
possible, but I think it's worth it in this case. Matt will alter the
install script so this change should be relatively seemless. If anyone has
any major objections, please raise them.
Matt, please chime in if I missed anything or presented anything
incorrectly.
Britton
Hi guys,
I've applied (in r1293) the patch to HOP for in-place access rather
than the copying of arrays that it currently does. We discussed this
a bit ago and it went through with no objections, but I had been
holding off on committing.
Let me know if there are any problems.
-Matt
Hi all,
here is an attempt to make preloading work with HaloFinding.
http://codereview.appspot.com/59041
Ignore hop_hop and hop_regroup, those changes aren't related to this issue and won't be added to trunk. I couldn't convince the upload script to ignore them.
Looking at the stderr log, it appears that it is reading multiple grids. I'm not sure that it's not reading them again later. Can someone (Matt) take a quick look and see what you think?
http://paste.enzotools.org/show/116/
Thanks!
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
Matt,
is there a reason why you used self.data_source in HaloFinding, but in other places you used self._data_source, like in Profiles? Is there a naming scheme you've got set up?
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
