Hi guys, I've been working on a new branch: http://svn.enzotools.org/yt/grid-optimization/ to pull down overhead on large hierarchies, specifically for Kraken. If you could give it a checkout and run some simple analysis with it, that'd be awesome -- right now it passes all the unit tests, so any failures *you* run into should be made into *new* unit tests. It's substantially faster and substantially lower memory (40% less for the L7 run) so I'd like to make sure it works, merge it back and kill. It also features a fairly important fix to the Covering Grids. -Matt
That link is broken, try: svn co http://svn.enzotools.org/yt/branches/grid-optimization/ yt-grid-op
thanks, sorry, was shooting from the hip. I guess I always thought of myself as William Munney, but evidently that was mistaken. On Fri, Jan 16, 2009 at 11:15 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
That link is broken, try:
svn co http://svn.enzotools.org/yt/branches/grid-optimization/ yt-grid-op
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
If you could give it a checkout and run some simple analysis with it, that'd be awesome
I ran parallel hop on Ranger with this an it was faster than the previous version. It gave identically sized HDF5 files for my halo data, which is good, but I'm guessing since things might have been computed and saved in a different order the md5 sums don't match. To be really sure, I'd have to do a group-by-group comparison, but the fact that the sizes are the same says that the same number of groups and particles in each group were found both times. login4% ll *hdf5 -rw-r--r-- 1 tg457850 G-80416 1611766912 Jan 15 19:41 HOPtables.hdf5 -rw-r--r-- 1 tg457850 G-80416 1611766912 Jan 16 14:42 HOPtables-new.hdf5 login4% md5sum HOPtables* 179e3765c8220db3c8aa2b91752c400c HOPtables.hdf5 8dd848895e7706043b081534f54de53d HOPtables-new.hdf5 _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Can you check the masses of the halos? If we assume that an hdf5 is composed of: sizeof(metadata) * n_arrays sizeof(particle) * n_particles then the only real datapoint we have here is that it's the same number of arrays and same number of *total* particles. I'm not sure this assumption is valid, but I'd like to ensure we aren't seeing particles moving between halos. But, it's great to hear that HOP doesn't need anything changed. On Fri, Jan 16, 2009 at 12:53 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
If you could give it a checkout and run some simple analysis with it, that'd be awesome
I ran parallel hop on Ranger with this an it was faster than the previous version. It gave identically sized HDF5 files for my halo data, which is good, but I'm guessing since things might have been computed and saved in a different order the md5 sums don't match. To be really sure, I'd have to do a group-by-group comparison, but the fact that the sizes are the same says that the same number of groups and particles in each group were found both times.
login4% ll *hdf5 -rw-r--r-- 1 tg457850 G-80416 1611766912 Jan 15 19:41 HOPtables.hdf5 -rw-r--r-- 1 tg457850 G-80416 1611766912 Jan 16 14:42 HOPtables-new.hdf5 login4% md5sum HOPtables* 179e3765c8220db3c8aa2b91752c400c HOPtables.hdf5 8dd848895e7706043b081534f54de53d HOPtables-new.hdf5
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Can you check the masses of the halos?
login4% h5ls -r HOPtables.hdf5 > old login4% h5ls -r HOPtables-new.hdf5 > new login4% diff old new login4% Huzzah! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Okay; this is all in the grid-opt branch, which I think is actually pretty close to being done. I've run a bunch of my calculations with it, too, so I'd say we're about go. I'll give it until maybe the end of the weekend, then I'll merge it back to trunk and kill the branch -- so if you're going to run stuff, maybe try running off of this branch for a few days? This should cut WAY down on overhead, but there's more that can be cut down, I'm just not sure if it'll be significant. Thanks guys. On Fri, Jan 16, 2009 at 1:29 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Can you check the masses of the halos?
login4% h5ls -r HOPtables.hdf5 > old login4% h5ls -r HOPtables-new.hdf5 > new login4% diff old new login4%
Huzzah! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey guys, We all have tons of stuff to do -- myself no exception! -- but did anybody get a chance since Friday or so to test out the grid-opt branch? Any problems? -Matt On Fri, Jan 16, 2009 at 1:31 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Okay; this is all in the grid-opt branch, which I think is actually pretty close to being done. I've run a bunch of my calculations with it, too, so I'd say we're about go. I'll give it until maybe the end of the weekend, then I'll merge it back to trunk and kill the branch -- so if you're going to run stuff, maybe try running off of this branch for a few days? This should cut WAY down on overhead, but there's more that can be cut down, I'm just not sure if it'll be significant.
Thanks guys.
On Fri, Jan 16, 2009 at 1:29 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Can you check the masses of the halos?
login4% h5ls -r HOPtables.hdf5 > old login4% h5ls -r HOPtables-new.hdf5 > new login4% diff old new login4%
Huzzah! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Okay, hearing no objections ... I'll merge it in and kill it this afternoon. :) -Matt On Tue, Jan 20, 2009 at 6:18 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hey guys,
We all have tons of stuff to do -- myself no exception! -- but did anybody get a chance since Friday or so to test out the grid-opt branch? Any problems?
-Matt
On Fri, Jan 16, 2009 at 1:31 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Okay; this is all in the grid-opt branch, which I think is actually pretty close to being done. I've run a bunch of my calculations with it, too, so I'd say we're about go. I'll give it until maybe the end of the weekend, then I'll merge it back to trunk and kill the branch -- so if you're going to run stuff, maybe try running off of this branch for a few days? This should cut WAY down on overhead, but there's more that can be cut down, I'm just not sure if it'll be significant.
Thanks guys.
On Fri, Jan 16, 2009 at 1:29 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Can you check the masses of the halos?
login4% h5ls -r HOPtables.hdf5 > old login4% h5ls -r HOPtables-new.hdf5 > new login4% diff old new login4%
Huzzah! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (2)
-
Matthew Turk -
Stephen Skory