Hi Stephen,
On Wed, Oct 19, 2011 at 1:18 PM, Stephen Skory wrote:
Hi Matt,
I've gone through what you started and I fixed the bugs. This diff below produces identical results to the old way. Could you run this on your dataset you used and see if I haven't ballooned the memory? I was using a 40^3 dataset (it's fast!) but it's hard to see big changes in memory using that. If we like this, I'll remove all the prints, and apply it to my yt fork, and ask Geoffrey to try it out. If he sees no big problems, I'll do a pull request.
http://paste.enzotools.org/show/1876/
* Initializing ParallelHOPHaloFinder with copies (by dividing) of position and mass fields
I replaced those with na.multiply and na.divide, which should be doing it in-place on the arrays.
* Copying position, mass into the fKD object without removing them from the halo finder (is there any reason they can't simply be moved in and removed from the halo finder, then copied back out?)
No, it should be possible, and I did that in the new diff. I don't know if it's helped.
* Rearrange = True is the default, which I believe copies the entire kD-tree inside fKD?
That's correct, it rearranges the position data so that the nearest neighbor searches are faster, and makes a copy to do this.
It also looks like some of this is because the fKD tree requires position to be (N,3) for memory access speed.
Yes, why I want a C/C++ kdtree someday that is either as fast at this fortran one, or nearly as fast. I have yet to find it.
Unfortunately, it's not entirely clear to me how we then copy back out of the Forthon kD-tree at the right time; I don't quite now the inner workings of pHOP. But I think this is valid, as I don't believe at any point information is lost.
I think I've found the right place, just before the fKD object is deleted and the tree cleared.
Your patch drops my memory usage as well. If you think this works, I say go for it! Thanks, Stephen! -Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org