Hi David,
On Mon, Mar 12, 2012 at 5:45 PM, david collins
Hi, All--
What's the current state of the art with regard to parallel clump finding?
It's not implemented.
Out of curiosity, I did a clumpfind blocked into octant, and I found super-linear scaling (non-scientific, my 512^3 wouldn't finish in 24 hours, but each octant ran in 30 minutes). Now I have these clumps, and I'd like to sew them together. So I have a couple questions:
1.) What's the easiest way to join two extracted sets? I figure I can take these 8 sets and sew them together pretty easy ex-post-facto. If there's a tool that does new_joined_thing = yt.join( clump1.data, clump2.data) then writing the machinery to sort out which clumps hit which faces is pretty easy, as is sewing together the various joined things into a clump hierarchy.
Extracted sets have a built-in method for this, which is supposed to intelligently join the indices to minimize overhead. "join" You can also use boolean regions, but joining boolean regions won't coalesce indices so they may be slightly higher overhead.
2.) Is there anything about the countour finder that would not block decompose like this?
Mmm, probably joins across boundaries and across corners, and any join that extends over an octant. Particularly, you will need to examine every cell on the boundary between octants. Almost invariably this will involve the entire dataset residing in memory, or an OOC structure that allows multiple r/w accesses.
3.) Has anyone made any progress on things like this?
There was a project going to insert an external level set identification algorithm. While I believe that project is progressing, I would instead recommend you investigate a method or algorithm yourself and work with a senior yt developer (Britton and I have both expressed interest in taking this on) to contribute this change back to the community. A parallel contouring algorithm would be very valuable, and while it would be challenging, it can certainly be accomplished. I would encourage you, if this is functionality you are interested in utilizing, to take up the reins of a project like this, identify collaborators, and push forward on developing this functionality and returning it to the community. As it stands, while there are interested parties, I doubt that they will be willing to devote substantial time to either private development of a feature for a specific project (rather than something that advances the yt project as a whole) or to work on it in isolation without a concrete use case. The clump finder Britton and I developed initially targeted a very specific project, but was developed not only in the open but designed to be a general, multi-purpose module. I would hope that efforts to extend it to function in parallel would share those goals. -Matt
d.
-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org