nickel instruction for using parallel HOP (not on Kraken)
Hi all, Matt asked me to outline how to get parallel HOP (in the hg devel version) working. In easy to follow list format: 0. Build yt like you did already (you're on this list, no?). 1. Get the latest hg, choose the 'yt' branch, which is the default. 2. Install Forthon in the standard python way. http://hifweb.lbl.gov/Forthon/ 3. Go into yt-devel/yt/extensions/kdtree and I think 'make' will work on most machines. If not, let me know. It should make 'fKDpy.so' in that same dir. 4. Go to the top level of yt-devel, and type 'python setup.py install' like usual, but then do 'cp -r yt/extensions/kdtree ../../lib/python2.6/site-packages/yt-version-num-sys-etc/yt/extensions/'. I haven't added the kdtree stuff to setup utils yet, so this step isn't automated. 5. One runs parallel HOP very similarly to old HOP: h = yt.lagos.HaloFinding.parallelHF(pf, threshold=160.0, safety=1.5, \ dm_only=False,resize=True, fancy_padding=True, rearrange=True) - safety - is a padding multiplier that adjusts the size of the padding around each subvolume. 1.5 is usually sufficient, and 1.35 probably is. There's no reason to try going lower from 1.5 unless memory is a concern. - dm_only - I currently have things a little messed up in HaloFinding.py in the dev, so this needs to be set unless you have a ['creation_time'] field. - resize - Load balancing. This is a recursive bisection that subdivides the domains like a kd tree, but it can do arbitrary numbers of cuts, so you can run with any number of tasks. Generally, I turn it on for datasets smaller than 300 Mpc, and for ones larger, the particles are already pretty evenly balanced and it doesn't help. - fancy_padding - Each subvolume has six different faces, and six different values for the padding, one for each face. There is really no reason to turn this off. - rearrange - The fortran kdtree can make an internal copy of the point data which increases the speed of the searches by almost 20%. It however increases the memory by 3 fields, so if memory is a concern, turn this off. 6. The usual halo convenience functions work as well, but the output is a bit different. h.write_out('dist-chain.out') - Just like the normal one. h.write_particle_lists("chain") - One hdf5 file per task, that contains the particle data for all the haloes on that task. h.write_particle_lists_txt("chain") - One text file that lists which tasks own a piece of a halo. Since haloes can exist on more than one task, each halo may have several entries on it's line. 7. There is a rough memory scaling law: it takes 1 MB of RAM per 4,500 particles, with rearrange turned on, regardless of task count. This does not include the yt hierarchy costs, so if you have a dataset like L7, which uses 1GB per task, you will need to take that into account, too. Good luck! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Hi Stephen, This is a bit off topic, but...
2. Install Forthon in the standard python way. http://hifweb.lbl.gov/Forthon/ Can I ask what made you use Forthon over f2py? I have been using f2py, and have found it quite useful for wrapping fortran code. Given that f2py comes with numpy, I'm wondering what the specific advantage for Forthon is, if any.
thanks, J. S.
J. S.,
2. Install Forthon in the standard python way. http://hifweb.lbl.gov/Forthon/ Can I ask what made you use Forthon over f2py? I have been using f2py, and have found it quite useful for wrapping fortran code. Given that f2py comes with numpy, I'm wondering what the specific advantage for Forthon is, if any.
I was unable to convince f2py to build static objects, which is required on Kraken. That is it's big advantage. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Hi Stephen, I'm having a problem with parallelHOP where one (and seemingly only one) of the processors is having an array out of bounds error with the densest_in_chain_real_index array. I printed out the size of this array and the densest_in_chain array and in the one processor where it fails the densest_in_chain_real_index array has one less element. Below is the traceback: The three numbers printed out at the top are the sizes of densest_in_chain, densest_in_chain_real_index, and sort, in that order. P005 yt INFO 2009-12-01 10:12:28,438 Locally sorting chains... 3794 3793 3794 Traceback (most recent call last): File "do_hop.py", line 9, in <module> fancy_padding=True, rearrange=True) File "/duanestorage/home/student/brittons/local/x86_64/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 958, in __init__ threshold=threshold, dm_only=dm_only, rearrange=rearrange) File "/duanestorage/home/student/brittons/local/x86_64/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 598, in __init__ HaloList.__init__(self, data_source, dm_only) File "/duanestorage/home/student/brittons/local/x86_64/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 339, in __init__ self._run_finder() File "/duanestorage/home/student/brittons/local/x86_64/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 609, in _run_finder self.threshold, rearrange=self.rearrange) File "/duanestorage/home/student/brittons/local/x86_64/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/parallelHOP/parallelHOP.py",line 56, in __init__ self._chain_hop() File "/duanestorage/home/student/brittons/local/x86_64/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/parallelHOP/parallelHOP.py",line 1423, in _chain_hop chain_count = self._preconnect_chains(chain_count) File "/duanestorage/home/student/brittons/local/x86_64/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/parallelHOP/parallelHOP.py",line 538, in _preconnect_chains self.densest_in_chain_real_index = self.densest_in_chain_real_index[sort] IndexError: index 3793 out of bounds 0<=index<3793 Do you have any idea what's going on here? Thanks, Britton On Wed, Nov 25, 2009 at 4:45 PM, Stephen Skory <stephenskory@yahoo.com>wrote:
J. S.,
2. Install Forthon in the standard python way. http://hifweb.lbl.gov/Forthon/ Can I ask what made you use Forthon over f2py? I have been using f2py, and have found it quite useful for wrapping fortran code. Given that f2py comes with numpy, I'm wondering what the specific advantage for Forthon is, if any.
I was unable to convince f2py to build static objects, which is required on Kraken. That is it's big advantage.
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ <http://physics.ucsd.edu/%7Esskory/> _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Britton,
I'm having a problem with parallelHOP where one (and seemingly only one) of the processors is having an array out of bounds error with the densest_in_chain_real_index array. I printed out the size of this array and the densest_in_chain array and in the one processor where it fails the densest_in_chain_real_index array has one less element. Below is the traceback:
Can you apply this patch to ParallelHOP.py, and tell me what you get? You should change the 'self.mine == -1' to the task ID that's giving you problems. http://paste.enzotools.org/show/273/ I'm glad you're trying to use it! Thank you! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Hi Stephen, Here's the output after adding the patch and choosing the process with the issue: before cleaning dic 10000 dicri 10000 count1 3794 count2+count3 3794 I'm definitely psyched to be using this. Britton On Tue, Dec 1, 2009 at 10:50 AM, Stephen Skory <stephenskory@yahoo.com>wrote:
Britton,
I'm having a problem with parallelHOP where one (and seemingly only one) of the processors is having an array out of bounds error with the densest_in_chain_real_index array. I printed out the size of this array and the densest_in_chain array and in the one processor where it fails the densest_in_chain_real_index array has one less element. Below is the traceback:
Can you apply this patch to ParallelHOP.py, and tell me what you get? You should change the 'self.mine == -1' to the task ID that's giving you problems.
http://paste.enzotools.org/show/273/
I'm glad you're trying to use it! Thank you!
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ <http://physics.ucsd.edu/%7Esskory/> _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Britton,
Here's the output after adding the patch and choosing the process with the issue: before cleaning dic 10000 dicri 10000 count1 3794 count2+count3 3794
It didn't print "Real index -1!!!" or "Padded index -1!!!"? Did you notice the other 'self.mine == -1'? The reason I ask, is because of the counts, it looks like the same number of elements are being added to self.dic and self.dicri. But when the array is cleaned up, the two have different lengths, which my first guess was that there was a particle_index == -1 that was being added. Let me think about this some more... _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Ooops! My fault. I didn't notice the others. I'll fix and rerun right away. On Tue, Dec 1, 2009 at 11:47 AM, Stephen Skory <stephenskory@yahoo.com>wrote:
Britton,
Here's the output after adding the patch and choosing the process with the issue: before cleaning dic 10000 dicri 10000 count1 3794 count2+count3 3794
It didn't print "Real index -1!!!" or "Padded index -1!!!"? Did you notice the other 'self.mine == -1'? The reason I ask, is because of the counts, it looks like the same number of elements are being added to self.dic and self.dicri. But when the array is cleaned up, the two have different lengths, which my first guess was that there was a particle_index == -1 that was being added.
Let me think about this some more...
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ <http://physics.ucsd.edu/%7Esskory/> _.>/ _Graduate Student ________________________________(_)_\(_)_______________
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Ok, I got one of these: Real index -1!!! I didn't get a "Padded index -1!!!". You're suspicion seems to have been right. Britton On Tue, Dec 1, 2009 at 11:49 AM, Britton Smith <brittonsmith@gmail.com>wrote:
Ooops! My fault. I didn't notice the others. I'll fix and rerun right away.
On Tue, Dec 1, 2009 at 11:47 AM, Stephen Skory <stephenskory@yahoo.com>wrote:
Britton,
Here's the output after adding the patch and choosing the process with the issue: before cleaning dic 10000 dicri 10000 count1 3794 count2+count3 3794
It didn't print "Real index -1!!!" or "Padded index -1!!!"? Did you notice the other 'self.mine == -1'? The reason I ask, is because of the counts, it looks like the same number of elements are being added to self.dic and self.dicri. But when the array is cleaned up, the two have different lengths, which my first guess was that there was a particle_index == -1 that was being added.
Let me think about this some more...
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ <http://physics.ucsd.edu/%7Esskory/>_.>/ _Graduate Student ________________________________(_)_\(_)_______________
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Britton,
Ok, I got one of these:
Real index -1!!!
I didn't get a "Padded index -1!!!". You're suspicion seems to have been right.
So it seems to me that you probably have a particle with index == -1. There are a few things we can try. If you have just one with -1, we can change the default 'empty' value in those arrays to something else. I suspect you won't have just one, however, at which point you're kind of up a creek, because that's absolutely required for parallel HOP. It may be worth your time to fix the -1 IDs to something unique in the enzo dataset. I should add a check that screens for bad particle index values, like -1. I've learned something today. Let me know if there's anything else I can help you with! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Stephen, Your suspicion was correct. For some reason, this simulation has a number of particles with ids that are -1. I went all the way back to the very first output of that simulation and found 126 out of 256^3 particles to have ids of -1. I have absolutely no idea why this is. This simulation data is more than a year old, so it is possible there was a bug in enzo at the time. I just tested ParallelHOP out on a much newer dataset and it worked without a problem. Thanks a lot for your help and thanks again for writing this code. Britton On Tue, Dec 1, 2009 at 12:22 PM, Stephen Skory <stephenskory@yahoo.com>wrote:
Britton,
Ok, I got one of these:
Real index -1!!!
I didn't get a "Padded index -1!!!". You're suspicion seems to have been right.
So it seems to me that you probably have a particle with index == -1. There are a few things we can try. If you have just one with -1, we can change the default 'empty' value in those arrays to something else. I suspect you won't have just one, however, at which point you're kind of up a creek, because that's absolutely required for parallel HOP.
It may be worth your time to fix the -1 IDs to something unique in the enzo dataset.
I should add a check that screens for bad particle index values, like -1. I've learned something today.
Let me know if there's anything else I can help you with!
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ <http://physics.ucsd.edu/%7Esskory/> _.>/ _Graduate Student ________________________________(_)_\(_)_______________
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (3)
-
Britton Smith -
j s oishi -
Stephen Skory