Hi Britton & Matt, first of all, thanks very much for working on the parallel halo profiler! I appreciate it very much. I am having some trouble running it on a L7 dataset. Specifically, I am getting the error below at all processor counts (this is a -np 1 run, rev 1226): http://paste.enzotools.org/show/83/ It's getting late so I'll take another whack at it tomorrow morning. I'm sending it out on the chance that one of you might have some insight. Thanks! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Hi Stephen, We'd be happy to help. It'd be great if you could help and do some detective work, too. :) Specifically, maybe you can figure out which field is being profiled at the time of the crash, maybe submit the parameter file so we can take a look at it, and then see what it guesses the limits are. Maybe you could try generating an AMRSphere yourself with the center/radius it's using (this should be visible inside the HOP output file) and see if the data being fed into the profiler (not the halo profiler, but Binned1DProfile) is meaningful or not -- unfortunately because the dataset is so big, and since neither of us have run into this before, we're going to have to rely on a bit of your sleuthing. Typically -- in my experience -- the specific problem you are bringing up is from a data source not having enough data points inside the binning range. I've made up a patch that should make this more explicit. Regardless, this is a flaw that needs to be addressed in yt. The patch below doesn't give *much* information, but it will tell you if what we're seeing is what I think it is. http://paste.enzotools.org/show/84/ -Matt On Sun, Mar 22, 2009 at 9:54 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Hi Britton & Matt,
first of all, thanks very much for working on the parallel halo profiler! I appreciate it very much.
I am having some trouble running it on a L7 dataset. Specifically, I am getting the error below at all processor counts (this is a -np 1 run, rev 1226):
http://paste.enzotools.org/show/83/
It's getting late so I'll take another whack at it tomorrow morning. I'm sending it out on the chance that one of you might have some insight.
Thanks!
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
We'd be happy to help. It'd be great if you could help and do some detective work, too. :)
Sure, that's what I was expecting to do! I just needed a bit of a push and I didn't want to work in isolation on this, it would have gone slower. I have some things to do this morning but I'm sure to try you patch out today. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Hi Stephen, Any luck? I'm out of contact tomorrow and maybe Wednesday, so it'd be great to hear if you had a chance to run the patch today. -Matt On Mon, Mar 23, 2009 at 6:24 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
We'd be happy to help. It'd be great if you could help and do some detective work, too. :)
Sure, that's what I was expecting to do! I just needed a bit of a push and I didn't want to work in isolation on this, it would have gone slower.
I have some things to do this morning but I'm sure to try you patch out today.
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
Any luck? I'm out of contact tomorrow and maybe Wednesday, so it'd be great to hear if you had a chance to run the patch today.
This is what I'm getting now, which isn't very helpful... hrm. File "./hp-para.py", line 8, in <module> q.makeProfiles() File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/extensions/HaloProfiler.py", line 161, in makeProfiles File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/Profiles.py", line 208, in __init__ File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/Profiles.py", line 47, in save_state File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/Profiles.py", line 245, in _get_bins yt.lagos.Profiles.EmptyProfileData _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Well, it indicates that my guess was correct -- the sphere has no values within the bounds supplied to the profiler. The next step would be to figure out the specifics; you might try getting the bounds from within the profile object (you can use the insert_ipython() function I just added to get yourself a prompt at that point in the execution) and then check the sphere object. My guess is that it's simply too small a sphere, and the radius binning is failing as a result. On Mon, Mar 23, 2009 at 10:03 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
Any luck? I'm out of contact tomorrow and maybe Wednesday, so it'd be great to hear if you had a chance to run the patch today.
This is what I'm getting now, which isn't very helpful... hrm.
File "./hp-para.py", line 8, in <module> q.makeProfiles() File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/extensions/HaloProfiler.py", line 161, in makeProfiles File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/Profiles.py", line 208, in __init__ File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/Profiles.py", line 47, in save_state File "/nics/c/home/sskory/yt-cnl//lib/python2.6/site-packages/yt-1.5dev-py2.6-linux-x86_64.egg/yt/lagos/Profiles.py", line 245, in _get_bins yt.lagos.Profiles.EmptyProfileData
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
The next step would be to figure out the specifics; you might try getting the bounds from within the profile object (you can use the insert_ipython() function I just added to get yourself a prompt at that point in the execution) and then check the sphere object. My guess is that it's simply too small a sphere, and the radius binning is failing as a result.
OK. I'll give that a shot. Thanks! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Well, it indicates that my guess was correct -- the sphere has no values within the bounds supplied to the profiler.
Yup, I identified the problem halo and confirmed it by manually cutting it out with pf.h.sphere and trying BinnedProfile1D following the usage in HaloProfiler, and I got the same error. That halo is one of the smallest and has a fairly small radius. So, how do I make it skip that halo? I see there is a len(sphere._grids) == 0 test, but this case there are two grids so it passes that. Thanks! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
So, how do I make it skip that halo? I see there is a len(sphere._grids) == 0 test, but this case there are two grids so it passes that.
I think the problem might be with the bounds. If no points within the sphere satisfy the radius bounds, it will fail. I will commit the changes in that patch I sent you and then we can insert a try/except block for the exception I added; that is the pragmatic solution, I think. But I really, *really* think we need to figure out why the radius bounds are not giving us any points. -Matt
Maybe you could try increasing the value of PROFILE_RADIUS_THRESHOLD in HaloProfiler.py. Right now, it's set to 2, meaning that it will allow profiles for halos whose outer radii are only 2 times the inner radii. That's probably too generous. Even halos of that size should probably be ignored. Britton On Tue, Mar 24, 2009 at 11:02 AM, Matthew Turk <matthewturk@gmail.com>wrote:
So, how do I make it skip that halo? I see there is a len(sphere._grids) == 0 test, but this case there are two grids so it passes that.
I think the problem might be with the bounds. If no points within the sphere satisfy the radius bounds, it will fail. I will commit the changes in that patch I sent you and then we can insert a try/except block for the exception I added; that is the pragmatic solution, I think. But I really, *really* think we need to figure out why the radius bounds are not giving us any points.
-Matt _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi guys,
Maybe you could try increasing the value of PROFILE_RADIUS_THRESHOLD in HaloProfiler.py. Right now, it's set to 2, meaning that it will allow profiles for halos whose outer radii are only 2 times the inner radii. That's probably too generous. Even halos of that size should probably be ignored.
The radius of this problem halo/pf.h.get_smallest_dx() = 26, so I don't think that's the issue. Looking at the hierarchy, below are the two grids listed in sphere._grids. The sphere being cut is sp = pf.h.sphere([6.983622952e-01, 3.006639125e-01, 4.988738710e-01],8.156776433e-04) I notice that the center of the halo isn't in these grids. Could that be the problem? Grid = 163293 GridRank = 3 GridDimension = 14 22 20 GridStartIndex = 3 3 3 GridEndIndex = 10 18 16 GridLeftEdge = 0.693359375 0.296875 0.4921875 GridRightEdge = 0.697265625 0.3046875 0.4990234375 Time = 584.25598986586 SubgridsAreStatic = 0 FileName = RD0035.cpu0150 GroupName = /Grid00163293 NumberOfBaryonFields = 7 FieldType = 0 1 2 4 5 6 22 CourantSafetyNumber = 0.100000 PPMFlatteningParameter = 0 PPMDiffusionParameter = 0 PPMSteepeningParameter = 0 NumberOfParticles = 273 GravityBoundaryType = 2 Pointer: Grid[163293]->NextGridThisLevel = 0 Pointer: Grid[163293]->NextGridNextLevel = 163294 Grid = 163917 GridRank = 3 GridDimension = 16 14 10 GridStartIndex = 3 3 3 GridEndIndex = 12 10 6 GridLeftEdge = 0.701171875 0.296875 0.498046875 GridRightEdge = 0.7060546875 0.30078125 0.5 Time = 584.25598986586 SubgridsAreStatic = 0 FileName = RD0035.cpu0150 GroupName = /Grid00163917 NumberOfBaryonFields = 7 FieldType = 0 1 2 4 5 6 22 CourantSafetyNumber = 0.100000 PPMFlatteningParameter = 0 PPMDiffusionParameter = 0 PPMSteepeningParameter = 0 NumberOfParticles = 84 GravityBoundaryType = 2 Pointer: Grid[163917]->NextGridThisLevel = 0 Pointer: Grid[163917]->NextGridNextLevel = 163918 _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Okay, I have pulled up some source code and the RD0035 dataset and looked at this. Here's what's going on. The sphere object gets all candidate grids, from a first pass: http://yt.enzotools.org/browser/trunk/yt/lagos/HierarchyType.py#L418 This is the first pass estimate at which grids could be members, which then gets cut down in the actual sphere object; these two grids are red herrings. Note that this checks the grid dx < radius, as this is important in a moment. There are 36 particles within this region, which I was able to find by turning off the criteria that grids have dx > radius of the sphere. The ids of the grids in which these particles live are: array([ 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162648., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757., 162757.]) and the dx's are: array([ 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656, 0.00097656]) I'm torn about this issue. On the one hand, I kind of don't like the idea of giving back data from grids with dx > radius; this doesn't sit well with me. On the other, particles are not necessarily supposed to behave the same way, so perhaps in this case it would be appropriate. Changing the requirement from gridI = na.where(na.logical_and((self.gridDxs<=radius)[:,0],(dist < (radius + long_axis))) == 1) to gridI = na.where(na.logical_and((self.gridDxs<=2.0*radius)[:,0],(dist < (radius + long_axis))) == 1) would satisfy me, and would work with this particular use case. What do you think? I guess a secondary question would be about the meaning of this halo; it's 36 particles, but they're all within a radius of less than the dx of their host grid. Once I've changed the requirement for getting grids to be 2*radius, we end up with three cells included. This profile would be uninformative. -Matt On Tue, Mar 24, 2009 at 11:03 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Hi guys,
Maybe you could try increasing the value of PROFILE_RADIUS_THRESHOLD in HaloProfiler.py. Right now, it's set to 2, meaning that it will allow profiles for halos whose outer radii are only 2 times the inner radii. That's probably too generous. Even halos of that size should probably be ignored.
The radius of this problem halo/pf.h.get_smallest_dx() = 26, so I don't think that's the issue.
Looking at the hierarchy, below are the two grids listed in sphere._grids. The sphere being cut is
sp = pf.h.sphere([6.983622952e-01, 3.006639125e-01, 4.988738710e-01],8.156776433e-04)
I notice that the center of the halo isn't in these grids. Could that be the problem?
Grid = 163293 GridRank = 3 GridDimension = 14 22 20 GridStartIndex = 3 3 3 GridEndIndex = 10 18 16 GridLeftEdge = 0.693359375 0.296875 0.4921875 GridRightEdge = 0.697265625 0.3046875 0.4990234375 Time = 584.25598986586 SubgridsAreStatic = 0 FileName = RD0035.cpu0150 GroupName = /Grid00163293 NumberOfBaryonFields = 7 FieldType = 0 1 2 4 5 6 22 CourantSafetyNumber = 0.100000 PPMFlatteningParameter = 0 PPMDiffusionParameter = 0 PPMSteepeningParameter = 0 NumberOfParticles = 273 GravityBoundaryType = 2 Pointer: Grid[163293]->NextGridThisLevel = 0 Pointer: Grid[163293]->NextGridNextLevel = 163294
Grid = 163917 GridRank = 3 GridDimension = 16 14 10 GridStartIndex = 3 3 3 GridEndIndex = 12 10 6 GridLeftEdge = 0.701171875 0.296875 0.498046875 GridRightEdge = 0.7060546875 0.30078125 0.5 Time = 584.25598986586 SubgridsAreStatic = 0 FileName = RD0035.cpu0150 GroupName = /Grid00163917 NumberOfBaryonFields = 7 FieldType = 0 1 2 4 5 6 22 CourantSafetyNumber = 0.100000 PPMFlatteningParameter = 0 PPMDiffusionParameter = 0 PPMSteepeningParameter = 0 NumberOfParticles = 84 GravityBoundaryType = 2 Pointer: Grid[163917]->NextGridThisLevel = 0 Pointer: Grid[163917]->NextGridNextLevel = 163918
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I guess a secondary question would be about the meaning of this halo; it's 36 particles, but they're all within a radius of less than the dx of their host grid. Once I've changed the requirement for getting grids to be 2*radius, we end up with three cells included. This profile would be uninformative.
The meaning of this profile would be muddled at best. Let's think about when this issue would pop up. We've seen it here, with a very large simulation with many haloes. In this case, the only time you're interested in the profile of the small haloes is for statistics, like me. If you're trying to do anything more with them you've done the wrong sim. And since I'm just doing statistics, throwing out bad ones, or having a few junky values, won't change things. What other cases can we think of where this would be an issue? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
I think for now, you can set the minimum virial mass for halo profiling, in the parameter file, to be higher. Let's ponder the long-term solution to this particular corner case, and see if anything comes up? On Tue, Mar 24, 2009 at 12:02 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
I guess a secondary question would be about the meaning of this halo; it's 36 particles, but they're all within a radius of less than the dx of their host grid. Once I've changed the requirement for getting grids to be 2*radius, we end up with three cells included. This profile would be uninformative.
The meaning of this profile would be muddled at best.
Let's think about when this issue would pop up. We've seen it here, with a very large simulation with many haloes. In this case, the only time you're interested in the profile of the small haloes is for statistics, like me. If you're trying to do anything more with them you've done the wrong sim. And since I'm just doing statistics, throwing out bad ones, or having a few junky values, won't change things. What other cases can we think of where this would be an issue?
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (3)
-
Britton Smith -
Matthew Turk -
Stephen Skory