Hi all, I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The problem was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator. What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems. -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)
Hi I would advocate, instead of changing the default, adding a message that says: halo profiler projections will be NxN, and preferably have it print somewhere just before it might die. That way when it does crash, it will be simpler to figure out why. Two Cents, Sam On Mon, Apr 11, 2011 at 8:11 PM, Stephen Skory <s@skory.us> wrote:
Hi all,
I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The problem was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator.
What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Sam, My feeling is that the HaloProfiler does a *lot*, and I would like to see it become feature parity with enzo_anyl. (A shame that we haven't yet fully recreated that enzo_anyl experience yet, with anything in yt...) As such, I think we should be positioning it as useful for a lot of domains. I guess the choice is really, do we want it to do something smart? Or do we want to let the code knowingly go down, and then tell the user "Sorry, but we warned you!" while shaking our heads and shrugging our shoulders. -Matt On Mon, Apr 11, 2011 at 8:20 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi I would advocate, instead of changing the default, adding a message that says: halo profiler projections will be NxN, and preferably have it print somewhere just before it might die. That way when it does crash, it will be simpler to figure out why. Two Cents, Sam On Mon, Apr 11, 2011 at 8:11 PM, Stephen Skory <s@skory.us> wrote:
Hi all,
I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The problem was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator.
What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Matt, Stephen, I think there is then an issue of how does the user know what the size of the image is. Perhaps we should load up some hdf5 attributes that include things like length, pixel scale, etc. The nice thing about the old behavior was that you could easily figure out what the size was from looking at what the default was. And if you changed it, then you know from the start. Thoughts? I would be in favor of this change assuming we add the hdf5 attributes. Sam On Mon, Apr 11, 2011 at 8:34 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Sam,
My feeling is that the HaloProfiler does a *lot*, and I would like to see it become feature parity with enzo_anyl. (A shame that we haven't yet fully recreated that enzo_anyl experience yet, with anything in yt...) As such, I think we should be positioning it as useful for a lot of domains.
I guess the choice is really, do we want it to do something smart? Or do we want to let the code knowingly go down, and then tell the user "Sorry, but we warned you!" while shaking our heads and shrugging our shoulders.
-Matt
On Mon, Apr 11, 2011 at 8:20 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi I would advocate, instead of changing the default, adding a message that says: halo profiler projections will be NxN, and preferably have it print somewhere just before it might die. That way when it does crash, it will be simpler to figure out why. Two Cents, Sam On Mon, Apr 11, 2011 at 8:11 PM, Stephen Skory <s@skory.us> wrote:
Hi all,
I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The problem was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator.
What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi guys, I am not in favor of changing the width to something related to the halo radius. One of the main ideas behind the halo profiler projection routine is to get images of halos with a constant width so that they can all be compared against each other. I agree with Sam that if we change this to something not constant then these images become a lot let useful as an analysis tool. I think just indicating that the image is going to be a certain size, or adding some sort of max image size argument is fine. Honestly, I think sometimes it's just fine to throw up your hands and say, "We warned you!" Maintaining usefulness is a lot more important to me than seeing everything just run without an error. Britton On Mon, Apr 11, 2011 at 8:47 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi Matt, Stephen,
I think there is then an issue of how does the user know what the size of the image is. Perhaps we should load up some hdf5 attributes that include things like length, pixel scale, etc. The nice thing about the old behavior was that you could easily figure out what the size was from looking at what the default was. And if you changed it, then you know from the start.
Thoughts? I would be in favor of this change assuming we add the hdf5 attributes. Sam
On Mon, Apr 11, 2011 at 8:34 PM, Matthew Turk <matthewturk@gmail.com>wrote:
Hi Sam,
My feeling is that the HaloProfiler does a *lot*, and I would like to see it become feature parity with enzo_anyl. (A shame that we haven't yet fully recreated that enzo_anyl experience yet, with anything in yt...) As such, I think we should be positioning it as useful for a lot of domains.
I guess the choice is really, do we want it to do something smart? Or do we want to let the code knowingly go down, and then tell the user "Sorry, but we warned you!" while shaking our heads and shrugging our shoulders.
-Matt
Hi I would advocate, instead of changing the default, adding a message that says: halo profiler projections will be NxN, and preferably have it
On Mon, Apr 11, 2011 at 8:20 PM, Sam Skillman <samskillman@gmail.com> wrote: print
somewhere just before it might die. That way when it does crash, it will be simpler to figure out why. Two Cents, Sam On Mon, Apr 11, 2011 at 8:11 PM, Stephen Skory <s@skory.us> wrote:
Hi all,
I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The problem was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator.
What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all, Okay, I have come around. Stephen, I think the code works as-is. It's not really designed for a single halo analysis task, which I think is what you're applying it to. I think a better solution to this would be to re-architecture things a bit, and I think that perhaps Britton and Sam might be on board if we took it to a different level. What is floating around in my head is that the Halo Profiler's responsibilities and actions can be divided into two camps, and maybe Britton and Sam can chime in on this. 1) Tasks that make sense from the perspective of detailed examination of a single halo 2) Tasks that make sense from the perspective of detailed comparison of multiple halos What I was thinking is that we could try to aim for a separation of responsibilities into, perhaps, a set of domains: Halo analysis Halo catalog analysis (This would also, ideally, be inserted into the astro_objects directory when that project really takes off.) The tasks that would replicate the large part of enzo_anyl functionality, a number of which are included, would be included into the former camp. The tasks that make more sense (like projecting at a fixed scale) for the comparison of multiple halos would be put into camp two. The HaloProfiler itself would then construct multiple HaloAnalysis objects, for each halo, and would itself be a HaloCatalogAnalysis object. Would this make sense? I guess this conversation has gotten a bit broader, by the injection of a discussion of enzo_anyl's functionality, and the idea of an all-in-one single halo profiler, but I think it's valid. What I'd really like to avoid here is either fragmenting into many different overlapping tools and utilities, as well as ensuring that tools that currently serve a good purpose continue to serve that purpose. I'd be happy to help with this kind of project, as it would serve a few of my own very-near term research purposes. -Matt On Mon, Apr 11, 2011 at 9:26 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi guys,
I am not in favor of changing the width to something related to the halo radius. One of the main ideas behind the halo profiler projection routine is to get images of halos with a constant width so that they can all be compared against each other. I agree with Sam that if we change this to something not constant then these images become a lot let useful as an analysis tool.
I think just indicating that the image is going to be a certain size, or adding some sort of max image size argument is fine. Honestly, I think sometimes it's just fine to throw up your hands and say, "We warned you!" Maintaining usefulness is a lot more important to me than seeing everything just run without an error.
Britton
On Mon, Apr 11, 2011 at 8:47 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi Matt, Stephen, I think there is then an issue of how does the user know what the size of the image is. Perhaps we should load up some hdf5 attributes that include things like length, pixel scale, etc. The nice thing about the old behavior was that you could easily figure out what the size was from looking at what the default was. And if you changed it, then you know from the start.
Thoughts? I would be in favor of this change assuming we add the hdf5 attributes. Sam On Mon, Apr 11, 2011 at 8:34 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Sam,
My feeling is that the HaloProfiler does a *lot*, and I would like to see it become feature parity with enzo_anyl. (A shame that we haven't yet fully recreated that enzo_anyl experience yet, with anything in yt...) As such, I think we should be positioning it as useful for a lot of domains.
I guess the choice is really, do we want it to do something smart? Or do we want to let the code knowingly go down, and then tell the user "Sorry, but we warned you!" while shaking our heads and shrugging our shoulders.
-Matt
On Mon, Apr 11, 2011 at 8:20 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi I would advocate, instead of changing the default, adding a message that says: halo profiler projections will be NxN, and preferably have it print somewhere just before it might die. That way when it does crash, it will be simpler to figure out why. Two Cents, Sam On Mon, Apr 11, 2011 at 8:11 PM, Stephen Skory <s@skory.us> wrote:
Hi all,
I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The problem was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator.
What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Matt, I think this sounds like a good plan. I think the halo profiler is already somewhat set up like this, in that the catalog of halos are all individual objects, but britton knows more about this. Perhaps we should attempt to add any new functionality that we want for a single halo and go from there? It may just require splitting up the objects a bit without changing much of the api. There is already a halos=single/multiple keyword, but I think you're right that we could make the split more explicit. As a side note, I've used the halo profiler a lot in my last couple of projects for the "catalog" type analysis. In doing so, I've written some 2D analysis scripts that act on the resulting images to mimic what an observation would do (e.g. 2D radial profiles, combining many of these into "average" profiles of many objects, etc). Perhaps during this time it would be the right time to clean it up and add it to work on single objects or catalogs. Sam On Mon, Apr 11, 2011 at 9:42 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
Okay, I have come around.
Stephen, I think the code works as-is. It's not really designed for a single halo analysis task, which I think is what you're applying it to. I think a better solution to this would be to re-architecture things a bit, and I think that perhaps Britton and Sam might be on board if we took it to a different level.
What is floating around in my head is that the Halo Profiler's responsibilities and actions can be divided into two camps, and maybe Britton and Sam can chime in on this.
1) Tasks that make sense from the perspective of detailed examination of a single halo 2) Tasks that make sense from the perspective of detailed comparison of multiple halos
What I was thinking is that we could try to aim for a separation of responsibilities into, perhaps, a set of domains:
Halo analysis Halo catalog analysis
(This would also, ideally, be inserted into the astro_objects directory when that project really takes off.)
The tasks that would replicate the large part of enzo_anyl functionality, a number of which are included, would be included into the former camp. The tasks that make more sense (like projecting at a fixed scale) for the comparison of multiple halos would be put into camp two. The HaloProfiler itself would then construct multiple HaloAnalysis objects, for each halo, and would itself be a HaloCatalogAnalysis object.
Would this make sense? I guess this conversation has gotten a bit broader, by the injection of a discussion of enzo_anyl's functionality, and the idea of an all-in-one single halo profiler, but I think it's valid.
What I'd really like to avoid here is either fragmenting into many different overlapping tools and utilities, as well as ensuring that tools that currently serve a good purpose continue to serve that purpose. I'd be happy to help with this kind of project, as it would serve a few of my own very-near term research purposes.
-Matt
Hi guys,
I am not in favor of changing the width to something related to the halo radius. One of the main ideas behind the halo profiler projection routine is to get images of halos with a constant width so that they can all be compared against each other. I agree with Sam that if we change this to something not constant then these images become a lot let useful as an analysis tool.
I think just indicating that the image is going to be a certain size, or adding some sort of max image size argument is fine. Honestly, I think sometimes it's just fine to throw up your hands and say, "We warned you!" Maintaining usefulness is a lot more important to me than seeing everything just run without an error.
Britton
On Mon, Apr 11, 2011 at 8:47 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi Matt, Stephen, I think there is then an issue of how does the user know what the size
of
the image is. Perhaps we should load up some hdf5 attributes that include things like length, pixel scale, etc. The nice thing about the old behavior was that you could easily figure out what the size was from looking at what the default was. And if you changed it, then you know from the start.
Thoughts? I would be in favor of this change assuming we add the hdf5 attributes. Sam On Mon, Apr 11, 2011 at 8:34 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Sam,
My feeling is that the HaloProfiler does a *lot*, and I would like to see it become feature parity with enzo_anyl. (A shame that we haven't yet fully recreated that enzo_anyl experience yet, with anything in yt...) As such, I think we should be positioning it as useful for a lot of domains.
I guess the choice is really, do we want it to do something smart? Or do we want to let the code knowingly go down, and then tell the user "Sorry, but we warned you!" while shaking our heads and shrugging our shoulders.
-Matt
On Mon, Apr 11, 2011 at 8:20 PM, Sam Skillman <samskillman@gmail.com> wrote:
Hi I would advocate, instead of changing the default, adding a message that says: halo profiler projections will be NxN, and preferably have it print somewhere just before it might die. That way when it does crash, it will be simpler to figure out why. Two Cents, Sam On Mon, Apr 11, 2011 at 8:11 PM, Stephen Skory <s@skory.us> wrote:
Hi all,
I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The
On Mon, Apr 11, 2011 at 9:26 PM, Britton Smith <brittonsmith@gmail.com> wrote: problem
was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator.
What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
All, I think that improving the Halo Profiler in these ways is a good idea. To address the original thread of this email (which is not to say that I'm not glad that I sparked a discussion of how to improve things), unless there are any objections, I'll put a mylog.info() in fixed_resolution.py just above the call to _MPL.Pixelize() that prints the size of the buffer that's about to be created. -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)
Hi Stephen, I think that's a fine idea. Britton On Tue, Apr 12, 2011 at 11:03 AM, Stephen Skory <s@skory.us> wrote:
All,
I think that improving the Halo Profiler in these ways is a good idea. To address the original thread of this email (which is not to say that I'm not glad that I sparked a discussion of how to improve things), unless there are any objections, I'll put a mylog.info() in fixed_resolution.py just above the call to _MPL.Pixelize() that prints the size of the buffer that's about to be created.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
+1 On Mon, Apr 11, 2011 at 8:11 PM, Stephen Skory <s@skory.us> wrote:
Hi all,
I just spent some time trying to diagnose a segfault when the HaloProfiler was trying to make projections of the haloes. The problem was in making the FixedResolutionBuffer, the image it was trying to make was 16K by 16K, which cannot fit in any normal machine's memory. I think this is because I am looking at a zoom-in simulation of a smallish cosmological size, with high resolution in the region of interest. The default projection width in the HaloProfiler is 8 Mpc, so I had a big numerator and a small denominator.
What do we think about changing this default to some multiple of the halo maximum radius? I think any constant value will make problems.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (4)
-
Britton Smith -
Matthew Turk -
Sam Skillman -
Stephen Skory