Shankar, In a box size of 200 Mpc/h^3 and 512^3 particles, your mass resolution is about 2e10M_solar. That means in a 10^13 M_solar halo, you have at most ~500 particles in your halos, which is well resolved. The main problem with what you're doing is putting an upper limit on the mass of the halos that you are profiling, which is opposite to what one normally does. Even if you want to be cavalier with what you call a halo, you definitely don't want to go much below 2-5e12 since then you're talking 100 particles. I personally don't trust profiles unless they have a few thousand particles, because then I might believe the hydro is resolved. My guess is that if you do: virial_filters=[['TotalMassMsun','>=','5e12']] you'll have more luck with them actually being virialized. Main point: If you want to study halos with masses less than 10^13, you're using the wrong simulation (i.e. too large of a physical box size or too small of a grid). Sam On Tue, Dec 8, 2009 at 3:04 PM, Agarwal, Shankar wrote:

Sam,

I am trying to get the virial masses of the halos (with mass upto 1e+13 Msun) with this filter...

hp.add_halo_filter(HP.VirialFilter, must_be_virialized=True, overdensity_field='ActualOverdensity', virial_overdensity=200, virial_filters=[['TotalMassMsun','<=','1e13']], virial_quantities=['TotalMassMsun','RadiusMpc'])

But when I look at the radial_profiles/Halo_xxxx_profile.dat files, I see that none of the halos are virialized. The peak halo densities are reaching about 20*mean_matter_density. I began my 200Mpc/h Box with 512^3 particles simulation at z=99. I have not looked at the literature in great detail but is there something fishy here ?

shankar

-----Original Message----- From: yt-users-bounces@lists.spacepope.org on behalf of Sam Skillman Sent: Tue 12/8/2009 3:23 PM To: Discussion of the yt analysis package Subject: Re: [yt-users] r_min in HaloProfiler

Shankar,

r_min is defined in your email: r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc']

it's twice the smallest dx in units of mpc, dx being the smallest cell size. it is being rejected because your halo has something like 3 cells (radial) in it, which is almost certainly not resolved.

sam

On Tue, Dec 8, 2009 at 2:11 PM, Agarwal, Shankar wrote:

...

Hi,

While running HaloProfiler, I am seeing this...

P001 yt.lagos ERROR 2009-12-08 15:53:22,403 Skipping halo with r_max / r_min = 1.885682.

I looked at yt/extensions/HaloProfiler.py...

r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc'] if (halo['r_max'] / r_min < PROFILE_RADIUS_THRESHOLD): mylog.error("Skipping halo with r_max / r_min = %f." % (halo['r_max']/r_min))

r_max would be the distance to the furthest particle in the halo as found by HaloFinder. But what is r_min ? And what is the basis for skipping this halo ?

shankar _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu

Eric and Sam, Yeah. that makes sense. It boils down to the resolution of the box. A question to Sam ... How did you get "mass resolution is about 2e10M_solar" ? I am getting 5e9M_solar as the mass of each dark matter particle. And the number of particles in a 1e13M_solar halo is 2000. ID mass particles density x y z centreofmass x centerofmass y centerofmass z 3700 9.853250284e+12 2000 1.136880784e+04 3.619545698e-01 4.593685567e-01 8.036487699e-01 3.618929449e-01 4.593253520e-01 8.041266519e-01 So I guess for 250 particles, I can try... virial_filters=[['TotalMassMsun','>=','1.25e12']] A question to Eric : Actually, I don't care for the resolution of the halos. I am analyzing the suppression of matter power spectrum as a function of neutrino mass. Which means I am interested only on the semi-linear scales of 20-60 Mpc/h. Towards this, I wish to extract small galaxy groups (upto 1e13M_solar) from my simulation. So, can I regard the HaloFinder halos as blobs, w/o caring for the resolution ? If yes, can I include halofinder halos with as low as 50-100 particles (since they are still halos, regardless of the resolution) ? I will appreciate if I can get some opinion on this. shankar -----Original Message----- From: yt-users-bounces@lists.spacepope.org on behalf of Eric Hallman Sent: Tue 12/8/2009 4:53 PM To: Discussion of the yt analysis package Subject: Re: [yt-users] r_min in HaloProfiler Shankar, sam is right on the money with his analysis. Although the halo finders identify a large number of halos in your simulation they are very poorly resolved, and so this filter will rule them out on the basis of not getting to OD of 200. Our analysis with enzo shows that you can not expect the halos to be resolved under a few hundred particles as sam suggests. I would take a look at how many halos are above 500 particles or so and only run your analysis on those. I suspect it is possible that one of the reasons you are not seeing speedup with the parallel runs is that you don't have enough work for the processors given the small number of virialized halos that you'll actually analyze with this filter. On Dec 8, 2009, at 3:30 PM, Sam Skillman wrote:

Shankar,

In a box size of 200 Mpc/h^3 and 512^3 particles, your mass resolution is about 2e10M_solar. That means in a 10^13 M_solar halo, you have at most ~500 particles in your halos, which is well resolved. The main problem with what you're doing is putting an upper limit on the mass of the halos that you are profiling, which is opposite to what one normally does. Even if you want to be cavalier with what you call a halo, you definitely don't want to go much below 2-5e12 since then you're talking 100 particles. I personally don't trust profiles unless they have a few thousand particles, because then I might believe the hydro is resolved. My guess is that if you do:

virial_filters=[['TotalMassMsun','>=','5e12']]

you'll have more luck with them actually being virialized.

Main point: If you want to study halos with masses less than 10^13, you're using the wrong simulation (i.e. too large of a physical box size or too small of a grid).

Sam

On Tue, Dec 8, 2009 at 3:04 PM, Agarwal, Shankar wrote: Sam,

I am trying to get the virial masses of the halos (with mass upto 1e +13 Msun) with this filter...

hp.add_halo_filter(HP.VirialFilter, must_be_virialized=True, overdensity_field='ActualOverdensity', virial_overdensity=200, virial_filters=[['TotalMassMsun','<=','1e13']], virial_quantities=['TotalMassMsun','RadiusMpc'])

But when I look at the radial_profiles/Halo_xxxx_profile.dat files, I see that none of the halos are virialized. The peak halo densities are reaching about 20*mean_matter_density. I began my 200Mpc/h Box with 512^3 particles simulation at z=99. I have not looked at the literature in great detail but is there something fishy here ?

shankar

-----Original Message----- From: yt-users-bounces@lists.spacepope.org on behalf of Sam Skillman Sent: Tue 12/8/2009 3:23 PM To: Discussion of the yt analysis package Subject: Re: [yt-users] r_min in HaloProfiler

Shankar,

r_min is defined in your email: r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc']

it's twice the smallest dx in units of mpc, dx being the smallest cell size. it is being rejected because your halo has something like 3 cells (radial) in it, which is almost certainly not resolved.

sam

On Tue, Dec 8, 2009 at 2:11 PM, Agarwal, Shankar wrote:

...

Hi,

While running HaloProfiler, I am seeing this...

P001 yt.lagos ERROR 2009-12-08 15:53:22,403 Skipping halo with r_max / r_min = 1.885682.

I looked at yt/extensions/HaloProfiler.py...

r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc'] if (halo['r_max'] / r_min < PROFILE_RADIUS_THRESHOLD): mylog.error("Skipping halo with r_max / r_min = %f." % (halo['r_max']/r_min))

r_max would be the distance to the furthest particle in the halo as found by HaloFinder. But what is r_min ? And what is the basis for skipping this halo ?

shankar _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Eric Hallman Google Voice: (312) 725-HMAN hallman13@gmail.com

Hi Shankar,

Actually, I don't care for the resolution of the halos. I am analyzing the suppression of matter power spectrum as a function of neutrino mass. Which means I am interested only on the semi-linear scales of 20-60 Mpc/h. Towards this, I wish to extract small galaxy groups (upto 1e13M_solar) from my simulation. So, can I regard the HaloFinder halos as blobs, w/o caring for the resolution ? If yes, can I include halofinder halos with as low as 50-100 particles (since they are still halos, regardless of the resolution) ?

I will appreciate if I can get some opinion on this.

I would strongly reccomend to read O'Shea et al. 2005, ApJS, 160, 1. The halo mass function in Enzo is not correctly reproduced below ~200-300 particles, even if you use AMR. You start missing a significant number of halos below that threshold. This is because the use of a PM scheme at very high z (at the beginning of your simulation) suppresses high frequency modes in the power spectrum. If you need to study the halo mass function/power spectrum, then I would suggest you to consider an alternative code. For example, Gadget2 (Springel 2005) can reproduce correctly the halo mass function down to ~30 particles and might be better suited to your needs. Like Enzo, it is public. Cheers, Michele Michele Trenti, Ph.D. Center for Astrophysics and Space Astronomy University of Colorado 389 UCB Phone: +1 303-492-8490 Boulder, Colorado 80309-0389 Fax: +1 303-492-7178 USA E-mail: michele.trenti@colorado.edu "It often happens that a player carries out a deep and complicated calculation, but fails to spot something elementary right at the first move." -- Chess Grandmaster Alexander Kotov On Tue, 8 Dec 2009, Agarwal, Shankar wrote:

Eric and Sam,

Yeah. that makes sense. It boils down to the resolution of the box.

A question to Sam ... How did you get "mass resolution is about 2e10M_solar" ?

I am getting 5e9M_solar as the mass of each dark matter particle. And the number of particles in a 1e13M_solar halo is 2000.

ID mass particles density x y z centreofmass x centerofmass y centerofmass z 3700 9.853250284e+12 2000 1.136880784e+04 3.619545698e-01 4.593685567e-01 8.036487699e-01 3.618929449e-01 4.593253520e-01 8.041266519e-01

So I guess for 250 particles, I can try...

virial_filters=[['TotalMassMsun','>=','1.25e12']]

A question to Eric :

Actually, I don't care for the resolution of the halos. I am analyzing the suppression of matter power spectrum as a function of neutrino mass. Which means I am interested only on the semi-linear scales of 20-60 Mpc/h. Towards this, I wish to extract small galaxy groups (upto 1e13M_solar) from my simulation. So, can I regard the HaloFinder halos as blobs, w/o caring for the resolution ? If yes, can I include halofinder halos with as low as 50-100 particles (since they are still halos, regardless of the resolution) ?

I will appreciate if I can get some opinion on this.

shankar

-----Original Message----- From: yt-users-bounces@lists.spacepope.org on behalf of Eric Hallman Sent: Tue 12/8/2009 4:53 PM To: Discussion of the yt analysis package Subject: Re: [yt-users] r_min in HaloProfiler

Shankar, sam is right on the money with his analysis. Although the halo finders identify a large number of halos in your simulation they are very poorly resolved, and so this filter will rule them out on the basis of not getting to OD of 200. Our analysis with enzo shows that you can not expect the halos to be resolved under a few hundred particles as sam suggests.

I would take a look at how many halos are above 500 particles or so and only run your analysis on those. I suspect it is possible that one of the reasons you are not seeing speedup with the parallel runs is that you don't have enough work for the processors given the small number of virialized halos that you'll actually analyze with this filter.

On Dec 8, 2009, at 3:30 PM, Sam Skillman wrote:

...

Shankar,

In a box size of 200 Mpc/h^3 and 512^3 particles, your mass resolution is about 2e10M_solar. That means in a 10^13 M_solar halo, you have at most ~500 particles in your halos, which is well resolved. The main problem with what you're doing is putting an upper limit on the mass of the halos that you are profiling, which is opposite to what one normally does. Even if you want to be cavalier with what you call a halo, you definitely don't want to go much below 2-5e12 since then you're talking 100 particles. I personally don't trust profiles unless they have a few thousand particles, because then I might believe the hydro is resolved. My guess is that if you do:

virial_filters=[['TotalMassMsun','>=','5e12']]

you'll have more luck with them actually being virialized.

Main point: If you want to study halos with masses less than 10^13, you're using the wrong simulation (i.e. too large of a physical box size or too small of a grid).

Sam

On Tue, Dec 8, 2009 at 3:04 PM, Agarwal, Shankar wrote: Sam,

I am trying to get the virial masses of the halos (with mass upto 1e +13 Msun) with this filter...

hp.add_halo_filter(HP.VirialFilter, must_be_virialized=True, overdensity_field='ActualOverdensity', virial_overdensity=200, virial_filters=[['TotalMassMsun','<=','1e13']], virial_quantities=['TotalMassMsun','RadiusMpc'])

But when I look at the radial_profiles/Halo_xxxx_profile.dat files, I see that none of the halos are virialized. The peak halo densities are reaching about 20*mean_matter_density. I began my 200Mpc/h Box with 512^3 particles simulation at z=99. I have not looked at the literature in great detail but is there something fishy here ?

shankar

-----Original Message----- From: yt-users-bounces@lists.spacepope.org on behalf of Sam Skillman Sent: Tue 12/8/2009 3:23 PM To: Discussion of the yt analysis package Subject: Re: [yt-users] r_min in HaloProfiler

Shankar,

r_min is defined in your email: r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc']

it's twice the smallest dx in units of mpc, dx being the smallest cell size. it is being rejected because your halo has something like 3 cells (radial) in it, which is almost certainly not resolved.

sam

On Tue, Dec 8, 2009 at 2:11 PM, Agarwal, Shankar wrote:

...

Hi,

While running HaloProfiler, I am seeing this...

P001 yt.lagos ERROR 2009-12-08 15:53:22,403 Skipping halo with r_max / r_min = 1.885682.

I looked at yt/extensions/HaloProfiler.py...

r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc'] if (halo['r_max'] / r_min < PROFILE_RADIUS_THRESHOLD): mylog.error("Skipping halo with r_max / r_min = %f." % (halo['r_max']/r_min))

r_max would be the distance to the furthest particle in the halo as found by HaloFinder. But what is r_min ? And what is the basis for skipping this halo ?

shankar _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Eric Hallman Google Voice: (312) 725-HMAN hallman13@gmail.com

Shankar, to your second point. The whole issue about resolution (mass resolution that is) is that unless the potential is resolved, you can't be sure if a) it is a real halo or just a chance aggregation of particles, and b) you certainly can't say you know it's mass accurately. In a statistical sense, you have some halo mass function down to 50 or so particles and that is ok, but for any individual halos, their properties are NOT accurately represented in the simulation at this level of mass resolution. On Dec 8, 2009, at 7:25 PM, Agarwal, Shankar wrote:

Eric and Sam,

Yeah. that makes sense. It boils down to the resolution of the box.

A question to Sam ... How did you get "mass resolution is about 2e10M_solar" ?

I am getting 5e9M_solar as the mass of each dark matter particle. And the number of particles in a 1e13M_solar halo is 2000.

ID mass particles density x y z centreofmass x centerofmass y centerofmass z 3700 9.853250284e+12 2000 1.136880784e+04 3.619545698e-01 4.593685567e-01 8.036487699e-01 3.618929449e-01 4.593253520e-01 8.041266519e-01

So I guess for 250 particles, I can try...

virial_filters=[['TotalMassMsun','>=','1.25e12']]

A question to Eric :

Actually, I don't care for the resolution of the halos. I am analyzing the suppression of matter power spectrum as a function of neutrino mass. Which means I am interested only on the semi-linear scales of 20-60 Mpc/h. Towards this, I wish to extract small galaxy groups (upto 1e13M_solar) from my simulation. So, can I regard the HaloFinder halos as blobs, w/o caring for the resolution ? If yes, can I include halofinder halos with as low as 50-100 particles (since they are still halos, regardless of the resolution) ?

I will appreciate if I can get some opinion on this.

shankar

-----Original Message----- From: yt-users-bounces@lists.spacepope.org on behalf of Eric Hallman Sent: Tue 12/8/2009 4:53 PM To: Discussion of the yt analysis package Subject: Re: [yt-users] r_min in HaloProfiler

Shankar, sam is right on the money with his analysis. Although the halo finders identify a large number of halos in your simulation they are very poorly resolved, and so this filter will rule them out on the basis of not getting to OD of 200. Our analysis with enzo shows that you can not expect the halos to be resolved under a few hundred particles as sam suggests.

I would take a look at how many halos are above 500 particles or so and only run your analysis on those. I suspect it is possible that one of the reasons you are not seeing speedup with the parallel runs is that you don't have enough work for the processors given the small number of virialized halos that you'll actually analyze with this filter.

On Dec 8, 2009, at 3:30 PM, Sam Skillman wrote:

...

Shankar,

In a box size of 200 Mpc/h^3 and 512^3 particles, your mass resolution is about 2e10M_solar. That means in a 10^13 M_solar halo, you have at most ~500 particles in your halos, which is well resolved. The main problem with what you're doing is putting an upper limit on the mass of the halos that you are profiling, which is opposite to what one normally does. Even if you want to be cavalier with what you call a halo, you definitely don't want to go much below 2-5e12 since then you're talking 100 particles. I personally don't trust profiles unless they have a few thousand particles, because then I might believe the hydro is resolved. My guess is that if you do:

virial_filters=[['TotalMassMsun','>=','5e12']]

you'll have more luck with them actually being virialized.

Main point: If you want to study halos with masses less than 10^13, you're using the wrong simulation (i.e. too large of a physical box size or too small of a grid).

Sam

On Tue, Dec 8, 2009 at 3:04 PM, Agarwal, Shankar wrote: Sam,

I am trying to get the virial masses of the halos (with mass upto 1e +13 Msun) with this filter...

hp.add_halo_filter(HP.VirialFilter, must_be_virialized=True, overdensity_field='ActualOverdensity', virial_overdensity=200, virial_filters=[['TotalMassMsun','<=','1e13']], virial_quantities=['TotalMassMsun','RadiusMpc'])

But when I look at the radial_profiles/Halo_xxxx_profile.dat files, I see that none of the halos are virialized. The peak halo densities are reaching about 20*mean_matter_density. I began my 200Mpc/h Box with 512^3 particles simulation at z=99. I have not looked at the literature in great detail but is there something fishy here ?

shankar

-----Original Message----- From: yt-users-bounces@lists.spacepope.org on behalf of Sam Skillman Sent: Tue 12/8/2009 3:23 PM To: Discussion of the yt analysis package Subject: Re: [yt-users] r_min in HaloProfiler

Shankar,

r_min is defined in your email: r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc']

it's twice the smallest dx in units of mpc, dx being the smallest cell size. it is being rejected because your halo has something like 3 cells (radial) in it, which is almost certainly not resolved.

sam

On Tue, Dec 8, 2009 at 2:11 PM, Agarwal, Shankar wrote:

...

Hi,

While running HaloProfiler, I am seeing this...

P001 yt.lagos ERROR 2009-12-08 15:53:22,403 Skipping halo with r_max / r_min = 1.885682.

I looked at yt/extensions/HaloProfiler.py...

r_min = 2 * self.pf.h.get_smallest_dx() * self.pf['mpc'] if (halo['r_max'] / r_min < PROFILE_RADIUS_THRESHOLD): mylog.error("Skipping halo with r_max / r_min = %f." % (halo['r_max']/r_min))

r_max would be the distance to the furthest particle in the halo as found by HaloFinder. But what is r_min ? And what is the basis for skipping this halo ?

shankar _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Eric Hallman Google Voice: (312) 725-HMAN hallman13@gmail.com

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Dr. Eric J. Hallman NSF Astronomy and Astrophysics Postdoctoral Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder hallman (at) casa.colorado.edu Phone: (312) 725-4626 http://solo.colorado.edu/~hallman/