"Overdensity" in yt 2.x and 3 and the Enzo Dark_Matter_Density field
For the past couple of days I've been trying to figure out halo profiles and virial radius calculations in yt, which so far has mostly consisted of trying to understand the differences in the "Overdensity" field between yt versions. The biggest difference is due to yt-2.x using an incorrect redshift scaling for the overdensity calculation. The field is defined as Matter_Density / (rho_crit_now * data.pf.hubble_constant**2 * (1 + data.pf.current_redshift)**3) which scales the critical density by (1+z)^3 instead of only the matter component. This error is fixed in yt 3, but I'm not sure if it's corrected for or known by people still using legacy versions of yt. After accounting for this, there is still a ~1.5% difference in the mean overdensity in my simulation between yt 2.x and 3 (the yt 3 value is lower). This difference comes from the particle_density component of the matter_density. The yt 2.x version uses the particle masses from Enzo and yt's CICDeposit_3. The yt 3 version uses the Dark_Matter_Density field from Enzo, which uses Enzo's build in cic method. As far as I understand, differences in cic methods could lead to different values over subdomains of a simulation, but shouldn't lead to different total masses over the entire simulation box. I went through each grid of my simulation individually, and in every one of them the sum of the Dark_Matter_Density values is less than the sum of the particle_mass values. Differences range from 0.4% to 3.5% across individual grids. My guess is that this means the Dark_Matter_Density field is losing mass that is deposited in the ghost zones outside of the grid boundaries. I'm not familiar enough with the internal workings of Enzo to know if this has any impact on the actual simulations, or if it is only an issue with the data outputs. Either way, it might be a good idea to have yt return to calculating the particle_density field from particle masses and do the cic deposition in yt, at least for Enzo data. - Josh _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Thanks for bringing this to our attention and being very thorough in your
analysis.
A couple things:
You may want to make a pull request for the 2.x fix to the overdensity
field. While I don't think we will be doing releases in the 2.x series, the
2.x branch is still open and if people install from the install script,
they will get your bugfix.
It might be worthwhile to raise this on the enzo-dev mailing list, since
there will be more expertise on Enzo internals over there.
I don't think yt's CIC deposited particle properly accounts for grid
boundaries. I opened an issue about it here:
https://bitbucket.org/yt_analysis/yt/issues?q=cloud+in+cell
Andrew Myers looked at this a while ago and didn't come to a satisfactory
conclusion. I believe the most accurate deposited particle density would
use nearest neighbor interpolation.
Is there any chance you can make a script (maybe two scripts - one for 2.x,
one for 3.x) that demonstrates this issue? That will make the issue a bit
more concrete.
I don't do cosmology simulations so I'm not sure which is necessarily the
correct thing to do.
-Nathan
On Tue, Jan 12, 2016 at 5:41 PM, Josh Moloney
For the past couple of days I've been trying to figure out halo profiles and virial radius calculations in yt, which so far has mostly consisted of trying to understand the differences in the "Overdensity" field between yt versions.
The biggest difference is due to yt-2.x using an incorrect redshift scaling for the overdensity calculation. The field is defined as
Matter_Density / (rho_crit_now * data.pf.hubble_constant**2 * (1 + data.pf.current_redshift)**3)
which scales the critical density by (1+z)^3 instead of only the matter component.
This error is fixed in yt 3, but I'm not sure if it's corrected for or known by people still using legacy versions of yt.
After accounting for this, there is still a ~1.5% difference in the mean overdensity in my simulation between yt 2.x and 3 (the yt 3 value is lower). This difference comes from the particle_density component of the matter_density. The yt 2.x version uses the particle masses from Enzo and yt's CICDeposit_3. The yt 3 version uses the Dark_Matter_Density field from Enzo, which uses Enzo's build in cic method. As far as I understand, differences in cic methods could lead to different values over subdomains of a simulation, but shouldn't lead to different total masses over the entire simulation box.
I went through each grid of my simulation individually, and in every one of them the sum of the Dark_Matter_Density values is less than the sum of the particle_mass values. Differences range from 0.4% to 3.5% across individual grids. My guess is that this means the Dark_Matter_Density field is losing mass that is deposited in the ghost zones outside of the grid boundaries.
I'm not familiar enough with the internal workings of Enzo to know if this has any impact on the actual simulations, or if it is only an issue with the data outputs. Either way, it might be a good idea to have yt return to calculating the particle_density field from particle masses and do the cic deposition in yt, at least for Enzo data. - Josh
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Thanks Josh — I think you are correct that the Enzo Dark_Matter_Density field does not include the boundary contribution to the CIC deposition (this is properly accounted for in the simulation itself, but not in the output field). Cheers, Greg
On Jan 12, 2016, at 6:41 PM, Josh Moloney
wrote: For the past couple of days I've been trying to figure out halo profiles and virial radius calculations in yt, which so far has mostly consisted of trying to understand the differences in the "Overdensity" field between yt versions.
The biggest difference is due to yt-2.x using an incorrect redshift scaling for the overdensity calculation. The field is defined as
Matter_Density / (rho_crit_now * data.pf.hubble_constant**2 * (1 + data.pf.current_redshift)**3)
which scales the critical density by (1+z)^3 instead of only the matter component.
This error is fixed in yt 3, but I'm not sure if it's corrected for or known by people still using legacy versions of yt.
After accounting for this, there is still a ~1.5% difference in the mean overdensity in my simulation between yt 2.x and 3 (the yt 3 value is lower). This difference comes from the particle_density component of the matter_density. The yt 2.x version uses the particle masses from Enzo and yt's CICDeposit_3. The yt 3 version uses the Dark_Matter_Density field from Enzo, which uses Enzo's build in cic method. As far as I understand, differences in cic methods could lead to different values over subdomains of a simulation, but shouldn't lead to different total masses over the entire simulation box.
I went through each grid of my simulation individually, and in every one of them the sum of the Dark_Matter_Density values is less than the sum of the particle_mass values. Differences range from 0.4% to 3.5% across individual grids. My guess is that this means the Dark_Matter_Density field is losing mass that is deposited in the ghost zones outside of the grid boundaries.
I'm not familiar enough with the internal workings of Enzo to know if this has any impact on the actual simulations, or if it is only an issue with the data outputs. Either way, it might be a good idea to have yt return to calculating the particle_density field from particle masses and do the cic deposition in yt, at least for Enzo data. - Josh _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I've attached a simple script that illustrates the differences between the
various particle_density fields in yt-3. It calculates the total particle
mass in the simulation for each field used. It should work for any Enzo
simulation, including the test outputs packaged with yt (I've currently set
it up to be run from the DD0010 test directory). The particle_density field
in yt-2.x gives results equivalent to the "all_cic" field (at least to
within a part in 1000) so I haven't included a second legacy script.
When I run this script on the DD0010 test simulation, the nearest grid cell
deposition field "all_density" matches the directly summed particle masses.
The Enzo supplied "dark_matter_density" field produces the largest
discrepancy, while the yt "all_cic" field is intermediate between the two.
For the unigrid (no AMR) simulation I was originally using, the "all_cic"
and "all_density" fields both match the summed particle masses, with the
"dark_matter_density" giving a smaller mass by ~1.2%.
I think this indicates that YT's CIC deposited particles have issues at the
boundaries of refined regions (the issue that Nathan found), while Enzo's
CIC field is losing mass at all grid boundaries. For my purposes, the YT
CIC field is accurate, but this does make it less clear what the
appropriate fix is. Replacing the "dark_matter_mass" field with YT's CIC
calculation is an improvement, but leaves residual inaccuracies in AMR
simulations. Switching the calculations to use nearest grid cell deposition
fixes mass conservation, but changes the smoothness of the density field in
a way that probably shouldn't be done behind the scenes.
The best solution might be to overwrite the "dark_matter_density" field
with "all_cic" (or "io_cic", I don't understand the difference) for Enzo
simulations only. This would improve the accuracy of the field for Enzo
without negatively impacting other frontends that may do their density
calculations in a way that is superior to yt's CIC version.
- Josh
On Wed, Jan 13, 2016 at 6:28 AM, Greg Bryan
Thanks Josh — I think you are correct that the Enzo Dark_Matter_Density field does not include the boundary contribution to the CIC deposition (this is properly accounted for in the simulation itself, but not in the output field).
Cheers, Greg
On Jan 12, 2016, at 6:41 PM, Josh Moloney
wrote: For the past couple of days I've been trying to figure out halo profiles and virial radius calculations in yt, which so far has mostly consisted of trying to understand the differences in the "Overdensity" field between yt versions.
The biggest difference is due to yt-2.x using an incorrect redshift scaling for the overdensity calculation. The field is defined as
Matter_Density / (rho_crit_now * data.pf.hubble_constant**2 * (1 + data.pf.current_redshift)**3)
which scales the critical density by (1+z)^3 instead of only the matter component.
This error is fixed in yt 3, but I'm not sure if it's corrected for or known by people still using legacy versions of yt.
After accounting for this, there is still a ~1.5% difference in the mean overdensity in my simulation between yt 2.x and 3 (the yt 3 value is lower). This difference comes from the particle_density component of the matter_density. The yt 2.x version uses the particle masses from Enzo and yt's CICDeposit_3. The yt 3 version uses the Dark_Matter_Density field from Enzo, which uses Enzo's build in cic method. As far as I understand, differences in cic methods could lead to different values over subdomains of a simulation, but shouldn't lead to different total masses over the entire simulation box.
I went through each grid of my simulation individually, and in every one of them the sum of the Dark_Matter_Density values is less than the sum of the particle_mass values. Differences range from 0.4% to 3.5% across individual grids. My guess is that this means the Dark_Matter_Density field is losing mass that is deposited in the ghost zones outside of the grid boundaries.
I'm not familiar enough with the internal workings of Enzo to know if this has any impact on the actual simulations, or if it is only an issue with the data outputs. Either way, it might be a good idea to have yt return to calculating the particle_density field from particle masses and do the cic deposition in yt, at least for Enzo data. - Josh _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (3)
-
Greg Bryan
-
Josh Moloney
-
Nathan Goldbaum