Re: [yt-dev] [continued] RAMSES units bug in yt-3.0

It should be the tip of the yt-3.0 branch of the yt repo. I'll double-check that I haven't somehow messed up my install of yt or something. If you don't find a similar issue it's probably on my end... On 21/05/14 18:47, Matthew Turk wrote:

Hi Sam,

What's teh changeset hash? All of my tests have been against the "tip" of the yt-3.0 branch. Note that the repository yt_analysis/yt-3.0 is no longer in use.

On Wed, May 21, 2014 at 12:19 PM, Sam Geen wrote:

...

Hi again,

I've downloaded the latest version of yt-3.0, and am finding total mass values of 0.002 g for a simulation of a cloud of mass 10^5 Msun. I'm also finding that the temperature values are way too high (a max of 10^9K or so, when the max should be more like 10^5K). Finally, the pressure units are listed as code_mass/(code_length*code_time**2) rather than cgs (I have no idea if this represents a problem, but it could be related to the temperature bug).

Out of interest, is there a recommended version of yt-3.0 to use before the units were added to compare against? (I believe the old units system gave the correct values for RAMSES data).

Thanks,

Sam

On 14/05/14 16:38, Sam Geen wrote:

...

Hi Matt,

Sorry for the confusion - I've cross-checked against another dataset and the cell mass should be more like 2.5e8Msun, as you find; I thought the density values seemed very low (as well as the mass fraction of gas to dark matter) and hence added a factor of boxlen**3, which seemed more reasonable, but checking against another run it seems that the other run just has very low densities, and so the density unit doesn't need converting by boxlen**3 after all, so the only thing to be careful of is to make sure that cell-based spatial units are multiplied by boxlen while particle-based spatial units aren't; this way both the lengths and masses should be consistent. The particle mass should still be 1.8e10Msun.

Thanks for your work on this!

Sam

On 14/05/14 14:22, Matthew Turk wrote:

...

Hi all,

I've got Sam's dataset here. As a quick note, with current yt-3.0 tip, all results are correct for cosmology.

With the outstanding pull request, here are the results:

Cell Mass: 2.474e+08 Msun (5.300e+13 desired, 4.669e-06 ratio) Part Mass: 3.865e+15 Msun (1.800e+10 desired, 2.147e+05 ratio)

Without it:

Cell Mass: 1.146e+03 Msun (5.300e+13 desired, 2.161e-11 ratio) Part Mass: 1.789e+10 Msun (1.800e+10 desired, 9.941e-01 ratio)

So the particle mass is correct in current yt-3.0 tip, and the cell mass isn't right in either the tip or the pull request. Now, if I use yt-3.0 tip and compare cell_mass*boxlen**6 against the known, I get the right answer.

To solve this asymmetry between the particles and the cell density, I've added a "code_density" unit and updated the pull request. I now get correct results for non-cosmo and cosmo runs. This is in changeset 8847f04588b1:

https://bitbucket.org/yt_analysis/yt/pull-request/895/fixing-ramses-density-...

-Matt

On Tue, May 13, 2014 at 8:59 AM, Sam Geen wrote:

...

I'll try to hunt down and upload a sample dataset shortly.

On 13/05/2014 14:52, Matthew Turk wrote:

...

On Tue, May 13, 2014 at 8:48 AM, Sam Geen wrote: > Ah, very good, thanks for the link to that old PR! I was worried I was > going > crazy... in any case, whichever is the easiest way to make the > position/length units consistent. I change the particle positions in the ingestion phase, so at no point should they appear incorrect. I'd really like to figure out the boxlen != 1.0 ...

> By the way, I took a quick look at reading the grav_ files recently, > although the output seems kind of strange, so I haven't issued a PR > yet > until I understand what it's doing (plus I still don't 100% understand > the > units). I suspect it might be reading in the wrong order or something. > It > also complicates the code, so it might be worth refactoring a bit to > make > a > universal "AMR-like" data_structures/io interface for the files called > hydro_, grav_, rt_, etc, rather than what I did which was to scatter a > bunch > of conditionals into the existing functions. That could be -- I think there was some discussion that they might also include ghost zones?

I've also run some checks on cosmology runs. Here's what I've found. With no changes to the current tip of yt-3.0, I have run this script:

-- ds = yt.load("output_00010/info_00010.txt") dd = ds.all_data()

# Let's find the total mass pmass = dd["particle_mass"].sum().in_units("Msun") cmass = dd["cell_mass"].sum().in_units("Msun") rho = ds.cosmology.critical_density(ds.current_redshift) crit_mass = (rho * dd["cell_volume"]).sum().in_units("Msun") print pmass print cmass print pmass / (pmass + cmass) print ((pmass + cmass) / crit_mass) --

The results are good. I get roughly 0.84 for pmass / (pmass + cmass), and I get 0.998 for the final check, which seems within the parameters of the simulation. So the cosmological units should be correct, and more to the point, they match up with what I'm expecting having read over the code carefully.

So it looks to me like there is in fact a boxlen missing, but that as it stands, the particle and density units are correct for boxlen = 1.0. Nick's message is worrisome because the factor isn't exactly 1e3 between his max density. I have a boxlen != 1.0 dataset, but I have no reference values to compare it again. Any chance there's one with *reference* results for total mass in particles, total mass in gas, etc?

-Matt

> On 13/05/2014 14:21, Matthew Turk wrote: >> Hi Sam, >> >> So, here's an old pull request I found: >> >> >> >> >> http://hg.yt-project.org/yt-3.0/pull-request/62/adding-boxlen-to-ramses-unit... >> >> and then there's the units.f90 from RAMSES itself: >> >> >> >> >> https://bitbucket.org/rteyssie/ramses/src/81ab29b8a405e7ccf6bf30fd67582b4eda... >> >> I'm going to take care of this, update the PR, and ping you both. >> (And maybe we should move to yt-dev? :) >> >> -Matt >> >> On Tue, May 13, 2014 at 8:17 AM, Sam Geen >> wrote: >>> Huh, that difference in the max density is very strange. I can't >>> think >>> why >>> you'd have that difference; it's not an obvious multiple. >>> >>> Actually, I'm looking at the code and it's possible that the >>> particle >>> positions are scaled from 0 to boxlen and the grid positions from 0 >>> to >>> 1, >>> though I'd need to open up the raw values in a RAMSES file to >>> confirm >>> this. >>> If this is true, I suppose one "fix" could be to hard-multiply the >>> cell >>> sizes by pf["boxlen"] when you load them so that the length units >>> are >>> consistent for all data. I *think* that the older version of yt-3.0 >>> gave >>> correct answers, though, so that should at least be a way to >>> calibrate >>> the >>> new version. >>> >>> >>> On 13/05/2014 14:01, nick moeckel wrote: >>> >>> I had a chance to test out an older and a newer version of yt-3 on a >>> small >>> dataset that has boxlen=10. This seems to confirm a boxlen**3 factor >>> somewhere, although there's a further difference in the max density. >>> >>> more recent version (hg id -i gives f4838a2165c0): >>> >>> after dd = ds.all_data(): >>> >>> In [11]: dd.quantities.extrema('density') >>> >>> Out[11]: (2.00739832795e-28 g/cm**3, 4.54771338163e-24 g/cm**3) >>> >>> >>> older version (hg id -i gives 3e8b733c9ee9): >>> >>> after dd = ds.h.all_data(): >>> >>> In [10]: dd.quantities.extrema('Density') >>> >>> Out[10]: [(2.0073983279492632e-25, 4.0079147772594363e-21)] >>> >>> >>> >>> On Fri, May 9, 2014 at 5:50 PM, Sam Geen >>> wrote: >>>> Yep, that looks like it should work. I'll try to run it on some >>>> particle >>>> data when I get the time, but like I said I'm 99% sure the mass >>>> units >>>> should >>>> be identical for both grid hydro and particle data. >>>> >>>> >>>> On 09/05/14 17:46, Matthew Turk wrote: >>>>> Hi Sam, >>>>> >>>>> Okay, I've looked over a bit, and I think the correct change would >>>>> be: >>>>> >>>>> mass_unit = rho_u * length_unit**3. >>>>> >>>>> That should include the correct length unit, and I think will >>>>> reduce >>>>> "density" back to the "unit_d" that's in the parameter file. If >>>>> this >>>>> looks okay to you, I will push it, but I really do want to make >>>>> sure >>>>> the particle masses are correct. Can Nick or Romain provide a bit >>>>> of >>>>> guidance here? >>>>> >>>>> -Matt >>>>> >>>>> On Fri, May 9, 2014 at 11:25 AM, Sam Geen >>>>> wrote: >>>>>> OK, interesting. In theory RAMSES should have identical units for >>>>>> both >>>>>> particles and gas. I can hunt down a run with particles to test >>>>>> if >>>>>> you >>>>>> like. >>>>>> >>>>>> Thanks! >>>>>> Sam >>>>>> >>>>>> >>>>>> On 09/05/14 17:22, Matthew Turk wrote: >>>>>>> Hi Sam, >>>>>>> >>>>>>> On reflection, I think this might be related to getting the >>>>>>> *particle* >>>>>>> masses correct. I will take a look at it as soon as I can. >>>>>>> >>>>>>> -Matt >>>>>>> >>>>>>> On Fri, May 9, 2014 at 11:00 AM, Sam Geen >>>>>>> >>>>>>> wrote: >>>>>>>> Hmm, that looks like it should be "mass_unit = rho_u * >>>>>>>> length_unit**3" >>>>>>>> in >>>>>>>> line 492. You're right that it mentions the boxlength issue, >>>>>>>> though. >>>>>>>> >>>>>>>> >>>>>>>> On 09/05/14 16:50, Matthew Turk wrote: >>>>>>>>> Hi Sam, >>>>>>>>> >>>>>>>>> Okay, sounds good. Looking at how code unit attributes are >>>>>>>>> set >>>>>>>>> up: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://bitbucket.org/yt_analysis/yt/src/a14a150c7c81850df81346162bdaff271e7... >>>>>>>>> >>>>>>>>> suggests to me that length_unit takes into account boxlen, and >>>>>>>>> mass_unit does not. The comments have some indication why >>>>>>>>> this >>>>>>>>> might >>>>>>>>> be. >>>>>>>>> >>>>>>>>> -Matt >>>>>>>>> >>>>>>>>> On Fri, May 9, 2014 at 10:46 AM, Sam Geen >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>> Yep; both this and dd["density"] give cgs values that are too >>>>>>>>>> small >>>>>>>>>> by >>>>>>>>>> roughly a factor of boxlen**3. >>>>>>>>>> >>>>>>>>>> One other thing I need to try is to make sure I'm using the >>>>>>>>>> very >>>>>>>>>> latest >>>>>>>>>> version of YT; I've been playing around with the Ramses >>>>>>>>>> frontend >>>>>>>>>> so >>>>>>>>>> it's >>>>>>>>>> possible my version is somehow out of sync. Will let you know >>>>>>>>>> if >>>>>>>>>> this >>>>>>>>>> fixes >>>>>>>>>> things. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 09/05/14 16:42, Matthew Turk wrote: >>>>>>>>>>> Hi Sam, >>>>>>>>>>> >>>>>>>>>>> Can you verify the units are in fact incorrect in *cgs*? >>>>>>>>>>> Something >>>>>>>>>>> like this would work: >>>>>>>>>>> >>>>>>>>>>> ds = load(...) >>>>>>>>>>> dd = ds.all_data() >>>>>>>>>>> print dd.quantities.total_quantitiy("cell_mass").in_cgs() >>>>>>>>>>> >>>>>>>>>>> Your second message makes me wonder if there's just a slipup >>>>>>>>>>> in >>>>>>>>>>> how >>>>>>>>>>> the units are returned. >>>>>>>>>>> >>>>>>>>>>> -Matt >>>>>>>>>>> >>>>>>>>>>> On Fri, May 9, 2014 at 10:37 AM, Sam Geen >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> Sorry for the spam; a second bug I've seen is that the >>>>>>>>>>>> density >>>>>>>>>>>> and >>>>>>>>>>>> pressure >>>>>>>>>>>> unit labels on figures appears to be broken; it seems to >>>>>>>>>>>> print >>>>>>>>>>>> a >>>>>>>>>>>> latex-mangled code names for the units rather than the cgs >>>>>>>>>>>> name >>>>>>>>>>>> of >>>>>>>>>>>> the >>>>>>>>>>>> units; see attached example. Temperature is fine. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 09/05/14 16:28, Sam Geen wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Just to say that in the latest version of yt-3.0 (i.e. >>>>>>>>>>>>> since >>>>>>>>>>>>> various >>>>>>>>>>>>> fields were renamed or re-implemented), I've found a bug >>>>>>>>>>>>> in >>>>>>>>>>>>> the >>>>>>>>>>>>> implementation of "cell_mass", which is giving results >>>>>>>>>>>>> that >>>>>>>>>>>>> are >>>>>>>>>>>>> too >>>>>>>>>>>>> low >>>>>>>>>>>>> in >>>>>>>>>>>>> my runs; I believe the issue is that it's missing a factor >>>>>>>>>>>>> of >>>>>>>>>>>>> pf["boxlen"]**3 (which is of course only a problem if >>>>>>>>>>>>> boxlen >>>>>>>>>>>>> is >>>>>>>>>>>>> not >>>>>>>>>>>>> 1). >>>>>>>>>>>>> The >>>>>>>>>>>>> previous "CellMass_Msun" worked fine. If I get time I >>>>>>>>>>>>> might >>>>>>>>>>>>> take >>>>>>>>>>>>> a >>>>>>>>>>>>> look >>>>>>>>>>>>> and >>>>>>>>>>>>> issue a pull request, but otherwise I'm just flagging this >>>>>>>>>>>>> in >>>>>>>>>>>>> case >>>>>>>>>>>>> someone >>>>>>>>>>>>> else runs into problems; you can just manually multiply >>>>>>>>>>>>> the >>>>>>>>>>>>> result >>>>>>>>>>>>> by >>>>>>>>>>>>> pf["boxlen"]**3 until it's fixed. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> >>>>>>>>>>>>> Sam >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> yt-users mailing list >>>>>>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> yt-users mailing list >>>>>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>>>>> >>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> yt-users mailing list >>>>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>>>> >>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> yt-users mailing list >>>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>> _______________________________________________ >>>>>>>>> yt-users mailing list >>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-users mailing list >>>>>>>> yt-users@lists.spacepope.org >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>> _______________________________________________ >>>>>>> yt-users mailing list >>>>>>> yt-users@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>> >>>>>> _______________________________________________ >>>>>> yt-users mailing list >>>>>> yt-users@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> _______________________________________________ >>>>> yt-users mailing list >>>>> yt-users@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

---

yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

---

yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org