Greetings!
I am extremely pleased to announce the release of ytree version 3.0!
ytree is an extension of yt for analyzing merger trees.
This is a major release that sees significant speed and memory use
improvements, support for 5 new data formats, and several new
features. New features and major enhancements are listed below. For a
complete list of all changes, see the changelog
<https://ytree.readthedocs.io/en/latest/changelog.html#version-3-0>.
A special thanks to Manodeep Sinha for inspiring the development of
two of the new frontends and graciously providing feedback and to
Benedikt Diemer for feedback on the development of the Moria
frontend. As always, ytree would be nothing without the wonderful yt
community. Thanks to everyone there.
Have a great day.
Britton
*New Featues*
* Halo selection and generation with yt data objects
* New functions: get_node, get_root_nodes, get_leaf_nodes,
add_vector_field
* Plotting and plot customization
*New Data Formats*
* consistent-trees hlist and locations.dat files
* consistent-trees-HDF5
* LHaloTree_hdf5 (IllustrisTNG)
* Moria
* TreeFrog
*Enhancements*
* All functions returning TreeNodes now return generators for a
significant speed and memory usage improvement.
* Speed and usability improvements to select_halos function
* Speed and usability updates to save_arbor function
* Add parallel analysis documentation
ytree can be installed with pip:
pip install ytree
Full documentation can be found at https://ytree.readthedocs.io/
Hello yt users,
I am having the exact same issues as was reported 1.5 years ago (see
https://github.com/ytree-project/ytree/issues/44) in that ytree does not
import because of yt.extern.six. I see this was resolved back then, but I
am just now experiencing this issue. This happened after I installed yt
4.0.1. Here is my output:
>>>import ytree
/users/carla/.local/lib/python3.8/site-packages/yt/utilities/logger.py:4:
VisibleDeprecationWarning: The configuration file
/users/carla/.config/yt/ytrc is deprecated in favor of
/users/carla/.config/yt/yt.toml. Currently, both are present. Please
manually remove the deprecated one to silence this warning.
Deprecated since v4.0.0. This feature will be removed in v4.1.0
from yt.config import ytcfg
>>> import ytree
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/opt/apps/pkgs/cpython/3.8.9/intel64/intel_2019.2/lib/python3.8/site-packages/ytree/__init__.py",
line 16, in <module>
from ytree.data_structures import \
File
"/opt/apps/pkgs/cpython/3.8.9/intel64/intel_2019.2/lib/python3.8/site-packages/ytree/data_structures/__init__.py",
line 16, in <module>
from ytree.data_structures.arbor import \
File
"/opt/apps/pkgs/cpython/3.8.9/intel64/intel_2019.2/lib/python3.8/site-packages/ytree/data_structures/arbor.py",
line 23, in <module>
from yt.extern.six import \
ModuleNotFoundError: No module named 'yt.extern'
Please let me know how I can get around this.
Thanks in advance,
--
Carla Bernhardt
PhD Student
Universität Heidelberg
ZAH Institut für Theoretische Astrophysik
Dear yt users,
I just updated yt to 4.0.1 and now I get this error message when trying to
run enzo-mrp-music.py from John Wise on
https://github.com/jwise77/enzo-mrp-music.
Traceback (most recent call last):
File "enzo-mrp-music/enzo-mrp-music.py", line 264, in <module>
params = find_lagrangian_region(params)
File "enzo-mrp-music/enzo-mrp-music.py", line 177, in
find_lagrangian_region
get_center_and_extent(params["halo_info"],
File "enzo-mrp-music/get_halo_initial_extent.py", line 147, in
get_center_and_extent
halo_indices, halo_com, shifted = get_halo_indices(my_halo,
final_dataset,
File "enzo-mrp-music/get_halo_initial_extent.py", line 119, in
get_halo_indices
get_halo_sphere_particles(my_halo, dataset, radius_factor=radius_factor)
File "enzo-mrp-music/get_halo_initial_extent.py", line 73, in
get_halo_sphere_particles
my_sphere = pf.h.sphere(my_halo_data['center'], radius_factor * r_200)
AttributeError: 'EnzoDataset' object has no attribute 'h'
I am honestly not sure what the attribute 'h' does in this case, so I
didn't know what to try to adjust. Any suggestions for getting around this?
Or how can I go back to yt 3?
Kind regards,
--
Carla Bernhardt
PhD Student
Universität Heidelberg
ZAH Institut für Theoretische Astrophysik
Hello yters!
yt 4.0.1 is now out!
We've included ~2.5 backports in this release so that users having issues
with our original 4.0 release can be supported. See PRs #3413
<http://Updating EWAH from upstream lemire/EWAHBoolArray #3413> and c821824
<https://github.com/yt-project/yt/pull/3447/commits/c821824860595b4a969d9b89…>
for
EWAH support to fix windows builds and #3445
<https://github.com/yt-project/yt/pull/3445> for WarpX support.
Binaries for yt 4.0.1 are available via pip and conda. If you installed via
the install script or use conda to manage your python installation, you can
update yt via:
$ conda update -c conda-forge yt
And via pip if you manage your python installation with pip:
$ pip install -U yt
As always, if you have any questions, concerns, or run into any trouble
updating please don’t hesitate to send a message to the mailing list or
stop by our Slack workspace.
Thanks for being in the community!
- the yt development team
** please note that the conda builds may be available slightly after this
message. If you have trouble updating via conda, please wait a few hours
and try again.
Hi dear all,
I want to plot a halo mass function, using a simulation's halo catalogue,
with differential number density as vertical axis, i.e. "dn/d(logm)".
But It seems that the only option is cumulative number density.
Is there any way to do that?
Regards,
Hamed
Dear Sir/Ma'am,
To introduce myself first, I am going to start my PhD at Max Planck
Institute of Astrophysics in the group of Dr. Benedetta Ciardi in few
months. For my ongoing project, I need to use yt package for which I
have installed yt 4.0.0 via pip pipeline. But, I am facing a difficulty
in processing the file successfully loaded and read by "yt.load". I am
sharing 2-3 lines of my code and the errors I get.
CODE:
import yt
import numpy as np
import h5py
ad=ds.all_data() ##[ERROR 1 here]
fields = ['density']
le = ds.domain_left_edge
re = ds.domain_right_edge
ag = ds.arbitrary_grid(le,re,[256]*3)
for f in fields:
print(f)
data[f] = ag[('gas', f)]
bbox = np.array(list(zip(le.in_units('Mpccm/h'),
re.in_units('Mpccm/h'))))
ds_gridded = yt.load_uniform_grid(data, [256]*3, length_unit =
ds.length_unit.to('Mpccm/h'), bbox=bbox)
sp = yt.SlicePlot(ds_gridded,'z',('gas','density'))
sp.save() ##[ERROR 2 here]
ERROR:1:
RuntimeError Traceback (most recent call
last)
<ipython-input-4-007efe033704> in <module>
----> 1 ad=ds.all_data()
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
all_data(self, find_max, **kwargs)
1039 which covers the entire simulation domain.
1040 """
-> 1041 self.index
1042 if find_max:
1043 c = self.find_max("density")[1]
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
index(self)
526 if self._index_class is None:
527 raise RuntimeError("You should not instantiate
Dataset.")
--> 528 self._instantiated_index = self._index_class(
529 self, dataset_type=self.dataset_type
530 )
~/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py
in __init__(self, ds, dataset_type)
22 self.dataset = weakref.proxy(ds)
23 self.float_type = np.float64
---> 24 super().__init__(ds, dataset_type)
25 self._initialize_index()
26
~/.local/lib/python3.8/site-packages/yt/geometry/geometry_handler.py in
__init__(self, ds, dataset_type)
42 # potentially quite expensive, and should be done with
the indexing.
43 mylog.debug("Detecting fields.")
---> 44 self._detect_output_fields()
45
46 @abc.abstractmethod
~/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py
in _detect_output_fields(self)
296 fl, _units = field_cache[dom.filename]
297 else:
--> 298 fl, _units = self.io._identify_fields(dom)
299 field_cache[dom.filename] = fl, _units
300 units.update(_units)
~/.local/lib/python3.8/site-packages/yt/frontends/gadget/ioKeyError
Traceback (most recent call last)
<ipython-input-6-97fe8b4b4ff8> in <module>
1 for f in fields:
2 print(f)
----> 3 data[f] = ag[('gas', f)]
4 bbox = np.array(list(zip(le.in_units('Mpccm/h'),
re.in_units('Mpccm/h'))))
5 ds_gridded = yt.load_uniform_grid(data, [256]*3, length_unit =
ds.length_unit.to('Mpccm/h'), bbox=bbox)
~/.local/lib/python3.8/site-packages/yt/data_objects/data_containers.py
in __getitem__(self, key)
250 Returns a single field. Will add if necessary.
251 """
--> 252 f = self._determine_fields([key])[0]
253 if f not in self.field_data and key not in
self.field_data:
254 if f in self._container_fields:
~/.local/lib/python3.8/site-packages/yt/data_objects/data_containers.py
in _determine_fields(self, fields)
1450 ftype, fname = self._tupleize_field(field)
1451 # print(field, " : ",ftype, fname)
-> 1452 finfo = self.ds._get_field_info(ftype, fname)
1453
1454 # really ugly check to ensure that this field really
does exist somewhere,
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
_get_field_info(self, ftype, fname)
817
818 def _get_field_info(self, ftype, fname=None):
--> 819 field_info, is_ambiguous =
self._get_field_info_helper(ftype, fname)
820
821 if is_ambiguous:
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
_get_field_info_helper(self, ftype, fname)
833
834 def _get_field_info_helper(self, ftype, fname=None):
--> 835 self.index
836
837 # store the original inputs in case we need to raise an
error
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
index(self)
526 if self._index_class is None:
527 raise RuntimeError("You should not instantiate
Dataset.")
--> 528 self._instantiated_index = self._index_class(
529 self, dataset_type=self.dataset_type
530 )
~/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py
in __init__(self, ds, dataset_type)
22 self.dataset = weakref.proxy(ds)
23 self.float_type = np.float64
---> 24 super().__init__(ds, dataset_type)
25 self._initialize_index()
26
~/.local/lib/python3.8/site-packages/yt/geometry/geometry_handler.py in
__init__(self, ds, dataset_type)
42 # potentially quite expensive, and should be done with
the indexing.
43 mylog.debug("Detecting fields.")
---> 44 self._detect_output_fields()
45
46 @abc.abstractmethod
~/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py
in _detect_output_fields(self)
296 fl, _units = field_cache[dom.filename]
297 else:
--> 298 fl, _units = self.io._identify_fields(dom)
299 field_cache[dom.filename] = fl, _units
300 units.update(_units)
~/.local/lib/python3.8/site-packages/yt/frontends/gadget/io.py in
_identify_fields(self, data_file)
290 else:
291 kk = k
--> 292 if not hasattr(g[kk], "shape"):
293 continue
294 if len(g[kk].shape) > 1:
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
~/.local/lib/python3.8/site-packages/h5py/_hl/group.py in
__getitem__(self, name)
286 raise ValueError("Invalid HDF5 object
reference")
287 else:
--> 288 oid = h5o.open(self.id, self._e(name),
lapl=self._lapl)
289
290 otype = h5i.get_type(oid)
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/h5o.pyx in h5py.h5o.open()
KeyError: 'Unable to open object (bad object header version number)'.py
in _identify_fields(self, data_file)
261 # --------------------------------------
262 g = f[key]
--> 263 if cname not in g:
264 continue
265
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
~/.local/lib/python3.8/site-packages/h5py/_hl/group.py in
__contains__(self, name)
437 def __contains__(self, name):
438 """ Test if a member name exists """
--> 439 return self._e(name) in self.id
440
441 def copy(self, source, dest, name=None,
h5py/h5g.pyx in h5py.h5g.GroupID.__contains__()
h5py/h5g.pyx in h5py.h5g.GroupID.__contains__()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/h5g.pyx in h5py.h5g._path_valid()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/h5l.pyx in h5py.h5l.LinkProxy.exists()
RuntimeError: Unable to get link info (free block size is zero?)
ERROR:2:
KeyError Traceback (most recent call
last)
<ipython-input-6-97fe8b4b4ff8> in <module>
1 for f in fields:
2 print(f)
----> 3 data[f] = ag[('gas', f)]
4 bbox = np.array(list(zip(le.in_units('Mpccm/h'),
re.in_units('Mpccm/h'))))
5 ds_gridded = yt.load_uniform_grid(data, [256]*3, length_unit =
ds.length_unit.to('Mpccm/h'), bbox=bbox)
~/.local/lib/python3.8/site-packages/yt/data_objects/data_containers.py
in __getitem__(self, key)
250 Returns a single field. Will add if necessary.
251 """
--> 252 f = self._determine_fields([key])[0]
253 if f not in self.field_data and key not in
self.field_data:
254 if f in self._container_fields:
~/.local/lib/python3.8/site-packages/yt/data_objects/data_containers.py
in _determine_fields(self, fields)
1450 ftype, fname = self._tupleize_field(field)
1451 # print(field, " : ",ftype, fname)
-> 1452 finfo = self.ds._get_field_info(ftype, fname)
1453
1454 # really ugly check to ensure that this field really
does exist somewhere,
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
_get_field_info(self, ftype, fname)
817
818 def _get_field_info(self, ftype, fname=None):
--> 819 field_info, is_ambiguous =
self._get_field_info_helper(ftype, fname)
820
821 if is_ambiguous:
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
_get_field_info_helper(self, ftype, fname)
833
834 def _get_field_info_helper(self, ftype, fname=None):
--> 835 self.index
836
837 # store the original inputs in case we need to raise an
error
~/.local/lib/python3.8/site-packages/yt/data_objects/static_output.py in
index(self)
526 if self._index_class is None:
527 raise RuntimeError("You should not instantiate
Dataset.")
--> 528 self._instantiated_index = self._index_class(
529 self, dataset_type=self.dataset_type
530 )
~/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py
in __init__(self, ds, dataset_type)
22 self.dataset = weakref.proxy(ds)
23 self.float_type = np.float64
---> 24 super().__init__(ds, dataset_type)
25 self._initialize_index()
26
~/.local/lib/python3.8/site-packages/yt/geometry/geometry_handler.py in
__init__(self, ds, dataset_type)
42 # potentially quite expensive, and should be done with
the indexing.
43 mylog.debug("Detecting fields.")
---> 44 self._detect_output_fields()
45
46 @abc.abstractmethod
~/.local/lib/python3.8/site-packages/yt/geometry/particle_geometry_handler.py
in _detect_output_fields(self)
296 fl, _units = field_cache[dom.filename]
297 else:
--> 298 fl, _units = self.io._identify_fields(dom)
299 field_cache[dom.filename] = fl, _units
300 units.update(_units)
~/.local/lib/python3.8/site-packages/yt/frontends/gadget/io.py in
_identify_fields(self, data_file)
290 else:
291 kk = k
--> 292 if not hasattr(g[kk], "shape"):
293 continue
294 if len(g[kk].shape) > 1:
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
~/.local/lib/python3.8/site-packages/h5py/_hl/group.py in
__getitem__(self, name)
286 raise ValueError("Invalid HDF5 object
reference")
287 else:
--> 288 oid = h5o.open(self.id, self._e(name),
lapl=self._lapl)
289
290 otype = h5i.get_type(oid)
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/h5o.pyx in h5py.h5o.open()
KeyError: 'Unable to open object (bad object header version number)'
What I feel is that there is a mismatch in the h5py version. What is
that exact problem here? I have h5py 3.0.0, matplotlib 3.4.1,python
3.8,yt 4.0.0 in my machine. Please make me understand and suggest me a
reasonable solution so that I can proceed smoothly to my next step. The
length of this email seems large although most of it contains some
outcomes of the codes.
Waiting for your early reply.
Regards,
Arghyadeep Basu
Hi Nick,
Right, I see now. Setting the particle mass manually may be the best way to
fix this. I believe most frontends store particle masses and yt just reads
them in and hands them to rockstar, so it may be tricky to do this
generally. I can also imagine scenarios where one might not want to
renormalize the particle mass to include baryons, for example with
baryon-poor mini-halos at high redshift. We could potentially allow the
user to pass a custom particle mass field with altered masses to rockstar.
For the box size issue, just let us know when you have a fix. You can
either issue a pull request or just post it here and I'd be happy to apply
it.
Britton
On Tue, Jul 13, 2021 at 2:33 PM Nick Gnedin <gnedin(a)fnal.gov> wrote:
>
> Britton,
>
> Thanks for the answer.
>
> > I would have said that the particle mass calculation looks correct as
> > is. If the simulation has gas, then the particle mass should be
> > calculated with omega_dm, right?
>
> No, RS only uses DM particles, so without correcting the particle mass
> all halos end up 15% less massive.
>
> > For a loaded snapshot or timeseries object, ds, you can try doing:
> > mass = ds.quan(5.02229e+06, 'Msun/h')
> > and passing that with the particle_mass argument. That said, it would be
> > useful to allow the user to specify a float and assume units of Msun/h.
> > We can do that.
>
> Thank you for the fix, that works.
>
> > It's not clear to me whether this is happening in the yt frontend or in
> > the halo finder. If you load any of the snapshots and do:
> > print (ds.domain_width.to
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__ds.domain-5Fwidth.to&d=…
> >('Mpc/h'))
> > do you get the correct answer at every snapshot? That will help narrow
> > down what needs fixing.
>
> I planned to look into that and had it fixed, but did not get a chance
> yet. I believe all that needs to be fixed is for Yt to use abox for the
> scale factor, not auni (which is just a label for convenience and in the
> majority of cases has no physical meaning).
>
>
>
>
>
>
> --
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
>
I am trying to change flux units to Jansky, but I run into an error. Mor specifically I am starting from a FLASH hdf5 output, computing some flux and trying to have it plotted in Jansky instead of SI units. Here are the relevant code lines:
> import yt
>
>...
> LFLY = YTArray(1.0/cosmo.hubble_constant.value,"Mpc").in_cgs()
> TFLY = (2./3./cosmo.hubble_constant).in_cgs()
> MFLY = YTArray(5.23e12/cosmo.hubble_constant.value,"Msun").in_cgs()
> uo = {"length_unit":LFLY, "time_unit":TFLY, "mass_unit":MFLY}
> ....
> yt.add_field(("gas", "i_nu"), function=_i_nu, units="W/(m**2)/Hz", sampling_type='cell', display_name="Synchr. emission", force_override=True, validators=[ValidateParameter(['lmin', 'l.o.s.', 'lmin origin', 'Background I_nu', 'Electron p-law index', 'b(p)', 'E_min', 'E_max', 'Magnetic field','nu (Ghz)'])])
> ....
> ds=yt.load(<hdf5 output>,units_override=uo)
> ds.define_unit("Jansky", (1.e-26, "W/(m**2)/Hz"))
> ....
> rg=ds.region([0.,0.,0.],[-0.05,-0.05,-0.1],[0.05,0.05,0.1])
> ....
> plt_synchr = yt.OffAxisSlicePlot(ds,n_dir,'i_nu', width=((40,'kpc'),(100,'kpc')),north_vector=north, data_source=rg)
> ...
> plt_synchr.set_unit('i_nu','Jansky')
Here I get the error:
> yt.units.unit_registry.UnitParseError: Could not find unit symbol 'Jansky' in the provided symbols.
I had understood that define_unit also registers the newly defined unit: what is going wrong here?
Many thanks in advance.
It appears that Yt does not run Rockstar correctly. Here are 2 issues we
discovered so far:
1. DM particle mass and hence halo masses are incorrect in a simulation
with baryons. Here are lines from one of out_*.list for a simulation
with 1024^3 particles in a 40 cMpc/h box:
#Om = 0.303600; Ol = 0.696400; h = 0.681400
#Particle mass: 4.22991e+06 Msun/h
The particle mass should be 2.77536623e11*0.3036/0.6814*40**3/1024**3 =
7370552 Msun
Instead, the reported value is 4.22991e+06 Msun/h = 6207675 Msun, which
is exactly 2.77536623e11*0.2557/0.6814*40**3/1024**3 and 0.2557 =
Omega_{DM}, not Omega_M.
The same incorrect value is reported independently of what I set for the
dm_only argument to RockstarHaloFinder.
In a DMO simulation with identical initial conditions the particle mass
is reported correctly:
#Particle mass: 5.02229e+06 Msun/h
This is for the snapshot read by the artio frontend.
2. I tried to circumvent the previous bug by setting
particle_mass=5.02229e+06 in the call to RockstarHaloFinder, but that
dies with the following error:
p['particle_mass'].convert_to_units("Msun / h")
P002 yt : [ERROR ] 2021-06-01 19:17:19,937 AttributeError: 'float'
object has no attribute 'convert_to_units'
It appears that the argument of particle_mass keyword must be some class
rather than a number as listed in the documentation.
3. When running on a simulation with a non-zero DC mode Yt sets the box
size incorrectly:
> fgrep '#Box size' out_*
out_0.list:#Box size: 40.216236 Mpc/h
out_1.list:#Box size: 40.216236 Mpc/h
...
out_14.list:#Box size: 40.216236 Mpc/h
It appears that Yt uses the value Lbox*abox/auni for the first snapshot,
and never recomputes it for the following snapshots. That corrupts halo
positions, and, what is even worse, applies periodic wrapping
incorrectly, so halos near the box boundaries are messed up.
Nick Gnedin
--
This email has been checked for viruses by AVG.
https://www.avg.com
Hi all,
I would like to make a light cone projection from hydrodynamic, not cosmological, data set.
What I mean by **light cone projection** is: Given the coordinate of camera in simulation box and opening angle of the light cone, we can project 3D gas distribution onto a 2D plane. I found there is a similar API in the yt manual can do this, but it is for cosmological analysis not for hydrodynamics.
Does anyone has suggestions? Thanks.
