I am post processing some MHD simulations done with FLASH using yt (yt version 4.3.0). Initially, the simulation has no particles, but over time, we add some (sink) particles to it. I would like to use the boolean ds.particles_exist to check if a given FLASH HDF5 particle file has any particles in it or not. It works fine when there are particles, but gives a key error when there are no particles, instead of returning False. The same issue also occurs with an earlier version of yt (4.0.4). Traceback below: In [1]: yt.__version__ Out[1]: '4.3.0' In [4]: ds=yt.load('FLASH_part_0050') yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: current_time = 50966272658115.96 yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: domain_dimensions = [1 1 1] yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: domain_left_edge = [-3.703e+18 -3.703e+18 -3.703e+18] yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: domain_right_edge = [3.703e+18 3.703e+18 3.703e+18] yt : [INFO ] 2023-10-24 11:01:22,509 Parameters: cosmological_simulation = 0 In [5]: ds.particles_exist yt : [INFO ] 2023-10-24 11:01:26,270 Allocating for 3 particles Out[5]: True In [6]: ds=yt.load('FLASH_part_0000') yt : [INFO ] 2023-10-24 11:01:41,713 Parameters: current_time = 0.0 yt : [INFO ] 2023-10-24 11:01:41,713 Parameters: domain_dimensions = [1 1 1] yt : [INFO ] 2023-10-24 11:01:41,714 Parameters: domain_left_edge = [-3.703e+18 -3.703e+18 -3.703e+18] yt : [INFO ] 2023-10-24 11:01:41,714 Parameters: domain_right_edge = [3.703e+18 3.703e+18 3.703e+18] yt : [INFO ] 2023-10-24 11:01:41,714 Parameters: cosmological_simulation = 0 In [7]: ds.particles_exist --------------------------------------------------------------------------- KeyError Traceback (most recent call last) Input In [7], in <cell line: 1>() ----> 1 ds.particles_exist File ~/.local/lib/python3.9/site-packages/yt/data_objects/static_output.py:1238, in Dataset.particles_exist(self) 1236 @property 1237 def particles_exist(self): -> 1238 for pt, f in itertools.product(self.particle_types_raw, self.field_list): 1239 if pt == f[0]: 1240 return True File ~/.local/lib/python3.9/site-packages/yt/data_objects/static_output.py:657, in Dataset.field_list(self) 655 @property 656 def field_list(self): --> 657 return self.index.field_list File ~/.local/lib/python3.9/site-packages/yt/data_objects/static_output.py:606, in Dataset.index(self) 603 @property 604 def index(self): 605 if self._instantiated_index is None: --> 606 self._instantiated_index = self._index_class( 607 self, dataset_type=self.dataset_type 608 ) 609 # Now we do things that we need an instantiated index for 610 # ...first off, we create our field_info now. 611 oldsettings = np.geterr() File ~/.local/lib/python3.9/site-packages/yt/geometry/particle_geometry_handler.py:72, in ParticleIndex.__init__(self, ds, dataset_type) 70 self.dataset = weakref.proxy(ds) 71 self.float_type = np.float64 ---> 72 super().__init__(ds, dataset_type) 73 self._initialize_index() File ~/.local/lib/python3.9/site-packages/yt/geometry/geometry_handler.py:42, in Index.__init__(self, ds, dataset_type) 39 self._setup_geometry() 41 mylog.debug("Initializing data grid data IO") ---> 42 self._setup_data_io() 44 # Note that this falls under the "geometry" object since it's 45 # potentially quite expensive, and should be done with the indexing. 46 mylog.debug("Detecting fields.") File ~/.local/lib/python3.9/site-packages/yt/geometry/geometry_handler.py:124, in Index._setup_data_io(self) 122 if getattr(self, "io", None) is not None: 123 return --> 124 self.io = io_registry[self.dataset_type](self.dataset) File ~/.local/lib/python3.9/site-packages/yt/frontends/flash/io.py:224, in IOHandlerFLASHParticle.__init__(self, ds) 222 self._handle = ds._handle 223 self._particle_fields = determine_particle_fields(self._handle) --> 224 self._position_fields = [ 225 self._particle_fields[f"particle_pos{ax}"] for ax in "xyz" 226 ] File ~/.local/lib/python3.9/site-packages/yt/frontends/flash/io.py:225, in <listcomp>(.0) 222 self._handle = ds._handle 223 self._particle_fields = determine_particle_fields(self._handle) 224 self._position_fields = [ --> 225 self._particle_fields[f"particle_pos{ax}"] for ax in "xyz" 226 ] KeyError: 'particle_posx'
Hi Piyush, Good catch. I'm looking over the source code and I think what's happening is that it's trying to load the data file as a 'particle' file, and in doing so it tries to read the fields listed in "/particle names". But, because there aren't any yet, it gets a key error. Before I suggest a fix (or issue a pull request) can I ask what the output of "h5ls -r" is on the data file you are attempting to load? Thanks, Matt On Tue, Oct 24, 2023 at 4:09 AM <piyush.suneet@gmail.com> wrote:
I am post processing some MHD simulations done with FLASH using yt (yt version 4.3.0). Initially, the simulation has no particles, but over time, we add some (sink) particles to it. I would like to use the boolean ds.particles_exist to check if a given FLASH HDF5 particle file has any particles in it or not. It works fine when there are particles, but gives a key error when there are no particles, instead of returning False. The same issue also occurs with an earlier version of yt (4.0.4). Traceback below:
In [1]: yt.__version__ Out[1]: '4.3.0'
In [4]: ds=yt.load('FLASH_part_0050') yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: current_time = 50966272658115.96 yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: domain_dimensions = [1 1 1] yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: domain_left_edge = [-3.703e+18 -3.703e+18 -3.703e+18] yt : [INFO ] 2023-10-24 11:01:22,508 Parameters: domain_right_edge = [3.703e+18 3.703e+18 3.703e+18] yt : [INFO ] 2023-10-24 11:01:22,509 Parameters: cosmological_simulation = 0
In [5]: ds.particles_exist yt : [INFO ] 2023-10-24 11:01:26,270 Allocating for 3 particles Out[5]: True
In [6]: ds=yt.load('FLASH_part_0000') yt : [INFO ] 2023-10-24 11:01:41,713 Parameters: current_time = 0.0 yt : [INFO ] 2023-10-24 11:01:41,713 Parameters: domain_dimensions = [1 1 1] yt : [INFO ] 2023-10-24 11:01:41,714 Parameters: domain_left_edge = [-3.703e+18 -3.703e+18 -3.703e+18] yt : [INFO ] 2023-10-24 11:01:41,714 Parameters: domain_right_edge = [3.703e+18 3.703e+18 3.703e+18] yt : [INFO ] 2023-10-24 11:01:41,714 Parameters: cosmological_simulation = 0
In [7]: ds.particles_exist --------------------------------------------------------------------------- KeyError Traceback (most recent call last) Input In [7], in <cell line: 1>() ----> 1 ds.particles_exist
File ~/.local/lib/python3.9/site-packages/yt/data_objects/static_output.py:1238, in Dataset.particles_exist(self) 1236 @property 1237 def particles_exist(self): -> 1238 for pt, f in itertools.product(self.particle_types_raw, self.field_list): 1239 if pt == f[0]: 1240 return True
File ~/.local/lib/python3.9/site-packages/yt/data_objects/static_output.py:657, in Dataset.field_list(self) 655 @property 656 def field_list(self): --> 657 return self.index.field_list
File ~/.local/lib/python3.9/site-packages/yt/data_objects/static_output.py:606, in Dataset.index(self) 603 @property 604 def index(self): 605 if self._instantiated_index is None: --> 606 self._instantiated_index = self._index_class( 607 self, dataset_type=self.dataset_type 608 ) 609 # Now we do things that we need an instantiated index for 610 # ...first off, we create our field_info now. 611 oldsettings = np.geterr()
File ~/.local/lib/python3.9/site-packages/yt/geometry/particle_geometry_handler.py:72, in ParticleIndex.__init__(self, ds, dataset_type) 70 self.dataset = weakref.proxy(ds) 71 self.float_type = np.float64 ---> 72 super().__init__(ds, dataset_type) 73 self._initialize_index()
File ~/.local/lib/python3.9/site-packages/yt/geometry/geometry_handler.py:42, in Index.__init__(self, ds, dataset_type) 39 self._setup_geometry() 41 mylog.debug("Initializing data grid data IO") ---> 42 self._setup_data_io() 44 # Note that this falls under the "geometry" object since it's 45 # potentially quite expensive, and should be done with the indexing. 46 mylog.debug("Detecting fields.")
File ~/.local/lib/python3.9/site-packages/yt/geometry/geometry_handler.py:124, in Index._setup_data_io(self) 122 if getattr(self, "io", None) is not None: 123 return --> 124 self.io = io_registry[self.dataset_type](self.dataset)
File ~/.local/lib/python3.9/site-packages/yt/frontends/flash/io.py:224, in IOHandlerFLASHParticle.__init__(self, ds) 222 self._handle = ds._handle 223 self._particle_fields = determine_particle_fields(self._handle) --> 224 self._position_fields = [ 225 self._particle_fields[f"particle_pos{ax}"] for ax in "xyz" 226 ]
File ~/.local/lib/python3.9/site-packages/yt/frontends/flash/io.py:225, in <listcomp>(.0) 222 self._handle = ds._handle 223 self._particle_fields = determine_particle_fields(self._handle) 224 self._position_fields = [ --> 225 self._particle_fields[f"particle_pos{ax}"] for ax in "xyz" 226 ]
KeyError: 'particle_posx' _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com
Hi Matt, Thanks for your response! If I do h5ls -r on a particle file that has no particles, I get: (base) RSAA-043527:Documents psharda$ h5ls -r FLASH_part_0003 / Group /integer\ runtime\ parameters Dataset {136} /integer\ scalars Dataset {17} /localnp Dataset {585} /logical\ runtime\ parameters Dataset {147} /logical\ scalars Dataset {2} /real\ runtime\ parameters Dataset {206} /real\ scalars Dataset {13} /sim\ info Dataset {1} /string\ runtime\ parameters Dataset {152} /string\ scalars Dataset {1} /unknown\ names Dataset {37, 1} If I do the same on a particle file that has particles: (base) RSAA-043527:Documents psharda$ h5ls -r FLASH_part_0044 / Group /integer\ runtime\ parameters Dataset {136} /integer\ scalars Dataset {17} /localnp Dataset {2753} /logical\ runtime\ parameters Dataset {147} /logical\ scalars Dataset {2} /particle\ names Dataset {37, 1} /real\ runtime\ parameters Dataset {206} /real\ scalars Dataset {13} /sim\ info Dataset {1} /string\ runtime\ parameters Dataset {152} /string\ scalars Dataset {1} /tracer\ particles Dataset {3, 37} /unknown\ names Dataset {37, 1}
Hi Piyush, OK -- I think the simplest solution *for now* would be to catch the exception in your loop. This isn't really a long term solution, but it should be able to get you to the next step. The longer term solution would be for yt to just return empty arrays -- that should allow you to query the metadata but not get the keyerrors. On Fri, Oct 27, 2023 at 4:07 AM <piyush.suneet@gmail.com> wrote:
Hi Matt,
Thanks for your response!
If I do h5ls -r on a particle file that has no particles, I get:
(base) RSAA-043527:Documents psharda$ h5ls -r FLASH_part_0003 / Group /integer\ runtime\ parameters Dataset {136} /integer\ scalars Dataset {17} /localnp Dataset {585} /logical\ runtime\ parameters Dataset {147} /logical\ scalars Dataset {2} /real\ runtime\ parameters Dataset {206} /real\ scalars Dataset {13} /sim\ info Dataset {1} /string\ runtime\ parameters Dataset {152} /string\ scalars Dataset {1} /unknown\ names Dataset {37, 1}
If I do the same on a particle file that has particles:
(base) RSAA-043527:Documents psharda$ h5ls -r FLASH_part_0044 / Group /integer\ runtime\ parameters Dataset {136} /integer\ scalars Dataset {17} /localnp Dataset {2753} /logical\ runtime\ parameters Dataset {147} /logical\ scalars Dataset {2} /particle\ names Dataset {37, 1} /real\ runtime\ parameters Dataset {206} /real\ scalars Dataset {13} /sim\ info Dataset {1} /string\ runtime\ parameters Dataset {152} /string\ scalars Dataset {1} /tracer\ particles Dataset {3, 37} /unknown\ names Dataset {37, 1} _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com
Hi Matt, Thanks! Since ds.particles_exist is a boolean, can it be such that it just returns false instead of returning empty arrays?
I *think* so, but I'm trying to figure out how we can make that happen without having to go through the IO functions. It's one of those things where the solution seems simple, but I am worried about side-effects. On Mon, Oct 30, 2023 at 3:05 AM <piyush.suneet@gmail.com> wrote:
Hi Matt,
Thanks! Since ds.particles_exist is a boolean, can it be such that it just returns false instead of returning empty arrays? _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address: matthewturk@gmail.com
participants (2)
-
Matthew Turk -
piyush.suneet＠gmail.com