Re: [yt-users] FLASH Time Series Analysis with Particle Files
Hi John & Nathan,
Yes, I'm having trouble using the time series analysis machinery with my FLASH simulations. I've checked that the plot files and particle files all have the same current_time variable, so there's no issue there. I'm using a development version of yt with a changeset ID of 5cf30893afdf. This changeset includes Nathan's changes supporting cut_region functionality for FLASH plot files. Here's what I see when I try to do a time series analysis:
###########################
In [9]: ts = yt.load("mysim_hdf5_plt_cnt_*", particle_filename="mysim_hdf5_part_*")
In [10]: for ds in ts: print ds.current_time ....: --------------------------------------------------------------------------- IOError Traceback (most recent call last) <ipython-input-10-fccd01bb5065> in <module>() ----> 1 for ds in ts: 2 print ds.current_time 3
/home/emrys/yt/yt/data_objects/time_series.pyc in __iter__(self) 179 for o in self._pre_outputs: 180 if isinstance(o, string_types): --> 181 ds = self._load(o, **self.kwargs) 182 self._setup_function(ds) 183 yield ds
/home/emrys/yt/yt/data_objects/time_series.pyc in _load(self, output_fn, **kwargs) 396 elif self._mixed_dataset_types: 397 return load(output_fn, **kwargs) --> 398 ds = load(output_fn, **kwargs) 399 self._dataset_cls = ds.__class__ 400 return ds
/home/emrys/yt/yt/convenience.pyc in load(*args, **kwargs) 84 candidates = find_lowest_subclasses(candidates) 85 if len(candidates) == 1: ---> 86 return candidates[0](*args, **kwargs) 87 if len(candidates) == 0: 88 if ytcfg.get("yt", "enzo_db") != '' \
/home/emrys/yt/yt/frontends/flash/data_structures.pyc in __init__(self, filename, dataset_type, storage_filename, particle_filename, units_override, unit_system) 209 self._particle_handle = HDF5FileHandler(self.particle_filename) 210 except: --> 211 raise IOError(self.particle_filename) 212 # Check if the particle file has the same time 213 if self._particle_handle != self._handle:
IOError: mysim_hdf5_part_*
###########################
It looks to me like HDF5FileHandler() just does not like the wildcard. I've also tried, unsuccessfully, to feed a list of particle files to yt.load(). I thought perhaps that I should be using the DatasetSeries framework directly, but I'm not clear on how to pass in the list of particle files, as DatasetSeries does not understand the particle_filename keyword argument.
Any ideas?
Thanks, Jason
On Wed, Nov 30, 2016 at 1:57 PM, yt-users-request@lists.spacepope.org wrote:
Send yt-users mailing list submissions to yt-users@lists.spacepope.org
To subscribe or unsubscribe via the World Wide Web, visit http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org or, via email, send a message with subject or body 'help' to yt-users-request@lists.spacepope.org
You can reach the person managing the list at yt-users-owner@lists.spacepope.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of yt-users digest..."
Today's Topics:
- FLASH Time Series Analysis with Particle Files (Jason Galyardt)
- Re: FLASH Time Series Analysis with Particle Files (Nathan Goldbaum)
- Re: FLASH Time Series Analysis with Particle Files (John Zuhone)
---------- Forwarded message ---------- From: Jason Galyardt jason.galyardt@gmail.com To: Discussion of the yt analysis package yt-users@lists.spacepope.org Cc: Date: Wed, 30 Nov 2016 10:16:44 -0500 Subject: [yt-users] FLASH Time Series Analysis with Particle Files Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?
Thanks, Jason
Jason Galyardt University of Georgia
---------- Forwarded message ---------- From: Nathan Goldbaum nathan12343@gmail.com To: Discussion of the yt analysis package yt-users@lists.spacepope.org Cc: Date: Wed, 30 Nov 2016 09:32:37 -0600 Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt jason.galyardt@gmail.com wrote:
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?
Thanks, Jason
Jason Galyardt University of Georgia
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
---------- Forwarded message ---------- From: John Zuhone jzuhone@gmail.com To: Discussion of the yt analysis package yt-users@lists.spacepope.org Cc: Date: Wed, 30 Nov 2016 11:32:09 -0500 Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files Yes, my understanding is that this is working, provided that your files have the same basename and the times of the files are the same.
On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt jason.galyardt@gmail.com wrote:
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?
Thanks, Jason
Jason Galyardt University of Georgia
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Jason,
Thanks for those details.
The FLASH frontend has a capability to automatically detect the presence of an accompanying particle file for a given plotfile. I have a test dataset (that I will be uploading to yt-project.org/data shortly for future questions about flash datasets that contain both particle and mesh fields) that I've written the following test script for:
import yt
ts = yt.load('fiducial_1to1_b0_hdf5_plt_cnt_*')
for ds in ts: print(ds.particle_type_counts)
This time series contains five outputs, with a plotfile and a particlefile for each output:
$ ls fiducial_1to1_b0_hdf5_part_0000 fiducial_1to1_b0_hdf5_plt_cnt_0001 fiducial_1to1_b0_hdf5_part_0001 fiducial_1to1_b0_hdf5_plt_cnt_0002 fiducial_1to1_b0_hdf5_part_0002 fiducial_1to1_b0_hdf5_plt_cnt_0003 fiducial_1to1_b0_hdf5_part_0003 fiducial_1to1_b0_hdf5_plt_cnt_0004 fiducial_1to1_b0_hdf5_part_0004 test.py fiducial_1to1_b0_hdf5_plt_cnt_0000
When I run the script, it outputs:
{'io': 9999998} {'io': 9999998} {'io': 9999998} {'io': 9999998} {'io': 9999998}
As you can see, my script only explicitly loads the plotfile but yt still recognizes that there are about 10 million particles in each ouput, since the particle file for each plotfile is automatically associated with the plotfile. You can see how this functionality is setup in the FLASH frontend here:
https://bitbucket.org/yt_analysis/yt/src/yt/yt/frontends/flash/data_structur...
This requires that the particle file associated with a plotfile has the same output number. We used to strictly enforce that the particle file has the same simulation time as the plotifle, but this ended up being too strict for real-world use cases so now we only print a warning (see around like 217 in the file I linked to above).
Is this sufficient for your use cases or do you need to explicitly specify the name of the particle file every time you load the dataset? If so, one hacky way around this would be to manually associate the particle file with each plotfile using the `setup_function` keyword argument for the DatasetSeries object. I could also look into making it so that if you pass a glob pattern as the `particle_filename` keyword argument to the time series object, it does the right thing, but I want to make sure that the existing functionality will not work for you before trying that.
Hope that clears things up,
Nathan
On Wed, Nov 30, 2016 at 1:47 PM, Jason Galyardt jason.galyardt@gmail.com wrote:
Hi John & Nathan,
Yes, I'm having trouble using the time series analysis machinery with my FLASH simulations. I've checked that the plot files and particle files all have the same current_time variable, so there's no issue there. I'm using a development version of yt with a changeset ID of 5cf30893afdf. This changeset includes Nathan's changes supporting cut_region functionality for FLASH plot files. Here's what I see when I try to do a time series analysis:
###########################
In [9]: ts = yt.load("mysim_hdf5_plt_cnt_*", particle_filename="mysim_hdf5_part_*")
In [10]: for ds in ts: print ds.current_time ....:
IOError Traceback (most recent call last) <ipython-input-10-fccd01bb5065> in <module>() ----> 1 for ds in ts: 2 print ds.current_time 3
/home/emrys/yt/yt/data_objects/time_series.pyc in __iter__(self) 179 for o in self._pre_outputs: 180 if isinstance(o, string_types): --> 181 ds = self._load(o, **self.kwargs) 182 self._setup_function(ds) 183 yield ds
/home/emrys/yt/yt/data_objects/time_series.pyc in _load(self, output_fn, **kwargs) 396 elif self._mixed_dataset_types: 397 return load(output_fn, **kwargs) --> 398 ds = load(output_fn, **kwargs) 399 self._dataset_cls = ds.__class__ 400 return ds
/home/emrys/yt/yt/convenience.pyc in load(*args, **kwargs) 84 candidates = find_lowest_subclasses(candidates) 85 if len(candidates) == 1: ---> 86 return candidates[0](*args, **kwargs) 87 if len(candidates) == 0: 88 if ytcfg.get("yt", "enzo_db") != '' \
/home/emrys/yt/yt/frontends/flash/data_structures.pyc in __init__(self, filename, dataset_type, storage_filename, particle_filename, units_override, unit_system) 209 self._particle_handle = HDF5FileHandler(self.particle_filename) 210 except: --> 211 raise IOError(self.particle_filename) 212 # Check if the particle file has the same time 213 if self._particle_handle != self._handle:
IOError: mysim_hdf5_part_*
###########################
It looks to me like HDF5FileHandler() just does not like the wildcard. I've also tried, unsuccessfully, to feed a list of particle files to yt.load(). I thought perhaps that I should be using the DatasetSeries framework directly, but I'm not clear on how to pass in the list of particle files, as DatasetSeries does not understand the particle_filename keyword argument.
Any ideas?
Thanks, Jason
On Wed, Nov 30, 2016 at 1:57 PM, yt-users-request@lists.spacepope.org wrote:
Send yt-users mailing list submissions to yt-users@lists.spacepope.org
To subscribe or unsubscribe via the World Wide Web, visit http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org or, via email, send a message with subject or body 'help' to yt-users-request@lists.spacepope.org
You can reach the person managing the list at yt-users-owner@lists.spacepope.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of yt-users digest..."
Today's Topics:
- FLASH Time Series Analysis with Particle Files (Jason Galyardt)
- Re: FLASH Time Series Analysis with Particle Files (Nathan Goldbaum)
- Re: FLASH Time Series Analysis with Particle Files (John Zuhone)
---------- Forwarded message ---------- From: Jason Galyardt jason.galyardt@gmail.com To: Discussion of the yt analysis package yt-users@lists.spacepope.org Cc: Date: Wed, 30 Nov 2016 10:16:44 -0500 Subject: [yt-users] FLASH Time Series Analysis with Particle Files Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?
Thanks, Jason
Jason Galyardt University of Georgia
---------- Forwarded message ---------- From: Nathan Goldbaum nathan12343@gmail.com To: Discussion of the yt analysis package yt-users@lists.spacepope.org Cc: Date: Wed, 30 Nov 2016 09:32:37 -0600 Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <jason.galyardt@gmail.com
wrote:
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?
Thanks, Jason
Jason Galyardt University of Georgia
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
---------- Forwarded message ---------- From: John Zuhone jzuhone@gmail.com To: Discussion of the yt analysis package yt-users@lists.spacepope.org Cc: Date: Wed, 30 Nov 2016 11:32:09 -0500 Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files Yes, my understanding is that this is working, provided that your files have the same basename and the times of the files are the same.
On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <jason.galyardt@gmail.com
wrote:
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?
Thanks, Jason
Jason Galyardt University of Georgia
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (2)
-
Jason Galyardt -
Nathan Goldbaum