Mailman 3 Re: [yt-users] FLASH Time Series Analysis with Particle Files - yt-users

1 Dec 2016

      Hi Nathan,

Thanks for the explanation. I thought I had tried calling yt.load() with
just the plot files at some point, which is why I was trying to give it the
particle_filename keyword. I tried it on my dataset, and it works. User
error. Apologies.

Jason

On Thu, Dec 1, 2016 at 10:47 AM, 
wrote:
...
Send yt-users mailing list submissions to
        yt-users@lists.spacepope.org
To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
or, via email, send a message with subject or body 'help' to
        yt-users-request@lists.spacepope.org
You can reach the person managing the list at
        yt-users-owner@lists.spacepope.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of yt-users digest..."
Today's Topics:
1. Re: FLASH Time Series Analysis with Particle Files (Alex Hill)
   2. Re: FLASH Time Series Analysis with Particle Files
      (Jason Galyardt)
   3. Re: FLASH Time Series Analysis with Particle Files
      (Nathan Goldbaum)
---------- Forwarded message ----------
From: Alex Hill 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 13:59:19 -0500
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Hi Jason,
Yes, I use Flash files with particles routinely now. As John alludes to,
there are issues when the plot files and particle files don’t match in both
file name (including the plot/particle file number) and timestamp. I now
avoid those issues by ensuring that the names are the same; I don’t recall
if the issue with different names/numbers was solved but could look into it
again if it would help.
-Alex
...
On Nov 30, 2016, at 11:32, John Zuhone  wrote:
Yes, my understanding is that this is working, provided that your files
have the same basename and the times of the files are the same.
...
On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum 
wrote:
Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <
jason.galyardt@gmail.com> wrote:
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in
time series analysis on FLASH simulations that include both plot files and
particle files. Does anyone have an update on the status of this issue?
Thanks,
Jason
------
Jason Galyardt
University of Georgia
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
---------- Forwarded message ----------
From: Jason Galyardt 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 14:47:06 -0500
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Hi John & Nathan,
Yes, I'm having trouble using the time series analysis machinery with my
FLASH simulations. I've checked that the plot files and particle files all
have the same current_time variable, so there's no issue there. I'm using a
development version of yt with a changeset ID of 5cf30893afdf. This
changeset includes Nathan's changes supporting cut_region functionality for
FLASH plot files.  Here's what I see when I try to do a time series
analysis:
###########################
In [9]: ts = yt.load("mysim_hdf5_plt_cnt_*",
    particle_filename="mysim_hdf5_part_*")
In [10]: for ds in ts:
    print ds.current_time
   ....:
------------------------------------------------------------
---------------
IOError                                   Traceback (most recent call
last)
<ipython-input-10-fccd01bb5065> in <module>()
----> 1 for ds in ts:
      2     print ds.current_time
      3
/home/emrys/yt/yt/data_objects/time_series.pyc in __iter__(self)
    179         for o in self._pre_outputs:
    180             if isinstance(o, string_types):
--> 181                 ds = self._load(o, **self.kwargs)
    182                 self._setup_function(ds)
    183                 yield ds
/home/emrys/yt/yt/data_objects/time_series.pyc in _load(self, output_fn,
**kwargs)
    396         elif self._mixed_dataset_types:
    397             return load(output_fn, **kwargs)
--> 398         ds = load(output_fn, **kwargs)
    399         self._dataset_cls = ds.__class__
    400         return ds
/home/emrys/yt/yt/convenience.pyc in load(*args, **kwargs)
     84     candidates = find_lowest_subclasses(candidates)
     85     if len(candidates) == 1:
---> 86         return candidates[0](*args, **kwargs)
     87     if len(candidates) == 0:
     88         if ytcfg.get("yt", "enzo_db") != '' \
/home/emrys/yt/yt/frontends/flash/data_structures.pyc in __init__(self,
filename, dataset_type, storage_filename, particle_filename,
units_override, unit_system)
    209                 self._particle_handle =
HDF5FileHandler(self.particle_filename)
    210             except:
--> 211                 raise IOError(self.particle_filename)
    212         # Check if the particle file has the same time
    213         if self._particle_handle != self._handle:
IOError: mysim_hdf5_part_*
###########################
It looks to me like HDF5FileHandler() just does not like the wildcard.
I've also tried, unsuccessfully, to feed a list of particle files to
yt.load(). I thought perhaps that I should be using the DatasetSeries
framework directly, but I'm not clear on how to pass in the list of
particle files, as DatasetSeries does not understand the particle_filename
keyword argument.
Any ideas?
Thanks,
Jason
On Wed, Nov 30, 2016 at 1:57 PM, 
wrote:
...
Send yt-users mailing list submissions to
        yt-users@lists.spacepope.org
To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
or, via email, send a message with subject or body 'help' to
        yt-users-request@lists.spacepope.org
You can reach the person managing the list at
        yt-users-owner@lists.spacepope.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of yt-users digest..."
Today's Topics:
1. FLASH Time Series Analysis with Particle Files (Jason Galyardt)
   2. Re: FLASH Time Series Analysis with Particle Files
      (Nathan Goldbaum)
   3. Re: FLASH Time Series Analysis with Particle Files (John Zuhone)
---------- Forwarded message ----------
From: Jason Galyardt 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 10:16:44 -0500
Subject: [yt-users] FLASH Time Series Analysis with Particle Files
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in
time series analysis on FLASH simulations that include both plot files and
particle files. Does anyone have an update on the status of this issue?
Thanks,
Jason
------
Jason Galyardt
University of Georgia
---------- Forwarded message ----------
From: Nathan Goldbaum 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 09:32:37 -0600
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt 
...
wrote:
...
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in
time series analysis on FLASH simulations that include both plot files and
particle files. Does anyone have an update on the status of this issue?
Thanks,
Jason
------
Jason Galyardt
University of Georgia
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
---------- Forwarded message ----------
From: John Zuhone 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 11:32:09 -0500
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Yes, my understanding is that this is working, provided that your files
have the same basename and the times of the files are the same.
On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum 
wrote:
Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt 
...
wrote:
...
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in
time series analysis on FLASH simulations that include both plot files and
particle files. Does anyone have an update on the status of this issue?
Thanks,
Jason
------
Jason Galyardt
University of Georgia
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
---------- Forwarded message ----------
From: Nathan Goldbaum 
To: Discussion of the yt analysis package 
Cc:
Date: Thu, 1 Dec 2016 09:46:54 -0600
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Hi Jason,
Thanks for those details.
The FLASH frontend has a capability to automatically detect the presence
of an accompanying particle file for a given plotfile.  I have a test
dataset (that I will be uploading to yt-project.org/data shortly for
future questions about flash datasets that contain both particle and mesh
fields) that I've written the following test script for:
import yt
ts = yt.load('fiducial_1to1_b0_hdf5_plt_cnt_*')
for ds in ts:
        print(ds.particle_type_counts)
This time series contains five outputs, with a plotfile and a particlefile
for each output:
$ ls
    fiducial_1to1_b0_hdf5_part_0000    fiducial_1to1_b0_hdf5_plt_cnt_0001
    fiducial_1to1_b0_hdf5_part_0001    fiducial_1to1_b0_hdf5_plt_cnt_0002
    fiducial_1to1_b0_hdf5_part_0002    fiducial_1to1_b0_hdf5_plt_cnt_0003
    fiducial_1to1_b0_hdf5_part_0003    fiducial_1to1_b0_hdf5_plt_cnt_0004
    fiducial_1to1_b0_hdf5_part_0004    test.py
    fiducial_1to1_b0_hdf5_plt_cnt_0000
When I run the script, it outputs:
{'io': 9999998}
    {'io': 9999998}
    {'io': 9999998}
    {'io': 9999998}
    {'io': 9999998}
As you can see, my script only explicitly loads the plotfile but yt still
recognizes that there are about 10 million particles in each ouput, since
the particle file for each plotfile is automatically associated with the
plotfile. You can see how this functionality is setup in the FLASH frontend
here:
https://bitbucket.org/yt_analysis/yt/src/yt/yt/frontends/flash/data_
structures.py?#data_structures.py-202
This requires that the particle file associated with a plotfile has the
same output number. We used to strictly enforce that the particle file has
the same simulation time as the plotifle, but this ended up being too
strict for real-world use cases so now we only print a warning (see around
like 217 in the file I linked to above).
Is this sufficient for your use cases or do you need to explicitly specify
the name of the particle file every time you load the dataset? If so, one
hacky way around this would be to manually associate the particle file with
each plotfile using the `setup_function` keyword argument for the
DatasetSeries object. I could also look into making it so that if you pass
a glob pattern as the `particle_filename` keyword argument to the time
series object, it does the right thing, but I want to make sure that the
existing functionality will not work for you before trying that.
Hope that clears things up,
Nathan
On Wed, Nov 30, 2016 at 1:47 PM, Jason Galyardt 
wrote:
...
Hi John & Nathan,
Yes, I'm having trouble using the time series analysis machinery with my
FLASH simulations. I've checked that the plot files and particle files all
have the same current_time variable, so there's no issue there. I'm using a
development version of yt with a changeset ID of 5cf30893afdf. This
changeset includes Nathan's changes supporting cut_region functionality for
FLASH plot files.  Here's what I see when I try to do a time series
analysis:
###########################
In [9]: ts = yt.load("mysim_hdf5_plt_cnt_*",
    particle_filename="mysim_hdf5_part_*")
In [10]: for ds in ts:
    print ds.current_time
   ....:
------------------------------------------------------------
---------------
IOError                                   Traceback (most recent call
last)
<ipython-input-10-fccd01bb5065> in <module>()
----> 1 for ds in ts:
      2     print ds.current_time
      3
/home/emrys/yt/yt/data_objects/time_series.pyc in __iter__(self)
    179         for o in self._pre_outputs:
    180             if isinstance(o, string_types):
--> 181                 ds = self._load(o, **self.kwargs)
    182                 self._setup_function(ds)
    183                 yield ds
/home/emrys/yt/yt/data_objects/time_series.pyc in _load(self, output_fn,
**kwargs)
    396         elif self._mixed_dataset_types:
    397             return load(output_fn, **kwargs)
--> 398         ds = load(output_fn, **kwargs)
    399         self._dataset_cls = ds.__class__
    400         return ds
/home/emrys/yt/yt/convenience.pyc in load(*args, **kwargs)
     84     candidates = find_lowest_subclasses(candidates)
     85     if len(candidates) == 1:
---> 86         return candidates[0](*args, **kwargs)
     87     if len(candidates) == 0:
     88         if ytcfg.get("yt", "enzo_db") != '' \
/home/emrys/yt/yt/frontends/flash/data_structures.pyc in __init__(self,
filename, dataset_type, storage_filename, particle_filename,
units_override, unit_system)
    209                 self._particle_handle =
HDF5FileHandler(self.particle_filename)
    210             except:
--> 211                 raise IOError(self.particle_filename)
    212         # Check if the particle file has the same time
    213         if self._particle_handle != self._handle:
IOError: mysim_hdf5_part_*
###########################
It looks to me like HDF5FileHandler() just does not like the wildcard.
I've also tried, unsuccessfully, to feed a list of particle files to
yt.load(). I thought perhaps that I should be using the DatasetSeries
framework directly, but I'm not clear on how to pass in the list of
particle files, as DatasetSeries does not understand the particle_filename
keyword argument.
Any ideas?
Thanks,
Jason
On Wed, Nov 30, 2016 at 1:57 PM, 
wrote:
...
Send yt-users mailing list submissions to
        yt-users@lists.spacepope.org
To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
or, via email, send a message with subject or body 'help' to
        yt-users-request@lists.spacepope.org
You can reach the person managing the list at
        yt-users-owner@lists.spacepope.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of yt-users digest..."
Today's Topics:
1. FLASH Time Series Analysis with Particle Files (Jason Galyardt)
   2. Re: FLASH Time Series Analysis with Particle Files
      (Nathan Goldbaum)
   3. Re: FLASH Time Series Analysis with Particle Files (John Zuhone)
---------- Forwarded message ----------
From: Jason Galyardt 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 10:16:44 -0500
Subject: [yt-users] FLASH Time Series Analysis with Particle Files
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in
time series analysis on FLASH simulations that include both plot files and
particle files. Does anyone have an update on the status of this issue?
Thanks,
Jason
------
Jason Galyardt
University of Georgia
---------- Forwarded message ----------
From: Nathan Goldbaum 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 09:32:37 -0600
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <
jason.galyardt@gmail.com> wrote:
...
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in
time series analysis on FLASH simulations that include both plot files and
particle files. Does anyone have an update on the status of this issue?
Thanks,
Jason
------
Jason Galyardt
University of Georgia
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
---------- Forwarded message ----------
From: John Zuhone 
To: Discussion of the yt analysis package 
Cc:
Date: Wed, 30 Nov 2016 11:32:09 -0500
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Yes, my understanding is that this is working, provided that your files
have the same basename and the times of the files are the same.
On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum 
wrote:
Hi Jason,
As far as I'm aware this should be working now. Are you having trouble?
-Nathan
On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <
jason.galyardt@gmail.com> wrote:
...
Dear yt-users,
In May 2015, there was some traffic on this list concerning errors in
time series analysis on FLASH simulations that include both plot files and
particle files. Does anyone have an update on the status of this issue?
Thanks,
Jason
------
Jason Galyardt
University of Georgia
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Re: [yt-users] FLASH Time Series Analysis with Particle Files

Jason Galyardt

tags

participants (1)