Hi Jason,

Thanks for those details.

The FLASH frontend has a capability to automatically detect the presence of an accompanying particle file for a given plotfile.  I have a test dataset (that I will be uploading to yt-project.org/data shortly for future questions about flash datasets that contain both particle and mesh fields) that I've written the following test script for:

    import yt

    ts = yt.load('fiducial_1to1_b0_hdf5_plt_cnt_*')

    for ds in ts:
        print(ds.particle_type_counts)

This time series contains five outputs, with a plotfile and a particlefile for each output:

    $ ls
    fiducial_1to1_b0_hdf5_part_0000    fiducial_1to1_b0_hdf5_plt_cnt_0001
    fiducial_1to1_b0_hdf5_part_0001    fiducial_1to1_b0_hdf5_plt_cnt_0002
    fiducial_1to1_b0_hdf5_part_0002    fiducial_1to1_b0_hdf5_plt_cnt_0003
    fiducial_1to1_b0_hdf5_part_0003    fiducial_1to1_b0_hdf5_plt_cnt_0004
    fiducial_1to1_b0_hdf5_part_0004    test.py
    fiducial_1to1_b0_hdf5_plt_cnt_0000

When I run the script, it outputs:

    {'io': 9999998}
    {'io': 9999998}
    {'io': 9999998}
    {'io': 9999998}
    {'io': 9999998}

As you can see, my script only explicitly loads the plotfile but yt still recognizes that there are about 10 million particles in each ouput, since the particle file for each plotfile is automatically associated with the plotfile. You can see how this functionality is setup in the FLASH frontend here:

https://bitbucket.org/yt_analysis/yt/src/yt/yt/frontends/flash/data_structures.py?#data_structures.py-202

This requires that the particle file associated with a plotfile has the same output number. We used to strictly enforce that the particle file has the same simulation time as the plotifle, but this ended up being too strict for real-world use cases so now we only print a warning (see around like 217 in the file I linked to above).

Is this sufficient for your use cases or do you need to explicitly specify the name of the particle file every time you load the dataset? If so, one hacky way around this would be to manually associate the particle file with each plotfile using the `setup_function` keyword argument for the DatasetSeries object. I could also look into making it so that if you pass a glob pattern as the `particle_filename` keyword argument to the time series object, it does the right thing, but I want to make sure that the existing functionality will not work for you before trying that.

Hope that clears things up,

Nathan

On Wed, Nov 30, 2016 at 1:47 PM, Jason Galyardt <jason.galyardt@gmail.com> wrote:
Hi John & Nathan,

Yes, I'm having trouble using the time series analysis machinery with my FLASH simulations. I've checked that the plot files and particle files all have the same current_time variable, so there's no issue there. I'm using a development version of yt with a changeset ID of 5cf30893afdf. This changeset includes Nathan's changes supporting cut_region functionality for FLASH plot files.  Here's what I see when I try to do a time series analysis:

###########################

In [9]: ts = yt.load("mysim_hdf5_plt_cnt_*",
    particle_filename="mysim_hdf5_part_*")

In [10]: for ds in ts:
    print ds.current_time
   ....:    
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-10-fccd01bb5065> in <module>()
----> 1 for ds in ts:
      2     print ds.current_time
      3

/home/emrys/yt/yt/data_objects/time_series.pyc in __iter__(self)
    179         for o in self._pre_outputs:
    180             if isinstance(o, string_types):
--> 181                 ds = self._load(o, **self.kwargs)
    182                 self._setup_function(ds)
    183                 yield ds

/home/emrys/yt/yt/data_objects/time_series.pyc in _load(self, output_fn, **kwargs)
    396         elif self._mixed_dataset_types:
    397             return load(output_fn, **kwargs)
--> 398         ds = load(output_fn, **kwargs)
    399         self._dataset_cls = ds.__class__
    400         return ds

/home/emrys/yt/yt/convenience.pyc in load(*args, **kwargs)
     84     candidates = find_lowest_subclasses(candidates)
     85     if len(candidates) == 1:
---> 86         return candidates[0](*args, **kwargs)
     87     if len(candidates) == 0:
     88         if ytcfg.get("yt", "enzo_db") != '' \

/home/emrys/yt/yt/frontends/flash/data_structures.pyc in __init__(self, filename, dataset_type, storage_filename, particle_filename, units_override, unit_system)
    209                 self._particle_handle = HDF5FileHandler(self.particle_filename)
    210             except:
--> 211                 raise IOError(self.particle_filename)
    212         # Check if the particle file has the same time
    213         if self._particle_handle != self._handle:

IOError: mysim_hdf5_part_*

###########################

It looks to me like HDF5FileHandler() just does not like the wildcard. I've also tried, unsuccessfully, to feed a list of particle files to yt.load(). I thought perhaps that I should be using the DatasetSeries framework directly, but I'm not clear on how to pass in the list of particle files, as DatasetSeries does not understand the particle_filename keyword argument.

Any ideas?

Thanks,
Jason


On Wed, Nov 30, 2016 at 1:57 PM, <yt-users-request@lists.spacepope.org> wrote:
Send yt-users mailing list submissions to
        yt-users@lists.spacepope.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
or, via email, send a message with subject or body 'help' to
        yt-users-request@lists.spacepope.org

You can reach the person managing the list at
        yt-users-owner@lists.spacepope.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of yt-users digest..."

Today's Topics:

   1. FLASH Time Series Analysis with Particle Files (Jason Galyardt)
   2. Re: FLASH Time Series Analysis with Particle Files
      (Nathan Goldbaum)
   3. Re: FLASH Time Series Analysis with Particle Files (John Zuhone)


---------- Forwarded message ----------
From: Jason Galyardt <jason.galyardt@gmail.com>
To: Discussion of the yt analysis package <yt-users@lists.spacepope.org>
Cc: 
Date: Wed, 30 Nov 2016 10:16:44 -0500
Subject: [yt-users] FLASH Time Series Analysis with Particle Files
Dear yt-users,

In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?

Thanks,
Jason

------
Jason Galyardt
University of Georgia



---------- Forwarded message ----------
From: Nathan Goldbaum <nathan12343@gmail.com>
To: Discussion of the yt analysis package <yt-users@lists.spacepope.org>
Cc: 
Date: Wed, 30 Nov 2016 09:32:37 -0600
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Hi Jason,

As far as I'm aware this should be working now. Are you having trouble?

-Nathan

On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <jason.galyardt@gmail.com> wrote:
Dear yt-users,

In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?

Thanks,
Jason

------
Jason Galyardt
University of Georgia


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org




---------- Forwarded message ----------
From: John Zuhone <jzuhone@gmail.com>
To: Discussion of the yt analysis package <yt-users@lists.spacepope.org>
Cc: 
Date: Wed, 30 Nov 2016 11:32:09 -0500
Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
Yes, my understanding is that this is working, provided that your files have the same basename and the times of the files are the same. 

On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:

Hi Jason,

As far as I'm aware this should be working now. Are you having trouble?

-Nathan

On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <jason.galyardt@gmail.com> wrote:
Dear yt-users,

In May 2015, there was some traffic on this list concerning errors in time series analysis on FLASH simulations that include both plot files and particle files. Does anyone have an update on the status of this issue?

Thanks,
Jason

------
Jason Galyardt
University of Georgia


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org



_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org