+! On Thu, Feb 23, 2012 at 11:46 AM, Cameron Hummels < chummels@astro.columbia.edu> wrote:

+1

On 2/23/12 11:43 AM, Matthew Turk wrote:

...

Hi all,

Long ago, I added the "lazy_reader" option to profiles. This then showed up also in DerivedQuantities. In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy. lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.

I'd like to propose we deprecate it as an option, and force all profiles to be "lazy". I would be implementing this in the geometry handling branch. In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded. By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways. For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like. I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.

[+-][01] on removing lazy_reader as optional?

-Matt ______________________________**_________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/**listinfo.cgi/yt-dev-spacepope.**orghttp://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

______________________________**_________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/**listinfo.cgi/yt-dev-spacepope.**orghttp://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

+0.98 For what it's worth-- in one script I have wherein I compute profiles with a covering grid, I have lazy_reader = false. I seem to recall that it had something to do with covering grids on fields that contain spatial derivatives, thus requiring ghost zones. The line in question looks like: pc.add_profile_object(covering_grid,fields, lazy_reader=False) d. On Thu, Feb 23, 2012 at 9:48 AM, Britton Smith wrote:

+!

On Thu, Feb 23, 2012 at 11:46 AM, Cameron Hummels wrote:

...

+1

On 2/23/12 11:43 AM, Matthew Turk wrote:

...

Hi all,

Long ago, I added the "lazy_reader" option to profiles.  This then showed up also in DerivedQuantities.  In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy.  lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.

I'd like to propose we deprecate it as an option, and force all profiles to be "lazy".  I would be implementing this in the geometry handling branch.  In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded.  By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways.  For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like.  I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.

[+-][01] on removing lazy_reader as optional?

-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer.

Okay, *this* is interesting, and I vaguely recall it.  Let's put fixing that without the lazy_reader as a use case.  When the merge request comes for geometry, we'll revisit this.

That sounds like a fine plan. I try to save everything I do, and that was the only instance I could find of lazy_reader=False. I'll try to track down and reproduce whatever error we had before and write it up. d.

The merge request will be a ways off, and will touch a lot of things, so hopefully we can get people on board with testing.

Any chance you could write up something that shows this behavior with existing spatial derivative fields, so we can add it to the test suite?

-Matt

...

On Thu, Feb 23, 2012 at 9:48 AM, Britton Smith wrote:

...

+!

On Thu, Feb 23, 2012 at 11:46 AM, Cameron Hummels wrote:

...

+1

On 2/23/12 11:43 AM, Matthew Turk wrote:

...

Hi all,

Long ago, I added the "lazy_reader" option to profiles.  This then showed up also in DerivedQuantities.  In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy.  lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.

I'd like to propose we deprecate it as an option, and force all profiles to be "lazy".  I would be implementing this in the geometry handling branch.  In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded.  By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways.  For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like.  I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.

[+-][01] on removing lazy_reader as optional?

-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer.