Hi all, Long ago, I added the "lazy_reader" option to profiles. This then showed up also in DerivedQuantities. In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy. lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory. I'd like to propose we deprecate it as an option, and force all profiles to be "lazy". I would be implementing this in the geometry handling branch. In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded. By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways. For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like. I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this. [+-][01] on removing lazy_reader as optional? -Matt
+1 On 2/23/12 11:43 AM, Matthew Turk wrote:
Hi all,
Long ago, I added the "lazy_reader" option to profiles. This then showed up also in DerivedQuantities. In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy. lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.
I'd like to propose we deprecate it as an option, and force all profiles to be "lazy". I would be implementing this in the geometry handling branch. In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded. By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways. For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like. I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.
[+-][01] on removing lazy_reader as optional?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
+! On Thu, Feb 23, 2012 at 11:46 AM, Cameron Hummels < chummels@astro.columbia.edu> wrote:
+1
On 2/23/12 11:43 AM, Matthew Turk wrote:
Hi all,
Long ago, I added the "lazy_reader" option to profiles. This then showed up also in DerivedQuantities. In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy. lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.
I'd like to propose we deprecate it as an option, and force all profiles to be "lazy". I would be implementing this in the geometry handling branch. In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded. By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways. For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like. I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.
[+-][01] on removing lazy_reader as optional?
-Matt ______________________________**_________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/**listinfo.cgi/yt-dev-spacepope.**org<http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org>
______________________________**_________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/**listinfo.cgi/yt-dev-spacepope.**org<http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org>
+0.98 For what it's worth-- in one script I have wherein I compute profiles with a covering grid, I have lazy_reader = false. I seem to recall that it had something to do with covering grids on fields that contain spatial derivatives, thus requiring ghost zones. The line in question looks like: pc.add_profile_object(covering_grid,fields, lazy_reader=False) d. On Thu, Feb 23, 2012 at 9:48 AM, Britton Smith <brittonsmith@gmail.com> wrote:
+!
On Thu, Feb 23, 2012 at 11:46 AM, Cameron Hummels <chummels@astro.columbia.edu> wrote:
+1
On 2/23/12 11:43 AM, Matthew Turk wrote:
Hi all,
Long ago, I added the "lazy_reader" option to profiles. This then showed up also in DerivedQuantities. In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy. lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.
I'd like to propose we deprecate it as an option, and force all profiles to be "lazy". I would be implementing this in the geometry handling branch. In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded. By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways. For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like. I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.
[+-][01] on removing lazy_reader as optional?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
Hi all, Looks like we're leading toward passage. On Thu, Feb 23, 2012 at 12:02 PM, david collins <antpuncher@gmail.com> wrote:
+0.98
For what it's worth-- in one script I have wherein I compute profiles with a covering grid, I have lazy_reader = false. I seem to recall that it had something to do with covering grids on fields that contain spatial derivatives, thus requiring ghost zones. The line in question looks like:
pc.add_profile_object(covering_grid,fields, lazy_reader=False)
d.
Okay, *this* is interesting, and I vaguely recall it. Let's put fixing that without the lazy_reader as a use case. When the merge request comes for geometry, we'll revisit this. The merge request will be a ways off, and will touch a lot of things, so hopefully we can get people on board with testing. Any chance you could write up something that shows this behavior with existing spatial derivative fields, so we can add it to the test suite? -Matt
On Thu, Feb 23, 2012 at 9:48 AM, Britton Smith <brittonsmith@gmail.com> wrote:
+!
On Thu, Feb 23, 2012 at 11:46 AM, Cameron Hummels <chummels@astro.columbia.edu> wrote:
+1
On 2/23/12 11:43 AM, Matthew Turk wrote:
Hi all,
Long ago, I added the "lazy_reader" option to profiles. This then showed up also in DerivedQuantities. In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy. lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.
I'd like to propose we deprecate it as an option, and force all profiles to be "lazy". I would be implementing this in the geometry handling branch. In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded. By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways. For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like. I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.
[+-][01] on removing lazy_reader as optional?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Okay, *this* is interesting, and I vaguely recall it. Let's put fixing that without the lazy_reader as a use case. When the merge request comes for geometry, we'll revisit this.
That sounds like a fine plan. I try to save everything I do, and that was the only instance I could find of lazy_reader=False. I'll try to track down and reproduce whatever error we had before and write it up. d.
The merge request will be a ways off, and will touch a lot of things, so hopefully we can get people on board with testing.
Any chance you could write up something that shows this behavior with existing spatial derivative fields, so we can add it to the test suite?
-Matt
On Thu, Feb 23, 2012 at 9:48 AM, Britton Smith <brittonsmith@gmail.com> wrote:
+!
On Thu, Feb 23, 2012 at 11:46 AM, Cameron Hummels <chummels@astro.columbia.edu> wrote:
+1
On 2/23/12 11:43 AM, Matthew Turk wrote:
Hi all,
Long ago, I added the "lazy_reader" option to profiles. This then showed up also in DerivedQuantities. In almost all cases used, this is set to True, and I believe that the only case when it cannot be is when calculating energy. lazy_reader = True means data is read on demand, grid by grid (mediated by preloading for those formats where multiple grids might be in a single file) and False means that all of the data necessary for the reduction is loaded immediately into memory.
I'd like to propose we deprecate it as an option, and force all profiles to be "lazy". I would be implementing this in the geometry handling branch. In the geometry handling branch, I have implemented a chunking scheme which is more flexible; in this way subsets of data objects can be loaded. By moving to this scheme, the chunks can either be the *entire* data object (i.e., the non-lazy method from before) or subsets divided in several different ways. For example, you could do grid-by-grid, you could do gridfile-by-gridfile, or whatever type of scheme you like. I believe that getting rid of lazy and moving to just always operating on chunks is the best way to accomplish this.
[+-][01] on removing lazy_reader as optional?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
participants (5)
-
Britton Smith -
Cameron Hummels -
david collins -
Matthew Turk -
Stephen Skory