Revising the skeleton frontend
Hi all, I just finish the GAMER frontend for YT, and below I want to summarize some issues and things that might be confusing in the current skeleton frontend. I appreciate if you could help validate these issues and/or clarify some points in the section "Things to be clarified". Hopefully it will make frontend implementation easier in the future. I'll collect all comments and issue a PR for the revised skeleton. It will be complementary to the recent PR #2130 by Jonah Miller, which provides an extremely helpful example frontend. *Things to be clarified:*============================ 1. Difference between *FieldInfoContainer.known_other_fields* and *GridIndex._detect_output_field*: => known_other_fields contains fields that *might* be in an output, and lists their units and aliases. _detect_output_fields finds the fields that are defined in the currently loaded dataset. 2. *Dataset.unique_identifier*: => ?? 3. Set *AMRGridPatch.Parent = None* for grids without parents, and *AMRGridPatch.Children = []* for grids without children 4. Field array returned by *BaseIOHandler._read_fluid_selection* should be contiguous along the z instead of x direction. Therefore, for C-like array with the dimension [x][y][z] and for Fortran-like array with the the dimension (z,y,x), a matrix transpose is required (e.g., using np_array.transpose() or np_array.swapaxes(0,2)) 5. *start_index, stop_index, *and *ActiveDimension*s in the AMRGridPatch objects => ?? (1) It looks like stop_index is not used anymore (2) ActiveDimensions will be set by AMRGridPatch._prepare_grid. So perhaps frontend does not need to set it explicitly anymore (3) It seems that start_index is also calculated automatically 6. *chunk, selector, *and *_read_chunk_data* in io.py. => ?? Also the following comment about caching is confusing: def _read_chunk_data(self, chunk, fields): # This reads the data from a single chunk, and is only used for caching. 7. float type: (1) *GridIndex.float_type* is the float type for left and right simulation edges and must be float64 now (2) *BaseIOHandler._read_fluid_selection* should always return float64 now even if the on-disk data are stored in single precision 8. Difference between *add_output_field *and *add_field *used in FieldInfoContainer.setup_fluid_fields: 9. Dataset._parse_parameter_file: what does the following comment mean "Note that these are all assumed to be in code units; domain_left_edge and domain_right_edge will be updated to be in code units at a later time" => Perhaps it means that "domain_left_edge and domain_right_edge will be converted to YTArray automatically at a later time"? *Missing procedure*============================ 1. Add the new frontend in *frontends/api.py* 2. Set *GridIndex.max_level* 3. Set *Dataset.refine_by* *Bugin the current skeleton frontend* ============================ 1. data_structures.py: *self.dataset* is not set in GridIndex.__init__ when calling self.index_filename = self.dataset.parameter_filename 2. data_structures.py: replace self.Parent=[] by *self.Parent = None* 3. fields.py: the *field_list *argument is missing in FieldInfoContainer.__init__. This issue has been fixed in Jonah's PR. def __init__(self, ds, field_list): super(SkeletonFieldInfo, self).__init__(ds, field_list) Sincerely, Hsi-Yu (Justin)
Hi Hsi-Yu,
Thanks for going through this!
On Wed, Apr 27, 2016 at 7:47 PM, Hsi-Yu Schive
Hi all,
I just finish the GAMER frontend for YT, and below I want to summarize some issues and things that might be confusing in the current skeleton frontend. I appreciate if you could help validate these issues and/or clarify some points in the section "Things to be clarified". Hopefully it will make frontend implementation easier in the future. I'll collect all comments and issue a PR for the revised skeleton. It will be complementary to the recent PR #2130 by Jonah Miller, which provides an extremely helpful example frontend.
*Things to be clarified:*============================ 1. Difference between *FieldInfoContainer.known_other_fields* and *GridIndex._detect_output_field*: => known_other_fields contains fields that *might* be in an output, and lists their units and aliases. _detect_output_fields finds the fields that are defined in the currently loaded dataset.
Sounds right.
2. *Dataset.unique_identifier*: => ??
This just needs to be unique; for some codes it's a UUID output at runtime. A combination of hash and ctime might work for your data, or even just a random number.
3. Set *AMRGridPatch.Parent = None* for grids without parents, and *AMRGridPatch.Children = []* for grids without children
Yes, but creating lots of lists can be slow, so some frontends also make it on-demand if Children is None. Some frontends also allow for multiple parents, so that can be a list.
4. Field array returned by *BaseIOHandler._read_fluid_selection* should be contiguous along the z instead of x direction. Therefore, for C-like array with the dimension [x][y][z] and for Fortran-like array with the the dimension (z,y,x), a matrix transpose is required (e.g., using np_array.transpose() or np_array.swapaxes(0,2))
Yes, unless you swap your coordinate systems in the indexing scheme, which is also possible. You may try doing this my supplying geometry = ("cartesian", ("z", "y", "x")) and not transposing. It may not work, but I think it should.
5. *start_index, stop_index, *and *ActiveDimension*s in the AMRGridPatch objects => ?? (1) It looks like stop_index is not used anymore
I think that's correct.
(2) ActiveDimensions will be set by AMRGridPatch._prepare_grid. So perhaps frontend does not need to set it explicitly anymore
Also correct, unless you can do it faster.
(3) It seems that start_index is also calculated automatically
Yes, but this can be overridden, as I think the boxlib frontends do.
6. *chunk, selector, *and *_read_chunk_data* in io.py. => ?? Also the following comment about caching is confusing: def _read_chunk_data(self, chunk, fields): # This reads the data from a single chunk, and is only used for caching.
That is confusing; it's designed to be a "read data from this chunk without doing any selection" -- then it can be used for caching data that might be used by multiple different selectors later. For instance, this can speed up ghost zone computation.
7. float type: (1) *GridIndex.float_type* is the float type for left and right simulation edges and must be float64 now (2) *BaseIOHandler._read_fluid_selection* should always return float64 now even if the on-disk data are stored in single precision
Yes, that's presently the case.
8. Difference between *add_output_field *and *add_field *used in FieldInfoContainer.setup_fluid_fields:
One means on-disk, the other is not necessarily on-disk.
9. Dataset._parse_parameter_file: what does the following comment mean "Note that these are all assumed to be in code units; domain_left_edge and domain_right_edge will be updated to be in code units at a later time" => Perhaps it means that "domain_left_edge and domain_right_edge will be converted to YTArray automatically at a later time"?
Yes, that's what it means.
*Missing procedure*============================ 1. Add the new frontend in *frontends/api.py* 2. Set *GridIndex.max_level* 3. Set *Dataset.refine_by*
*Bugin the current skeleton frontend* ============================ 1. data_structures.py: *self.dataset* is not set in GridIndex.__init__ when calling self.index_filename = self.dataset.parameter_filename 2. data_structures.py: replace self.Parent=[] by *self.Parent = None* 3. fields.py: the *field_list *argument is missing in FieldInfoContainer.__init__. This issue has been fixed in Jonah's PR. def __init__(self, ds, field_list): super(SkeletonFieldInfo, self).__init__(ds, field_list)
Sincerely, Hsi-Yu (Justin)
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Matt,
Thanks for the comments. I 'll add them to the revised skeleton frontend.
Sincerely,
Justin
On Fri, May 6, 2016 at 11:02 AM, Matthew Turk
Hi Hsi-Yu,
Thanks for going through this!
On Wed, Apr 27, 2016 at 7:47 PM, Hsi-Yu Schive
wrote: Hi all,
I just finish the GAMER frontend for YT, and below I want to summarize some issues and things that might be confusing in the current skeleton frontend. I appreciate if you could help validate these issues and/or clarify some points in the section "Things to be clarified". Hopefully it will make frontend implementation easier in the future. I'll collect all comments and issue a PR for the revised skeleton. It will be complementary to the recent PR #2130 by Jonah Miller, which provides an extremely helpful example frontend.
*Things to be clarified:*============================ 1. Difference between *FieldInfoContainer.known_other_fields* and *GridIndex._detect_output_field*: => known_other_fields contains fields that *might* be in an output, and lists their units and aliases. _detect_output_fields finds the fields that are defined in the currently loaded dataset.
Sounds right.
2. *Dataset.unique_identifier*: => ??
This just needs to be unique; for some codes it's a UUID output at runtime. A combination of hash and ctime might work for your data, or even just a random number.
3. Set *AMRGridPatch.Parent = None* for grids without parents, and *AMRGridPatch.Children = []* for grids without children
Yes, but creating lots of lists can be slow, so some frontends also make it on-demand if Children is None. Some frontends also allow for multiple parents, so that can be a list.
4. Field array returned by *BaseIOHandler._read_fluid_selection* should be contiguous along the z instead of x direction. Therefore, for C-like array with the dimension [x][y][z] and for Fortran-like array with the the dimension (z,y,x), a matrix transpose is required (e.g., using np_array.transpose() or np_array.swapaxes(0,2))
Yes, unless you swap your coordinate systems in the indexing scheme, which is also possible. You may try doing this my supplying geometry = ("cartesian", ("z", "y", "x")) and not transposing. It may not work, but I think it should.
5. *start_index, stop_index, *and *ActiveDimension*s in the AMRGridPatch objects => ?? (1) It looks like stop_index is not used anymore
I think that's correct.
(2) ActiveDimensions will be set by AMRGridPatch._prepare_grid. So perhaps frontend does not need to set it explicitly anymore
Also correct, unless you can do it faster.
(3) It seems that start_index is also calculated automatically
Yes, but this can be overridden, as I think the boxlib frontends do.
6. *chunk, selector, *and *_read_chunk_data* in io.py. => ?? Also the following comment about caching is confusing: def _read_chunk_data(self, chunk, fields): # This reads the data from a single chunk, and is only used for caching.
That is confusing; it's designed to be a "read data from this chunk without doing any selection" -- then it can be used for caching data that might be used by multiple different selectors later. For instance, this can speed up ghost zone computation.
7. float type: (1) *GridIndex.float_type* is the float type for left and right simulation edges and must be float64 now (2) *BaseIOHandler._read_fluid_selection* should always return float64 now even if the on-disk data are stored in single precision
Yes, that's presently the case.
8. Difference between *add_output_field *and *add_field *used in FieldInfoContainer.setup_fluid_fields:
One means on-disk, the other is not necessarily on-disk.
9. Dataset._parse_parameter_file: what does the following comment mean "Note that these are all assumed to be in code units; domain_left_edge and domain_right_edge will be updated to be in code units at a later time" => Perhaps it means that "domain_left_edge and domain_right_edge will be converted to YTArray automatically at a later time"?
Yes, that's what it means.
*Missing procedure*============================ 1. Add the new frontend in *frontends/api.py* 2. Set *GridIndex.max_level* 3. Set *Dataset.refine_by*
*Bugin the current skeleton frontend* ============================ 1. data_structures.py: *self.dataset* is not set in GridIndex.__init__ when calling self.index_filename = self.dataset.parameter_filename 2. data_structures.py: replace self.Parent=[] by *self.Parent = None* 3. fields.py: the *field_list *argument is missing in FieldInfoContainer.__init__. This issue has been fixed in Jonah's PR. def __init__(self, ds, field_list): super(SkeletonFieldInfo, self).__init__(ds, field_list)
Sincerely, Hsi-Yu (Justin)
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (2)
-
Hsi-Yu Schive
-
Matthew Turk