Mailman 3 April 2016 - yt-dev

Revising the skeleton frontend
by Hsi-Yu Schive May 6, 2016

May 6, 2016

Hi all, I just finish the GAMER frontend for YT, and below I want to summarize some issues and things that might be confusing in the current skeleton frontend. I appreciate if you could help validate these issues and/or clarify some points in the section "Things to be clarified". Hopefully it will make frontend implementation easier in the future. I'll collect all comments and issue a PR for the revised skeleton. It will be complementary to the recent PR #2130 by Jonah Miller, which provides an extremely helpful example frontend. *Things to be clarified:*============================ 1. Difference between *FieldInfoContainer.known_other_fields* and *GridIndex._detect_output_field*: => known_other_fields contains fields that *might* be in an output, and lists their units and aliases. _detect_output_fields finds the fields that are defined in the currently loaded dataset. 2. *Dataset.unique_identifier*: => ?? 3. Set *AMRGridPatch.Parent = None* for grids without parents, and *AMRGridPatch.Children = []* for grids without children 4. Field array returned by *BaseIOHandler._read_fluid_selection* should be contiguous along the z instead of x direction. Therefore, for C-like array with the dimension [x][y][z] and for Fortran-like array with the the dimension (z,y,x), a matrix transpose is required (e.g., using np_array.transpose() or np_array.swapaxes(0,2)) 5. *start_index, stop_index, *and *ActiveDimension*s in the AMRGridPatch objects => ?? (1) It looks like stop_index is not used anymore (2) ActiveDimensions will be set by AMRGridPatch._prepare_grid. So perhaps frontend does not need to set it explicitly anymore (3) It seems that start_index is also calculated automatically 6. *chunk, selector, *and *_read_chunk_data* in io.py. => ?? Also the following comment about caching is confusing: def _read_chunk_data(self, chunk, fields): # This reads the data from a single chunk, and is only used for caching. 7. float type: (1) *GridIndex.float_type* is the float type for left and right simulation edges and must be float64 now (2) *BaseIOHandler._read_fluid_selection* should always return float64 now even if the on-disk data are stored in single precision 8. Difference between *add_output_field *and *add_field *used in FieldInfoContainer.setup_fluid_fields: 9. Dataset._parse_parameter_file: what does the following comment mean "Note that these are all assumed to be in code units; domain_left_edge and domain_right_edge will be updated to be in code units at a later time" => Perhaps it means that "domain_left_edge and domain_right_edge will be converted to YTArray automatically at a later time"? *Missing procedure*============================ 1. Add the new frontend in *frontends/api.py* 2. Set *GridIndex.max_level* 3. Set *Dataset.refine_by* *Bugin the current skeleton frontend* ============================ 1. data_structures.py: *self.dataset* is not set in GridIndex.__init__ when calling self.index_filename = self.dataset.parameter_filename 2. data_structures.py: replace self.Parent=[] by *self.Parent = None* 3. fields.py: the *field_list *argument is missing in FieldInfoContainer.__init__. This issue has been fixed in Jonah's PR. def __init__(self, ds, field_list): super(SkeletonFieldInfo, self).__init__(ds, field_list) Sincerely, Hsi-Yu (Justin)

2 2

Issue #1214: vector annotation of off-axis plot generates inconsistent result (yt_analysis/yt)
by Suoqing Ji April 29, 2016

April 29, 2016

New issue 1214: vector annotation of off-axis plot generates inconsistent result https://bitbucket.org/yt_analysis/yt/issues/1214/vector-annotation-of-off-a… Suoqing Ji: Test dataset is [here](https://dl.dropboxusercontent.com/u/83265784/data.npy) Test script: ``` #!python import yt import numpy as np dims = (200,200,200) x = np.load('data.npy') field1 = x[:,2].reshape(dims, order="F") field2 = x[:,1].reshape(dims, order="F") field3 = x[:,0].reshape(dims, order="F") data = dict(velocity_x = field1, velocity_y = field2, velocity_z = field3) bbox = np.array([[-100, 99], [-100, 99], [-100, 99]]) ds = yt.load_uniform_grid(data, dims, 'km', bbox=bbox, nprocs=1) L = [0, -1, 0] north_vector = [0, 0, 1] slc1 = yt.OffAxisSlicePlot(ds, L, 'velocity_x', center=[0,0,0], width=(180,'km'), north_vector=north_vector) slc1.annotate_velocity(factor=40, normalize=True) slc1.save() L = [0, 1, 0] north_vector = [0, 0, 1] slc2 = yt.SlicePlot(ds, L, 'velocity_x', center=[0,0,0], width=(180,'km'), north_vector=north_vector) slc2.annotate_velocity(factor=40, normalize=True) slc2.save() ``` On-axis plot: ![UniformGridData_Slice_y_velocity_x.png](https://bitbucket.org/repo/BXbAb/images/3764369440-UniformGridData_Slice_y_velocity_x.png) Off-axis plot: ![UniformGridData_OffAxisSlice_velocity_x.png](https://bitbucket.org/repo/BXbAb/images/829906186-UniformGridData_OffAxisSlice_velocity_x.png)

1 0

Revising the skeleton frontend
by Hsi-Yu Schive April 27, 2016

April 27, 2016

Hi all, I just finish the GAMER frontend for YT, and below I want to summarize some issues and things that might be confusing in the current skeleton frontend. I appreciate if you could help validate these issues and/or clarify some points in the section "Things to be clarified". Hopefully it will make frontend implementation easier in the future. I'll collect all comments and issue a PR for the revised skeleton. It will be complementary to the recent PR #2130 by Jonah Miller, which provides an extremely helpful example frontend. *Things to be clarified:*============================ 1. Difference between *FieldInfoContainer.known_other_fields* and *GridIndex._detect_output_field*: => known_other_fields contains fields that *might* be in an output, and lists their units and aliases. _detect_output_fields finds the fields that are defined in the currently loaded dataset. 2. *Dataset.unique_identifier*: => ?? 3. Set *AMRGridPatch.Parent = None* for grids without parents, and *AMRGridPatch.Children = []* for grids without children 4. Field array returned by *BaseIOHandler._read_fluid_selection* should be contiguous along the z instead of x direction. Therefore, for C-like array with the dimension [x][y][z] and for Fortran-like array with the the dimension (z,y,x), a matrix transpose is required (e.g., using np_array.transpose() or np_array.swapaxes(0,2)) 5. *start_index, stop_index, *and *ActiveDimension*s in the AMRGridPatch objects => ?? (1) It looks like stop_index is not used anymore (2) ActiveDimensions will be set by AMRGridPatch._prepare_grid. So perhaps frontend does not need to set it explicitly anymore (3) It seems that start_index is also calculated automatically 6. *chunk, selector, *and *_read_chunk_data* in io.py. => ?? Also the following comment about caching is confusing: def _read_chunk_data(self, chunk, fields): # This reads the data from a single chunk, and is only used for caching. 7. float type: (1) *GridIndex.float_type* is the float type for left and right simulation edges and must be float64 now (2) *BaseIOHandler._read_fluid_selection* should always return float64 now even if the on-disk data are stored in single precision 8. Difference between *add_output_field *and *add_field *used in FieldInfoContainer.setup_fluid_fields: 9. Dataset._parse_parameter_file: what does the following comment mean: "Note that these are all assumed to be in code units; domain_left_edge and domain_right_edge will be updated to be in code units at a later time"? => Perhaps it means that "domain_left_edge and domain_right_edge will be converted to YTArray automatically at a later time"? *Missing procedure*============================ 1. Add the new frontend in *frontends/api.py* 2. Set *GridIndex.max_level* 3. Set *Dataset.refine_by* *Bugin the current skeleton frontend* ============================ 1. data_structures.py: *self.dataset* is not set in GridIndex.__init__ when calling self.index_filename = self.dataset.parameter_filename 2. data_structures.py: replace self.Parent=[] by *self.Parent = None* 3. fields.py: the *field_list *argument is missing in FieldInfoContainer.__init__. This issue has been fixed in Jonah's PR. def __init__(self, ds, field_list): super(SkeletonFieldInfo, self).__init__(ds, field_list) Sincerely, Hsi-Yu (Justin)

2 1

Issue #1213: annotate_magnetic_field fails with off-axis slice plot (yt_analysis/yt)
by Suoqing Ji April 26, 2016

April 26, 2016

New issue 1213: annotate_magnetic_field fails with off-axis slice plot https://bitbucket.org/yt_analysis/yt/issues/1213/annotate_magnetic_field-fa… Suoqing Ji: Test script: ``` #!python import yt import numpy as np nn = 100 data = dict(magnetic_field_x = np.random.random((nn,nn,nn)), magnetic_field_y = np.random.random((nn,nn,nn)), magnetic_field_z = np.random.random((nn,nn,nn))) bbox = np.array([[-1, 1], [-1, 1], [-1, 1]]) ds = yt.load_uniform_grid(data, (nn,nn,nn), length_unit="cm", bbox=bbox, nprocs=1) L = [1, 1, 0] north_vector = [0, 0, 1] slc = yt.SlicePlot(ds, L, 'magnetic_field_x', width=(1 ,'cm'), north_vector=north_vector) slc.annotate_magnetic_field(factor=40, normalize=True) slc.save() ``` Error: ``` #!python Traceback (most recent call last): File "yt_magnetic_fields.py", line 16, in <module> slc.save() File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/visualization/plot_container.py", line 78, in newfunc args[0]._setup_plots() File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/visualization/plot_window.py", line 936, in _setup_plots self.run_callbacks() File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/visualization/plot_window.py", line 989, in run_callbacks sys.exc_info()[2]) File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/visualization/plot_window.py", line 985, in run_callbacks callback(cbw) File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/visualization/plot_modifications.py", line 316, in __call__ return qcb(plot) File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/visualization/plot_modifications.py", line 838, in __call__ plot.data[self.field_x], File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 263, in __getitem__ f = self._determine_fields([key])[0] File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 987, in _determine_fields finfo = self.ds._get_field_info("unknown", fname) File "/Users/jisuoqing/Workspace/code/yt-x86_64/src/yt-hg/yt/data_objects/static_output.py", line 626, in _get_field_info raise YTFieldNotFound((ftype, fname), self) yt.utilities.exceptions.YTPlotCallbackError: annotate_magnetic_field callback failed with the following error: Could not find field '('io', 'cutting_plane_bx')' in UniformGridData. ```

1 0

Issue #1212: SPH and Octree AMR Datasets Failing in volume rendering, OAProj, Interactive Data Viz (yt_analysis/yt)
by Cameron Hummels April 21, 2016

April 21, 2016

New issue 1212: SPH and Octree AMR Datasets Failing in volume rendering, OAProj, Interactive Data Viz https://bitbucket.org/yt_analysis/yt/issues/1212/sph-and-octree-amr-dataset… Cameron Hummels: This similar to Issues #788, #986, #1008, #1183, but I think this description generalizes and distills all of these problems. It appears that all particle-based (or Octree-based) datasets are segfaulting when one attempts to do anything involving the KDTree with them. The following three scripts all fail with the same result: ``` #!python import yt ds = yt.load("gizmo_cosmology_plus/snap_N128L16_151.hdf5") L = ds.arr([1,0,0], 'unitary') yt.OffAxisProjectionPlot(ds, L, 'density').save() ``` ``` #!python import yt ds = yt.load("gizmo_cosmology_plus/snap_N128L16_151.hdf5") im, sc = yt.volume_render(ds, 'density', fname='rendering.png') ``` ``` #!python import yt ds = yt.load("gizmo_cosmology_plus/snap_N128L16_151.hdf5") yt.interactive_render(ds) ``` These all fail with some variation on this output: ``` #!python Failed to split grids. Failed to split grids. Segmentation fault ``` The above scripts each use the publicly available dataset `gizmo_cosmology_plus`, but one gets similar results by using the same simple commands on any of the SPH datasets on http://yt-project.org/data/ e.g. `TipsyGalaxy`, `snapshot_033`, `GadgetDiskGalaxy`. Notably, however, when you run the same script with Octree-based AMR outputs, you get similar results. Example datasets that do this are Ramses (e.g. `output_00080`), and ART (e.g. `D9p_500`). These scripts were run on an OS X installation with the tip of yt-dev (843a342ee510).

1 0

PR review hangout
by Nathan Goldbaum April 20, 2016

April 20, 2016

Hi all, Since the review hangout didn't happen last week, I think we're due for one this Wednesday at 1:00 PM CDT: https://plus.google.com/u/0/b/107728486871834552760/events/cbcf6icgj879gij0… -Nathan

1 1

vertex-centered data
by Jonah Miller April 20, 2016

April 20, 2016

Hi yt-dev, Nathan Goldbaum and I were discussing the possibility of handling vertex-centered data natively in yt. We think that roughly this is quite do-able and that most of the pieces are already in place. Nathan suggested I summarize our discussion and email it to all of you to ask for your opinions. Internally, all that's required is using the generated ghost cells to convert between cell- and vertex-centered data. (For non-periodic boundaries, linear extrapolation, which would need to be implemented, should be good enough.) In the userspace side, we need a concept for this type of data so that it can be extracted and manipulated. One option would be to make centering an attribute for datasets. It could be immutable or mutable, depending on design choices. In both cases, we would create a method to go between centerings: ds.to_centering(centering_type) in the former case, this method would return a new dataset. In the latter case, it would reset the dataset. In principle, centering is a metadata property. The input data can be interpolated when it's explored and not before. So for this reason, it makes some sense to leave the centering as a dataset property rather than as, say, a field property. I think that's about it. Nathan can correct me if I forgot something. Thanks! Best, Jonah Miller

4 3

Issue #1211: HDF5FileHandler is missing close() (yt_analysis/yt)
by Kacper Kowalik April 18, 2016

April 18, 2016

New issue 1211: HDF5FileHandler is missing close() https://bitbucket.org/yt_analysis/yt/issues/1211/hdf5filehandler-is-missing… Kacper Kowalik: ``` /home/yt/WindTunnel $ yt search ( 0 candidates) Examining . ( 0/ 1) Evaluating ./windtunnel_4lev_hdf5_plt_cnt_0030 yt : [INFO ] 2016-04-18 13:54:14,762 Parameters: current_time = 3.00091455609 yt : [INFO ] 2016-04-18 13:54:14,762 Parameters: domain_dimensions = [120 40 1] yt : [INFO ] 2016-04-18 13:54:14,763 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2016-04-18 13:54:14,763 Parameters: domain_right_edge = [ 3. 1. 1.] yt : [INFO ] 2016-04-18 13:54:14,764 Parameters: cosmological_simulation = 0.0 Traceback (most recent call last): File "/home/xarth/local/bin/yt", line 9, in <module> load_entry_point('yt', 'console_scripts', 'yt')() File "/home/xarth/codes/yt-my/yt/utilities/command_line.py", line 1193, in run_main args.func(args) File "/home/xarth/codes/yt-my/yt/utilities/command_line.py", line 115, in run self(args) File "/home/xarth/codes/yt-my/yt/utilities/command_line.py", line 1176, in __call__ ds.close() File "/home/xarth/codes/yt-my/yt/frontends/flash/data_structures.py", line 425, in close self._handle.close() AttributeError: 'HDF5FileHandler' object has no attribute 'close' ``` Version: ``` Version = 3.3-dev Changeset = b10ac4289001 (yt) tip issue_1206 ``` Responsible: xarthisius

1 0

trouble with a frontend
by Jonah Miller April 17, 2016

April 17, 2016

Hi yt-dev, I'm trying to write a frontend for a patch-based AMR code. (The file format is more general and experimental.) I'm running into a strange problem where slice plots come out fine but projection plots come out looking very strange. I attach a pdf explaining my situation and the problem I'm encountering. If you have any suggestions, I'd appreciate all the help you can offer. Thanks very much! Best, Jonah Miller P.S. I have an open pull request (WIP) for my frontend, which you can find here: https://bitbucket.org/yt_analysis/yt/pull-requests/2121/wip-simulationio-fr… It contains information on the frontend, the file format, and of course, my code. I have prepared some test cases here: https://bitbucket.org/Yurlungur/simulationio-yt-tests/overview And the frontend relies on this code: https://github.com/eschnett/SimulationIO

1 0

Call for testing: updated install script
by Nathan Goldbaum April 14, 2016

April 14, 2016

Hi all, I'd appreciate it if people could try running my updated install script on a variety of platforms. With my changes, the install script sets up a conda environment by default, but the option to create a bootstrapped from-source python environment is still implemented. See PR 2009 for details: https://bitbucket.org/yt_analysis/yt/pull-requests/2009 You can either grab the raw install_script.sh file from my fork: $ wget https://bitbucket.org/ngoldbaum/yt/raw/install-script-updates/doc/install_s… Or pull from my fork and run the "test_install_script.py" that I've added. The latter will take a while, because it runs the install script six times with a variety of permutations in options. Thanks for your help! -Nathan

8 12