Here's the same test, but with 'particle_mass' replaced with 'Density' and 'Msun' replaced with 'g/cm**3'. yt 2.x (dd1ca98) 20.84s user 1.27s system 99% cpu 22.171 total yt 3.0 (1b17b7e) 21.23s user 0.86s system 99% cpu 22.180 total yt-3.0 unitrefactor (8131901d) 49.88s user 2.08s system 90% cpu 57.584 total Here are the top functions in the profile for the unitrefactor test: http://paste.yt-project.org/show/4313/ So still about the same 30 second slowdown compared to the other two, with a lot of the overhead inside of new unitrefactor stuff. I'm not sure what the best strategy for further optimizations is. I suspect most of the YTArray and YTquantity instances are being created inside of field detection. Would it be possible to alter the field detection algorithm to more efficiently store random test data? On Fri, Feb 14, 2014 at 4:47 PM, Matthew Turk wrote:

Hi Nathan,

Thanks for running this! I suspect the particles might be particularly bad right now because of caching being disabled. What is the timing for gas density?

-Matt

On Feb 14, 2014 7:38 PM, "Nathan Goldbaum" wrote:

...

Hi all,

Matt asked me to look at the performance of unitrefactor relative to 2.X and the current 3.0 codebase. I've done that using the following test script: http://paste.yt-project.org/show/4312

The script cycles through a time series, printing out particle masses for each dataset. In a perfect world, this should be an i/o-bound operation.

Currently the timings for running this test are as follows:

yt 2.x (dd1ca98) 18.37s user 1.43s system 99% cpu 19.997 total

yt 3.0 (1b17b7e) 48.76s user 1.81s system 99% cpu 50.743 total

yt-3.0 unitrefactor (8131901df14c) 78.72s user 2.00s system 99% cpu 1:20.85 total

So about a thirty second difference between each of the versions.

Clearly this isn't the best news given that we're moving to 3.0 going forward. That said, I don't think there has been a lot of focus on 3.0 performance, so it's possible (likely even) there's some low-hanging fruit for performance optimizations. There was also a big refactor of how i/o works for Enzo datasets (moving away from hdf5_light_reader toward h5py) - maybe that explains some of the difference between 2.x and 3.0.

Hope this is helpful,

-Nathan _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Hi Hsi-Yu, Sorry for the delay in replying. On Friday, February 21, 2014, wrote:

Dear all,

I'm a new user of yt and is trying to use it for analyzing our octree AMR data, which is similar to FLASH. We are stilling testing yt version 2.x, and currently we are able to create a slice image together with the underlying AMR grid distribution WITHOUT providing any parent-children relation between each grid. However, after we further set up the parent-children relation, the plot (e.g., the SlicePlot command) failed with the following error messages:

===============Error Message======================= =========================

ValueError Traceback (most recent call last)

<ipython-input-3-e98bfe3813b1> in <module>()

----> 1 p=SlicePlot(pf,2,"Density",center=(0.501,0.501,0.501))

/work1/ftd/yt-x86_64/src/yt-hg/yt/visualization/plot_window.pyc in __init__(self, pf, axis, fields, center, width, origin)

805 slc = pf.h.slice(axis, center[axis], fields=fields)

806 print slc,bounds,origin

--> 807 PWViewerMPL.__init__(self, slc, bounds, origin=origin)

808

809 class ProjectionPlot(PWViewerMPL):

/work1/ftd/yt-x86_64/src/yt-hg/yt/visualization/plot_window.pyc in __init__(self, *args, **kwargs)

446 self._field_transform[field] = linear_transform

447

--> 448 if setup: self._setup_plots()

449

450 @invalidate_plot

/work1/ftd/yt-x86_64/src/yt-hg/yt/visualization/plot_window.pyc in _setup_plots(self)

617 print self.plots[f].image,self.plots[f].cax

618 self.plots[f].cb = self.plots[f].figure.colorbar(

--> 619 self.plots[f].image, cax = self.plots[f].cax)

620

621 if self.oblique == False:

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/matplotlib/figure.pyc in colorbar(self, mappable, cax, ax, **kw)

1194 cax, kw = cbar.make_axes(ax, **kw)

1195 cax.hold(True)

-> 1196 cb = cbar.Colorbar(cax, mappable, **kw)

1197

1198 def on_changed(m):

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/matplotlib/colorbar.pyc in __init__(self, ax, mappable, **kw)

741 kw['alpha'] = mappable.get_alpha()

742

--> 743 ColorbarBase.__init__(self, ax, **kw)

744

745

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/matplotlib/colorbar.pyc in __init__(self, ax, cmap, norm, alpha, values, boundaries, orientation, extend, spacing, ticks, format, drawedges, filled)

256 # The rest is in a method so we can recalculate when clim changes.

257 self.config_axis()

--> 258 self.draw_all()

259

260 def _patch_ax(self):

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/matplotlib/colorbar.pyc in draw_all(self)

270 and do all the drawing.

271 '''

--> 272 self._process_values()

273 self._find_range()

274 X, Y = self._mesh()

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/matplotlib/colorbar.pyc in _process_values(self, b)

568 self.norm.vmin = 0

569 self.norm.vmax = 1

--> 570 b = self.norm.inverse(self._uniform_y(self.cmap.N+1))

571 if self.extend in ('both', 'min'):

572 b[0] = b[0] - 1

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/matplotlib/colors.pyc in inverse(self, value)

946 if cbook.iterable(value):

947 val = ma.asarray(value)

--> 948 return vmin * ma.power((vmax/vmin), val)

949 else:

950 return vmin * pow((vmax/vmin), value)

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/numpy/ma/core.pyc in __mul__(self, other)

3640 def __mul__(self, other):

3641 "Multiply other by self, and return a new masked array."

-> 3642 return multiply(self, other)

3643 #

3644 def __rmul__(self, other):

/work1/ftd/yt-x86_64/lib/python2.7/site-packages/numpy/ma/core.pyc in __call__(self, a, b, *args, **kwargs)

934 # Case 1. : scalar

935 if not result.ndim:

--> 936 if m:

937 return masked

938 return result

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

==========End of Error Message======================= ===========================

So, my questions are as follows. 1. What is the necessary information for constructing the tree structure in yt?Currently the SlicePlot command failed only after we set up the "Parent" and "Children" variables for each grid in the function "_populate_grid_objects".

Typically the information is necessary just to check the masking of grid objects; Parent is considerably less important for this than Children. There is a diagnostic that can be run to examine the volume: vol = 0.0 dv = pf.domain_width.prod(dtype="float64") for g in pf.h.grids: vol += g.dds.prod(dtype="float64") * g.child_mask.sum() print np.abs(vol - dv)/dv What it looks like is happening, based on the error message, is that somehow the total values in a slice are being set to the same thing everywhere. This diagnostic -- which should produce a value of 0.0 -- should help track that down.

2. Does the SlicePlot command need to know the parent-children relation of each grid? If no, why it will fail after we set up the relation?

It only needs to know the children of a grid, so that it can mask out overlapping regions.

3. Will yt automatically construct the "neighbor" information for nearby grids at the same AMR refinement level? If no, how does it handle the operations requiring spatial derivative (e.g., curl and divergence)?

It will do so, yup.

Any suggestion will be welcome, and thanks for the help in advance :)

I suspect something might be going wrong with the way the children pointers are set up. Can you show how you're constructing them? It's possible that there is an off-by-one or something that is creeping in. -Matt

Sincerely, Hsi-Yu

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org