Hi Stephanie,

Well, this particular thing is definitely a memory issue. It looks like it failed when trying to open the x-coordinate dataset for all_data(). By itself, this is about 7 GB, given the domain dimensions that you gave. By the way, do those look right (1048,1048,950)?

By the way, is this uniform grid or does this have SMR? I don't think that we've verified that SMR works. 

I'm sorry I'm not able to be more definitive yet--let me think on it a bit more. 

Best,

John

On Dec 18, 2012, at 4:27 PM, Stephanie Tonnesen <stonnes@gmail.com> wrote:

Hi John,

The weighted average quantity call of specific_scalar[0] worked.

The domain info:

pf.domain_right_edge
array([ 3.2,  3.2,  3.8])
>>> pf.domain_dimensions
array([1048, 1048,  950])

And then using id0 and id1:

 pf2.domain_right_edge
array([-1.349618, -2.259542, -1.145263])
>>> pf2.domain_dimensions
array([303, 154, 140])

When I combined the vtk files into a single 25 GB file, I get an error when creating the PlotCollection:

 pc = PlotCollection(pf)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/visualization/plot_collection.py", line 121, in __init__
    v,self.c = pf.h.find_max("Density") # @todo: ensure no caching
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 61, in find_max
    mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 74, in find_max_cell_location
    source.quantities["MaxLocation"]( field, lazy_reader=True)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 92, in __call__
    return self._call_func_lazy(args, kwargs)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 99, in _call_func_lazy
    rv = self.func(GridChildMaskWrapper(g, self._data_source), *args, **kwargs)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 649, in _MaxLocation
    mx, my, mz = [data[ax][maxi] for ax in 'xyz']
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 60, in __getitem__
    data = self.data_source._get_data_from_grid(self.grid, item)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 93, in save_state
    tr = func(self, grid, field, *args, **kwargs)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 2630, in _get_data_from_grid
    tr = grid[field]
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 157, in __getitem__
    self.get_data(key)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 200, in get_data
    self._generate_field(field)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 145, in _generate_field
    self[field] = self.pf.field_info[field](self)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.py", line 378, in __call__
    dd = self._function(self, data)
  File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/universal_fields.py", line 94, in _coordX
    return (np.ones(data.ActiveDimensions, dtype='float64')
  File "/u/stonnes/Installs/yt-x86_64/lib/python2.7/site-packages/numpy/core/numeric.py", line 1819, in ones
    a = empty(shape, dtype, order)
MemoryError


On Tue, Dec 18, 2012 at 1:49 PM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Stephanie,

I doubt that 25 GB is too big.

Can you also try a weighted average quantity of specific_scalar[0] itself?

When you use only id0 and id1, you are potentially also changing domain_right_edge, since for Athena yt has to go through all of the id*/* to find the largest right edge for any grid to determine domain_right_edge. 

Can you figure out what pf.domain_right_edge and pf.domain_dimensions are for the two different cases (where you have all the data and also where you have just id0 and id1)?

I'm not sure at the moment what that would tell us, but I'm curious about it.

As another possible way to get some clues, I am aware that Athena has a tool that joins the vtk files into one. Is it possible to run that and then try the same script on the merged file?

I'm also wondering if Matt and Sam have any thoughts on this.

Best,

John

On Dec 18, 2012, at 1:21 PM, Stephanie Tonnesen <stonnes@gmail.com> wrote:

Hi John,

So, what I mean by "expected" variables are basically all variables BUT specific_scalar[0].  When I read in the vtk file, you can see that the bottom line says 

yt : [INFO     ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields
Yes, I can do any of the "quantities" commands, for example:

alld = pf.h.all_data()
tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25", "grid['z'] > 0.01"])
radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")

Slicing also works:

p = SlicePlot(pf,"x","specific_scalar[0]")

Now I am starting to worry that it is because output is so big--each vtk file is 72 MB, so a full output is almost 25 GB.  When I just use two outputs (moving them so yt only sees id0 and id1 folders) the phase plot works.  But, after running the add_phase_object command on the whole data set for more than 1.5 hours I still have no result and my computer keeps freezing--although top shows that only 42% of memory is being used. 

Is it possible that my output is just too big?

Stephanie


On Tue, Dec 18, 2012 at 11:00 AM, John Zuhone <jzuhone@gmail.com> wrote:
Hi Stephanie,

Ok, thanks for checking that. 

I guess I'm not sure what you mean by the "expected" set of variables. Do you mean the ones that yt defines, or the ones that are in the field list? It looks like "velocity_z" works for one call and doesn't for another. 

Are you able to do anything else with "specific_scalar[0]"? Like make a slice, or a projection of some sort? Or maybe take the weighted average of it in a sphere. I'd be interested in seeing if the problem was restricted to phase plotting. 

How big is the dataset, by the way? Is there something that fails in the same way that the both of us can test on? 

Best,

John


On Tue, Dec 18, 2012 at 10:53 AM, Stephanie Tonnesen <stonnes@gmail.com> wrote:

Hi,

Yup.  Just to be sure I updated again and reattempted to make the phase_object and python was Killed again.

Thanks!

Stephanie



On Tue, Dec 18, 2012 at 12:35 AM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Stephanie,

Just want to make sure that you're using the latest version of the yt development tree--it contains some important Athena fixes that were put in a few weeks back.

If so, then we'll work from there to figure out what's gone wrong. 

Best,

John

On Dec 17, 2012, at 5:44 PM, Stephanie Tonnesen <stonnes@gmail.com> wrote:

Hi yt-users,

I am trying to make phase_objects of Athena data.  I am able to make phase_objects if I use the "expected" set of variables (will show below), but kill my kernel or am kicked out of python if I try and use an "unknown field" read in from a set of vtk files. 



from yt.mods import *

pf = load("id0/rps.0080.vtk")

yt : [INFO     ] 2012-12-13 23:27:08,678 Temporarily setting domain_right_edge = -domain_left_edge. This will be corrected automatically if it is not the case.
yt : [INFO     ] 2012-12-13 23:27:08,974 Parameters: current_time              = 40.00096
yt : [INFO     ] 2012-12-13 23:27:08,974 Parameters: domain_dimensions         = [1048 1048  655]
yt : [INFO     ] 2012-12-13 23:27:08,975 Parameters: domain_left_edge          = [-3.2 -3.2 -2. ]
yt : [INFO     ] 2012-12-13 23:27:08,976 Parameters: domain_right_edge         = [ 3.2  3.2  2. ]
yt : [INFO     ] 2012-12-13 23:27:08,977 Parameters: cosmological_simulation   = 0.0
yt : [INFO     ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields

pf.h.field_list

['density',
 'pressure',
 'velocity_z',
 'velocity_x',
 'velocity_y',
 'specific_scalar[0]']
alld = pf.h.all_data()
tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25", "grid['z'] > 0.01"])

##These commands works:
radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")

pc = PlotCollectionInteractive(pf)

pc.add_phase_object(alld,["z","velocity_z","CellMass"],weight=None)

#The calls below fail.  There is no error message.  If I am in iPython,
#the kernel dies.  If I am running on a computer, it freezes and
#eventually I just get the message
#Killed

pc.add_phase_object(alld,["z","specific_scalar[0]","CellMass"],weight=None)
pc.add_phase_object(tracerp25,["z","velocity_z","CellMass"],weight=None)

#If I read in a single vtk file ("id122/rps-id122.0080.vtk") the above
#commands work
Any help would be much appreciated!  Thanks,

Stephanie
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org



_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org




--
John ZuHone

Postdoctoral Researcher
NASA/Goddard Space Flight Center

_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org