I'd like to make a derived field that is the gradient of a primitive field. Is there an example of how I can do this? I'm not sure how to handle ghost cells for something like this. I spoke with Jeff and he indicated that there was a vorticity example somewhere, but I cannot find it. Thanks.
The best place to look is in yt/lagos/UniversalFields.py, and look at the DivV example. That takes one ghost zone on the 3 velocity components. I've pasted it below for convenience The HydroMethod business at the top is to switch between the cell centered PPM (HydroMethod==0) or the staggered face-centered Zeus data (HydroMethod == 2) The object that 'data' then returns is larger than the grid patch it represents in each direction by the number passed to ValidateSpatial. ValidateSpatial gets passed to add_field with the Validators argument, and expects the number of ghost zones and the list of fields to get ghost zones for. Hope that helps, d. def _DivV(field, data): # We need to set up stencils if data.pf["HydroMethod"] == 2: sl_left = slice(None,-2,None) sl_right = slice(1,-1,None) div_fac = 1.0 else: sl_left = slice(None,-2,None) sl_right = slice(2,None,None) div_fac = 2.0 ds = div_fac * data['dx'].flat[0] f = data["x-velocity"][sl_right,1:-1,1:-1]/ds f -= data["x-velocity"][sl_left ,1:-1,1:-1]/ds ds = div_fac * data['dy'].flat[0] f += data["y-velocity"][1:-1,sl_right,1:-1]/ds f -= data["y-velocity"][1:-1,sl_left ,1:-1]/ds ds = div_fac * data['dz'].flat[0] f += data["z-velocity"][1:-1,1:-1,sl_right]/ds f -= data["z-velocity"][1:-1,1:-1,sl_left ]/ds new_field = na.zeros(data["x-velocity"].shape, dtype='float64') new_field[1:-1,1:-1,1:-1] = f return new_field def _convertDivV(data): return data.convert("cm")**-1.0 add_field("DivV", function=_DivV, validators=[ValidateSpatial(1, ["x-velocity","y-velocity","z-velocity"])], units=r"\rm{s}^{-1}", take_log=False, convert_function=_convertDivV) On Thu, May 27, 2010 at 3:48 PM, Andrew Cunningham <ajc4@pas.rochester.edu> wrote:
I'd like to make a derived field that is the gradient of a primitive field. Is there an example of how I can do this? I'm not sure how to handle ghost cells for something like this.
I spoke with Jeff and he indicated that there was a vorticity example somewhere, but I cannot find it.
Thanks.
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Sent from my Stone Tablet and carried by my Pterodactyl.
Thanks, David. That does help. On Thu, 27 May 2010, David Collins wrote:
The best place to look is in yt/lagos/UniversalFields.py, and look at the DivV example. That takes one ghost zone on the 3 velocity components. I've pasted it below for convenience
The HydroMethod business at the top is to switch between the cell centered PPM (HydroMethod==0) or the staggered face-centered Zeus data (HydroMethod == 2)
The object that 'data' then returns is larger than the grid patch it represents in each direction by the number passed to ValidateSpatial. ValidateSpatial gets passed to add_field with the Validators argument, and expects the number of ghost zones and the list of fields to get ghost zones for.
Hope that helps, d.
def _DivV(field, data): # We need to set up stencils if data.pf["HydroMethod"] == 2: sl_left = slice(None,-2,None) sl_right = slice(1,-1,None) div_fac = 1.0 else: sl_left = slice(None,-2,None) sl_right = slice(2,None,None) div_fac = 2.0 ds = div_fac * data['dx'].flat[0] f = data["x-velocity"][sl_right,1:-1,1:-1]/ds f -= data["x-velocity"][sl_left ,1:-1,1:-1]/ds ds = div_fac * data['dy'].flat[0] f += data["y-velocity"][1:-1,sl_right,1:-1]/ds f -= data["y-velocity"][1:-1,sl_left ,1:-1]/ds ds = div_fac * data['dz'].flat[0] f += data["z-velocity"][1:-1,1:-1,sl_right]/ds f -= data["z-velocity"][1:-1,1:-1,sl_left ]/ds new_field = na.zeros(data["x-velocity"].shape, dtype='float64') new_field[1:-1,1:-1,1:-1] = f return new_field def _convertDivV(data): return data.convert("cm")**-1.0 add_field("DivV", function=_DivV, validators=[ValidateSpatial(1, ["x-velocity","y-velocity","z-velocity"])], units=r"\rm{s}^{-1}", take_log=False, convert_function=_convertDivV)
On Thu, May 27, 2010 at 3:48 PM, Andrew Cunningham <ajc4@pas.rochester.edu> wrote:
I'd like to make a derived field that is the gradient of a primitive field. Is there an example of how I can do this? I'm not sure how to handle ghost cells for something like this.
I spoke with Jeff and he indicated that there was a vorticity example somewhere, but I cannot find it.
Thanks.
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Sent from my Stone Tablet and carried by my Pterodactyl. _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
By default, Orion data defaults to 'orion', which is equivalent to PPM DE in enzo speak. j On Thu, May 27, 2010 at 4:12 PM, Andrew Cunningham <ajc4@pas.rochester.edu> wrote:
Thanks, David. That does help.
On Thu, 27 May 2010, David Collins wrote:
The best place to look is in yt/lagos/UniversalFields.py, and look at the DivV example. That takes one ghost zone on the 3 velocity components. I've pasted it below for convenience
The HydroMethod business at the top is to switch between the cell centered PPM (HydroMethod==0) or the staggered face-centered Zeus data (HydroMethod == 2)
The object that 'data' then returns is larger than the grid patch it represents in each direction by the number passed to ValidateSpatial. ValidateSpatial gets passed to add_field with the Validators argument, and expects the number of ghost zones and the list of fields to get ghost zones for.
Hope that helps, d.
def _DivV(field, data): # We need to set up stencils if data.pf["HydroMethod"] == 2: sl_left = slice(None,-2,None) sl_right = slice(1,-1,None) div_fac = 1.0 else: sl_left = slice(None,-2,None) sl_right = slice(2,None,None) div_fac = 2.0 ds = div_fac * data['dx'].flat[0] f = data["x-velocity"][sl_right,1:-1,1:-1]/ds f -= data["x-velocity"][sl_left ,1:-1,1:-1]/ds ds = div_fac * data['dy'].flat[0] f += data["y-velocity"][1:-1,sl_right,1:-1]/ds f -= data["y-velocity"][1:-1,sl_left ,1:-1]/ds ds = div_fac * data['dz'].flat[0] f += data["z-velocity"][1:-1,1:-1,sl_right]/ds f -= data["z-velocity"][1:-1,1:-1,sl_left ]/ds new_field = na.zeros(data["x-velocity"].shape, dtype='float64') new_field[1:-1,1:-1,1:-1] = f return new_field def _convertDivV(data): return data.convert("cm")**-1.0 add_field("DivV", function=_DivV, validators=[ValidateSpatial(1, ["x-velocity","y-velocity","z-velocity"])], units=r"\rm{s}^{-1}", take_log=False, convert_function=_convertDivV)
On Thu, May 27, 2010 at 3:48 PM, Andrew Cunningham <ajc4@pas.rochester.edu> wrote:
I'd like to make a derived field that is the gradient of a primitive field. Is there an example of how I can do this? I'm not sure how to handle ghost cells for something like this.
I spoke with Jeff and he indicated that there was a vorticity example somewhere, but I cannot find it.
Thanks.
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Sent from my Stone Tablet and carried by my Pterodactyl. _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- ---------------- i am dot org: www.jsoishi.org
Hi David, Andrew, As a quick note, the next version of the docs (in preparation, coming very soon) will have all the fields with source code. But for now you can call get_source() on DerivedField objects:
pf = load("RD0005-mine/RedshiftOutput0005") pf.field_info <yt.lagos.EnzoFields.EnzoFieldContainer object at 0x103d02ed0> print pf.field_info["DivV"].get_source() def _DivV(field, data): # We need to set up stencils if data.pf["HydroMethod"] == 2: sl_left = slice(None,-2,None) sl_right = slice(1,-1,None) div_fac = 1.0 else: sl_left = slice(None,-2,None) sl_right = slice(2,None,None) div_fac = 2.0 ds = div_fac * data['dx'].flat[0] f = data["x-velocity"][sl_right,1:-1,1:-1]/ds f -= data["x-velocity"][sl_left ,1:-1,1:-1]/ds ds = div_fac * data['dy'].flat[0] f += data["y-velocity"][1:-1,sl_right,1:-1]/ds f -= data["y-velocity"][1:-1,sl_left ,1:-1]/ds ds = div_fac * data['dz'].flat[0] f += data["z-velocity"][1:-1,1:-1,sl_right]/ds f -= data["z-velocity"][1:-1,1:-1,sl_left ]/ds new_field = na.zeros(data["x-velocity"].shape, dtype='float64') new_field[1:-1,1:-1,1:-1] = f return new_field
I thought there was an example of how to do a spatially-defined derived field, but I looked in the docs and I don't see it. I'll add this as well. -Matt On Thu, May 27, 2010 at 4:12 PM, Andrew Cunningham <ajc4@pas.rochester.edu> wrote:
Thanks, David. That does help.
On Thu, 27 May 2010, David Collins wrote:
The best place to look is in yt/lagos/UniversalFields.py, and look at the DivV example. That takes one ghost zone on the 3 velocity components. I've pasted it below for convenience
The HydroMethod business at the top is to switch between the cell centered PPM (HydroMethod==0) or the staggered face-centered Zeus data (HydroMethod == 2)
The object that 'data' then returns is larger than the grid patch it represents in each direction by the number passed to ValidateSpatial. ValidateSpatial gets passed to add_field with the Validators argument, and expects the number of ghost zones and the list of fields to get ghost zones for.
Hope that helps, d.
def _DivV(field, data): # We need to set up stencils if data.pf["HydroMethod"] == 2: sl_left = slice(None,-2,None) sl_right = slice(1,-1,None) div_fac = 1.0 else: sl_left = slice(None,-2,None) sl_right = slice(2,None,None) div_fac = 2.0 ds = div_fac * data['dx'].flat[0] f = data["x-velocity"][sl_right,1:-1,1:-1]/ds f -= data["x-velocity"][sl_left ,1:-1,1:-1]/ds ds = div_fac * data['dy'].flat[0] f += data["y-velocity"][1:-1,sl_right,1:-1]/ds f -= data["y-velocity"][1:-1,sl_left ,1:-1]/ds ds = div_fac * data['dz'].flat[0] f += data["z-velocity"][1:-1,1:-1,sl_right]/ds f -= data["z-velocity"][1:-1,1:-1,sl_left ]/ds new_field = na.zeros(data["x-velocity"].shape, dtype='float64') new_field[1:-1,1:-1,1:-1] = f return new_field def _convertDivV(data): return data.convert("cm")**-1.0 add_field("DivV", function=_DivV, validators=[ValidateSpatial(1, ["x-velocity","y-velocity","z-velocity"])], units=r"\rm{s}^{-1}", take_log=False, convert_function=_convertDivV)
On Thu, May 27, 2010 at 3:48 PM, Andrew Cunningham <ajc4@pas.rochester.edu> wrote:
I'd like to make a derived field that is the gradient of a primitive field. Is there an example of how I can do this? I'm not sure how to handle ghost cells for something like this.
I spoke with Jeff and he indicated that there was a vorticity example somewhere, but I cannot find it.
Thanks.
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Sent from my Stone Tablet and carried by my Pterodactyl. _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Related to this, I'd also like to query the group for thoughts the best approach to a specific kind of analysis I have in mind with this. I'd like to know what the azimuthally averaged radiation flux is at some fixed distance r from the star plotted as a function of polar angle. My plan is to create a radiation flux derived quantity (which is why I need a derivative), then create a flux dot rhat quantity and then extract to a fixed interpolated grid and then proceed with the extracting, averaging, plotting. Does that sound like the correct approach or is there some yt functionality that I don't yet know about that might be more elegant? - ie) some way to extract the field on a spherical count our and return that as a list of values and positions.
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump? Thanks, Charles /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str): /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882 EOFError:
Hi Chris, YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding. Britton On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu>wrote:
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump?
Thanks, Charles
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882
EOFError:
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Britton, Charles, Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does. So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated. For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same. When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation. The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol. Best, Matt On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Chris,
YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding.
Britton
On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump?
Thanks, Charles
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882
EOFError:
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
pf.h.save_object does work as advertised for me. Thanks for the pointer. For what it is worth, clumps do not have the attribute save_object, so masterclump.save_object is not a valid function. I tried pickling again and I can no longer even dump the clumps. I don't know why the dumping 'worked' before and not now, though I have been trying it on more complicated clumps than before. I pasted the error message if you are interested. In [6]: pickle.dump(masterclump, outfile) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) /nics/c/home/chansen/lib/yt/src/yt-trunk-svn/scripts/iyt in <module>() ----> 1 2 3 4 5 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(obj, file, protocol) 1360 1361 def dump(obj, file, protocol=None): -> 1362 Pickler(file, protocol).dump(obj) 1363 1364 def dumps(obj, protocol=None): /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(self, obj) 222 if self.proto >= 2: 223 self.write(PROTO + chr(self.proto)) --> 224 self.save(obj) 225 self.write(STOP) 226 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object --> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj): /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo: /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object --> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj): /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo: /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object --> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj): /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo: /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288 /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object --> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj): /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 403 404 if obj is not None: --> 405 self.memoize(obj) 406 407 # More new special cases (that work with older protocols as /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in memoize(self, obj) 242 if self.fast: 243 return --> 244 assert id(obj) not in self.memo 245 memo_len = len(self.memo) 246 self.write(self.put(memo_len)) AssertionError: Matthew Turk wrote:
Hi Britton, Charles,
Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does.
So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated.
For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same.
When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation.
The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol.
Best,
Matt
On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Chris,
YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding.
Britton
On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump?
Thanks, Charles
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882
EOFError:
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Charles, What's "outfile" in this case? Can you send a bit more of your setup of the problem, or at least the output of "type(outfile)"? -Matt On Fri, May 28, 2010 at 11:32 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
pf.h.save_object does work as advertised for me. Thanks for the pointer. For what it is worth, clumps do not have the attribute save_object, so masterclump.save_object is not a valid function.
I tried pickling again and I can no longer even dump the clumps. I don't know why the dumping 'worked' before and not now, though I have been trying it on more complicated clumps than before. I pasted the error message if you are interested.
In [6]: pickle.dump(masterclump, outfile) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last)
/nics/c/home/chansen/lib/yt/src/yt-trunk-svn/scripts/iyt in <module>() ----> 1 2 3 4 5
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(obj, file, protocol) 1360 1361 def dump(obj, file, protocol=None): -> 1362 Pickler(file, protocol).dump(obj) 1363 1364 def dumps(obj, protocol=None):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(self, obj) 222 if self.proto >= 2: 223 self.write(PROTO + chr(self.proto)) --> 224 self.save(obj) 225 self.write(STOP) 226
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 403 404 if obj is not None: --> 405 self.memoize(obj) 406 407 # More new special cases (that work with older protocols as
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in memoize(self, obj) 242 if self.fast: 243 return --> 244 assert id(obj) not in self.memo 245 memo_len = len(self.memo) 246 self.write(self.put(memo_len))
AssertionError:
Matthew Turk wrote:
Hi Britton, Charles,
Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does.
So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated.
For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same.
When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation.
The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol.
Best,
Matt
On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Chris,
YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding.
Britton
On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump?
Thanks, Charles
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882
EOFError:
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
sorry, the preceding line was outfile = open("dummy.pkl", "wb") The type is... In [73]: type(outfile) Out[73]: <type 'file'> Matthew Turk wrote:
Hi Charles,
What's "outfile" in this case? Can you send a bit more of your setup of the problem, or at least the output of "type(outfile)"?
-Matt
On Fri, May 28, 2010 at 11:32 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
pf.h.save_object does work as advertised for me. Thanks for the pointer. For what it is worth, clumps do not have the attribute save_object, so masterclump.save_object is not a valid function.
I tried pickling again and I can no longer even dump the clumps. I don't know why the dumping 'worked' before and not now, though I have been trying it on more complicated clumps than before. I pasted the error message if you are interested.
In [6]: pickle.dump(masterclump, outfile) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last)
/nics/c/home/chansen/lib/yt/src/yt-trunk-svn/scripts/iyt in <module>() ----> 1 2 3 4 5
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(obj, file, protocol) 1360 1361 def dump(obj, file, protocol=None): -> 1362 Pickler(file, protocol).dump(obj) 1363 1364 def dumps(obj, protocol=None):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(self, obj) 222 if self.proto >= 2: 223 self.write(PROTO + chr(self.proto)) --> 224 self.save(obj) 225 self.write(STOP) 226
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 403 404 if obj is not None: --> 405 self.memoize(obj) 406 407 # More new special cases (that work with older protocols as
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in memoize(self, obj) 242 if self.fast: 243 return --> 244 assert id(obj) not in self.memo 245 memo_len = len(self.memo) 246 self.write(self.put(memo_len))
AssertionError:
Matthew Turk wrote:
Hi Britton, Charles,
Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does.
So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated.
For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same.
When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation.
The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol.
Best,
Matt
On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Chris,
YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding.
Britton
On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump?
Thanks, Charles
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882
EOFError:
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Charles, This feels a bit more like a file system errorl. What happens if you use the cPickle module instead of the Pickle module? -Matt On Fri, May 28, 2010 at 11:46 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
sorry, the preceding line was
outfile = open("dummy.pkl", "wb")
The type is...
In [73]: type(outfile) Out[73]: <type 'file'>
Matthew Turk wrote:
Hi Charles,
What's "outfile" in this case? Can you send a bit more of your setup of the problem, or at least the output of "type(outfile)"?
-Matt
On Fri, May 28, 2010 at 11:32 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
pf.h.save_object does work as advertised for me. Thanks for the pointer. For what it is worth, clumps do not have the attribute save_object, so masterclump.save_object is not a valid function.
I tried pickling again and I can no longer even dump the clumps. I don't know why the dumping 'worked' before and not now, though I have been trying it on more complicated clumps than before. I pasted the error message if you are interested.
In [6]: pickle.dump(masterclump, outfile)
--------------------------------------------------------------------------- AssertionError Traceback (most recent call last)
/nics/c/home/chansen/lib/yt/src/yt-trunk-svn/scripts/iyt in <module>() ----> 1 2 3 4 5
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(obj, file, protocol) 1360 1361 def dump(obj, file, protocol=None): -> 1362 Pickler(file, protocol).dump(obj) 1363 1364 def dumps(obj, protocol=None):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(self, obj) 222 if self.proto >= 2: 223 self.write(PROTO + chr(self.proto)) --> 224 self.save(obj) 225 self.write(STOP) 226
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 403 404 if obj is not None: --> 405 self.memoize(obj) 406 407 # More new special cases (that work with older protocols as
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in memoize(self, obj) 242 if self.fast: 243 return --> 244 assert id(obj) not in self.memo 245 memo_len = len(self.memo) 246 self.write(self.put(memo_len))
AssertionError:
Matthew Turk wrote:
Hi Britton, Charles,
Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does.
So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated.
For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same.
When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation.
The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol.
Best,
Matt
On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Chris,
YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding.
Britton
On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from
http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump?
Thanks, Charles
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882
EOFError:
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi, Charles-- I save my clumps directly with cPickle, and it works fine. So, in principle, at least, it's possible. I do:
cPickle.dump(SomeClump,file('ctest.pickle','wb'),protocol=-1) c = cPickle.load(file('ctest.pickle','rb'))
Using protocol=-1 stores the data in binary, so it's faster on both read and write, and smaller by quite a lot. Have you reloaded anything since you created the object? Pickle tends to choke if you've reloaded the module since you've created the object. d. On Fri, May 28, 2010 at 11:49 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Charles,
This feels a bit more like a file system errorl. What happens if you use the cPickle module instead of the Pickle module?
-Matt
On Fri, May 28, 2010 at 11:46 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
sorry, the preceding line was
outfile = open("dummy.pkl", "wb")
The type is...
In [73]: type(outfile) Out[73]: <type 'file'>
Matthew Turk wrote:
Hi Charles,
What's "outfile" in this case? Can you send a bit more of your setup of the problem, or at least the output of "type(outfile)"?
-Matt
On Fri, May 28, 2010 at 11:32 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
pf.h.save_object does work as advertised for me. Thanks for the pointer. For what it is worth, clumps do not have the attribute save_object, so masterclump.save_object is not a valid function.
I tried pickling again and I can no longer even dump the clumps. I don't know why the dumping 'worked' before and not now, though I have been trying it on more complicated clumps than before. I pasted the error message if you are interested.
In [6]: pickle.dump(masterclump, outfile)
--------------------------------------------------------------------------- AssertionError Traceback (most recent call last)
/nics/c/home/chansen/lib/yt/src/yt-trunk-svn/scripts/iyt in <module>() ----> 1 2 3 4 5
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(obj, file, protocol) 1360 1361 def dump(obj, file, protocol=None): -> 1362 Pickler(file, protocol).dump(obj) 1363 1364 def dumps(obj, protocol=None):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(self, obj) 222 if self.proto >= 2: 223 self.write(PROTO + chr(self.proto)) --> 224 self.save(obj) 225 self.write(STOP) 226
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 403 404 if obj is not None: --> 405 self.memoize(obj) 406 407 # More new special cases (that work with older protocols as
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in memoize(self, obj) 242 if self.fast: 243 return --> 244 assert id(obj) not in self.memo 245 memo_len = len(self.memo) 246 self.write(self.put(memo_len))
AssertionError:
Matthew Turk wrote:
Hi Britton, Charles,
Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does.
So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated.
For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same.
When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation.
The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol.
Best,
Matt
On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Chris,
YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding.
Britton
On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
> > The find_clumps function is working correctly for me I believe, but > the > analysis takes a very long time and I'd like to save the master_clump > object > (from > > http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) > for > further analysis. I've tried pickling master_clump. It dumps > correctly, > but gives an EOF error (below) on loading. Is there another way to > save > master_clump? > > Thanks, > Charles > > /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) > 1368 > 1369 def load(file): > -> 1370 return Unpickler(file).load() > 1371 > 1372 def loads(str): > > /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) > 856 while 1: > 857 key = read(1) > --> 858 dispatch[key](self) > 859 except _Stop, stopinst: > 860 return stopinst.value > > /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) > 878 > 879 def load_eof(self): > --> 880 raise EOFError > 881 dispatch[''] = load_eof > 882 > > EOFError: > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Sent from my Stone Tablet and carried by my Pterodactyl.
cPickle works. I should probably be using that instead of pickle anyway. I did reload the first time I tried to pickle, but not the second time. David Collins wrote:
Hi, Charles--
I save my clumps directly with cPickle, and it works fine. So, in principle, at least, it's possible. I do:
cPickle.dump(SomeClump,file('ctest.pickle','wb'),protocol=-1) c = cPickle.load(file('ctest.pickle','rb'))
Using protocol=-1 stores the data in binary, so it's faster on both read and write, and smaller by quite a lot.
Have you reloaded anything since you created the object? Pickle tends to choke if you've reloaded the module since you've created the object.
d.
On Fri, May 28, 2010 at 11:49 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Charles,
This feels a bit more like a file system errorl. What happens if you use the cPickle module instead of the Pickle module?
-Matt
On Fri, May 28, 2010 at 11:46 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
sorry, the preceding line was
outfile = open("dummy.pkl", "wb")
The type is...
In [73]: type(outfile) Out[73]: <type 'file'>
Matthew Turk wrote:
Hi Charles,
What's "outfile" in this case? Can you send a bit more of your setup of the problem, or at least the output of "type(outfile)"?
-Matt
On Fri, May 28, 2010 at 11:32 AM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
pf.h.save_object does work as advertised for me. Thanks for the pointer. For what it is worth, clumps do not have the attribute save_object, so masterclump.save_object is not a valid function.
I tried pickling again and I can no longer even dump the clumps. I don't know why the dumping 'worked' before and not now, though I have been trying it on more complicated clumps than before. I pasted the error message if you are interested.
In [6]: pickle.dump(masterclump, outfile)
--------------------------------------------------------------------------- AssertionError Traceback (most recent call last)
/nics/c/home/chansen/lib/yt/src/yt-trunk-svn/scripts/iyt in <module>() ----> 1 2 3 4 5
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(obj, file, protocol) 1360 1361 def dump(obj, file, protocol=None): -> 1362 Pickler(file, protocol).dump(obj) 1363 1364 def dumps(obj, protocol=None):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in dump(self, obj) 222 if self.proto >= 2: 223 self.write(PROTO + chr(self.proto)) --> 224 self.save(obj) 225 self.write(STOP) 226
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 399 else: 400 save(func) --> 401 save(args) 402 write(REDUCE) 403
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_tuple(self, obj) 560 write(MARK) 561 for element in obj: --> 562 save(element) 563 564 if id(obj) in memo:
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 284 f = self.dispatch.get(t) 285 if f: --> 286 f(self, obj) # Call unbound method with explicit self 287 return 288
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_list(self, obj) 598 599 self.memoize(obj) --> 600 self._batch_appends(iter(obj)) 601 602 dispatch[ListType] = save_list
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in _batch_appends(self, items) 613 if not self.bin: 614 for x in items: --> 615 save(x) 616 write(APPEND) 617 return
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save(self, obj) 329 330 # Save the reduce() output and finally memoize the object
--> 331 self.save_reduce(obj=obj, *rv) 332 333 def persistent_id(self, obj):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj) 403 404 if obj is not None: --> 405 self.memoize(obj) 406 407 # More new special cases (that work with older protocols as
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in memoize(self, obj) 242 if self.fast: 243 return --> 244 assert id(obj) not in self.memo 245 memo_len = len(self.memo) 246 self.write(self.put(memo_len))
AssertionError:
Matthew Turk wrote:
Hi Britton, Charles,
Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does.
So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated.
For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same.
When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation.
The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol.
Best,
Matt
On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
> Hi Chris, > > YT has its own routine designed to pickle data objects for saving and > reloading. For a given pf, you can do pf.h.save_object(object, name) > and > the object will be saved to the .yt file associated with the dataset. > You > can then get it back with object = pf.h.load_object(name). I have used > this > successfully to save the mast_clump structure from clump finding. > > Britton > > On Thu, May 27, 2010 at 9:26 PM, Charles Hansen > <chansen@astro.berkeley.edu> > wrote: > > > >> The find_clumps function is working correctly for me I believe, but >> the >> analysis takes a very long time and I'd like to save the master_clump >> object >> (from >> >> http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) >> for >> further analysis. I've tried pickling master_clump. It dumps >> correctly, >> but gives an EOF error (below) on loading. Is there another way to >> save >> master_clump? >> >> Thanks, >> Charles >> >> /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) >> 1368 >> 1369 def load(file): >> -> 1370 return Unpickler(file).load() >> 1371 >> 1372 def loads(str): >> >> /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) >> 856 while 1: >> 857 key = read(1) >> --> 858 dispatch[key](self) >> 859 except _Stop, stopinst: >> 860 return stopinst.value >> >> /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) >> 878 >> 879 def load_eof(self): >> --> 880 raise EOFError >> 881 dispatch[''] = load_eof >> 882 >> >> EOFError: >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >> > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > > > _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Is there a way to call add_projection on a plot collection without projecting through the entire simulation domain? I'd like to do projections on small sections, which should be much faster. Currently, the only way I know to make a small projection is pf = load(file) pc = PlotCollection(pf) p = pc.add_projection(field, center=x) p.set_width(something small) This wastes a lot of computational time projecting area I don't want to plot. Ideally, I'd like to be able to do something like pf = load(file) sp = pf.h.sphere(center, radius) pc = PlotCollection(pf) p = pc.add_projection(field, object=sp) I could probably put something together using the volume renderer if there aren't other options, but that doesn't sound very fast either. Charles
Hi Charles,
Ideally, I'd like to be able to do something like
pf = load(file) sp = pf.h.sphere(center, radius) pc = PlotCollection(pf) p = pc.add_projection(field, object=sp)
Instead of "object=sp", try "source=sp". That should do what you're looking for.
I could probably put something together using the volume renderer if there aren't other options, but that doesn't sound very fast either.
Results from the volume render will vary depending on the amount of work you ask it to do, but you can also spin it in arbitrary directions, which is nice. -Matt
Thanks Matt, that works. Matthew Turk wrote:
Hi Charles,
Ideally, I'd like to be able to do something like
pf = load(file) sp = pf.h.sphere(center, radius) pc = PlotCollection(pf) p = pc.add_projection(field, object=sp)
Instead of "object=sp", try "source=sp". That should do what you're looking for.
I could probably put something together using the volume renderer if there aren't other options, but that doesn't sound very fast either.
Results from the volume render will vary depending on the amount of work you ask it to do, but you can also spin it in arbitrary directions, which is nice.
-Matt _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Andrew,
Related to this, I'd also like to query the group for thoughts the best approach to a specific kind of analysis I have in mind with this.
I'd like to know what the azimuthally averaged radiation flux is at some fixed distance r from the star plotted as a function of polar angle.
My plan is to create a radiation flux derived quantity (which is why I need a derivative), then create a flux dot rhat quantity and then extract to a fixed interpolated grid and then proceed with the extracting, averaging, plotting.
I think that definitely generating the radiation flux in the manner you describe will work. I'm not sure that you need to extract to a fixed interpolated grid -- I think you should be able to generate the radiation flux inside a given cell, if you presuppose that the ghost zones are being generated correctly. You can manually verify that the ghost zones are looking like you expect them to if you look at the output of grid.retrieve_ghost_zones( ... ). This should perform a cascading interpolation that fills in the boundary values. More complicated calculations of the flux, through isocontours or the like, would be a bit harder but doable using the clump object. But I think for what you're looking at, this should do it.
Does that sound like the correct approach or is there some yt functionality that I don't yet know about that might be more elegant? - ie) some way to extract the field on a spherical count our and return that as a list of values and positions.
I believe that you should be able to use something like the field DiskAngle. If you look at the source, you can see that it grabs the field_parameter s height_vector and center, which you can set on an object with set_field_parameter. This then calculates a declination from that -- I think you may be able to use this as input to either a 1D or 2D profile, to get the average value of any arbitrary field, in this case your newly defined flux field versus the declination from the axis. Let me know if I'm crazy, or if this'll do what you're looking for! Best, Matt
participants (6)
-
Andrew Cunningham -
Britton Smith -
Charles Hansen -
David Collins -
j s oishi -
Matthew Turk