Hi Britton, Charles, Britton is correct, YT has its own routine for pickling. This is a couple step process, because specifically the pickling of parameter files is somewhat ill-defined as a problem. When you initialize a parameter file, it's given a (likely) unique hash that corresponds to its location, some of its contents, last modified time (or in Enzo's case, the CurrentTimeIdentifier in the parameter file) and this is stored in ~/.yt/parameter_files.csv. By this mechanism, one can store the pointer to a parameter file that exists on disk with a bidirectional hash. When a new parameter file is opened, it checks to see if the path location in the .csv file needs to be updated, and if so, it does. So when pickling a YT object, it stores the minimum set of information necessary to recreate that object. For a sphere, region, etc etc, this is really just the necessary arguments to recreate the empty object, along with the hash of the parameter file that it belongs to. So the actual information necessary to pickle simple objects is very small: hash, arguments. (Called "_con_args" in the source, for Constructor Arguments.) When this is unpickled, the parameter file is recreated (or grabbed from in memory, if it has already been instantiated) and then the object is recreated. For more complicated objects, like Clumps and ExtractedSets, we have to store more information -- because the constructor arguments are much larger and more complicated. But the basic idea is the same. When using save_object, the same pickling routine gets called, but instead of being loosely affiliated and put into a separate file, it gets stored as a dataset in a the corresponding .yt file, which is itself an HDF5 file. So the results should be the same, except save_object and load_object remove the extra file from the equation. The error you're seeing above seems to be related to an incompletely saved or corrupted file; can you replicate this in a very simple, quick fashion? I would suggest getting a clump that is very, very small, taken from a small data region, and seeing if that will pickle and unpickle correctly. If it does, then we know that perhaps the previous pickle file was corrupted, or that something is wrong with the yt pickling protocol. If it doesn't work, then we *know* something is wrong with the yt pickling protocol. Best, Matt On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Chris,
YT has its own routine designed to pickle data objects for saving and reloading. For a given pf, you can do pf.h.save_object(object, name) and the object will be saved to the .yt file associated with the dataset. You can then get it back with object = pf.h.load_object(name). I have used this successfully to save the mast_clump structure from clump finding.
Britton
On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen@astro.berkeley.edu> wrote:
The find_clumps function is working correctly for me I believe, but the analysis takes a very long time and I'd like to save the master_clump object (from http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for further analysis. I've tried pickling master_clump. It dumps correctly, but gives an EOF error (below) on loading. Is there another way to save master_clump?
Thanks, Charles
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file) 1368 1369 def load(file): -> 1370 return Unpickler(file).load() 1371 1372 def loads(str):
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self) 856 while 1: 857 key = read(1) --> 858 dispatch[key](self) 859 except _Stop, stopinst: 860 return stopinst.value
/nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self) 878 879 def load_eof(self): --> 880 raise EOFError 881 dispatch[''] = load_eof 882
EOFError:
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org