[AstroPy] new pyfits version deletes NP_pyfits, breaking pickle

[Joe Harrington]

Erik, thanks for the unpickler code.  We saw the note about those but
don't offhand know how to use it to fix the problem.

Perry, thanks for looking into it.

Also, if you know of *any* other way to save an object, please say.
It seems pretty clear to me that to load objects from any kind of save
file, you have to import the classes of the object and any objects it
contains that are not standard Python objects.  So even if we had
other methods for saving, they would have the same problem as pickle.
But we have to be able to save objects!  Perhaps saving the definitions
of the types rather than importing them would be the way to go?  I bet
there's a long thread about this somewhere.

--jh--

On Thu, 11 Nov 2010 14:38:38 -0500, Perry Greenfield <perry at stsci.edu> wrote:

> We'll look into it. This is a general problem with pickles (and one  
> reason I've been hesitant to avoid using them like save files). I  
> wonder if there is a better solution than that. In this case we had to  
> clean out the previous numarray interface.
> 
> Perry
> 
> On Nov 10, 2010, at 7:24 PM, Joe Harrington wrote:
> 
> > My research group uses Python pickles to save data as it goes through
> > our pipeline (.npy and .npz do not save objects, and neither does HDF,
> > etc.).  These need to be loadable forever, as we often compare work to
> > work done much earlier.  Some of the objects we save contain pyfits
> > header objects.  Pickles have to import all classes used in the
> > pickled objects before they load, and we are getting an ImportError
> > about NP_pyfits.  The file NP_pyfits.py existed in stsci_python 2.8
> > but is gone in 2.10.  The pickles refer to this object explicitly:
> >
> > ....
> > sS'photchan'
> > p494
> > I3
> > sS'header'
> > p495
> > (ipyfits.NP_pyfits
> > Header
> > p496
> > (dp497
> > S'_hdutype'
> > p498
> > cpyfits.NP_pyfits
> > PrimaryHDU
> > p499
> > sS'ascard'
> > p500
> > ccopy_reg
> > _reconstructor
> > p501
> > (cpyfits.NP_pyfits
> > CardList
> > p502
> > c__builtin__
> > list
> > p503
> > (lp504
> > g501
> > (cpyfits.NP_pyfits
> > Card
> > p505
> > c__builtin__
> > object
> > p506
> > NtRp507
> > (dp508
> > S'_valuestring'
> > p509
> > S'T'
> > ....
> >
> > Is there any way to make our pickles readable again, other than
> > running the old version of pyfits forever?  Can you provide a pickle
> > converter that replaces the old names in the file with whatever is
> > new?
> >
> > Please (everyone, not just STScI) be aware of this issue going
> > forward.  Pickles are the only way we know of to save objects.  You
> > can add things to your classes, but if you change what they import (or
> > otherwise break pickle), nobody can restore your class across
> > releases.
> >
> > Thanks,
> >
> > --jh--