[AstroPy] new pyfits version deletes NP_pyfits, breaking pickle

Erik Tollerud erik.tollerud at gmail.com
Thu Nov 11 14:43:45 EST 2010

The pickle module supports customized unpicklers that let you control
how modules and classes are loaded.  Have a look at
http://docs.python.org/library/pickle.html#subclassing-unpicklers for
details.  If you're using cPickle, the snippet below should fix it.
It probably would make sense to have this in pyfits, though...

from cPickle import Unpickler
import imp,sys

def fg(modname,classname):
    if 'NP_pyfits' in modname:
        ... do whatever is needed to get the correct pyfits module
after whatever changes were done...
        if modname in sys.modules:
            mod = sys.modules[modname]
            fp, pathname, description = imp.find_module(modname)
                mod = imp.load_module(modname, fp, pathname, description)
        return getattr(mod,modname)

with open(filename) as f:
    u = Unpickler(f)
    u.find_globals = fg
    obj = u.load()
    (load however many objects you need...)

On Wed, Nov 10, 2010 at 4:24 PM, Joe Harrington <jh at physics.ucf.edu> wrote:
> My research group uses Python pickles to save data as it goes through
> our pipeline (.npy and .npz do not save objects, and neither does HDF,
> etc.).  These need to be loadable forever, as we often compare work to
> work done much earlier.  Some of the objects we save contain pyfits
> header objects.  Pickles have to import all classes used in the
> pickled objects before they load, and we are getting an ImportError
> about NP_pyfits.  The file NP_pyfits.py existed in stsci_python 2.8
> but is gone in 2.10.  The pickles refer to this object explicitly:
> ....
> sS'photchan'
> p494
> I3
> sS'header'
> p495
> (ipyfits.NP_pyfits
> Header
> p496
> (dp497
> S'_hdutype'
> p498
> cpyfits.NP_pyfits
> PrimaryHDU
> p499
> sS'ascard'
> p500
> ccopy_reg
> _reconstructor
> p501
> (cpyfits.NP_pyfits
> CardList
> p502
> c__builtin__
> list
> p503
> (lp504
> g501
> (cpyfits.NP_pyfits
> Card
> p505
> c__builtin__
> object
> p506
> NtRp507
> (dp508
> S'_valuestring'
> p509
> S'T'
> ....
> Is there any way to make our pickles readable again, other than
> running the old version of pyfits forever?  Can you provide a pickle
> converter that replaces the old names in the file with whatever is
> new?
> Please (everyone, not just STScI) be aware of this issue going
> forward.  Pickles are the only way we know of to save objects.  You
> can add things to your classes, but if you change what they import (or
> otherwise break pickle), nobody can restore your class across
> releases.
> Thanks,
> --jh--
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

Erik Tollerud

More information about the AstroPy mailing list