[AstroPy] new pyfits version deletes NP_pyfits, breaking pickle

Joe Harrington jh at physics.ucf.edu
Fri Nov 12 12:22:11 EST 2010

> A more reliable pickling system works by making a common format for
> specifying objects, coupled with a custom pickler/unpickler for each
> data type that might be stored.

Mark, Python's objects are mutable, and this is a *good* thing.  We
can do much more with mutable objects as we develop a new analysis
than we ever could with IDL's awful static objects, and this was a big
motivator for me to switch from IDL to Python.  A user can add to an
object just by uttering the name of a new attribute and assigning
something to it.  They then carry that "basket" of associated
variables through an analysis in an organized fashion, with one
handle.  They can add to it, save it, load it, copy it, pass it to
functions, and so on, without having to track and separately treat
hundreds of minor variables separately.  It's the whole point of
using OOP for interactive data analysis.

But this makes writing a new pickler/unpickler for "each data type
that must be stored" (which is all of them, when saving a session) an
impossible solution.  Every possible mutation that a user might make
would need its own pickler/unpickler, and a general save/load-session
command would be impossible to write.  So, I think the pickle authors
realized that and did the best they could.

So, it's incumbent on software developers to realize that anything
pickle needs is part of their public interface, like it or not.  Even
if you provide your own pickler/unpickler, that doesn't cover the case
where one of your objects is an attribute of a higher-level object,
since pickle won't know where to find it when it encounters that
object.  Or actually it does.  It's in your import.  Oops, that's the
current problem, when the import changes.

Perhaps a possible pickle PEP might allow you to register your
pickler/unpickler when you define an object, so that this could be
part of the solution.  But then the unpickler would have to be saved
in the pickle to ensure that it will be available.


More information about the AstroPy mailing list