[AstroPy] new pyfits version deletes NP_pyfits, breaking pickle

Stefan Schwarzburg stefan.schwarzburg at googlemail.com
Fri Nov 12 04:29:53 EST 2010


Dear Joe,

I know that you did not ask for this advice, but I feel the need to share
this with you anyway.
I've seen projects in the past that had similar problems like you describe
(although they did not use python but root or R), and I think there is only
one real solution to this:
You should try to separate the data and logic and store the data in a
standard file format that will be readable in the far future. In astronomy
this is the FITS format. Of cause it is not able to do a lot of the cool
things you can do with pickled objects and it has some strange restrictions
that come from the time when FITS was supposed to be saved onto tapes, but
this is actually its strenght. Just as you are able to open FITS files that
were saved 20 years ago, you will be able to open these files in 20 years.

Probably it will not be the same software (I guess there is a reason why you
chose python and not algol or fortran77) but the data will still be there.

With pickled objects you will definitely get the same problems over and over
in the future. Pickled objects are great, because you can send them over a
network and so on, but they are not a way to archive data.

FITS files are good enough for all data of all astronomy projects I've seen
so far, although you sometimes have to give up a certain style of thinking,
but in the end you are always able to save what you need in images, tables
and meta-data in the headers.

As I said, this was not what you asked for, but I think I needed to tell you
that large astronomy collaborations started with something similar like your
setup and in the end had to learn the hard way that this does not work and
ties them to old software that they would like to give up (and will switch
to FITS in the future...).

Best regards,
Stefan

-- 
Institut für Astronomie und Astrophysik
Eberhard Karls Universität Tübingen
Sand 1   -  D-72076 Tübingen
-----------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20101112/4925bfc9/attachment.html>


More information about the AstroPy mailing list