[Numpy-discussion] cPickle.loads and Numeric

Julian Taylor jtaylor.debian at googlemail.com
Tue Feb 25 11:57:29 EST 2014

On Tue, Feb 25, 2014 at 5:41 PM, Alexander Belopolsky <ndarray at mac.com> wrote:
> On Tue, Feb 25, 2014 at 11:29 AM, Benjamin Root <ben.root at ou.edu> wrote:
>> I seem to recall reading somewhere that pickles are not intended to be
>> long-term archives as there is no guarantee that a pickle made in one
>> version of python would work in another version, much less between different
>> versions of the same (or similar) packages.
> That's not true about Python core and stdlib.  Python developers strive to
> maintain backward compatibility and any instance of newer python failing to
> read older pickles would be considered a bug.  This is even true across 2.x
> / 3.x line.
> You mileage with 3rd party packages, especially 10+ years old ones may vary.

The promise to keep compatibility does still not make it a good format
for long term storage. pickles are a processing format bound to one
specific tool and it is not trivial to read it with any other.
The same applies to HDF5, it may work well now but there is no
guarantee anyone will be able to read it in 50 years when we have
moved on to the next generation of data storage formats.

For long term storage simpler formats like FITS [0] are much more suitable.
Writing a basic FITS parser in any language is easy. But in return it
is not the best format for data processing.

[0] http://fits.gsfc.nasa.gov/fits_standard.html

More information about the NumPy-Discussion mailing list