[Numpy-discussion] saving groups of numpy arrays to disk

[David Warde-Farley]

On 2011-08-25, at 2:42 PM, Chris.Barker wrote:

> On 8/24/11 9:22 AM, Anthony Scopatz wrote:
>>    You can use Python pickling, if you do *not* have a requirement for:
> 
> I can't recall why, but it seem pickling of numpy arrays has been 
> fragile and not very performant.
> 
> I like the npy / npz format, built in to numpy, if you don't need:
> 
>>    - access from non-Python programs

While I'm not aware of reader implementations for any other language, NPY is a dirt-simple and well-documented format designed by Robert Kern, and should be readable without too much trouble from any language that supports binary I/O. The full spec is at

https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt

It should be especially trivial to read arrays of simple scalar numeric dtypes, but reading compound dtypes is also doable.

For NPZ, use a standard zip file reading library to access individual files in the archive, which are in .npy format (or just unzip it by hand first -- it's a normal .zip file with a special extension).

David