[Numpy-discussion] ENH IncrementalWriter for .npy files

Gabor Kovacs kgabor79 at gmail.com
Mon Sep 1 11:23:20 EDT 2014

Dear All,

I would like to add a class for writing one (possibly big) .npy file
saving multiple (same dtype, compatible shape) arrays. My use case was
the saving of slowly accumulating data regularly for a long time into
one file.

Please find a first implementation under
https://github.com/numpy/numpy/pull/4987 . It currently supports
writing a new file only and only in C order in the file. Opening an
existing file for append and reading back parts from a very big .npy
file would be straightforward next steps for a full featured class.

The .npy file format is only affected by leaving some extra space for
re-writing the header later with a possibly bigger "shape" field,
respecting the 16-byte alignment.

with np.IncrementalWriter("testfile.npy",hdrupdate=True,flush=True) as W:

Feel free to comment this idea.


More information about the NumPy-Discussion mailing list