[Numpy-discussion] ENH IncrementalWriter for .npy files
kgabor79 at gmail.com
Mon Sep 1 11:23:20 EDT 2014
I would like to add a class for writing one (possibly big) .npy file
saving multiple (same dtype, compatible shape) arrays. My use case was
the saving of slowly accumulating data regularly for a long time into
Please find a first implementation under
https://github.com/numpy/numpy/pull/4987 . It currently supports
writing a new file only and only in C order in the file. Opening an
existing file for append and reading back parts from a very big .npy
file would be straightforward next steps for a full featured class.
The .npy file format is only affected by leaving some extra space for
re-writing the header later with a possibly bigger "shape" field,
respecting the 16-byte alignment.
with np.IncrementalWriter("testfile.npy",hdrupdate=True,flush=True) as W:
Feel free to comment this idea.
More information about the NumPy-Discussion