ENH IncrementalWriter for .npy files
Dear All, I would like to add a class for writing one (possibly big) .npy file saving multiple (same dtype, compatible shape) arrays. My use case was the saving of slowly accumulating data regularly for a long time into one file. Please find a first implementation under https://github.com/numpy/numpy/pull/4987 . It currently supports writing a new file only and only in C order in the file. Opening an existing file for append and reading back parts from a very big .npy file would be straightforward next steps for a full featured class. The .npy file format is only affected by leaving some extra space for re-writing the header later with a possibly bigger "shape" field, respecting the 16-byte alignment. Example: ``` A=np.array([[0,1,2,3,4,5,6,7],[8,9,10,11,12,13,14,15]]) with np.IncrementalWriter("testfile.npy",hdrupdate=True,flush=True) as W: W.save(A) W.save(A) ``` Feel free to comment this idea. Cheers, Gabor
Hi, This feature is very similar to what is available in hdf5 and exposed under h5py using chunks and max_size ... Cheers, -- Jérôme Kieffer tel +33 476 882 445
participants (2)
-
Gabor Kovacs -
Jerome Kieffer