https://github.com/numpy/numpy/pull/20321/
but I have been busy and did have not received any feedback recently. A more direct integration into Numpy would allow to skip or ease the header update part, e.g. by introducing a new file format version. This could turn .npy into a sort of binary CSV equivalent where the size of the array is determined by the file size.
Best, Michael
On 24. Aug 2022, at 03:04, Robert Kern <robert.kern@gmail.com> wrote:
_______________________________________________On Tue, Aug 23, 2022 at 8:47 PM <bross_phobrain@sonic.net> wrote:`item.tofile(fp)` is more efficient, but yes, that's the basic scheme. There is no footer after the data.I want to calc multiple ndarrays at once and lack memory, so want to write in chunks (here sized to GPU batch capacity). It seems there should be an interface to write the header, then write a number of elements cyclically, then add any closing rubric and close the file.
Is it as simple as lib.format.write_array_header_2_0(fp, d)
then writing multiple shape(N,) arrays of float by fp.write(item.tobytes())?The alternative is to use `np.lib.format.open_memmap(filename, mode='w+', dtype=dtype, shape=shape)`, then assign slices sequentially to the returned memory-mapped array. A memory-mapped array is usually going to be friendlier to whatever memory limits you are running into than a nominally "in-memory" array.--Robert Kern
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-leave@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: michael.siebert2k@gmail.com