On Tue, Aug 23, 2022 at 8:47 PM <bross_phobrain@sonic.net> wrote:
I want to calc multiple ndarrays at once and lack memory, so want to write in chunks (here sized to GPU batch capacity). It seems there should be an interface to write the header, then write a number of elements cyclically, then add any closing rubric and close the file.

Is it as simple as lib.format.write_array_header_2_0(fp, d)
then writing multiple shape(N,) arrays of float by fp.write(item.tobytes())?
 
`item.tofile(fp)` is more efficient, but yes, that's the basic scheme. There is no footer after the data.

The alternative is to use `np.lib.format.open_memmap(filename, mode='w+', dtype=dtype, shape=shape)`, then assign slices sequentially to the returned memory-mapped array. A memory-mapped array is usually going to be friendlier to whatever memory limits you are running into than a nominally "in-memory" array.

--
Robert Kern