Thanks, np.lib.format.open_memmap() works great! With prediction procs using minimal sys memory, I can get twice as many on GPU, with fewer optimization warnings. Why even have the number of records in the header? Shouldn't record size plus system-reported/growable file size be enough? I'd love to have a shared-mem analog for smaller-scale data; now I load data and fork to emulate that effect. My file sizes will exceed memory, so I'm hoping to get the most out of memmap. Will this in-loop assignment to predsum work to avoid loading all to memory? predsum = np.lib.format.open_memmap(outfile, mode='w+', shape=(ids_sq,), dtype=np.float32) for i in range(len(IN_FILES)): pred = numpy.lib.format.open_memmap(IN_FILES[i]) predsum = np.add(predsum, pred) ################# <- del pred del predsum -- Phobrain.com On 2022-08-23 18:02, Robert Kern wrote:
On Tue, Aug 23, 2022 at 8:47 PM <bross_phobrain@sonic.net> wrote:
I want to calc multiple ndarrays at once and lack memory, so want to write in chunks (here sized to GPU batch capacity). It seems there should be an interface to write the header, then write a number of elements cyclically, then add any closing rubric and close the file.
Is it as simple as lib.format.write_array_header_2_0(fp, d) then writing multiple shape(N,) arrays of float by fp.write(item.tobytes())?
`item.tofile(fp)` is more efficient, but yes, that's the basic scheme. There is no footer after the data.
The alternative is to use `np.lib.format.open_memmap(filename, mode='w+', dtype=dtype, shape=shape)`, then assign slices sequentially to the returned memory-mapped array. A memory-mapped array is usually going to be friendlier to whatever memory limits you are running into than a nominally "in-memory" array. -- Robert Kern _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: bross_phobrain@sonic.net