[Numpy-discussion] numpy.savez(_compressed) in loop

[Pauli Virtanen]

Radek Machulka <radek.machulka <at> gmail.com> writes:
> is there a way how to save more arrays into single npy (npz if possible) file 
> in loop? Something like:
> 
> fw = open('foo.bar', 'wb')
> while foo:
> 	arr = np.array(bar)
> 	np.savez_compressed(fw, arr)
> fw.close()
> 
> Or some workaround maybe? I go through hundreds of thousands arrays and can 
> not keep them in memory. Yes, I can save each array into single file, but I 
> would better have them all in single one.
[clip]

You can take a look at the implementation of savez_compressed.
The npz files are nothing but zip files with npy format files inside
them, so you should be able to build them up by stuffing files saved
by np.save() into a zipfile.ZipFile.

However, a better option could be to use a another data format such
as hdf5.

-- 
Pauli Virtanen