[Numpy-discussion] About the npz format
jtaylor.debian at googlemail.com
Thu Apr 17 15:51:32 EDT 2014
On 17.04.2014 21:30, onefire wrote:
> Hi Nathaniel,
> Thanks for the suggestion. I did profile the program before, just not
> using Python.
one problem of npz is that the zipfile module does not support streaming
data in (or if it does now we aren't using it).
So numpy writes the file uncompressed to disk and then zips it which is
horrible for performance and disk usage.
It would be nice if we could add support for different compression
modules like gzip or xz which allow streaming data directly into a file
without an intermediate.
More information about the NumPy-Discussion