Hi,
* Julian Taylor
On 17.04.2014 21:30, onefire wrote:
Hi Nathaniel,
Thanks for the suggestion. I did profile the program before, just not using Python.
one problem of npz is that the zipfile module does not support streaming data in (or if it does now we aren't using it). So numpy writes the file uncompressed to disk and then zips it which is horrible for performance and disk usage.
As a workaround may also be possible to write the temporary NPY files to cStringIO instances and then use ``ZipFile.writestr`` with the ``getvalue()`` of the cStringIO object. However that approach may require some memory. In python 2.7, for each array: one copy inside the cStringIO instance and then another copy of when calling getvalue on the cString, I believe. best, V-