[Numpy-discussion] bad CRC errors when using np.savez, only sometimes though!

[Isaac Gerg]

I am using 1.19.5 on Windows 10 using Python 3.8.6 (tags/v3.8.6:db45529,
Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)].

I have two python processes running (i.e. no threads) which do
independent processing jobs and NOT writing to the same directories.  Each
process runs for 5-10 hours and then writes out a ~900MB npz file
containing 4 arrays.

When I go back to read in the npz files, I will sporadically get bad CRC
errors which are related to npz using ziplib.  I cannot figure out why this
is happening.  Looking through online forums, other folks have had CRC
problems but they seem to be isolated to specifically using ziblib, not
numpy.  I have found a few mentions though of ziplib causing headaches if
the same file pointer is used across calls when one uses the file handle
interface to ziblib as opposed to passing in a filename.'

I have verified with 7zip that the files do in fact have a CRC error so its
not an artifact of the ziblib.  I have also used the file handle interface
to np.load and still get the error.

Aside from writing my own numpy storage file container, I am stumped as to
how to fix this, or reproduce this in a consistent manner.  Any suggestions
would be greatly appreciated!

Thank you,
Isaac
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/numpy-discussion/attachments/20210514/cd4076ad/attachment.html>