CRC-checksum failed in gzip
andrea.crotti.0 at gmail.com
Thu Aug 2 11:26:51 CEST 2012
2012/8/1 Steven D'Aprano <steve+comp.lang.python at pearwood.info>:
> When you start using threads, you have to expect these sorts of
> intermittent bugs unless you are very careful.
> My guess is that you have a bug where two threads read from the same file
> at the same time. Since each read shares state (the position of the file
> pointer), you're going to get corruption. Because it depends on timing
> details of which threads do what at exactly which microsecond, the effect
> might as well be random.
> Example: suppose the file contains three blocks A B and C, and a
> checksum. Thread 8 starts reading the file, and gets block A and B. Then
> thread 2 starts reading it as well, and gets half of block C. Thread 8
> gets the rest of block C, calculates the checksum, and it doesn't match.
> I recommend that you run a file system check on the remote disk. If it
> passes, you can eliminate file system corruption. Also, run some network
> diagnostics, to eliminate corruption introduced in the network layer. But
> I expect that you won't find anything there, and the problem is a simple
> thread bug. Simple, but really, really hard to find.
> Good luck.
One last thing I would like to do before I add this fix is to actually
be able to reproduce this behaviour, and I thought I could just do the
fz = gzip.open('out2.txt.gz')
if __name__ == '__main__':
for i in range(100):
But no matter how many threads I start, I can't reproduce the CRC
error, any idea how I can try to help it happening?
The code in run should be shared by all the threads since there are no
More information about the Python-list