[New-bugs-announce] [issue10276] zlib crc32/adler32 buffer length truncation (64-bit)

[Nadeem Vawda]

New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

zlib.crc32() and zlib.adler32() in Modules/zlibmodule.c don't handle buffers of >=4GB correctly. The length of a Py_buffer is of type Py_ssize_t, while the C zlib functions take length as an unsigned integer. This means that on a 64-bit build, the buffer length gets silently truncated to 32 bits, which results in incorrect output for large inputs.

Attached is a patch that fixes this by computing the checksum incrementally, using small-enough chunks of the buffer.

A better fix might be to have Modules/zlib/crc32.c use 64-bit lengths. I tried this, but I couldn't get it to work. It seems that if the system already has zlib installed, Python will link against the existing version instead of compiling its own.

Testing this might be a bit tricky. Allocating a 4+GB regular buffer isn't practical. Using a memory-mapped file would work, but I'm not sure having a unit test create a multi-gigabyte file is a great thing to do.

----------
components: Library (Lib)
files: zlib-checksum-truncation.diff
keywords: patch
messages: 120114
nosy: nvawda
priority: normal
severity: normal
status: open
title: zlib crc32/adler32 buffer length truncation (64-bit)
type: behavior
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file19453/zlib-checksum-truncation.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10276>
_______________________________________