James Hutchison added the comment: I should clarify that this is the zipfile constructor I am using: zipfile.ZipFile(filename, mode='r', allowZip64=True); ---------- _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Changes by Xuanji Li : ---------- nosy: +xuanji _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Serhiy Storchaka added the comment: Actually reading from the zip file is buffered (at least 4 KiB of uncompressed data at a time). Can you give tests, scripts and data, which show the problem? ---------- _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Serhiy Storchaka added the comment: This is not because zipfile module is unbuffered. This is the difference between expensive function call and cheap bytes slicing. Replace `zf.open(namelist [0])` to `io.BufferedReader(zf.open(namelist [0]))` to see the effect of a good buffering. In 3.2 zipfile read() implemented not optimal, so it slower (twice), but in 3.3 it will be almost as fast as using io.BufferedReader. It is still several times more slowly than bytes slicing, but there's nothing you can do with it. Here is a patch, which is speeds up (+20%) the reading from a zip file by small chunks. Microbenchmark: ./python -m zipfile -c test.zip python ./python -m timeit -n 1 -s "import zipfile;zf=zipfile.ZipFile('test.zip')" "with zf.open('python') as f:" " while f.read(1):pass" Python 3.3 (vanilla): 1 loops, best of 3: 36.4 sec per loop Python 3.3 (patched): 1 loops, best of 3: 30.1 sec per loop Python 3.3 (with io.BufferedReader): 1 loops, best of 3: 30.2 sec per loop And, for comparison, Python 3.2: 1 loops, best of 3: 74.5 sec per loop ---------- components: -Documentation keywords: +patch versions: -Python 2.7, Python 3.2 Added file: http://bugs.python.org/file25530/zipfile_optimize_read.patch _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Changes by Martin v. Löwis : Added file: http://bugs.python.org/file25565/zipfile_optimize_read.patch _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Serhiy Storchaka added the comment: The patch updated to reflect Martin's stylistic comments. Sorry for the delay, Martin. I have not received an email with your review from 2012-05-13, and only today accidentally discovered your comments in Rietveld. It seems to have been some bug in Rietveld. ---------- Added file: http://bugs.python.org/file25769/zipfile_optimize_read_2.patch _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Serhiy Storchaka added the comment: Any chance to commit the patch before final feature freeze? ---------- _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Nadeem Vawda added the comment: Patch looks fine to me. Antoine, can you commit this? I'm currently away from the computer that has my SSH key on it. ---------- _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________

Antoine Pitrou added the comment:

Antoine, can you commit this?

Ok, done. ---------- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed _______________________________________ Python tracker http://bugs.python.org/issue10376 _______________________________________