[New-bugs-announce] [issue7471] gzip module too slow

[asnakelover]

New submission from asnakelover <a3277638 at uggsrock.com>:

It's not a big problem because we can just shell to zcat but compare
these two:

$ time python ./b.py >/dev/null

real    0m10.977s
user    0m7.128s
sys     0m0.888s
$ time python ./a.py >/dev/null

real    1m19.015s
user    1m18.185s
sys     0m0.072s

$ # Notice that the gzip module (a.py) had the benefit of the files
being in a disk cache by too now...

$ cat a.py 
import gzip
import os

apt_cache_dir = "/var/cache/apt/apt-file"

for apt_cache_file in os.listdir(apt_cache_dir):
    if not apt_cache_file.endswith(".gz"):
        continue
    f = gzip.open(os.path.join(apt_cache_dir, apt_cache_file))
    for line in f:
        print line

$ cat b.py 
import os
import subprocess
from cStringIO import StringIO

apt_cache_dir = "/var/cache/apt/apt-file"

for apt_cache_file in os.listdir(apt_cache_dir):
    if not apt_cache_file.endswith(".gz"):
        continue
    p = subprocess.Popen(["zcat", os.path.join(apt_cache_dir,
apt_cache_file)],
        stdout = subprocess.PIPE)
    f = StringIO(p.communicate()[0])
    assert p.returncode == 0 
    for line in f:
        print line

Also tried this one just for "completeness":

$ cat c.py 
import gzip
import os
from cStringIO import StringIO

apt_cache_dir = "/var/cache/apt/apt-file"

for apt_cache_file in os.listdir(apt_cache_dir):
    if not apt_cache_file.endswith(".gz"):
        continue
    f = gzip.open(os.path.join(apt_cache_dir, apt_cache_file))
    f = StringIO(f.read())
    for line in f:
        print line

But after it had ran (with some thrashing) for 3 and a half minutes I
killed it.

----------
components: Library (Lib)
messages: 96204
nosy: asnakelover
severity: normal
status: open
title: gzip module too slow
type: performance
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7471>
_______________________________________