[ python-Bugs-849046 ] gzip.GzipFile is slow
SourceForge.net
noreply at sourceforge.net
Tue Nov 25 16:03:38 EST 2003
Bugs item #849046, was opened at 2003-11-25 16:45
Message generated for change (Comment added) made by ronaldoussoren
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=849046&group_id=5470
Category: Python Library
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Ronald Oussoren (ronaldoussoren)
Assigned to: Nobody/Anonymous (nobody)
Summary: gzip.GzipFile is slow
Initial Comment:
gzip.GzipFile is significantly (an order of a magnitude)
slower than using the gzip binary. I've been bitten by this
several times, and have replaced "fd = gzip.open('somefile',
'r')" by "fd = os.popen('gzcat somefile', 'r')" on several
occassions.
Would a patch that implemented GzipFile in C have any
change of being accepted?
----------------------------------------------------------------------
>Comment By: Ronald Oussoren (ronaldoussoren)
Date: 2003-11-25 22:03
Message:
Logged In: YES
user_id=580910
The files are created using GzipFile. That speed is acceptable
because it happens in a batch-job, reading back is the problem
because that happens on demand and a user is waiting for the
results.
gzcat is a *uncompress* utility (specifically it is "gzip -dc"), the
compression level is irrelevant for this discussion.
The python code seems to do quite some string manipulation,
maybe that is causing the slowdown (I'm using fd.readline() in a
fairly tight loop). I'll do some profiling to check what is taking so
much time.
BTW. I'm doing this on Unix systems (Sun Solaris and Mac OS X).
----------------------------------------------------------------------
Comment By: Jim Jewett (jimjjewett)
Date: 2003-11-25 18:35
Message:
Logged In: YES
user_id=764593
Which compression level are you using?
It looks like most of the work is already done by zlib (which is in C), but GzipFile defaults to compression level 9. Many other zips (including your gzcat?) default to a lower (but much faster) compression level.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=849046&group_id=5470
More information about the Python-bugs-list
mailing list