speed problems

Antonio Cavallo cavallo at biochem.ucl.ac.uk
Wed Jun 9 01:16:32 CEST 2004


> I've become interested in Python a while ago and just converted a simple
> perl script to python.
> I've found that there's a huge difference in execution time for the scripts,
> in favor of perl and I can't pinpoint what's going wrong;

I had the same problem in dealing with a large (compressed) file using
python vs c++ (using the gzip library to open/read a file): the
results were in favour of python against c++ this time;)

I think the problem is in the flow: 

gzip -> file/pipe -> perl
file -> zlib -> python

The decompression through zlib is wfar slower because it is carried
using data chunks: there is no mean to control where a chunk will
terminate (like in correspondence to an EOL).
Try using: zcat <myfile> | myprogram.py
In my case it solved all the puzzling speed problems: zcat
uncoditionally dumps the data irregarding where the EOL happens (so
the upper layer should not wait for the next chunk to be
decompressed).
regards,
antonio cavallo



More information about the Python-list mailing list