Python "compiler" is too slow for processing large data files???

Terry Reedy tjreedy at udel.edu
Fri Sep 6 08:49:49 EDT 2002


> list1 = [
>     (323, 870, 46, ),
>     (810, 336, 271, ),
>     (572, 55, 596, ),
>     (337, 256, 629, ),
>     (31, 702, 16, ),
> ]

 > Anyway, as my data files went from just a few lines, up to about
8000 lines
 > (with 10 values in each line for total of about 450KB of text), the
time to
 > 'exec' the file became too slow (e.g. 15 seconds) and used too much
memory
 > (e.g. 50MB) (using ms-windows, python 2.2.1).  It is the "compile"
phase,
 > because if I re-run, and there is *.pyc file available, the import
goes very
 > fast (no compilation required).

I am curious whether any C/C++ compilers (or those for other
languages) would have any problems with literal array of 10000
10-member structs?

Would the following go faster?  Leave off outer list lines (first and
last) to make more data like.  Then read file twice: once to count
lines and allocate list; second to populate list.

f = file('data')
lenf = len(f.readlines())
data = [0]*lenf
f.rewind()
for i in range(lenf):
  data[i] = maketuple(f.readline()) # substitute code for maketuple

Terry J. Reedy







More information about the Python-list mailing list