Python "compiler" is too slow for processing large data files???
Terry Reedy
tjreedy at udel.edu
Fri Sep 6 08:49:49 EDT 2002
> list1 = [
> (323, 870, 46, ),
> (810, 336, 271, ),
> (572, 55, 596, ),
> (337, 256, 629, ),
> (31, 702, 16, ),
> ]
> Anyway, as my data files went from just a few lines, up to about
8000 lines
> (with 10 values in each line for total of about 450KB of text), the
time to
> 'exec' the file became too slow (e.g. 15 seconds) and used too much
memory
> (e.g. 50MB) (using ms-windows, python 2.2.1). It is the "compile"
phase,
> because if I re-run, and there is *.pyc file available, the import
goes very
> fast (no compilation required).
I am curious whether any C/C++ compilers (or those for other
languages) would have any problems with literal array of 10000
10-member structs?
Would the following go faster? Leave off outer list lines (first and
last) to make more data like. Then read file twice: once to count
lines and allocate list; second to populate list.
f = file('data')
lenf = len(f.readlines())
data = [0]*lenf
f.rewind()
for i in range(lenf):
data[i] = maketuple(f.readline()) # substitute code for maketuple
Terry J. Reedy
More information about the Python-list
mailing list