Python "compiler" is too slow for processing large data files???
tjreedy at udel.edu
Fri Sep 6 08:49:49 EDT 2002
> list1 = [
> (323, 870, 46, ),
> (810, 336, 271, ),
> (572, 55, 596, ),
> (337, 256, 629, ),
> (31, 702, 16, ),
> Anyway, as my data files went from just a few lines, up to about
> (with 10 values in each line for total of about 450KB of text), the
> 'exec' the file became too slow (e.g. 15 seconds) and used too much
> (e.g. 50MB) (using ms-windows, python 2.2.1). It is the "compile"
> because if I re-run, and there is *.pyc file available, the import
> fast (no compilation required).
I am curious whether any C/C++ compilers (or those for other
languages) would have any problems with literal array of 10000
Would the following go faster? Leave off outer list lines (first and
last) to make more data like. Then read file twice: once to count
lines and allocate list; second to populate list.
f = file('data')
lenf = len(f.readlines())
data = *lenf
for i in range(lenf):
data[i] = maketuple(f.readline()) # substitute code for maketuple
Terry J. Reedy
More information about the Python-list