Python "compiler" is too slow for processing large data files???
rchii at lycos.com
Thu Aug 29 21:15:49 CEST 2002
Thanks for trying it out, Martin and Wolfgang.
I re-ran the test on a different machine (still Windows 2000, but faster CPU
and more RAM), and can conclude only that the standard prebuilt install
(from www.python.org) for python 2.2.1, for Win32 is slower on compiles than
the linix and mac version you are using. I still got over 9 seconds on the
8000x10 test. Here are my numbers for the 8000x10 (and also an 8000x20)
test on my Windows 2000 machine, 1.5GHz, Pentium 4:
C:\test>python compiletest.py 8000 10 comp.py
C:\test>timer r "python -c \"import comp\""
Time passed: 00:00:09.39
C:\test>python compiletest.py 8000 20 comp2.py
C:\test>timer r "python -c \"import comp2\""
Time passed: 00:00:26.47
The second case, 8000x20, is more realistic as to the amount of data I'm
working with, and I also did a quick check of the memory used to compile
that amount of data. It was not scientific (I just watched the windows task
manager for the python process while it ran), but it seemed to need about
80MB to compile it. Could not even run 8000x20 on my other mahine (python
ran out of memory).
Thanks for looking at it. I'm glad the python community is out there! For
now, I've changed my data format so that I can "exec" each line of the file
on its own, so I'm doing 8000 execs (one line of code each) , instead of
just 1 exec (with 8000 lines of data). Seems unintuitive, but this runs in
about 3 seconds, instead of 26 seconds on the machine above.
Thanks again, maybe someone with more time and interest than me can
determine why the win32 compile function is slow and and eating so much
memory on larger-sized compiles?
"Martin Franklin" <mfranklin1 at gatwick.westerngeco.slb.com> wrote in message
news:mailman.1030614584.8635.python-list at python.org...
> Are you sure there is nothing else going on I generated a test module
> with 8000 x 10 and the result of importing it is :-
> [scripts]$ python compiletest.py 8000 10 comp.py
> [scripts]$ time python -c "import comp"
> real 0m2.658s
> user 0m2.490s
> sys 0m0.140s
> OK so I am not using Windows I am running on a 700Mh PIII laptop running
> RedHat 7.2, but even so 15 seconds sounds way to long.....
More information about the Python-list