Speeding up application startup

Delaney, Timothy tdelaney at avaya.com
Fri May 11 01:18:42 EDT 2001


> On startup my application reads a text file that contains around 15000
> regular expressions which i then need to compile for use 
> during the course
> of my program.  It takes around 15 seconds for this to be 
> done on my 900mhz
> machine.  Any suggestions for speeding this up?

Without seeing the code, there are only a few general suggestions we can
make. If you post the relevant bits of code we can probably help a lot more
...

First suggestion is to profile the app to see what is taking the most time -
reading in the expressions or compiling them, or possibly something else.
This will tell you what is more important to optimise.

Second suggestion is to not compile the expressions at startup. Read them in
at startup, but only compile each one as you need it. Just for fun, keep
track of which ones actually get compiled during an invocation of the
program, and how many times ;) This is a simple case of spreading the
compilation time over longer periods.

Another question is whether you can get away with not reading the entire
file in at startup, but I expect that realistically you do need to. In that
case, consider the method you are using to read in the file. I presume you
are using readlines() or xreadlines() (one expression per line).

Can you can get away with fewer expressions. It may be quite possible to
combine many expressions into one (that's the power of regular expressions).
I presume the list of 15000 has been automatically generated from elsewhere.
In that case they are probably fairly simplistic. However, I wouldn't want
to go through a list of 15000 regexes and try to combine them.

How are you storing the regexes in memory? Are they in a list, appended one
at a time? A dictionary? An array? Heaven forbid, are you adding tuples
together?

Try a few different methods for each action and profile them.

Tim Delaney




More information about the Python-list mailing list