Speeding up application startup

[Fredrik Lundh]

Fahd Khan wrote:
> This is a guess, and I haven't bothered to look up if it is even possible,
> but perhaps you could pickle the compiled regex objects into a cache file of
> some sort?

you can pickle regexps, but that stores the original textual representation,
and recompiles them on the way in (the internal data format is not designed
to be portable).

possible workarounds include:

- if you're using 1.6 or newer, you can use "from pre import re" instead
of "re" -- the new engine is usually faster when it comes to running the
compiled expression, but pre's compiler is written in C, not Python.

- rethink your design.  do you really need 15000 regular expressions?  do
you really need to compile all of them on the way in?  do you really use
all of them during a typical program run?

- try storing them as text strings instead, and use re.match(p, s) instead
of p.match(s).  you can bump the SRE cache like this:

    import sre
    sre._MAXCACHE = 15000

adding a "print len(sre._cache)" to the end of your program will give you
an idea of how many expressions you're using during a typical run.

Cheers /F