[Python-ideas] re.compile_lazy - on first use compiled regexes

[M.-A. Lemburg]

On 23.03.2013 13:18, Antoine Pitrou wrote:
> On Fri, 22 Mar 2013 15:42:49 -0700
> "Gregory P. Smith" <greg at krypto.org> wrote:
>> On Fri, Mar 22, 2013 at 3:31 PM, Ronny Pfannschmidt <
>> Ronny.Pfannschmidt at gmx.de> wrote:
>>
>>> Hi,
>>>
>>> while reviewing urllib.parse i noticed a pretty ugly pattern
>>>
>>> many functions had an attached global and in their own code they would
>>> compile an regex on first use and assign it to that global
>>>
>>> its clear that compiling a regex is expensive, so having them be compiled
>>> later at first use would be of some benefit
>>>
>>
>> It isn't expensive to do, it is expensive to do repeatedly for no reason.
>>  Thus the use of compiled regexes.  Code like this would be better off
>> refactored to reference a precompiled global rather than conditionally
>> check if it needs compiling every time it is called.
> 
> Precompiled regexes were a major contributor in Python startup time:
> http://bugs.python.org/issue13150
> http://hg.python.org/cpython/rev/df950158dc33/
> 
> In the stdlib we should be rather careful about this. Third-party
> libraries can ignore such concerns if they aren't meant to use in
> interactive applications.

Wouldn't it make sense to add a way to pickle or marshal compiled REs ?

The precompiled REs could then be loaded directly from the
pickle, avoiding the compiling overhead on startup.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 23 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2013-03-13: Released eGenix pyOpenSSL 0.13 ...    http://egenix.com/go39

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/