[Python-Dev] SRE should release the GIL (was: no subject)

[Duncan Grisby]

On Monday 28 November, Guido van Rossum wrote:

> On 11/24/05, Duncan Grisby <duncan-pythondev at grisby.org> wrote:

> > I have encountered a problem with the re module. I have a
> > multi-threaded program that does lots of regular expression searching,
> > with some relatively complex regular expressions. Occasionally, events
> > can conspire to mean that the re search takes minutes. That's bad
> > enough in and of itself, but the real problem is that the re engine
> > does not release the interpreter lock while it is running. All the
> > other threads are therefore blocked for the entire time it takes to do
> > the regular expression search.
> 
> Rather than trying to fight the GIL, I suggest that you let a regex
> expert look at your regex(es) and the input that causes the long
> running times. As Fredrik suggested, certain patterns are just
> inefficient but can be rewritten more efficiently. There are plenty of
> regex experts on c.l.py.

Part of the problem is certainly inefficient regexes, and we have
improved things to some extent by changing some of them. Unfortunately,
the regexes come from user input, so we can't be certain that our users
aren't going to do stupid things. It's not too bad if a stupid regex
slows things down for a bit, but it is bad if it causes the whole
application to freeze for minutes at a time.

> Unless you have a multi-CPU box, the performance of your app isn't
> going to improve by releasing the GIL -- it only affects the
> responsiveness of other threads.

We do have a multi-CPU box. Even with good regexes, regex matching takes
up a significant proportion of the time spent processing in our
application, so being able to release the GIL will hopefully increase
performance overall as well as increasing responsiveness.

We are currently testing our application with the patch to sre that Eric
posted. Once we get on to some performance tests, we'll post the results
of whether releasing the GIL does make a measurable difference for us.

Cheers,

Duncan.

-- 
 -- Duncan Grisby         --
  -- duncan at grisby.org     --
   -- http://www.grisby.org --