[New-bugs-announce] [issue1160] Medium size regexp crashes python

Guido Ostkamp report at bugs.python.org
Thu Sep 13 10:00:57 CEST 2007

New submission from 
					Guido Ostkamp


a medium size regular expression crashes Python 2.5.1 as follows:

Traceback (most recent call last):
  File "./regtest.py", line 14, in <module>
    m = rematch(pats)
  File "./regtest.py", line 12, in rematch
    return re.compile(pat).match
  File "/export/home/ostkamp/local/lib/python2.5/re.py", line 180, in
    return _compile(pattern, flags)
  File "/export/home/ostkamp/local/lib/python2.5/re.py", line 231, in
    p = sre_compile.compile(pattern, flags)
  File "/export/home/ostkamp/local/lib/python2.5/sre_compile.py", line
530, in compile
    groupindex, indexgroup
OverflowError: regular expression code size limit exceeded

This is apparently caused by some code in Modules/_sre.c and
Modules/sre.h as follows:

        self->code[i] = (SRE_CODE) value;
        if ((unsigned long) self->code[i] != value) {
                            "regular expression code size limit exceeded");

An 'unsigned int' value is unnecessarily squeezed into an 'unsigned
short' field defined in sre.h:

#define SRE_CODE Py_UCS4
#define SRE_CODE unsigned short

On all systems I'm working on (SuSE Linux SLES 9, Solaris 8 etc.) the
else case of the ifdef applies which chooses 'unsigned short'.

I don't understand the relationship between 'unicode' and what is
apparently the size of the regular expression stack here.

Some experiments have shown that changing the 'unsigned short' to
'unsigned long' and rebuilding Python fixes the problem.

Here is a test program to reproduce the error:

#!/usr/bin/env python
import re, random, sys
def randhexstring():
    return "".join(["%04x" % random.randint(0, 0xffff) for x in range(20)])
pats = [randhexstring() for x in range(1000)]
def rematch(pats):
    pat = '(?:%s)' % '|'.join(pats)
    return re.compile(pat).match
m = rematch(pats)
count = 0
for s in pats * 100:
    if m(s):
        count += 1
print count



components: Regular Expressions
messages: 55885
nosy: ostkamp
severity: normal
status: open
title: Medium size regexp crashes python
type: crash
versions: Python 2.5

Tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list