[pypy-dev] Regular Expression in PyPy

[Seo Sanghyeon]

In the checkin message of r8719, hpk wrote:
> Ok, it seems we had a mix of python 2.4 and 2.3 moduels with
> slight changes and were borrowing _sre from CPython anyway.
> So let's back out our changes in pypy/lib and see how far we
> get (if nobody minds, Seo?)

But I mind! Let me explain this...

Just before the checkin (r8718), there were following files in
pypy/lib related to regular expression: re.py, dumbre.py, plexre.py,
sre_adapt.py, sre_parse.py. You deleted re.py, sre_adapt.py,
sre_parse.py.

re.py contained (intended to contain) backend-neutral interface and
utilities, like this:

def match(pattern, string, flags=0):
    return compile(pattern, flags).match(string)

And compile() does memoizing, just as CPython, etc. And re.escape()
copied from CPython. These are backend-neutral.

re.py imported Pattern class from the backend. Pattern's initializer
has signature __init__(self, pattern, flags), and has methods of
C type _sre.SRE_Pattern, like match, search, etc.

And there were three backends (draft of backends) providing this
Pattern class: dumbre.py, plexre.py, sre_adapt.py. Only sre_adapt.py
imports CPython's _sre, others don't!

You can read dumbre.py and plexre.py now, since you didn't delete
them by mistake. :-)

See, "dumbre.py" is exactly what you're referring as "see how far
we get". It exits as soon as regular expression is used, and reports
what was used. And this was the default re.py imported. (see below)

"plexre.py" is more interesting, and it uses Plex to implement (for now)
match(), but returns bool rather than full Match object. This is
sufficient for all "if re.match(pattern, string):" tests, and this let
pickle import and run, unmodified. (As I wrote in the log of r8636...)

pickle uses regular expression only once, as following:

__all__.extend([x for x in dir() if re.match("[A-Z][A-Z0-9_]+$",x)])

This also means PyPy interprets all of Plex just fine. Since Plex does
regular expression parsing, NFA->DFA transformation, state machine all
in pure Python, this is actually quite cool.

"sre_adapt.py" is, as you see, cheating. It contains a single line:

Pattern = sre_compile.compile

And this does not work. Currently C types are not succesfully faked,
so _sre.SRE_Pattern instances are created (well, see below) but all
method calls will result in long traceback. I asked why is this on IRC
long time ago, and I heard that it's because this C type lacks __dict__.

Same applies for _random.Random, I think. That's why we have pure Python
random.py from 2.2.3 in lib directory now.

And why is sre_parse.py patched? Because, without those, no
_sre.SRE_Pattern instances will be created, because PyPy fails to
interpret some part of sre_parse.py. How to reproduce this problem:

(with current PyPy)
>>>> re.compile('a*')
Traceback (application level):
(snip)
AttributeError: getwidth

Why is this? I wrote my reasoning on the log of this file... r3332 and
r3456. To show what's going on, I will give an example:

class A:
    def getwidth(self):
        print 'hahaha'
class B:
    def getwidth(self):
        print 'lalala'
class C:
    def __getitem__(self, index):
        return A()
    def __getslice__(self, start, stop):
        return B()
c = C()
c[0].getwidth()
c[0:1].getwidth()

CPython prints hahaha and lalala. PyPy prints both hahaha. And sre_parse
uses __getslice__... And this is the only place getslice is used, in entire
Python standard library! (except those in UserList and UserString)

And getslice is deprecated since release 2.0, as officially announced in
Language Reference 3.3.6.

Okay, can I back your deletion now? :-)

Seo Sanghyeon