[Python-Dev] should sre.Scanner be exposed through re and documented?

[Gary Herron]

On Friday 25 April 2003 12:38 pm, Guido van Rossum wrote:
> > While moving tests from test_sre to test_re I stumbled upon a simple test
> > for sre.Scanner.  This looks fairly cool.  Should it be exposed through
> > re and documented?
>
> What's Scanner?
>

You create a Scanner instance with a list of re's and associated
functions, then you use it to scan a string, returning a list of parts
which match the given re's. (Actually the matches are run through the
associated functions, and their output is what forms the returned
list.)


Here's the single test case Skip refereed to:


def s_ident(scanner, token): return token
def s_operator(scanner, token): return "op%s" % token
def s_float(scanner, token): return float(token)
def s_int(scanner, token): return int(token)

scanner = sre.Scanner([
    (r"[a-zA-Z_]\w*", s_ident),
    (r"\d+\.\d*", s_float),
    (r"\d+", s_int),
    (r"=|\+|-|\*|/", s_operator),
    (r"\s+", None),
    ])

# sanity check
test('scanner.scan("sum = 3*foo + 312.50 + bar")',
     (['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5, 'op+', 'bar'], ''))

Gary Herron