python lexer

[Fredrik Lundh]

Alessio Pace wrote:
> Hi, I was wondering which is a good lexer written in Python, I heard about
> SPARK and PLY. I need only to tokenize texts, not to build trees on their
> structure.
> string.split() I guess it is not enough, because I can't use reg exp as
> token separators..How could I do otherwise?

it may be easier (and sometimes more efficient) to use re.findall
to find the tokens, instead of using split to find the separators.

more here:

    http://effbot.org/guides/xml-scanner.htm

</F>

<!-- (the eff-bot guide to) the python standard library (redux):
http://effbot.org/zone/librarybook-index.htm
-->