python lexer

Fredrik Lundh fredrik at pythonware.com
Wed Apr 23 11:45:57 EDT 2003


Alessio Pace wrote:
> Hi, I was wondering which is a good lexer written in Python, I heard about
> SPARK and PLY. I need only to tokenize texts, not to build trees on their
> structure.
> string.split() I guess it is not enough, because I can't use reg exp as
> token separators..How could I do otherwise?

it may be easier (and sometimes more efficient) to use re.findall
to find the tokens, instead of using split to find the separators.

more here:

    http://effbot.org/guides/xml-scanner.htm

</F>

<!-- (the eff-bot guide to) the python standard library (redux):
http://effbot.org/zone/librarybook-index.htm
-->








More information about the Python-list mailing list