[pypy-issue] [issue340] rewrite PyPy's tokenizer

Carl Friedrich Bolz tracker at bugs.pypy.org
Wed Mar 20 12:06:01 CET 2013

Carl Friedrich Bolz <cfbolz at gmx.de> added the comment:

I am not really sure that using rply's tokenizer would improve things (I should
really raise that with Alex, too). rply's tokenizer uses rlib.rsre to tokenize
stuff, which is overkill, given that the regular expressions needed for Python's
tokens are really regular, not Perl-style regular expressions.

nosy: +cfbolz

PyPy bug tracker <tracker at bugs.pypy.org>

More information about the pypy-issue mailing list