[Python-Dev] tokenize.py (generate_tokens yielding @ as OP)
Guido van Rossum
guido at python.org
Mon Aug 9 22:21:57 CEST 2004
> This is a question regarding the tokenize.generate_tokens function.
> As it is in CVS HEAD, generate_tokens yields the '@' token as
> having type OP, not type AT. Is this how it should be?
Without looking at the code, I believe tokenize treats all operators
as OP, for a rather liberal definition of "operator".
(A more serious problem that I just encountered in a different
context: when you feed it input containing unsupported characters, the
spaces before those unsupported characters are also returned as
tokens. I think that's a bug.)
--Guido van Rossum (home page: http://www.python.org/~guido/)
More information about the Python-Dev