[Python-Dev] tokenize.py (generate_tokens yielding @ as OP)

[Guido van Rossum]

> This is a question regarding the tokenize.generate_tokens function.
> 
> As it is in CVS HEAD, generate_tokens yields the '@' token as
> having type OP, not type AT. Is this how it should be?

Without looking at the code, I believe tokenize treats all operators
as OP, for a rather liberal definition of "operator".

(A more serious problem that I just encountered in a different
context: when you feed it input containing unsupported characters, the
spaces before those unsupported characters are also returned as
tokens.  I think that's a bug.)

--Guido van Rossum (home page: http://www.python.org/~guido/)