[Python-Dev] Small tweak to tokenize.py?

[python at rcn.com]

> It would be trivial to add another yield to tokenize.py when
> the backslah is detected

+1



> I think that it should probably yield a single NL pseudo-token
> whose value is a backslash followed by a newline; or perhaps it
> should yield the backslash as a comment token, or as a new token.

The first option is likely the most compatible with existing uses of tokenize.  If a comment token were emitted, an existing colorizer or pretty-printer would markup the continuation as a comment (possibly not what the tool author intended).  If a new token were created, it might break if-elif-else chains in tools that thought they knew the universe of possible token types.


Raymond