[Python-Dev] Small tweak to tokenize.py?

[Phillip J. Eby]

At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote:
>Are you opposed changing tokenize? If so, why (apart from
>compatibility)?

Nothing apart from compatibility.  I think you should have to explicitly 
request the new behavior(s), since tools (like detokenize) written to work 
around the old behavior might behave oddly with the change.

Mainly, though, I thought you might find the code useful, given the nature 
of your project.  (Although I suppose you've probably already written 
something similar.)