[Python-Dev] Small tweak to tokenize.py?

Phillip J. Eby pje at telecommunity.com
Thu Nov 30 19:55:44 CET 2006

At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote:
>Are you opposed changing tokenize? If so, why (apart from

Nothing apart from compatibility.  I think you should have to explicitly 
request the new behavior(s), since tools (like detokenize) written to work 
around the old behavior might behave oddly with the change.

Mainly, though, I thought you might find the code useful, given the nature 
of your project.  (Although I suppose you've probably already written 
something similar.)

More information about the Python-Dev mailing list