[issue719888] tokenize module w/ coding cookie

[Martin v. Löwis]

Martin v. Löwis <martin at v.loewis.de> added the comment:

In 3k, the tokenize module should definitely return strings, and, in 
doing so, it should definitely consider the encoding declaration (and 
also the default encoding in absence of the encoding declaration).

For 2.6, I wouldn't mind if it were changed incompatibly so that it 
returns Unicode strings, or else that it parses in Unicode, and then 
encodes back to the source encoding before returning anything.

____________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue719888>
____________________________________