[issue719888] tokenize module w/ coding cookie

[Michael Foord]

Michael Foord <fuzzyman at voidspace.org.uk> added the comment:

*Full* patch (excluding the new dependent test text files) for Python 3.
Includes fixes for standard library and tools usage of tokenize.

If it breaks anything blame Trent... ;-)

----------
versions:  -Python 2.6
Added file: http://bugs.python.org/file9741/tokenize_patch.diff

____________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue719888>
____________________________________