[issue719888] tokenize module w/ coding cookie

[Michael Foord]

Michael Foord <fuzzyman at voidspace.org.uk> added the comment:

Made quite extensive changes to tokenize.py (with tests) for Py3k. This
migrates it to a 'bytes' API so that it can correctly decode Python
source files following PEP-0263.

----------
nosy: +fuzzyman
Added file: http://bugs.python.org/file9735/tokenize.zip

____________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue719888>
____________________________________