[issue12486] tokenize module should have a unicode API

[Thomas Kluyver]

Change by Thomas Kluyver <thomas at kluyver.me.uk>:


----------
pull_requests: +6616

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue12486>
_______________________________________