[issue12486] tokenize module should have a unicode API

[Serhiy Storchaka]

Change by Serhiy Storchaka <storchaka+cpython at gmail.com>:


----------
nosy: +barry, mark.dickinson, michael.foord, trent
versions: +Python 3.8 -Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue12486>
_______________________________________