[issue9969] tokenize: add support for tokenizing 'str' objects

[Abhay Saxena]

Abhay Saxena <ark3 at email.com> added the comment:

If the goal is tokenize(...) accepting a text I/O readline, we already have the (undocumented) generate_tokens(readline).

----------
nosy: +ark3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9969>
_______________________________________