[issue9969] tokenize: add support for tokenizing 'str' objects
report at bugs.python.org
Tue Sep 28 16:04:59 CEST 2010
Michael Foord <michael at voidspace.org.uk> added the comment:
Note from Nick Coghlan from the Python-dev discussion:
A very quick scan of _tokenize suggests it is designed to support
detect_encoding returning None to indicate the line iterator will
return already decoded lines. This is confirmed by the fact the
standard library uses it that way (via generate_tokens).
An API that accepts a string, wraps a StringIO around it, then calls
_tokenise with an encoding of None would appear to be the answer here.
A feature request on the tracker is the best way to make that happen.
Python tracker <report at bugs.python.org>
More information about the Python-bugs-list