[issue12691] tokenize.untokenize is broken
report at bugs.python.org
Fri Aug 5 22:42:41 CEST 2011
Gareth Rees <gdr at garethrees.org> added the comment:
I think I can make these changes independently and issue two patches, one fixing the problems with untokenize listed here, and another improving tokenize.
I've just noticed a third bug in untokenize: in full mode, it doesn't handle backslash-continued lines correctly.
Python 3.3.0a0 (default:c099ba0a278e, Aug 2 2011, 12:35:03)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from io import BytesIO
>>> from tokenize import tokenize, untokenize
>>> untokenize(tokenize(BytesIO('1 and \\\n not 2'.encode('utf8')).readline))
b'1 andnot 2'
Python tracker <report at bugs.python.org>
More information about the Python-bugs-list