tokenize.untokenize adding line continuation characters
Peter Otten
__peter__ at web.de
Tue Jan 17 06:10:37 EST 2017
Rotwang wrote:
> Here's something odd I've found with the tokenize module: tokenizing 'if
> x:\n y' and then untokenizing the result adds '\\\n' to the end.
> Attempting to tokenize the result again fails because of the backslash
> continuation with nothing other than a newline after it. On the other
> hand, if the original string ends with a newline then it works fine. Can
> anyone explain why this happens?
> I'm using Python 3.4.3 on Windows 8. Copypasted from iPython:
This looks like a bug...
$ python3.4 -c 'import tokenize as t, io;
print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n y").readline)))'
b'if x:\n y\\\n'
...that is fixed now:
$ python3.6 -c 'import tokenize as t, io;
print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n y").readline)))'
b'if x:\n y'
A quick search brought up multiple bug reports, among them
http://bugs.python.org/issue12691
More information about the Python-list
mailing list