tokenize.untokenize adding line continuation characters
Rotwang
sg552 at hotmail.co.uk
Tue Jan 17 06:20:40 EST 2017
On Tuesday, January 17, 2017 at 11:11:27 AM UTC, Peter Otten wrote:
> Rotwang wrote:
>
> > Here's something odd I've found with the tokenize module: tokenizing 'if
> > x:\n y' and then untokenizing the result adds '\\\n' to the end.
> > Attempting to tokenize the result again fails because of the backslash
> > continuation with nothing other than a newline after it. On the other
> > hand, if the original string ends with a newline then it works fine. Can
> > anyone explain why this happens?
>
> > I'm using Python 3.4.3 on Windows 8. Copypasted from iPython:
>
> This looks like a bug...
>
> $ python3.4 -c 'import tokenize as t, io;
> print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n y").readline)))'
> b'if x:\n y\\\n'
>
> ...that is fixed now:
>
> $ python3.6 -c 'import tokenize as t, io;
> print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n y").readline)))'
> b'if x:\n y'
>
> A quick search brought up multiple bug reports, among them
>
> http://bugs.python.org/issue12691
Ah, thanks. I did search for bug reports before I started this thread but didn't find anything.
More information about the Python-list
mailing list