[issue20387] tokenize/untokenize roundtrip fails with tabs
Jason R. Coombs
report at bugs.python.org
Fri Jan 24 23:51:15 CET 2014
New submission from Jason R. Coombs:
Consider this simple unit test:
def test_tokenize():
input = "if False:\n\tx=3\n\ty=3\n"
g = list(generate_tokens(io.StringIO(input).readline))
assert untokenize(g) == input
According to the docs, untokenize guarantees the output equals the input to tokenize: http://docs.python.org/2/library/tokenize.html#tokenize.untokenize
As this test fails on Python 2 and Python 3 (use io.BytesIO on Python 2), I believe the test illustrates a violation of this guarantee. The second line in the if block gets its tab replaced by a space.
As the input is valid Python Syntax, this behavior is particularly surprising.
----------
messages: 209137
nosy: jason.coombs
priority: normal
severity: normal
status: open
title: tokenize/untokenize roundtrip fails with tabs
versions: Python 2.7, Python 3.4
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20387>
_______________________________________
More information about the Python-bugs-list
mailing list