[issue12691] tokenize.untokenize is broken

[Gareth Rees]

Gareth Rees <gdr at garethrees.org> added the comment:

I think I can make these changes independently and issue two patches, one fixing the problems with untokenize listed here, and another improving tokenize.

I've just noticed a third bug in untokenize: in full mode, it doesn't handle backslash-continued lines correctly.

    Python 3.3.0a0 (default:c099ba0a278e, Aug  2 2011, 12:35:03) 
    [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from io import BytesIO
    >>> from tokenize import tokenize, untokenize
    >>> untokenize(tokenize(BytesIO('1 and \\\n not 2'.encode('utf8')).readline))
    b'1 andnot 2'

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12691>
_______________________________________