[New-bugs-announce] [issue7922] Python 3's 2to3 does not handle non-ascii source files

[mike bayer]

New submission from mike bayer <mike_mp at zzzcomputing.com>:

given the following Python 2 source file:

    # -*- encoding: utf-8

    print 'bien mangé'

It can be converted to Python 3 using 2's 2to3 tool:

    classic$ 2to3 test.py
     ... omitted ...
    --- test.py (original)
    +++ test.py (refactored)
    @@ -1,3 +1,3 @@
     # -*- encoding: utf-8
 
    -print 'bien mangé'
    +print('bien mangé')

However that of Python 3.1.1 fails:

    classic$ 2to3-3.1 test.py
       ... omitted ...
    --- test.py (original)
    +++ test.py (refactored)
    @@ -1,3 +1,3 @@
     # -*- encoding: utf-8
 
    Traceback (most recent call last):
      File "/usr/local/bin/2to3-3.1", line 6, in <module>
        sys.exit(main("lib2to3.fixes"))
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/main.py", line 159, in main
        options.processes)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 616, in refactor
        items, write, doctests_only)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 276, in refactor
        self.refactor_file(dir_or_file, write, doctests_only)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 656, in refactor_file
        *args, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 332, in refactor_file
        write=write, encoding=encoding)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 432, in processed_file
        self.print_output(old_text, new_text, filename, equal)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/main.py", line 64, in print_output
        print(line)
    UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 17: ordinal not in range(128)

----------
components: 2to3 (2.x to 3.0 conversion tool)
messages: 99298
nosy: zzzeek
severity: normal
status: open
title: Python 3's 2to3 does not handle non-ascii source files
type: crash
versions: Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7922>
_______________________________________