[issue3574] compile() cannot decode Latin-1 source encodings

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

@brett.cannon: I found it: ast.c used a hack for iso-8859-1! Since 
this hack introduces a bug (your compile(...) example), I prefer to 
remove it to simplify to code. The new patch just removes the hack in 
tokenizer.c and ast.c. It does also document encoding and enc 
attributes of tok_state in tokenizer.h.

Using tokenizer_iso-8859-1-patch2.patch, all tests (test_pep3120.py, 
iso.py, test_shlex.py, etc.) are OK.

Added file: http://bugs.python.org/file11700/tokenizer_iso-8859-1-patch2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3574>
_______________________________________