[issue3574] compile() cannot decode Latin-1 source encodings

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

It looks like the problem of fix_latin.diff is the decoding_state: 
it's set to STATE_NORMAL whereas current behaviour is to stay in state 
STATE_RAW.

I wrote another patch which is a mix of case 1 (utf-8: just set 
tok->encoding) and case 2 (another charset: set tok->enc, 
tok->encoding and tok>decoding_state): a new case 3 which set enc, 
encoding but stay a the state STATE_RAW. I don't understand my patch, 
so review it (twice or more :-D). Using my patch:
 - compile(...) works
 - test_shlex.py works
 - test_pep3120.py
 - iso.py works

Added file: http://bugs.python.org/file11698/tokenizer_iso-8859-1.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3574>
_______________________________________