[issue4626] compile() doesn't ignore the source encoding when a string is passed in

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> I don't like the change of API to PyTokenizer_FromString. 
> I would prefer another function like 
PyTokenizer_IgnoreCodingCookie()

Ok, I created a new function PyTokenizer_FromUnicode(). I 
choosed "FromUnicode" because the string is encoded in unicode (as 
UTF-8, even if it's not the wchar_t* type).

> The (char *) cast in PyTokenizer_FromString is unneeded.

The cast on the decode_str() result? It was already present in the 
original code. I removed it in my new patch.

> You need to indent the "else" clause after you test for 
ignore_cookie.

Ooops, I always have problems to generate a diff because my editor 
removes trailing spaces and then I have to ignore space changes to 
create  the diff.

> I'd like to see a test that shows that byte strings still have their
cookies examined.

test_pep263 has already two tests using a "#coding:" header.

Added file: http://bugs.python.org/file12889/tokenizer_ignore_cookie-4.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4626>
_______________________________________