[issue2301] [Py3k] No text shown when SyntaxError (when not UTF8)

[Hirokazu Yamamoto]

Hirokazu Yamamoto <ocean-city at users.sourceforge.net> added the comment:

Hello. I tracked down source code and found where err->text is set.

Index: Parser/parsetok.c
===================================================================
--- Parser/parsetok.c	(revision 61411)
+++ Parser/parsetok.c	(working copy)
@@ -218,7 +218,7 @@
 			assert(tok->cur - tok->buf < INT_MAX);
 			err_ret->offset = (int)(tok->cur - tok->buf);
 			len = tok->inp - tok->buf;
-			text = PyTokenizer_RestoreEncoding(tok, len, &err_ret->offset);
+/*			text = PyTokenizer_RestoreEncoding(tok, len, &err_ret->offset); */
 			if (text == NULL) {
 				text = (char *) PyObject_MALLOC(len + 1);
 				if (text != NULL) {

It seems tok->buf is encoded with UTF-8, and
PyTokenizer_RestoreEncoding() resotores it to original encoding of
source file. So I tried above patch, output was expected on cp932/euc_jp
source files.

Maybe this function is not needed in py3k? I cannot find other place
where this function is used.

# Probably PyErr_ProgramText() needs more effort to be fixed.

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue2301>
__________________________________