[New-bugs-announce] [issue11168] UnicodeEncodeError on recusion limit if the script filename is undecodable

STINNER Victor report at bugs.python.org
Thu Feb 10 13:21:42 CET 2011

New submission from STINNER Victor <victor.stinner at haypocalc.com>:

If the script filename is not decodable from the filesystem encoding, Python fails with a UnicodeEncodeError when we reach the recursion limit. The problem doesn't come from the user script, but from Python internals. It is difficult to understand and the user expects a 

A longer explanation:

test_runpy fails with a pydebug build if the script filename is not encodable to UTF-8. In pydebug build only, PyEval_EvalFrameEx() encodes the frame filename to UTF-8. If the filename contains a surrogate character (which only occurs on UNIX with undecodable filename),the encoding function fails. PyEval_EvalFrameEx() ignores the error except if we hit the recusion limit (if the overflowed attribute of the thread state if set): in this case, the error is not ignored.

To reproduce the problem, change the Python directory (your local repository) to an undecodable filename (eg. b'py3k\xe9\xff' with UTF-8 locale encoding) and run: ./python Lib/test/regrtest.py test_py.

Solutions :

I propose to remove the filename variable from PyEval_EvalFrameEx() because it is only used by the old gdb macros: python-gdb.py doesn't need it anymore.

=> see attached patch

Or if you really want to keep it, tstate->overflowed should be reinitialized. But on overflow, other variables are changed, like _Py_CheckRecursionLimit. I don't know this code enough to write a correct patch, but the minimal patch is:

--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -1234,7 +1234,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
         filename = _PyUnicode_AsString(co->co_filename);
         if (filename == NULL && tstate->overflowed) {
             /* maximum recursion depth exceeded */
-            goto exit_eval_frame;
+            tstate->overflowed = 0;
         PyErr_Restore(error_type, error_value, error_traceback);

components: Unicode
files: ceval_filename.patch
keywords: patch
messages: 128285
nosy: haypo
priority: normal
severity: normal
status: open
title: UnicodeEncodeError on recusion limit if the script filename is undecodable
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file20728/ceval_filename.patch

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list