[issue3996] PyOS_CheckStack does not work

[Amaury Forgeot d'Arc]

New submission from Amaury Forgeot d'Arc <amauryfa at gmail.com>:

On Windows, PyOS_CheckStack is supposed to protect the interpreter from
stack overflow. But doing this, it always crashes when the stack is
nearly full.

The reason is a bad check of the return value of _resetstkoflw():
according to MSDN, the return value is "Nonzero if the function
succeeds, zero if it fails.":
http://msdn.microsoft.com/en-us/library/89f73td2.aspx

The patch below is enough to replace the "Fatal Python error: Could not
reset the stack!" into a "MemoryError: stack overflow" exception.

Tested with:
>>> loop = None,
>>> for x in xrange(100000): loop = {'x': loop}
...
>>> len(repr(loop))



Index: Python/pythonrun.c
===================================================================
--- Python/pythonrun.c  (revision 66486)
+++ Python/pythonrun.c  (working copy)
@@ -1749,7 +1755,7 @@
                        EXCEPTION_EXECUTE_HANDLER :
                        EXCEPTION_CONTINUE_SEARCH) {
                int errcode = _resetstkoflw();
-               if (errcode)
+               if (errcode == 0)
                {
                        Py_FatalError("Could not reset the stack!");
                }

----------
assignee: loewis
components: Windows
keywords: patch
messages: 74024
nosy: amaury.forgeotdarc, loewis
severity: normal
status: open
title: PyOS_CheckStack does not work
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3996>
_______________________________________