[New-bugs-announce] [issue40789] C-level destructor in PySide2 breaks gen_send_ex, which assumes it's safe to call Py_DECREF with a live exception

Nathaniel Smith report at bugs.python.org
Tue May 26 23:56:31 EDT 2020


New submission from Nathaniel Smith <njs at pobox.com>:

Consider the following short program. demo() is a trivial async function that creates a QObject instance, connects a Python signal, and then exits. When we call `send(None)` on this object, we expect to get a StopIteration exception.

-----

from PySide2 import QtCore

class MyQObject(QtCore.QObject):
    sig = QtCore.Signal()

async def demo():
    myqobject = MyQObject()
    myqobject.sig.connect(lambda: None)
    return 1

coro = demo()
try:
    coro.send(None)
except StopIteration as exc:
    print(f"OK: got {exc!r}")
except SystemError as exc:
    print(f"WTF: got {exc!r}")

-----

Actual output (tested on 3.8.2, but I think the code is present on all versions):

-----
StopIteration: 1
WTF: got SystemError("<method 'send' of 'coroutine' objects> returned NULL without setting an error")
-----

So there are two weird things here: the StopIteration exception is being printed on the console for some reason, and then the actual `send` method is raising SystemError instead of StopIteration.

Here's what I think is happening:

In genobject.c:gen_send_ex, when the coroutine finishes, we call _PyGen_SetStopIterationValue to raise the StopIteration exception:

https://github.com/python/cpython/blob/404b23b85b17c84e022779f31fc89cb0ed0d37e8/Objects/genobject.c#L241

Then, after that, gen_send_ex clears the frame object and drops references to it:

https://github.com/python/cpython/blob/404b23b85b17c84e022779f31fc89cb0ed0d37e8/Objects/genobject.c#L266-L273

At this point, the reference count for `myqobject` drops to zero, so its destructor is invoked. And this destructor ends up clearing the current exception again. Here's a stack trace:

-----
#0  0x0000000000677eb7 in _PyErr_Fetch (p_traceback=0x7ffd9fda77d0, 
    p_value=0x7ffd9fda77d8, p_type=0x7ffd9fda77e0, tstate=0x2511280)
    at ../Python/errors.c:399
#1  _PyErr_PrintEx (tstate=0x2511280, set_sys_last_vars=1) at ../Python/pythonrun.c:670
#2  0x00007f1afb455967 in PySide::GlobalReceiverV2::qt_metacall(QMetaObject::Call, int, void**) ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/libpyside2.abi3.so.5.14
#3  0x00007f1afaf2f657 in void doActivate<false>(QObject*, int, void**) ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/Qt/lib/libQt5Core.so.5
#4  0x00007f1afaf2a37f in QObject::destroyed(QObject*) ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/Qt/lib/libQt5Core.so.5
#5  0x00007f1afaf2d742 in QObject::~QObject() ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/Qt/lib/libQt5Core.so.5
#6  0x00007f1afb852681 in QObjectWrapper::~QObjectWrapper() ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/PySide2/QtCore.abi3.so
#7  0x00007f1afbf785bb in SbkDeallocWrapperCommon ()
   from /home/njs/.user-python3.8/lib/python3.8/site-packages/shiboken2/libshiboken2.abi3.so.5.14
#8  0x00000000005a4fbc in subtype_dealloc (self=<optimized out>)
    at ../Objects/typeobject.c:1289
#9  0x00000000005e8c08 in _Py_Dealloc (op=<optimized out>) at ../Objects/object.c:2215
#10 _Py_DECREF (filename=0x881795 "../Objects/frameobject.c", lineno=430, 
    op=<optimized out>) at ../Include/object.h:478
#11 frame_dealloc (f=Frame 0x7f1afc572dd0, for file qget-min.py, line 12, in demo ())
    at ../Objects/frameobject.c:430
#12 0x00000000004fdf30 in _Py_Dealloc (
    op=Frame 0x7f1afc572dd0, for file qget-min.py, line 12, in demo ())
    at ../Objects/object.c:2215
#13 _Py_DECREF (filename=<synthetic pointer>, lineno=279, 
    op=Frame 0x7f1afc572dd0, for file qget-min.py, line 12, in demo ())
    at ../Include/object.h:478
#14 gen_send_ex (gen=0x7f1afbd08440, arg=<optimized out>, exc=<optimized out>, 
    closing=<optimized out>) at ../Objects/genobject.c:279

------

We can read the source for PySide::GlobalReceiverV2::qt_metacall here: https://sources.debian.org/src/pyside2/5.13.2-3/sources/pyside2/libpyside/globalreceiverv2.cpp/?hl=310#L310

And we see that it (potentially) runs some arbitrary Python code, and then handles any exceptions by doing:

if (PyErr_Occurred()) {
    PyErr_Print();
}

This is intended to catch exceptions caused by the code it just executed, but in this case, gen_send_ex ends up invoking it with an exception already active, so PySide2 gets confused and clears the StopIteration.

-----------------------------------

OK so... what to do. I'm actually not 100% certain whether this is a CPython bug or a PySide2 bug.

In PySide2, it could be worked around by saving the exception state before executing that code, and then restoring it afterwards.

In gen_send_ex, it could be worked around by dropping the reference to the frame before setting the StopIteration exception.

In CPython in general, it could be worked around by not invoking deallocators with a live exception... I'm actually pretty surprised that this is even possible! It seems like having a live exception when you start executing arbitrary Python code would be bad. So maybe that's the real bug? Adding both "asyncio" and "memory management" interest groups to the nosy.

----------
messages: 370046
nosy: asvetlov, lemburg, njs, tim.peters, twouters, yselivanov
priority: normal
severity: normal
status: open
title: C-level destructor in PySide2 breaks gen_send_ex, which assumes it's safe to call Py_DECREF with a live exception
versions: Python 3.10, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40789>
_______________________________________


More information about the New-bugs-announce mailing list