[Python-Dev] bpo-36829: Add sys.unraisablehook()

Victor Stinner vstinner at redhat.com
Wed May 15 21:23:55 EDT 2019


Hi,

I recently modified Python 3.8 to start logging I/O error in io
destructors when using the development mode (-X dev) or when Python is
built in debug mode. This change allows to debug bpo-18748 very
strange crash in PyThread_exit_thread(), when a thread closes an
arbitrary file descriptor by mistake.

The problem is that exceptions raised in a destructor cannot be passed
to the current function call: the exception is logged into sys.stderr
with a message like "Exception ignored in: ...". It's easy to miss
such message in stderr.

Thomas Grainger opened 36829 to propose to add -X abortunraisable
command line option: abort immediately the current process (by calling
Py_FatalError()) at the first PyErr_WriteUnraisable() call. Thomas had
like a way to easily detect these in CI. I'm not satisfied by his
proposal, since it only gives little control to the user on how
"unraisable exceptions" are handled: the process dies, that's all.

https://bugs.python.org/issue36829

I proposed a different approach: add a new sys.unraisablehook hook
which is called to handle an "unraisable exception". To handle them
differently, replace the hook. For example, I wrote a custom hook to
log these exceptions into a file (the output on the Python test suite
is interesting!). It also becomes trivial to reimplement Thomas's idea
(kill the process):

    import signal
    def hook(unraisable): signal.raise_signal(signal.SIGABRT)
    sys.unraisablehook = hook

I plan to merge my implementation soon, are you fine with that?

https://github.com/python/cpython/pull/13187

--

The first implementation of my API used sys.unraisablehook(exc_type,
exc_value, exc_tb, obj). The problem is that Serhiy Storchaka asked me
to add a new error message field which breaks the API: the API is not
future-proof.

I modified my API to create an object to pack arguments. The new API
becomes sys.unraisablehook(unraisable) where unraisable has 4 fields:
exc_type, exc_value, exc_tb, obj. This API is now future-proof: adding
a new field will not break existing custom hooks!

By the way, I like this idea of adding an optional error message, and
I plan to implement it once sys.unraisablehook is merged ;-) I already
implemented it previously, but I reverted my changes to make the PR
simpler to review.


Extract of the documentation:

"""
Add new :func:`sys.unraisablehook` function which can be overriden to control
how "unraisable exceptions" are handled. It is called when an exception has
occurred but there is no way for Python to handle it. For example, when a
destructor raises an exception or during garbage collection
(:func:`gc.collect`).
"""

My implementation has a limitation: if PyErr_WriteUnraisable() is
called after the Python finalization cleared sys attributes (almost
the last function call of Py_Finalize), the default hook is called
instead of the custom hook. In this case, sys.stderr is None and so
the default hook does nothing.

These late calls to PyErr_WriteUnraisable() cannot be catched with my
approached, whereas Thomas Grainger's command line option "-X
abortunraisable" allows that.

My concern with Thomas's approach is that if the problem is killed
with SIGABRT by such late PyErr_WriteUnraisable(), a low-level
debugger like gdb is needed to investigate the crash, since Python is
already finalized and so cannot be used to investigate.

I prefer to allow arbitrary hook with the limitation, rather than
always kill the process with SIGABRT at the first
PyErr_WriteUnraisable() and require to use a low-level debugger.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.


More information about the Python-Dev mailing list