[Python-Dev] PEP 490: Chain exceptions at C level

M.-A. Lemburg mal at egenix.com
Sat Jun 20 21:16:48 CEST 2015


On 20.06.2015 09:30, Victor Stinner wrote:
> Hi,
> 
> I didn't get much feedback on this PEP. Since the Python 3.6 branch is
> open (default), it's probably better to push such change in the
> beginning of the 3.6 cycle, to catch issues earlier.
> 
> Are you ok to chain exceptions at C level by default?

I think it's a good idea to make C APIs available to
simplify chaining exceptions at the C level, but don't
believe that always doing this by default is a good idea.
It should really be a case-by-case decision, IMO.

Note that Python exceptions are cheap to raise in C
(very much unlike in Python), so making this more
expensive by default would introduce a significant
overhead - without much proven benefit.

More below...

> PEP: 490
> Title: Chain exceptions at C level
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner <victor.stinner at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 25-March-2015
> Python-Version: 3.6
> 
> 
> Abstract
> ========
> 
> Chain exceptions at C level, as already done at Python level.
> 
> 
> Rationale
> =========
> 
> Python 3 introduced a new killer feature: exceptions are chained by
> default, PEP 3134.
> 
> Example::
> 
>     try:
>         raise TypeError("err1")
>     except TypeError:
>         raise ValueError("err2")
> 
> Output::
> 
>     Traceback (most recent call last):
>       File "test.py", line 2, in <module>
>         raise TypeError("err1")
>     TypeError: err1
> 
>     During handling of the above exception, another exception occurred:
> 
>     Traceback (most recent call last):
>       File "test.py", line 4, in <module>
>         raise ValueError("err2")
>     ValueError: err2
> 
> Exceptions are chained by default in Python code, but not in
> extensions written in C.
> 
> A new private ``_PyErr_ChainExceptions()`` function was introduced in
> Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called
> explicitly to chain exceptions and its usage is not trivial.
> 
> Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport``
> module to chain the previous ``OSError`` to a new ``ZipImportError``
> exception::
> 
>     PyObject *exc, *val, *tb;
>     PyErr_Fetch(&exc, &val, &tb);
>     PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
>     _PyErr_ChainExceptions(exc, val, tb);
> 
> This PEP proposes to also chain exceptions automatically at C level to
> stay consistent and give more information on failures to help
> debugging. The previous example becomes simply::
> 
>     PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
> 
> 
> Proposal
> ========
> 
> Modify PyErr_*() functions to chain exceptions
> ----------------------------------------------
> 
> Modify C functions raising exceptions of the Python C API to
> automatically chain exceptions: modify ``PyErr_SetString()``,
> ``PyErr_Format()``, ``PyErr_SetNone()``, etc.
> 
> 
> Modify functions to not chain exceptions
> ----------------------------------------
> 
> Keeping the previous exception is not always interesting when the new
> exception contains information of the previous exception or even more
> information, especially when the two exceptions have the same type.
> 
> Example of an useless exception chain with ``int(str)``::
> 
>     TypeError: a bytes-like object is required, not 'type'
> 
>     During handling of the above exception, another exception occurred:
> 
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     TypeError: int() argument must be a string, a bytes-like object or
> a number, not 'type'
> 
> The new ``TypeError`` exception contains more information than the
> previous exception. The previous exception should be hidden.
> 
> The ``PyErr_Clear()`` function can be called to clear the current
> exception before raising a new exception, to not chain the current
> exception with a new exception.
> 
> 
> Modify functions to chain exceptions
> ------------------------------------
> 
> Some functions save and then restore the current exception. If a new
> exception is raised, the exception is currently displayed into
> sys.stderr or ignored depending on the function.  Some of these
> functions should be modified to chain exceptions instead.
> 
> Examples of function ignoring the new exception(s):
> 
> * ``ptrace_enter_call()``: ignore exception
> * ``subprocess_fork_exec()``: ignore exception raised by enable_gc()
> * ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised
>   by trying to display the bootstrap function to ``sys.stderr``
> * ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore
>   exception raised by looking for a key in the dictionary
> * ``_PyErr_TrySetFromCause()``: ignore exception
> * ``PyFrame_LocalsToFast()``: ignore exception raised by
>   ``dict_to_map()``
> * ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used
>   to debug, to inspect a running process, it should not modify the
>   Python state.
> * ``Py_ReprLeave()``: ignore exception "because there is no way to
>   report them"
> * ``type_dealloc()``: ignore exception raised by
>   ``remove_all_subclasses()``
> * ``PyObject_ClearWeakRefs()``: ignore exception?
> * ``call_exc_trace()``, ``call_trace_protected()``: ignore exception
> * ``remove_importlib_frames()``: ignore exception
> * ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example:
>   ignore exception?
> * ``flush_io()``: ignore exception
> * ``sys_write()``, ``sys_format()``: ignore exception
> * ``_PyTraceback_Add()``: ignore exception
> * ``PyTraceBack_Print()``: ignore exception

Which of these do you think would benefit from chaining exceptions ?

> Examples of function displaying the new exception to ``sys.stderr``:
> 
> * ``atexit_callfuncs()``: display exceptions with
>   ``PyErr_Display()`` and return the latest exception, the function
>   calls multiple callbacks and only returns the latest exception
> * ``sock_dealloc()``: log the ``ResourceWarning`` exception with
>   ``PyErr_WriteUnraisable()``
> * ``slot_tp_del()``: display exception with
>   ``PyErr_WriteUnraisable()``
> * ``_PyGen_Finalize()``: display ``gen_close()`` exception with
>   ``PyErr_WriteUnraisable()``
> * ``slot_tp_finalize()``: display exception raised by the
>   ``__del__()`` method with ``PyErr_WriteUnraisable()``
> * ``PyErr_GivenExceptionMatches()``: display exception raised by
>   ``PyType_IsSubtype()`` with ``PyErr_WriteUnraisable()``

Same here.

In many cases, there's no way to report exceptions at all,
so chaining them make things better :-)

> Backward compatibility
> ======================
> 
> A side effect of chaining exceptions is that exceptions store
> traceback objects which store frame objects which store local
> variables.  Local variables are kept alive by exceptions. A common
> issue is a reference cycle between local variables and exceptions: an
> exception is stored in a local variable and the frame indirectly
> stored in the exception. The cycle only impacts applications storing
> exceptions.

It's not only about reference cycles. Tracebacks can also keep
files, sockets and other external resources open as well as
prevent large memory areas from being freed.

> The reference cycle can now be fixed with the new
> ``traceback.TracebackException`` object introduced in Python 3.5. It
> stores informations required to format a full textual traceback without
> storing local variables.
> 
> The ``asyncio`` is impacted by the reference cycle issue. This module
> is also maintained outside Python standard library to release a
> version for Python 3.3.  ``traceback.TracebackException`` will maybe
> be backported in a private ``asyncio`` module to fix reference cycle
> issues.
> 
> 
> Alternatives
> ============
> 
> No change
> ---------
> 
> A new private ``_PyErr_ChainExceptions()`` function is enough to chain
> manually exceptions.
> 
> Exceptions will only be chained explicitly where it makes sense.
> 
> 
> New helpers to chain exceptions
> -------------------------------
> 
> Functions like ``PyErr_SetString()`` don't chain automatically
> exceptions. To make the usage of ``_PyErr_ChainExceptions()`` easier,
> new private functions are added:
> 
> * ``_PyErr_SetStringChain(exc_type, message)``
> * ``_PyErr_FormatChain(exc_type, format, ...)``
> * ``_PyErr_SetNoneChain(exc_type)``
> * ``_PyErr_SetObjectChain(exc_type, exc_value)``
> 
> Helper functions to raise specific exceptions like
> ``_PyErr_SetKeyError(key)`` or ``PyErr_SetImportError(message, name,
> path)`` don't chain exceptions.  The generic
> ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` should be used
> to chain exceptions with these helper functions.
> 
> 
> Appendix
> ========
> 
> PEPs
> ----
> 
> * `PEP 3134 -- Exception Chaining and Embedded Tracebacks
>   <https://www.python.org/dev/peps/pep-3134/>`_ (Python 3.0):
>   new ``__context__`` and ``__cause__`` attributes for exceptions
> * `PEP 415 - Implement context suppression with exception attributes
>   <https://www.python.org/dev/peps/pep-0415/>`_ (Python 3.3):
>   ``raise exc from None``
> * `PEP 409 - Suppressing exception context
>   <https://www.python.org/dev/peps/pep-0409/>`_
>   (superseded by the PEP 415)
> 
> 
> Python C API
> ------------
> 
> The header file ``Include/pyerror.h`` declares functions related to
> exceptions.
> 
> Functions raising exceptions:
> 
> * ``PyErr_SetNone(exc_type)``
> * ``PyErr_SetObject(exc_type, exc_value)``
> * ``PyErr_SetString(exc_type, message)``
> * ``PyErr_Format(exc, format, ...)``
> 
> Helpers to raise specific exceptions:
> 
> * ``PyErr_BadArgument()``
> * ``PyErr_BadInternalCall()``
> * ``PyErr_NoMemory()``
> * ``PyErr_SetFromErrno(exc)``
> * ``PyErr_SetFromWindowsErr(err)``
> * ``PyErr_SetImportError(message, name, path)``
> * ``_PyErr_SetKeyError(key)``
> * ``_PyErr_TrySetFromCause(prefix_format, ...)``
> 
> Manage the current exception:
> 
> * ``PyErr_Clear()``: clear the current exception,
>   like ``except: pass``
> * ``PyErr_Fetch(exc_type, exc_value, exc_tb)``
> * ``PyErr_Restore(exc_type, exc_value, exc_tb)``
> * ``PyErr_GetExcInfo(exc_type, exc_value, exc_tb)``
> * ``PyErr_SetExcInfo(exc_type, exc_value, exc_tb)``
> 
> Others function to handle exceptions:
> 
> * ``PyErr_ExceptionMatches(exc)``: check to implement
>   ``except exc:  ...``
> * ``PyErr_GivenExceptionMatches(exc1, exc2)``
> * ``PyErr_NormalizeException(exc_type, exc_value, exc_tb)``
> * ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)``
> 
> 
> Python Issues
> -------------
> 
> Chain exceptions:
> 
> * `Issue #23763: Chain exceptions in C
>   <http://bugs.python.org/issue23763>`_
> * `Issue #23696: zipimport: chain ImportError to OSError
>   <http://bugs.python.org/issue23696>`_
> * `Issue #21715: Chaining exceptions at C level
>   <http://bugs.python.org/issue21715>`_: added
>   ``_PyErr_ChainExceptions()``
> * `Issue #18488: sqlite: finalize() method of user function may be
>   called with an exception set if a call to step() method failed
>   <http://bugs.python.org/issue18488>`_
> * `Issue #23781: Add private _PyErr_ReplaceException() in 2.7
>   <http://bugs.python.org/issue23781>`_
> * `Issue #23782: Leak in _PyTraceback_Add
>   <http://bugs.python.org/issue23782>`_
> 
> Changes preventing to loose exceptions:
> 
> * `Issue #23571: Raise SystemError if a function returns a result with an
>   exception set <http://bugs.python.org/issue23571>`_
> * `Issue #18408: Fixes crashes found by pyfailmalloc
>   <http://bugs.python.org/issue18408>`_
> 
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.
> 
> 
> 
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 20 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78
2015-06-10: Released mxODBC Plone/Zope DA 2.2.2   http://egenix.com/go76
2015-07-20: EuroPython 2015, Bilbao, Spain ...             30 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


More information about the Python-Dev mailing list