[New-bugs-announce] [issue39873] Debug mode: check if objects are valid

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

One of the worst issue that I had to debug is a crash in the Python garbage collector. It is usually a crash in visit_decref(). See my notes:
https://pythondev.readthedocs.io/debug_tools.html#debug-crash-in-garbage-collection-visit-decref

My previous attempt to help debugging such issue failed: bpo-36389 "Add gc.enable_object_debugger(): detect corrupted Python objects in the GC". The idea was to check frequently if all objects tracked by the GC are valid. The problem is that even if the check looked trivial, checking all objects made Python way slower. Even when I tried to only check objects of the "young" GC generation (generation 0), it was still too slow.

Here I propose a different approach: attempt to only check objects when they are accessed. Recently, I started to replace explicit cast to (PyObject *) type with an indirection: a new _PyObject_CAST() macro which should be the only way to cast any object pointer to (PyObject *).

/* Cast argument to PyObject* type. */
#define _PyObject_CAST(op) ((PyObject*)(op))

This macro is used in many "core" macros like Py_TYPE(op), Py_REFCNT(op), Py_SIZE(op), Py_SETREF(op, op2), Py_VISIT(op), etc.

The idea here is to inject code in _PyObject_CAST(op) when Python is built in debug mode to ensure that the object is valid. The intent is to detect corrupted objects earlier than a garbage collection, to ease debugging C extensions.

The checks should be limited to reduce the performance overhead.

Attached PR implemnts this idea.

----------
components: Interpreter Core
messages: 363499
nosy: vstinner
priority: normal
severity: normal
status: open
title: Debug mode: check if objects are valid
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39873>
_______________________________________