[Python-Dev] Problem with PyObject_DEL in pydebug mode

[Victor Stinner]

Hi,

I filled an issue about the crash: "import re; re.finditer("a", {})"
   http://bugs.python.org/issue3299

It appears quickly that the bug is specific to Python compiled in pydebug 
mode, or to be exact: when Py_TRACE_REFS is defined.

== The Py_TRACE_REFS option ==

The problem is that PyObject_Del(obj) and PyObject_DEL(obj) don't remove obj 
from the "object list" (_ob_prev and _ob_next attributes of all objects when 
Py_TRACE_REFS is defined). And so obj will be reused later (maybe removed by 
garbage collector? or something like that) whereas it's invalid (memory 
freed).

PyObject_NEW(obj) and PyObject_NEW_VAR(obj) call PyObject_Init() which 
registers the obj to the object list. So a developer can expect that 
PyObject_DEL(obj) will remove the object from this list.

PyObject_Del(obj) and PyObject_DEL(obj) are used on object initialization 
failure, but also in "dealloc" callbacks.

Classic object destruction is done by Py_DECREF(): when reference count is 
zero, dealloc() is called. PyObject_Del/DEL are different because they just 
free memory but don't call dealloc() whereas some attributes may be set (and 
some other may be uninitialized or NULL).


== Solutions ==

(a) Replace PyObject_Del/PyObject_DEL by Py_DECREF to call dealloc callback 
which will call PyObject_Del/PyObject_DEL. I prefer this solution because 
dealloc is called and so we make that all attributes are deinitialized. 

New problem: dealloc expects that the object is fully initialized (all 
attributes are set and are not NULL), which is wrong is initialization fails. 
Eg. with re module,scanner_dealloc() calls Py_DECREF(self->pattern); whereas 
pattern attribute is NULL. Fix: replace Py_DECREF by Py_XDECREF. But all 
dealloc have to be reviewed.

(b) Fix PyObject_Del/PyObject_DEL to remove object from the object list

(c) Create new macro which is PyObject_Del/PyObject_DEL + remove the object 
from the list

(d) Never use Py_TRACE_REFS :-)


I wrote many informations in http://bugs.python.org/issue3299.

-- 
Victor Stinner aka haypo
http://fusil.hachoir.org/