[New-bugs-announce] [issue37347] Reference-counting problem in sqlite

Aleksandr Balezin report at bugs.python.org
Thu Jun 20 07:14:59 EDT 2019


New submission from Aleksandr Balezin <gescheit12 at gmail.com>:

There are a couple of bugs in sqlite bindings have been found related to reference-counting.
Short info:
sqlite connection class uses a dict to keep references to callbacks. This mechanism is not suitable for objects which equals but not the same. 
con = sqlite3.connect()
con.set_trace_callback(logger.debug) 
con.set_trace_callback(logger.debug)  # logger.debug == logger.debug is True, but logger.debug is logger.debug is False because logger.debug bound method is creating in every call
leads to segmentation fault during calling of trace_callback.
My patch fixes this behavior by using a dedicated variable for keeping references to each callback and using dict indexed by function name in case of named callbacks(e.g. create_function()).
Also, due to keeping objects in a variable or in a dict value, it is possible to use unhashable objects as callbacks. e.g. issue7478

Long version:
Sqlite under the hood use dict(called function_pinboard) to keep references to callbacks like progress_handler. 
It needs to decref callbacks objects after closing sqlite connection. 
This mechanism works tolerably(see bug with leaks) with functions but if you try to use bounded methods it causes a segmentation fault.
Let see how it works.

static PyObject *
Custom_set_callback(CustomObject *self, PyObject* args)
{
	PyObject* display_str;
	display_str = PyUnicode_FromFormat("set_callback called with cb=%R id=%i ob_refcnt=%i\n", args, args, args->ob_refcnt);
	PyObject_Print(display_str, stdout, Py_PRINT_RAW);
	if (PyDict_SetItem(self->function_pinboard, args, Py_None) == -1) return NULL;
	//sqlite3_trace(self->db, _trace_callback, trace_callback);
	self->callback_handler = args;
	display_str = PyUnicode_FromFormat("set_callback done for cb=%R id=%i ob_refcnt=%i\n", args, args, args->ob_refcnt);
	PyObject_Print(display_str, stdout, Py_PRINT_RAW);
	Py_RETURN_NONE;
}
static PyObject *
Custom_call_callback(CustomObject *self)
{
	PyObject* display_str;
	display_str = PyUnicode_FromFormat("call with id=%i ob_refcnt=%i\n", self->callback_handler ,
    self->callback_handler->ob_refcnt);
	PyObject_Print(display_str, stdout, Py_PRINT_RAW);
	Py_RETURN_NONE;
}

Python code:
>>>> class TEST:
        def log(self, msg=""):
            pass
>>>> t = TEST()
>>>> conn = Custom()
>>>> conn.set_trace_callback(t.log)
set_callback called with cb=<bound method TEST.log of <__main__.TEST object at 0x10bc60128>> id=196094408 ob_refcnt=1
set_callback done for cb=<bound method TEST.log of <__main__.TEST object at 0x10bc60128>> id=196094408 ob_refcnt=2
>>>> conn.set_trace_callback(t.log)
set_callback called with cb=<bound method TEST.log of <__main__.TEST object at 0x10bc60128>> id=196095112 ob_refcnt=1
set_callback done for cb=<bound method TEST.log of <__main__.TEST object at 0x10bc60128>> id=196095112 ob_refcnt=1
conn.call()
call with id=196095112 ob_refcnt=0

After second conn.set_trace_callback(t.log) call, object t.log reference-count is not increased because 't.log in self->function_pinboard' returns True thus self->function_pinboard[t.log] is not replaced and t.log is not increfed, but it replaces old object in self->callback_handler.
In the end, self->callback_handler keeps a pointer to t.log with ob_refcnt = 0.

Also, there is no cleaning of self->function_pinboard. This leads to leaks every object passed as callback(see test_leak() in bug.py).

----------
components: Extension Modules
messages: 346114
nosy: gescheit, ghaering
priority: normal
severity: normal
status: open
title: Reference-counting problem in sqlite
type: crash
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37347>
_______________________________________


More information about the New-bugs-announce mailing list