[Python-Dev] C API for gc.enable() and gc.disable()

[Amaury Forgeot d'Arc]

Hello,

Andrey Zhmoginov wrote:
> I don't know if the following question is relevant, but it seems that many
> people here are familiar with Python cyclic garbage collector.
> I see Python [v2.5.2 (r252:60911, Jul 31 2008, 17:28:52)] crashing with
> Segment fault when I extend Python with a very simple module. This behavior
> is observed when I create a thousand of lists (it does not crash with
> 10-100) in the module with the garbage collector turned on. When I turn it
> off - everything is perfect. I suspect that it is my module, but if it is a
> Python bug ( _GC_Malloc? memory leak somewhere?), it may be worth reporting.
>
> The gdb "where" reply is the following:
>
>   #0  0x080d8de9 in PyErr_Occurred ()
>   #1  0x080f508f in _PyObject_GC_Malloc ()
>   #2  0x080f5155 in _PyObject_GC_New ()
>   #3  0x08079c98 in PyList_New ()
>   #4  0xb7f53519 in draw_d_simple () from ./rt/rt_core.so
>   #5  0xb7cf7833 in ffi_call_SYSV () from
> /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #6  0xb7cf766a in ffi_call () from
> /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #7  0xb7cf2534 in _CallProc () from
> /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #8  0xb7cec02a in ?? () from /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #9  0x0805cb97 in PyObject_Call ()
>   #10 0x080c7aa7 in PyEval_EvalFrameEx ()
>   #11 0x080c96e5 in PyEval_EvalFrameEx ()
>   #12 0x080cb1f7 in PyEval_EvalCodeEx ()
>   #13 0x080cb347 in PyEval_EvalCode ()
>   #14 0x080ea818 in PyRun_FileExFlags ()
>   #15 0x080eaab9 in PyRun_SimpleFileExFlags ()
>   #16 0x08059335 in Py_Main ()
>   #17 0x080587f2 in main ()
>
> The crashing python code is:
>
>   from ctypes import *
>   import gc
>   core = CDLL( "./tst.so" )
>   core.a.argtypes = []
>   core.a.restype = py_object
>   #gc.disable()
>   core.a()
>   #gc.enable()
>
> And tst.cpp is:
>
>   #include <Python.h>
>   extern "C" { PyObject *a(); }
>   PyObject *a()
>   {
>       int n = 1000;
>       PyObject *item;
>       for ( int i = 0; i < n; i++ ) item = PyList_New( 0 );      // Crashes
> here (somewhere in between).
>       return item;
>   }
>
> tst.cpp is compiled with:
>
>   g++ -shared -Wl,-soname,tst.tmp.so -o tst.so tst.o
>   g++ -I /usr/include/python2.5 -Wall -fPIC -o tst.o -c tst.cpp

This has nothing to do with the garbage collection, but with the GIL
(the famous Global Interpreter Lock http://docs.python.org/api/threads.html )

When ctypes calls a CDLL function, it releases the GIL (to let
eventual other threads run)
Your example crashes because it calls python API functions when the
GIL is not held,
and is so invalid.

There are two solutions:
- re-acquire the GIL in your C functions with PyGILState_Ensure() & co
- use ctypes.PyDLL( "./tst.so" ), which does not release the GIL.

Hope this helps,

-- 
Amaury Forgeot d'Arc