[Cython] Bug: Extension Type inheriting from int cause a MemoryError

Stefan Behnel stefan_ml at behnel.de
Fri Jun 19 08:38:32 CEST 2015


Stephen LARROQUE schrieb am 15.06.2015 um 12:34:
> I am trying to make an extension type inheriting from int or cython.int (to
> do arithmetic operations in Galois Fields). However, this causes a
> MemoryError because it seems such extension type is not freed correctly.
> Other than that, it works perfectly well.
>
> cdef class ExtendedInt(int): pass
> 
> for j in xrange(10000000):
>     ExtendedInt(j)

It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this
code in intobject.c:

"""
static void
int_dealloc(PyIntObject *v)
{
    if (PyInt_CheckExact(v)) {
        Py_TYPE(v) = (struct _typeobject *)free_list;
        free_list = v;
    }
    else
        Py_TYPE(v)->tp_free((PyObject *)v);
}

static void
int_free(PyIntObject *v)
{
    Py_TYPE(v) = (struct _typeobject *)free_list;
    free_list = v;
}
"""

Your extension type automatically inherits the "int_free" slot function
from its base type, so the "else" case in "int_dealloc()" will in fact call
"int_free()" and append the object to the free list despite *not* being
exactly of type PyInt. Then, when creating a new ExtendedInt instance,
Python's int-subtype instantiation code *ignores* the free list and instead
creates a completely new object.

Thus, the free list keeps growing until it fills all memory and the process
dies. I created a CPython ticket.

https://bugs.python.org/issue24469

I guess we could hack up a work around for this in Cython somehow (Python
2.7 is an easy target being a dead end, after all), but let's wait what the
CPython devs have to say about this. Note, however, that any fix in a
future CPython 2.7.x release will not fix it in earlier Python 2.x
versions, so my guess is that we'd end up having to add a work-around on
our side anyway.

Stefan



More information about the cython-devel mailing list