[Cython] Metaclass bug corrupts the heap

[Dénes Vadász]

Hello,

according to valgrind the following Cython fragment causes a heap 
corruption (with Python 2.7.5 and Cython 0.19.1 on Ubuntu):

    cdef class MetaClass(type):
         cdef int i

    class MyClass(object):
         __metaclass__ = MetaClass



Please find below the result of a many hour investigation (originally 
triggered by random crashes and heap corruptions of a bigger Cython 
program that uses metaclasses).

MetaClass is compiled to a C structure that extends PyTypeObject (a cc. 
200 byte structure on a 32-bit architecture):

    struct __pyx_obj_4meta_MetaClass {
       PyTypeObject __pyx_base;
       int i;
    };


Instances of PyTypeObject are supposed to be allocated statically when 
initializing extension modules, because the structure does not support 
(among others) garbage collection. However, when MyClass is created, an 
instance of struct __pyx_obj_4meta_MetaClass (cc. 200 + 4 bytes) is 
dynamically allocated by the Python memory management machinery. The 
machinery then tries to initialize the allocated memory. The problem is 
that it expects that the first member of struct 
__pyx_obj_4meta_MetaClass is of type PyHeapTypeObject (a cc. 500 byte 
structure) and after a type cast it writes to the tail members of the 
assumed PyHeapTypeObject, i.e. way beyond the allocated cc. 200 + 4 
bytes, corrupting the object heap. This corruption is nicely reported by 
valgrind.

The proposed fix is to make __pyx_obj_4meta_MetaClass extend 
PyHeapTypeObject:

    struct __pyx_obj_4meta_MetaClass {
       PyHeapTypeObject __pyx_base;
       int i;
    };


This can be achieved by adding the below 2 lines to Compiler/Builtin.py:

    382a383,384
     >         elif name == 'type':
     >             objstruct_cname = 'PyHeapTypeObject'


So that the init_builtin_types function becomes:

    def init_builtin_types():
         global builtin_types
         for name, cname, methods in builtin_types_table:
             utility = builtin_utility_code.get(name)
             if name == 'frozenset':
                 objstruct_cname = 'PySetObject'
             elif name == 'bool':
                 objstruct_cname = None
    *       elif name == 'type':*
    *           objstruct_cname = 'PyHeapTypeObject'*
             else:
                 objstruct_cname = 'Py%sObject' % name.capitalize()
             the_type = builtin_scope.declare_builtin_type(name, cname,
    utility, objstruct_cname)
             builtin_types[name] = the_type
             for method in methods:
                 method.declare_in_type(the_type)


After patching my Cython installation with the above, valgrind stopped 
to complain and there were no more crashes.

Please consider adding this fix in the next release of Cython.

Regards

Dénes Vadász
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20130911/289cceae/attachment.html>