[Python-Dev] Heap-allocated StructSequences

[Eddie Elizondo]

PEP-384 talks about defining a Stable ABI by making PyTypeObject opaque. Thus, banning the use of static PyTypeObjects.

Specifically, I’d like to focus on those static PyTypeObjects that are initialized as StructSequences. As of today, there are 14 instances of these types (in timemodule.c, posixmodule.c, etc.) within cpython/Modules. These are all initialized through PyStructSequence_InitType2. This is very problematic when trying to make these types conform to PEP-384 as they are used through static PyTypeObjects.

Problems:

  *   PyStructSequence_InitType2 overrides the PyTypeObject:

This C-API does a direct memcpy from a “prototype” structure. This effectively overrides anything set within the PyTypeObject. For example, if we were to initialize a heap allocated PyTypeObject and pass it on to this function, the C-API would just get rid of the Py_TPFLAG_HEAPTYPE flag, causing issues with the GC.


  *   PyStructSequence_InitType2 does not work with heap allocated PyTypeObjects:

Even if the function is fixed to preserve the state of the PyTypeObject and only overriding the specific slots (i.e. tp_new, tp_repr, etc.), it is expected that PyStructSequence_InitType2 will call PyType_Ready on the object. That means that the incoming object shouldn’t be initialized by a function such as PyType_FromSpec, as that would have already called PyType_Ready on it. Therefore, PyStructSequence_InitType2 will now have the responsibility of setting all the slots and properties of the PyHeapTypeObject, which is not feasible.


  *   PyStructSequence_NewType is non-functional:

This function was meant to be used as a way of creating a heap-allocated PyTypeObject that be passed to PyStructSequence_InitType2, effectively returning a heap allocated PyTypeObject. The current implementation doesn’t work in practice. Given that this struct is created in the heap, the GC has control over it. Thus, when the GC tries to traverse the type it complains with: “Error: type_traverse() called for non-heap type”, since it doesn’t have the Py_TPFLAG_HEAPTYPE flag. If we add the flag, we run into bullet point 1, if we are able to preserve the flag then we will still run into the problem of bullet point 2. Extra note: This C-API is not being used anywhere within CPython itself.

Solution:

  *   Fix the implementation of PyStructSequence_NewType:

The best solution would be to fix the implementation of this function. This can easily be done by dynamically creating a PyType_Spec and calling PyType_FromSpec

```

PyObject*

PyStructSequence_NewType(PyStructSequence_Desc *desc)

{

    // …

    PyType_Spec* spec = PyMem_NEW(PyType_Spec, 1);

    spec->name = desc->name;

    spec->basicsize = sizeof(PyStructSequence) - sizeof(PyObject *);

    spec->itemsize = sizeof(PyObject *);

    spec->flags = Py_TPFLAGS_DEFAULT;

    spec->slots = PyMem_NEW(PyType_Slot, 6);

    spec->slots[0].slot = Py_tp_dealloc;

    spec->slots[0].pfunc = (destructor)structseq_dealloc;

    // …

    bases = PyTuple_Pack(1, &PyTuple_Type);

   type = PyType_FromSpecWithBases(spec, bases);

    // …

```



This will cleanly create a heap allocated PyStructSequence which can be used just like any stack allocated PyTypeObject initialized through PyStructSequence_InitType2. This means that any C-Extension should be using PyStructSequence_NewType and only built-in types should be calling PyStructSequence_InitType2. This will enable these types to comply with PEP-384


As an extra, I already have patches for this proposal. They can be found here:

Branch: https://github.com/eduardo-elizondo/cpython/tree/heap-structseq
Reimplement PyStructSequence_NewType:  https://github.com/eduardo-elizondo/cpython/commit/413f8ca5bc008d84b3397ca1c9565c604d54b661
Patch timemodule with NewType: https://github.com/eduardo-elizondo/cpython/commit/0a35ea263a531cb03c06be9efc9e96d68162b308

Thoughts?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180904/dd580b85/attachment-0001.html>