On 2019-03-04, Carl Shapiro wrote:
[...] instances of types created in the C-API can be allocated outside of the Python heap. Would that feature be preserved? You could keep it with PyObject* but it might be harder to do with PyHandle.
I can only speak for myself but I would like to kill off non-heap allocated types. That's not easy because nearly every extension module that defines a type does so using a static structure for the new type (not heap allocated). Some discussion here:
https://bugs.python.org/issue35810We have PyType_FromSpec() but converting an extension module to use it is non-trivial. I was wondering if we can make an easier change. Can we just make a version of PyType_Ready() that copies the static structure into a heap allocated type and then returns that? Then, fixing the extension modules is pretty easy. Instead of:
PyType_Ready(&MyType);you do:
MyType = PyType_FromStatic(&MyTypeDef);The related thing I would like to change is to force all PyObject structures to be allocated by CPython memory allocators. Aside from statically allocated types, I believe that is already the case for objects with the GC flag. The object memory has to come from _PyObject_GC_New() or _PyObject_GC_NewVar(). That is not the case for non-GC objects, as far as I'm aware. At least, it was the case years ago that extension types could use their own malloc implementation, for example.
There are some legitimate reasons to want to use a special allocator. However, I don't think those are good enough reasons for what we are giving up for supporting that. I suspect most people are not even aware that is a thing. I'm not sure it even works anymore. When we implemented obmalloc, it was a considerable challenge to keep it working, as I recall.
BTW, in the above example, "MyType" is nearly always a static variable in the extension module. All those static PyObject variables are a contributing factor to making CPython shutdown complicated and flakey. Look at Py_FinilizeEx() if you are brave. It is a pile of doggy dodo. Dirty hacks on hacks, slow, and doesn't really work correctly. Instead of keeping a static variable reference to the type, you can add the new type of the globals of the new extension module, e.g.
PyObject *mytype = PyType_FromStatic(&MyTypeDef);
PyModule_AddObject(module, "MyType", mytype);That way, you don't have an extra ref to MyType keeping it alive longer than the module it is contained within. If you need quick access to the type object, it really should be stored in the per-interpreter module data.
Regards,
Neil