Hi Joannah,
IMO you would save time by investing on HPy. It exists and solves your issue.
On Tue, Jun 8, 2021 at 1:09 PM Joannah Nanjekye <nanjekyejoannah@gmail.com> wrote:
Where now PyObject becomes an index into an internal array that stored all values that had to be given to. This means that when a value is in that array, it would not be collected by Python. When the C function returns its whole array is erased, and the values used by the function are collected.
The C API doesn't make any assumption about object lifetime. You should assume that objects are still used after the function exit.
Example (pseudo-code):
void store(PyObject list) { PyObject *obj = PyLong_FromLong(1); // obj is used after PyLong_FromLong() exited
PyList_Append(list, obj); Py_DECREF(obj);
// oh oh, obj is stored in list and must now remain valid // until list is destroyed }
By the way, how do you magically clear your "array" in store()? How do you inject code to clear it? In HPy, the array is "always there", it's not cleared: HPy_Close() simply punchs holes in it.
Changing PyObject structure to avoid direct access into the PyObject is the purpose of my PEP 620. This PEP remains controversial (it's not accepted) and introduces many incompatible changes in the C API. Until *all* structures are made opaque, I'm not sure that you can consider changing PyObject definition (to "unsigned int", or anything else). The practical problem is that the PyObject *structure* is "leaked" into all sub-types. For example, the PyUnicodeObject structure (str type) starts with "PyObject ob_base;". Right now, it remains possible to access directly the PyObject.ob_type member of a PyUnicodeObject using "((PyObject*)obj)->ob_type" (please don't do that, use Py_TYPE(obj) ;-)).
What is some feedback on this approach and am I overconfident of having reasonable backward compatibility?
It is simply totally backward incompatible and so impossible practically today.
Tons of code still access directly into the PyObject structure: see https://bugs.python.org/issue39573
By the way, my change to deny "Py_TYPE(obj) = new_type;" (direct access to PyObject.ob_type) has been reverted for the 2nd time ;-) (It broke Windows buildbots building Python in debug mode. Issue with the stack usage and function inlining.)
Victor
Night gathers, and now my watch begins. It shall not end until my death.