[New-bugs-announce] [issue46836] [C API] Move PyFrameObject to the internal C API

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

I propose to move the PyFrameObject structure to the internal C API.


--


Between Python 3.10 and Python 3.11, the work on optimizing ceval.c modified deeply the PyFrameObject structure. Examples:

* The f_code member has been removed by in bpo-44032 by the commit b11a951f16f0603d98de24fee5c023df83ea552c.
* The f_frame member has been added in bpo-44590 by the commit ae0a2b756255629140efcbe57fc2e714f0267aa3.

Most members have been moved to a new PyFrameObject.f_frame member which has the type "struct _interpreter_frame*". Problem: this type is only part of the *internal* C API.

Moreover, accessing the few remaining members which "didn't change" became dangerous. For example, f_back can be NULL even if the frame has a previous frame: the PyFrame_GetBack() function *must* now be called. See bpo-46356 "[C API] Enforce usage of PyFrame_GetBack()".

Reading directly f_lineno was already dangerous since Python 2.3: the value is only valid if the value is greater than 0. It's way safer to use the clean PyFrame_GetLineNumber() API instead.

PyFrame_GetBack() was added to Python 3.9. You can use the pythoncapi_compat project to get this function on Python 3.8 and older:

=> https://pythoncapi-compat.readthedocs.io/

PyFrame_GetLineNumber() was added to the limited API in Python 3.10.

=> Documentation: https://docs.python.org/dev/c-api/reflection.html#c.PyFrame_GetBack


--


There *are* projects accessing directly PyFrameObject like the gevent project which sets the f_code member (moved to f_frame.f_code in Python 3.11). It's broken on Python 3.11:
https://bugs.python.org/issue40421#msg413719

Debuggers and profilers also want to read PyFrameObject directly. IMO for these *specific* use cases, using the *internal* C API is a legit use case and it's fine.

Moving PyFrameObject to the internal C API would clarify the situation. Currently, What's New in Python 3.11 documents the change this with warning:

"While the documentation notes that the fields of PyFrameObject are subject to change at any time, they have been stable for a long time and were used in several popular extensions. "


--


I'm mostly worried about Cython which still get and set many PyFrameObject members directly (ex: f_lasti, f_lineno, f_localsplus, f_trace), since there are no public functions for that.

=> https://bugs.python.org/issue40421#msg367550

Right now, I would suggest Cython to use the internal C API, and *later* consider adding new getter and setter functions. I don't think that we can solve all problems at once: it takes take to design clean API and use them in Cython.

Python 3.11 already broke Cython since most PyFrameObject members moved into the new "internal" PyFrameObject.f_frame API which requires using the internal C API to get "struct _interpreter_frame".

=> https://github.com/cython/cython/issues/4500


--


Using a frame using the *public* C API was and remains supported. Short example:
--
PyThreadState *tstate = PyThreadState_Get();
PyFrameObject* frame = PyThreadState_GetFrame(tstate);
int lineno = PyFrame_GetLineNumber(frame);
---

The PyFrameObject structure is opaque and members are not accessed directly: it's fine.

----------
components: C API
messages: 413795
nosy: vstinner
priority: normal
severity: normal
status: open
title: [C API] Move PyFrameObject to the internal C API
versions: Python 3.11

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue46836>
_______________________________________