PyFrameObject structure has been moved to the internal C API
Hi,
tl; dr the C API of PyFrameObject changed. If you have issues, please speak up!
I moved the PyFrameObject structure to the internal C API headers. You should now use the public C API to access its members:
f_back: use :c:func:PyFrame_GetBack.f_blockstack: removed.f_builtins: usePyObject_GetAttrString((PyObject*)frame, "f_builtins").f_code: use :c:func:PyFrame_GetCode.f_gen: removed.f_globals: usePyObject_GetAttrString((PyObject*)frame, "f_globals").f_iblock: removed.f_lasti: usePyObject_GetAttrString((PyObject*)frame, "f_lasti"). Code usingf_lastiwithPyCode_Addr2Line()must use :c:func:PyFrame_GetLineNumberinstead.f_lineno: use :c:func:PyFrame_GetLineNumberf_locals: usePyObject_GetAttrString((PyObject*)frame, "f_locals").f_stackdepth: removed.f_state: no public API (renamed tof_frame.f_state).f_trace: no public API.f_trace_lines: usePyObject_GetAttrString((PyObject*)frame, "f_trace_lines")(it also be modified).f_trace_opcodes: usePyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes")(it also be modified).f_localsplus: no public API (renamed tof_frame.localsplus).f_valuestack: removed.
See What's New in Python 3.11 for more details (I completed the doc, but it will only be updated online tomorrow, there is a cron task once per day): https://docs.python.org/dev/whatsnew/3.11.html#id2
You can use the pythoncapi_compat project to get PyFrame_GetBack() and PyFrame_GetCode() on Python 3.8 and older: https://pythoncapi-compat.readthedocs.io/
If there is no public C API to access the PyFrameObject members used by your project, please speak up!
See for example my PR to use the public C API in coverage: https://github.com/nedbat/coveragepy/pull/1331
There is an on-going work for 2 years to add new getter functions for PyFrameObject: https://bugs.python.org/issue40421
In Python 3.10, there was no rush to make the structure internal. Things changed quickly in Python 3.11: a heavy rework on Python internals is on-going to optimize Python (especially ceval.c and how frames are created and used). For example, the _PyEval_EvalFrameDefault() function now gets a new InterpreterFrame structure, it no longer takes a PyFrameObject. Python 3.11 now creates Python frame objects on demand.
Because of these changes, reading directly PyFrameObject.f_back is now an error since the member is not always filled, but it wasn't a compiler error. You must now call the public PyFrame_GetBack() function rather than accessing directly to the member.
Since Python 2.3 (that's quite old ;-)), reading directly PyFrameObject.f_lineno is unsafe since it depends if the Python code is being traced or not: you must use PyFrame_GetLineNumber() instead.
See also the issue for the longer rationale: https://bugs.python.org/issue46836
I know that these changes are inconvenient, but I'm here if you need my help for updating your project! Moreover, Python 3.11 is scheduled for October, so there are still a few months to adapt your project, and maybe add new getter functions. The best would be to add all required getter functions before Python 3.11 beta1 scheduled for the beginning of May (2022-05-06).
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 25.02.2022 13:15, Victor Stinner wrote:
Hi,
tl; dr the C API of PyFrameObject changed. If you have issues, please speak up!
I moved the PyFrameObject structure to the internal C API headers. You should now use the public C API to access its members:
f_back: use :c:func:PyFrame_GetBack.
You only mention C APIs, but inspection tools are often written in Python. How can they access and walk the frames on the stack ?
Are these members still available as Python attributes of the frame objects ?
f_blockstack: removed.f_builtins: usePyObject_GetAttrString((PyObject*)frame, "f_builtins").f_code: use :c:func:PyFrame_GetCode.f_gen: removed.f_globals: usePyObject_GetAttrString((PyObject*)frame, "f_globals").f_iblock: removed.f_lasti: usePyObject_GetAttrString((PyObject*)frame, "f_lasti"). Code usingf_lastiwithPyCode_Addr2Line()must use :c:func:PyFrame_GetLineNumberinstead.f_lineno: use :c:func:PyFrame_GetLineNumberf_locals: usePyObject_GetAttrString((PyObject*)frame, "f_locals").f_stackdepth: removed.f_state: no public API (renamed tof_frame.f_state).f_trace: no public API.f_trace_lines: usePyObject_GetAttrString((PyObject*)frame, "f_trace_lines")(it also be modified).f_trace_opcodes: usePyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes")(it also be modified).f_localsplus: no public API (renamed tof_frame.localsplus).f_valuestack: removed.See What's New in Python 3.11 for more details (I completed the doc, but it will only be updated online tomorrow, there is a cron task once per day): https://docs.python.org/dev/whatsnew/3.11.html#id2
You can use the pythoncapi_compat project to get PyFrame_GetBack() and PyFrame_GetCode() on Python 3.8 and older: https://pythoncapi-compat.readthedocs.io/
If there is no public C API to access the PyFrameObject members used by your project, please speak up!
See for example my PR to use the public C API in coverage: https://github.com/nedbat/coveragepy/pull/1331
There is an on-going work for 2 years to add new getter functions for PyFrameObject: https://bugs.python.org/issue40421
In Python 3.10, there was no rush to make the structure internal. Things changed quickly in Python 3.11: a heavy rework on Python internals is on-going to optimize Python (especially ceval.c and how frames are created and used). For example, the _PyEval_EvalFrameDefault() function now gets a new InterpreterFrame structure, it no longer takes a PyFrameObject. Python 3.11 now creates Python frame objects on demand.
Because of these changes, reading directly PyFrameObject.f_back is now an error since the member is not always filled, but it wasn't a compiler error. You must now call the public PyFrame_GetBack() function rather than accessing directly to the member.
Since Python 2.3 (that's quite old ;-)), reading directly PyFrameObject.f_lineno is unsafe since it depends if the Python code is being traced or not: you must use PyFrame_GetLineNumber() instead.
See also the issue for the longer rationale: https://bugs.python.org/issue46836
I know that these changes are inconvenient, but I'm here if you need my help for updating your project! Moreover, Python 3.11 is scheduled for October, so there are still a few months to adapt your project, and maybe add new getter functions. The best would be to add all required getter functions before Python 3.11 beta1 scheduled for the beginning of May (2022-05-06).
Victor
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Feb 25 2022)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Fri, Feb 25, 2022 at 1:48 PM Marc-Andre Lemburg mal@egenix.com wrote:
On 25.02.2022 13:15, Victor Stinner wrote:
Hi,
tl; dr the C API of PyFrameObject changed. If you have issues, please speak up!
I moved the PyFrameObject structure to the internal C API headers. You should now use the public C API to access its members:
f_back: use :c:func:PyFrame_GetBack.You only mention C APIs, but inspection tools are often written in Python. How can they access and walk the frames on the stack ?
Are these members still available as Python attributes of the frame objects ?
As far as I know, frame object members which were available on Python 3.10 are still available in Python 3.11 using the Python API. For example, reading "frame.f_back" in Python remains perfectly fine.
That's why the What's New in Python 3.11 document now recommends to simply use PyObject_GetAttrString(): use the Python API from C :-)
There are 8 frame members accessible in Python. They are listed in the inspect doc: https://docs.python.org/dev/library/inspect.html
- f_back
- f_builtins
- f_code
- f_globals
- f_lasti
- f_lineno
- f_locals
- f_trace
In fact, there are two more attributes which are not documented. I don't know if it's made on purpose:
- f_trace_lines (can be modified)
- f_trace_opcodes (can be modified)
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 25.02.2022 14:29, Victor Stinner wrote:
On Fri, Feb 25, 2022 at 1:48 PM Marc-Andre Lemburg mal@egenix.com wrote:
On 25.02.2022 13:15, Victor Stinner wrote:
Hi,
tl; dr the C API of PyFrameObject changed. If you have issues, please speak up!
I moved the PyFrameObject structure to the internal C API headers. You should now use the public C API to access its members:
f_back: use :c:func:PyFrame_GetBack.You only mention C APIs, but inspection tools are often written in Python. How can they access and walk the frames on the stack ?
Are these members still available as Python attributes of the frame objects ?
As far as I know, frame object members which were available on Python 3.10 are still available in Python 3.11 using the Python API. For example, reading "frame.f_back" in Python remains perfectly fine.
That's why the What's New in Python 3.11 document now recommends to simply use PyObject_GetAttrString(): use the Python API from C :-)
Good to know, thanks.
Esp. walking the frame stack is important in Python for e.g. logging local variables in tracebacks or accessing context from the call stack further up.
Do you happen to know whether the current work on optimizations will break this approach by e.g. not always creating frames on the stack ?
Thanks,
Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Feb 25 2022)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Fri, Feb 25, 2022 at 2:35 PM Marc-Andre Lemburg mal@egenix.com wrote:
Esp. walking the frame stack is important in Python for e.g. logging local variables in tracebacks or accessing context from the call stack further up.
Do you happen to know whether the current work on optimizations will break this approach by e.g. not always creating frames on the stack ?
IMO if it no longer works in Python 3.11, it is a bug and it should be fixed.
I'm not aware of such regression. It should still work. Well, there are alpha releases of Python, you can already test it :-)
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On Sat, 26 Feb 2022 at 00:02, Victor Stinner vstinner@python.org wrote:
On Fri, Feb 25, 2022 at 2:35 PM Marc-Andre Lemburg mal@egenix.com wrote:
Esp. walking the frame stack is important in Python for e.g. logging local variables in tracebacks or accessing context from the call stack further up.
Do you happen to know whether the current work on optimizations will break this approach by e.g. not always creating frames on the stack ?
IMO if it no longer works in Python 3.11, it is a bug and it should be fixed.
I'm not aware of such regression. It should still work. Well, there are alpha releases of Python, you can already test it :-)
Belated follow-up mostly for the sake of the online mailing list archive: while CPython's frame objects have been split in Python 3.11 into a low level C data struct and lazily created full frame objects, all the Python level frame introspection APIs trigger the lazy creation step.
This provides most of the runtime performance gains of skipping frame object creation (as only the low level C struct is needed for code execution), without any Python level backwards compatibility issues (since the full frame objects are still created when needed).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19.03.2022 17:08, Nick Coghlan wrote:
On Sat, 26 Feb 2022 at 00:02, Victor Stinner vstinner@python.org wrote:
On Fri, Feb 25, 2022 at 2:35 PM Marc-Andre Lemburg mal@egenix.com wrote:
Esp. walking the frame stack is important in Python for e.g. logging local variables in tracebacks or accessing context from the call stack further up.
Do you happen to know whether the current work on optimizations will break this approach by e.g. not always creating frames on the stack ?
IMO if it no longer works in Python 3.11, it is a bug and it should be fixed.
I'm not aware of such regression. It should still work. Well, there are alpha releases of Python, you can already test it :-)
Belated follow-up mostly for the sake of the online mailing list archive: while CPython's frame objects have been split in Python 3.11 into a low level C data struct and lazily created full frame objects, all the Python level frame introspection APIs trigger the lazy creation step.
This provides most of the runtime performance gains of skipping frame object creation (as only the low level C struct is needed for code execution), without any Python level backwards compatibility issues (since the full frame objects are still created when needed).
Thanks for the clarification, Nick.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Mar 21 2022)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
participants (3)
-
Marc-Andre Lemburg -
Nick Coghlan -
Victor Stinner