Hi Gabriele!

I hope you would indulge me in asking for some details about the new CFrame structure, even in the form of existing literature (e.g. PEP) where the idea behind it is explained.

There isn't too much documentation on this, unfortunately (since these are all very unstable, low-level interpreter details), but a good place to start would be https://bugs.python.org/issue46090. Based on my own understanding (from reading the source code): - There are three relevant structures: CFrame, InterpreterFrame, and PyFrameObject. - PyFrameObjects are just PyObject wrappers around an InterpreterFrame, where all of the *actual* frame state for the Python stack is maintained. - They are created lazily (for example, when sys._getframe() is called). - See https://github.com/python/cpython/pull/27077. - InterpreterFrames live in a "datastack" for fast allocation and deallocation. - This "datastack" lives on the PyThreadState. - Because of how it is designed, InterpreterFrames must be allocated/deallocated "in order". - If an InterpreterFrame is cleared, but still has a live PyFrameObject that points to it, it will copy itself *into* the PyFrameObject first (to guarantee that the PyFrameObject keeps working). - See https://github.com/python/cpython/pull/26076. - A single CFrame is statically allocated inside of each _PyEval_EvalFrameDefault call, so it corresponds to the C stack, not the Python stack. - It links to a chain of one or more InterpreterFrames. - Multiple InterpreterFrames can correspond to a single CFrame! - This is a performance optimization in 3.11: rather than enter a new call to _PyEval_EvalFrameDefault, calls to pure-Python code just create a new InterpreterFrame, set it as the current one, and continue execution. - You can see how many InterpreterFrames correspond to the current CFrame by reading the "depth" member of the current InterpreterFrame. - A value of 0 indicates that this is the only InterpreterFrame for this CFrame. - A value of 42 means that this optimization has been performed 42 times (and there are currently 43 InterpreterFrames executing in this CFrame).

Also, I'd like to a quick question, if I may. There now appear to be two ways of unwinding the frame stack: either iterate over CFrame.previous, or the more traditional PyFrameObject.f_back. I suspect there are reasons why these are perhaps not actually equivalent, and indeed this is mainly what I'd like to read in the literature I've requested above.

The above outline probably makes the differences clear: - PyFrameObject.f_back just gives you a dummy wrapper around the previous frame object. - It's not really useful for unwinding anything. - InterpreterFrame.previous gives you the previous interpreter frame (duh!). - This is probably what you want. - CFrame.previous gives you the previous call to _PyEval_EvalFrameDefault. - It's not really useful for unwinding anything. - This is only really useful to maintain the current tracing state when returning. Hopefully this helps! Somebody (Pablo or Mark?) will probably jump in here if I got anything wrong. Brandt

Hi Gabriele, I think the code is currently the only documentation, since this is considered a CPython internal issue. I'm CC'ing Mark Shannon, since he designed this for 3.11. For a bit of background, see this issue in the Faster CPython tracker: https://github.com/faster-cpython/ideas/issues/31 -- maybe this will help you understand the background (though what was implemented doesn't match the discussion there precisely). Since you seem to be the author of Austin, a sampling profiler for Python code (which I presume itself is written in C or C++), I'm guessing you're asking because you're looking to make Austin work with 3.11. (Actually, the CFrame structure appears in 3.10, but it's not usable for unwinding stack frames in 3.10, since the 3.10 version doesn't have a pointer to the interpreter frame -- it's only used to speed up access to the `use-tracing` flag. So I'm assuming you're asking about 3.10.) One of the goals for the Faster CPython project that has only become in focus for the team recently is helping tools like Austin (and others like Cython, PyDev and Greenlets) that use internal interpreter state figure out how to get their code working in 3.11, without making too many commitments to stable APIs -- we're already thinking about more performance improvements in 3.12 and beyond, which most likely will require us to change the internal data structures again. (We do commit to leaving these internal data structures unchanged for bugfix release cycles, so that once you've got it working for 3.11.0, it should work without changes for 3.11.1, 3.11.2, etc. In fact we hope to stabilize everything starting with the release of 3.11b1, May 2020.) Oh, I see that Brandt has already given a better overview of how this stuff actually works (he's a faster typist than me :-), so I'll just sign off here. If you have more specific questions, you can continue this thread, or you can create a new Discussion item in the https://github.com/faster-cpython/ideas repo. --Guido On Mon, Dec 20, 2021 at 10:25 AM Gabriele wrote:

Hi there

I hope you would indulge me in asking for some details about the new CFrame structure, even in the form of existing literature (e.g. PEP) where the idea behind it is explained.

Also, I'd like to a quick question, if I may. There now appear to be two ways of unwinding the frame stack: either iterate over CFrame.previous, or the more traditional PyFrameObject.f_back. I suspect there are reasons why these are perhaps not actually equivalent, and indeed this is mainly what I'd like to read in the literature I've requested above.

Cheers, Gabriele

-- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto."

-- G. Galilei, Il saggiatore. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KQOQTLR5... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...