Add a way to execute code in the context of different frame
The idea is inspired by the following StackOverflow question: https://stackoverflow.com/questions/40945752/inspect-who-imported-me and the corresponding BPO issue: https://bugs.python.org/issue39643. In the older Python versions the f_back attribute of the frame in which the module code is executed pointed to the frame which executes the import statement or calls __import__() or import_module(). But after rewriting the import machinery in Python, there are other frames between the caller frame and the calling frame. This caused problems with tracebacks which was worked around by removing intermediate frames related to the import machinery when the exception floats to the caller. But there is still unresolved problem with warnings, and all frames of the import machinery are visible from the imported code. I propose to add possibility to execute the code in the context of different frame. Either add a frame argument in exec() and eval(), which will allow to pass an arbitrary frame. Or add a stacklevel argument in exec() and eval() (similar to warnings.warn()), which will limit possible frames to the parent frames of the current frame. I do not know what is more feasible. This will automatically fix problems with warnings. This will allow to get rid the workaround for exceptions, and may make the code simpler. This will allow the imported code to do miscellaneous magic things with the code which performs the import. What do you think about this?
On 18.02.2020 08:33, Serhiy Storchaka wrote:
The idea is inspired by the following StackOverflow question: https://stackoverflow.com/questions/40945752/inspect-who-imported-me and the corresponding BPO issue: https://bugs.python.org/issue39643.
In the older Python versions the f_back attribute of the frame in which the module code is executed pointed to the frame which executes the import statement or calls __import__() or import_module(). But after rewriting the import machinery in Python, there are other frames between the caller frame and the calling frame. This caused problems with tracebacks which was worked around by removing intermediate frames related to the import machinery when the exception floats to the caller. But there is still unresolved problem with warnings, and all frames of the import machinery are visible from the imported code.
I propose to add possibility to execute the code in the context of different frame. Either add a frame argument in exec() and eval(), which will allow to pass an arbitrary frame. Or add a stacklevel argument in exec() and eval() (similar to warnings.warn()), which will limit possible frames to the parent frames of the current frame. I do not know what is more feasible.
This will automatically fix problems with warnings. This will allow to get rid the workaround for exceptions, and may make the code simpler. This will allow the imported code to do miscellaneous magic things with the code which performs the import.
What do you think about this?
This sounds like a nice idea, but it may make sense to limit the frame to one already on the stack. Otherwise, the code you're executing may never return to your current stack... OTOH, perhaps that could be regarded and used as a feature, similar to what PyPy exposes: https://doc.pypy.org/en/latest/stackless.html -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 18 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
18.02.20 11:28, M.-A. Lemburg пише:
This sounds like a nice idea, but it may make sense to limit the frame to one already on the stack. Otherwise, the code you're executing may never return to your current stack...
I do not think there will be such problem. exec() will temporary replace the stack of frames before executing the code and restore the original stack after it be finished. Since the stack is implemented as a single-linked list, and multiple lists can share the same tail, it all should be zero-overhead.
OTOH, perhaps that could be regarded and used as a feature, similar to what PyPy exposes:
Thank you for the link. It is interesting. The difference with PyPy is that I do not propose to create cycles. And since CPython uses the single C stack it has nothing with stackless and do not allow to get rid of the recursion depth limit. But there is something common in these ideas -- both work with f_back. Maybe more.
I am a little confused how you get from "there are extra frames in the traceback" to "modify exec() to run code in another frame". Also, with PEP 558 (Defined semantics for locals(), by Nick Coghlan) we might be able to just pass the frame's globals and locals to exec() or eval() without further changes (or perhaps just with changes to allow passing a proxy instead of a true dict). On Mon, Feb 17, 2020 at 11:36 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
The idea is inspired by the following StackOverflow question: https://stackoverflow.com/questions/40945752/inspect-who-imported-me and the corresponding BPO issue: https://bugs.python.org/issue39643.
In the older Python versions the f_back attribute of the frame in which the module code is executed pointed to the frame which executes the import statement or calls __import__() or import_module(). But after rewriting the import machinery in Python, there are other frames between the caller frame and the calling frame. This caused problems with tracebacks which was worked around by removing intermediate frames related to the import machinery when the exception floats to the caller. But there is still unresolved problem with warnings, and all frames of the import machinery are visible from the imported code.
I propose to add possibility to execute the code in the context of different frame. Either add a frame argument in exec() and eval(), which will allow to pass an arbitrary frame. Or add a stacklevel argument in exec() and eval() (similar to warnings.warn()), which will limit possible frames to the parent frames of the current frame. I do not know what is more feasible.
This will automatically fix problems with warnings. This will allow to get rid the workaround for exceptions, and may make the code simpler. This will allow the imported code to do miscellaneous magic things with the code which performs the import.
What do you think about this? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ETBOLQ... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
18.02.20 20:13, Guido van Rossum пише:
I am a little confused how you get from "there are extra frames in the traceback" to "modify exec() to run code in another frame".
Sorry for skipping several steps. I'll try to expand them here. PEP 558 does not have relation to this, as frame is not the part of locals(). exec() is used by the import machinery to execute the module code. Currently frames look like this: import foo # user code # many frames in the import machinery # ... # ... exec(code, module.__dict__) # still in the import machinery # user code in foo.py I propose to add a parameter to exec() which will allow to execute the code in the imported mode as exec() was called immediately from the importer code: import foo # user code # user code in foo.py As it was when the import machinery was implemented in C. I did not propose to modify exec() to run code in another frame, I proposde to modify exec() to run code in new frame whose parent is a specified frame (either arbitrary frame or on of ancestors of the caller frame). I think it may be not difficult to do. I ask whether there are other applications of this or similar feature and what form of the feature will be more convenient. It just occurred to me that it may be not a feature of exec(), but a separate function which will allow to execute an arbitrary function, not just a Python code, as it was called from the different frame. apply_in_frame(frame, func, /, *args, **kwargs) or apply_with_frames_skipped(stacklevel, func, /, *args, **kwargs) The question is how to call this function and where to put it. In the first variant it could be a method of frame.
On Tue, Feb 18, 2020 at 11:09 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
18.02.20 20:13, Guido van Rossum пише:
I am a little confused how you get from "there are extra frames in the traceback" to "modify exec() to run code in another frame".
Sorry for skipping several steps. I'll try to expand them here. PEP 558 does not have relation to this, as frame is not the part of locals().
Ah, fair. exec() is used by the import machinery to execute the module code.
Currently frames look like this:
import foo # user code # many frames in the import machinery # ... # ... exec(code, module.__dict__) # still in the import machinery # user code in foo.py
I propose to add a parameter to exec() which will allow to execute the code in the imported mode as exec() was called immediately from the importer code:
import foo # user code # user code in foo.py
As it was when the import machinery was implemented in C.
This makes sense (except when debugging the import machinery, which is also a valid use case. :-). The caller would still have to provide a dict for the globals (the module dict), and I guess it would have to point to the *previous* frame (the one executing `import foo`) so that exec() can create a new frame pointing back there. And somehow we would need to set things up so that after exec() completes or raises we automatically return or raise to the parent frame, rather than passing control to exec()'s caller. Or alternatively, we'd need exec()'s caller (deep down inside importlib) to be able to say "pretend to return or raise from this (new) frame".
I did not propose to modify exec() to run code in another frame, I proposde to modify exec() to run code in new frame whose parent is a specified frame (either arbitrary frame or on of ancestors of the caller frame).
I understand that now. I think it may be not difficult to do. I ask whether there are other
applications of this or similar feature and what form of the feature will be more convenient.
My gut feeling says that the only other use cases will be alternatives to importlib, perhaps other kinds of import hooks.
It just occurred to me that it may be not a feature of exec(), but a separate function which will allow to execute an arbitrary function, not just a Python code, as it was called from the different frame.
apply_in_frame(frame, func, /, *args, **kwargs)
or
apply_with_frames_skipped(stacklevel, func, /, *args, **kwargs)
The question is how to call this function and where to put it. In the first variant it could be a method of frame.
So either way it would create a new frame, right? And it would not pass control to the caller, but to the indicated frame. I'm not that keen on passing a stacklevel -- that could easily cause confusion, and to calculate the right level the caller would have to crawl up the stack anyway. A problem I see with either solution is that it looks like except and finally blocks in the intervening frames would be skipped. Or perhaps not, but at that point I suspect that the implementation won't be all that easy after all. So *if* we end up doing this optimization I think it may be best to make it something that's clearly meant for the importlib use case. Sine we already have sys._getframe(), maybe it could be another magic thing in sys? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
participants (3)
-
Guido van Rossum -
M.-A. Lemburg -
Serhiy Storchaka