Virtual machine bleeds into generator implementation?
This is more an observation and question than anything else, but perhaps it will stimulate some ideas from the experts. Consider this trivial generator function: def gen(a): yield a When the YIELD_VALUE instruction is executed, it executes (in the non-async case): retval = POP(); f->f_stacktop = stack_pointer; goto exiting; This is fine as far as it goes. However, execution eventually leads to Objects/genobject.c where we hit this code (I think after falling off the YIELD_VALUE instruction, but perhaps virtual machine execution reaches RETURN_VALUE): /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ if (result && f->f_stacktop == NULL) { There are several other references to f->f_stacktop in genobject.c. I've not yet investigated all of them. As I'm working on a register-based virtual machine implementation, I don't fiddle with the stack at all, so it's a bit problematic that the generator implementation is so intimate with the stack. As this is an area of the core which is completely new to me, I wonder if someone can suggest alternate ways of achieving the same effect without relying on the state of the stack. It seems to me that from within PyEval_EvalFrameDefault the implementations of relevant instructions could reference the generator object via f->f_gen and call some (new?) private API on generators which toggles the relevant bit of state in the generator. I think it's worse that this though, as it seems that in gen_send_ex() it actually pushes a value onto the stack. That can't be solved by simply adding a state attribute to the generator object struct. Skip
It looks it is using this to save a separate, explicit bit that says "we left this frame by yielding rather than by returning". Note that f_stacktop is cleared when the frame is re-entered: stack_pointer = f->f_stacktop; assert(stack_pointer != NULL); f->f_stacktop = NULL; /* remains NULL unless yield suspends frame */ f->f_executing = 1; On Sun, Apr 26, 2020 at 7:57 PM Skip Montanaro <skip.montanaro@gmail.com> wrote:
This is more an observation and question than anything else, but perhaps it will stimulate some ideas from the experts. Consider this trivial generator function:
def gen(a): yield a
When the YIELD_VALUE instruction is executed, it executes (in the non-async case):
retval = POP(); f->f_stacktop = stack_pointer; goto exiting;
This is fine as far as it goes. However, execution eventually leads to Objects/genobject.c where we hit this code (I think after falling off the YIELD_VALUE instruction, but perhaps virtual machine execution reaches RETURN_VALUE):
/* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ if (result && f->f_stacktop == NULL) {
There are several other references to f->f_stacktop in genobject.c. I've not yet investigated all of them.
As I'm working on a register-based virtual machine implementation, I don't fiddle with the stack at all, so it's a bit problematic that the generator implementation is so intimate with the stack. As this is an area of the core which is completely new to me, I wonder if someone can suggest alternate ways of achieving the same effect without relying on the state of the stack. It seems to me that from within PyEval_EvalFrameDefault the implementations of relevant instructions could reference the generator object via f->f_gen and call some (new?) private API on generators which toggles the relevant bit of state in the generator.
I think it's worse that this though, as it seems that in gen_send_ex() it actually pushes a value onto the stack. That can't be solved by simply adding a state attribute to the generator object struct.
Skip _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q7JIWXV7... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Hello, On Sun, 26 Apr 2020 19:51:18 -0700 Skip Montanaro <skip.montanaro@gmail.com> wrote: []
I think it's worse that this though, as it seems that in gen_send_ex() it actually pushes a value onto the stack. That can't be solved by simply adding a state attribute to the generator object struct.
At the higher level, "it doesn't push value on stack", it "sets value of the yield operator to return". CPython uses stack slots to keep data, so puts it in a stack slot, you use registers, so would put it in a ("return value") register. Overall, that seems like minor patching detail comparing to patching which would be required to implement register-based calling convention for functions. -- Best regards, Paul mailto:pmiscml@gmail.com
I think it's worse that this though, as it seems that in gen_send_ex() it actually pushes a value onto the stack. That can't be solved by simply adding a state attribute to the generator object struct.
At the higher level, "it doesn't push value on stack", it "sets value of the yield operator to return".
Potatoes, potahtoes. :-) The current implementation "sets the value of the yield operator to return" by pushing a value onto the stack: /* Push arg onto the frame's value stack */ result = arg ? arg : Py_None; Py_INCREF(result); *(f->f_stacktop++) = result; Thanks for the replies. I will cook up some private API in my cpython fork. Whether or not my new vm ever sees the light of day, I think it would be worthwhile to consider a proper API (even a _PyEval macro or two) for the little dance the two subsystems do. Skip
Thanks for the replies. I will cook up some private API in my cpython fork. Whether or not my new vm ever sees the light of day, I think it would be worthwhile to consider a proper API (even a _PyEval macro or two) for the little dance the two subsystems do.
I committed a change to my fork: https://github.com/smontanaro/cpython/commit/305758a42ec92dcd1d0a181f454af63... This moves direct stack manipulation out of genobject.c into ceval.c and allows me to work on a non-stack way to deal with these tasks (note all the calls to Py_FatalError in the CO_REGISTER branches). I am specifically not holding this up as a proposal for how to do this (I am largely ignorant of many of the internal or CPython-specific aspects of the C API). Still, the tests pass and I can start to address those fatal errors. Skip .
participants (3)
-
Guido van Rossum -
Paul Sokolovsky -
Skip Montanaro