Christian Tismer wrote:
Right, CPython still keeps unneccessary crap on the C stack.
It's not just Python leaving stuff on the stack that's a problem, it's external C code that calls back into Python.
But that's not the point right now, because on the other hand, in the context of a possible yield (from or not), the C stack is clean, and this enables switching.
And actually in such clean positions, Stackless Python (as opposed to Greenlets) does soft-switching, which is very similar to what the generators are doing - there is no assembly stuff involved at all.
But the assembly code still needs to be there to handle the cases where you *can't* do soft switching. It's the presence of the code that's the issue here, not how frequently it gets called.
I have begun studying the code for YIELD_FROM. As it is written, every next iteration elevates the chain of generators once up and down. Maybe that can be avoided by changing the frame chain, so this can become a cheaper O(1) operation.
My original implementation of yield-from actually *did* avoid this, by keeping a C-level pointer chain of yielding-from frames. But that part was ripped out at the last minute when someone discovered that it had a detrimental effect on tracebacks. There are probably other ways the traceback problem could be fixed, so maybe we will get this optimisation back one day. -- Greg