Hi Greg, coming back to this after quite a storm in my brain... On 16.10.12 02:44, Greg Ewing wrote:
Christian Tismer wrote:
Right, CPython still keeps unneccessary crap on the C stack.
It's not just Python leaving stuff on the stack that's a problem, it's external C code that calls back into Python.
That one is something that I think to ignore. Of course there are quite some situations where callbacks into Python are a problem, but I don't want to put this into Python. There are ways to cope with this, for instance using greenlet as an optional extension module that handles these cases. Alternatively, Python itself can do it with strictly controlled threads. But I think leaving that out will simplify matters a lot and keeps Python clean. In the end, I want to model something that is likely to be accepted.
But that's not the point right now, because on the other hand, in the context of a possible yield (from or not), the C stack is clean, and this enables switching.
And actually in such clean positions, Stackless Python (as opposed to Greenlets) does soft-switching, which is very similar to what the generators are doing - there is no assembly stuff involved at all.
But the assembly code still needs to be there to handle the cases where you *can't* do soft switching. It's the presence of the code that's the issue here, not how frequently it gets called.
No, I'm intending to really rip that out. Or better, I want to do a rewrite of a subset of Stackless, actually the functionality that allows to implement greenlets or multi-level generators, task scheduling and so on. In effect, I want to find something that enables some extended switching. Emulated, without hacking the kernel in the first place. Generators are restricted to call/yield/return positions, and I thing that's fine. What can be switched is totally clear by definitions, and I like that. I'm talking of exactly that. What I dislike is a different topic ;-)
I have begun studying the code for YIELD_FROM. As it is written, every next iteration elevates the chain of generators once up and down. Maybe that can be avoided by changing the frame chain, so this can become a cheaper O(1) operation.
My original implementation of yield-from actually *did* avoid this, by keeping a C-level pointer chain of yielding-from frames. But that part was ripped out at the last minute when someone discovered that it had a detrimental effect on tracebacks.
There are probably other ways the traceback problem could be fixed, so maybe we will get this optimisation back one day.
Ok, let's ignore this O(n) problem for now. _yield from_ is anyway probably faster by more than an order of magnitude, so it will serve your purpose (nesting generators) pretty well. My problem is different because I want a scaling building block for building higher level structures, and I would love to build them using _yield from_ . There are a few things which contradict completely my thinking: - _yield from_ works from the top. That is, if I have five nested iterators and I want to drive them, then I have to call the root generator?! I see that that works, but it is against all what I'm used to. How can I inject the info that I want to switch context? - generators always yield to the caller, and also return values to the caller. What I'm looking for is to express a switch so something else? - generators are able to free the stack, when they yield. But when they are active, they use the full stack. At least when I follow the pattern "generator is calling sub-generator". A deeply nested recursion is therefore something to avoid. :-( Now I'm playing around with different approaches to model something flexible that gives me more freedom. Right now I'm trying a slightly pervert approach to give me an _unwindable_, merely a frame-like object that can vanish on demand. I'm also experimenting with emulating different kinds of _yield_". Since there is only one kind of yield to one target, I get the problem to distinguish that for different purposes. Example: I can take a set of nested functions in their native form. Then replacing ordinary calls by _yield from_ and inserting proper yields before actually returning, I now have the equivalent of a nested function call, that I can drive with another _yield from_ . This is now a function that permanently releases the stack. Now I would like to give one of the nested function the ability to transfer execution somewhere else. The support is insofar there, as the stack is freed all the time. But this function that wants to switch needs to pass the fact that it wants to switch, plus the target somewhere. As I understood it, I would need to yield that to the driver function. In order to do that, I would need to yield a tuple or a bound object. This is a different purpose than the simple driver functionality. Do you see it? In my understanding, a switch would not be driven from the top and then dispatched upon, but a called function below the function to be switched would modify something that leads to a switch as a result. In this example, the generator-emulated function would be driven by thousands of yields, kind of polled to catch the one event that needs to be supported by a switching action. This looks wrong for me, like doing things upside down. To shorten this: I have the problem that I have your very efficient yield collector, but I need to dispatch on what is intended by the yield, instead of initiating a reaction from where I am. All in all, I can't get rid of the thought "un-pythonic". So I'm still thinking of a frame-like object that allows me to control its execution, let it vanish, and so on, and use it as a building block. As always, I'm feeling bad when going this road, because I want to use the eficient _yield from_ as much as possible. But it may be my missing experience. Do you understand, and maybe see where I have the wrong brain shortcuts? How do you write something composable that scales? Cheers -- Chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/