[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Christian Tismer]

Hi Greg,

coming back to this after quite a storm in my brain...

On 16.10.12 02:44, Greg Ewing wrote:
> Christian Tismer wrote:
>
>> Right, CPython still keeps unneccessary crap on the C stack.
>
> It's not just Python leaving stuff on the stack that's a
> problem, it's external C code that calls back into Python.
>

That one is something that I think to ignore.
Of course there are quite some situations where callbacks
into Python are a problem, but I don't want to put this into Python.

There are ways to cope with this, for instance using greenlet as
an optional extension module that handles these cases.
Alternatively, Python itself can do it with strictly controlled threads.

But I think leaving that out will simplify matters a lot and keeps Python
clean. In the end, I want to model something that is likely to be
accepted.

>> But that's not the point right now, because on the other hand,
>> in the context of a possible yield (from or not), the C stack
>> is clean, and this enables switching.
>
>> And actually in such clean positions, Stackless Python (as opposed to
>> Greenlets) does soft-switching, which is very similar to what the 
>> generators
>> are doing - there is no assembly stuff involved at all.
>
> But the assembly code still needs to be there to handle the
> cases where you *can't* do soft switching. It's the presence
> of the code that's the issue here, not how frequently it
> gets called.

No, I'm intending to really rip that out.
Or better, I want to do a rewrite of a subset of Stackless,
actually the functionality that allows to implement greenlets
or multi-level generators, task scheduling and so on.

In effect, I want to find something that enables some extended
switching. Emulated, without hacking the kernel in the first place.

Generators are restricted to call/yield/return positions, and
I thing that's fine. What can be switched is totally clear by
definitions, and I like that. I'm talking of exactly that.

What I dislike is a different topic ;-)

>> I have begun studying the code for YIELD_FROM. As it is written, every
>> next iteration elevates the chain of generators once up and down.
>> Maybe that can be avoided by changing the frame chain, so this can 
>> become
>> a cheaper O(1) operation.
>
> My original implementation of yield-from actually *did* avoid
> this, by keeping a C-level pointer chain of yielding-from frames.
> But that part was ripped out at the last minute when someone
> discovered that it had a detrimental effect on tracebacks.
>
> There are probably other ways the traceback problem could be
> fixed, so maybe we will get this optimisation back one day.

Ok, let's ignore this O(n) problem for now. _yield from_ is anyway
probably faster by more than an order of magnitude, so it will
serve your purpose (nesting generators) pretty well.

My problem is different because I want a scaling building block
for building higher level structures, and I would love to build them
using _yield from_ .

There are a few things which contradict completely my thinking:

- _yield from_ works from the top. That is, if I have five nested iterators
    and I want to drive them, then I have to call the root generator?!
    I see that that works, but it is against all what I'm used to.

    How can I inject the info that I want to switch context?

- generators always yield to the caller, and also return values to the
    caller. What I'm looking for is to express a switch so something
    else?

- generators are able to free the stack, when they yield. But when they
    are active, they use the full stack. At least when I follow the pattern
    "generator is calling sub-generator".
    A deeply nested recursion is therefore something to avoid. :-(

Now I'm playing around with different approaches to model something
flexible that gives me more freedom. Right now I'm trying a slightly
pervert approach to give me an _unwindable_, merely a frame-like
object that can vanish on demand.

I'm also experimenting with emulating different kinds of _yield_".
Since there is only one kind of yield to one target, I get the problem
to distinguish that for different purposes.

Example:

I can take a set of nested functions in their native form.
Then replacing ordinary calls by _yield from_ and inserting proper
yields before actually returning, I now have the equivalent of a nested
function call, that I can drive with another _yield from_ .
This is now a function that permanently releases the stack.

Now I would like to give one of the nested function the ability to
transfer execution somewhere else. The support is insofar there,
as the stack is freed all the time. But this function that wants to
switch needs to pass the fact that it wants to switch, plus the target
somewhere. As I understood it, I would need to yield that to the
driver function.
In order to do that, I would need to yield a tuple or a bound object.
This is a different purpose than the simple driver functionality.

Do you see it? In my understanding, a switch would not be driven from
the top and then dispatched upon, but a called function below the
function to be switched would modify something that leads to a
switch as a result.
In this example, the generator-emulated function would be driven
by thousands of yields, kind of polled to catch the one event that
needs to be supported by a switching action.

This looks wrong for me, like doing things upside down.

To shorten this: I have the problem that I have your very efficient yield
collector, but I need to dispatch on what is intended by the yield,
instead of initiating a reaction from where I am.
All in all, I can't get rid of the thought "un-pythonic".

So I'm still thinking of a frame-like object that allows me to control
its execution, let it vanish, and so on, and use it as a building block.
As always, I'm feeling bad when going this road, because I want to use
the eficient _yield from_ as much as possible.

But it may be my missing experience.

Do you understand, and maybe see where I have the wrong
brain shortcuts?
How do you write something composable that scales?

Cheers -- Chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/