[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Christian Tismer]

Hey Nick,

On 15.10.12 14:18, Nick Coghlan wrote:
> On Mon, Oct 15, 2012 at 8:38 PM, Christian Tismer <tismer at stackless.com> wrote:
>> Just one thing that I don't get.
>> What do you mean by 'implicit taskswitching' ?
>> There is no such thing in greenlet, if you really meant that
>> Library from Armin Rigo.
>>
>> greenlets do everything explicitly, no pre-emption at all.
>>
>> So, is there a general understanding what a greenlet is and what not?
>> Just to make sure that the discussed terms are clearly defined.
> With greenlets, your potential switching points are every function
> call (because you can call switch() from anywhere, and you can't
> reliably know the name of *every* IO operation, or operation that
> implicitly invokes an IO operation).

That's true, and you will wonder: I never liked that!
See below (you'll wonder even more)
> With generators, there is always an explicit *local* marker within the
> generator body of the potential switching points: yield expressions
> (including yield from). Ordinary function calls cannot cause the
> function to be suspended.
>
> So greenlets give you the scalability benefits of microthreading (as
> almost any OS supports a couple of orders of magnitude more sockets
> than it can threads), but without the same benefits of locally visible
> suspension points that are provided by generators and explicit
> callbacks.

Yes, I understood that a lot better now.
The nice trick of the (actually a bit ugly) explicit down-chaining
of the locally visible switching points is the one thing that makes
a huge difference, both for Stackless and Greenlets.
Because we could never know the exact switching points, things became
so difficult to handle.

> That's the philosophical reason. As a *practical* matter, there's
> still the problem you described in more detail elsewhere that CPython
> relies too much on the C stack to support suspension of arbitrary call
> chains without the stack switching assembly code in
> Stackless/greenlets.
>

Right, CPython still keeps unneccessary crap on the C stack.
But that's not the point right now, because on the other hand,
in the context of a possible yield (from or not), the C stack
is clean, and this enables switching.
And actually in such clean positions, Stackless Python (as opposed to
Greenlets) does soft-switching, which is very similar to what the generators
are doing - there is no assembly stuff involved at all.

So in the context of switching, CPython is presumably more efficient
than greenlet (because of stack slicing), and a bit less efficient than
stackless because of the generator chaining.

I have begun studying the code for YIELD_FROM. As it is written, every
next iteration elevates the chain of generators once up and down.
Maybe that can be avoided by changing the frame chain, so this can become
a cheaper O(1) operation.

Alternatively I could also imagine to write real generators or coroutines
as an extension module. It would use the same concept as generators,
internally. No big deal, not changing the interpreter, maybe adding a bit.

I think this would make Greenlet and even Stackless obsolete in most
cases which are of real use.

I would like to discuss this and maybe do a prototype.

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/