[Python-Dev] PEP 380 (yield from a subgenerator) comments

[Stefan Rank]

on 2009-03-27 05:17 P.J. Eby said the following:
> At 04:08 PM 3/27/2009 +1300, Greg Ewing wrote:
>> You can't expect to improve something like that by
>> stuffing yield-from into the existing framework, because
>> the point of yield-from is to render the framework
>> itself unnecessary.
> 
> But it doesn't.  You still need *something* that processes the yielded 
> values, since practical frameworks have various things to yield "to" - 
> i/o, time, mouse clicks, whatever.  Correctly dealing with the call 
> stack part is tedious to implement, sure, but it's not really the focal 
> point of a microthreading framework.

I can chime in here with a use case, if an unusual one. I implemented 
just such a framework based on generator syntax for my thesis work to 
model the behaviour of software agents as a collection of interacting 
activities (microprocesses). The top layer is based on Twisted (similar 
to its _inlineCallbacks) and different schedulers decide on what to do 
with yielded values.
This is really very similar to Philip Eby's code, the main difference 
being that one uses a generic function yield_to as an extension point 
and the other one uses (subclasses of) Deferreds.
You can handle the call stack in the Deferred-based case just as 
clumsily as in the other :-).

And in my system, the "call stack" (i.e. the hierarchy of active 
microprocesses) and how it can be manipulated by the agent is actually 
the interesting part.

> Usually, you need to have some way to control which microthreads are 
> actually to be executing, vs. the ones that are waiting for a particular 
> time, an I/O operation, or some other sort of event.  None of that stuff 
> goes away just by taking care of the call stack.

Yes. However, the valuable addition that an explicit yield from syntax 
would provide for my use case is a way to explicitly distinguish between 
subgenerators just for the sake of refactoring code vs. sub-"processes".
I could remove quite some duplication from my current code.

Additionally, as noted in the PEP, it would open the path for 
optimisations of the refactoring cases.
I also think that a separation of handling the generator call stack and 
handling yielded values improves the situation for scheduling/trampoline 
authors conceptually.

Just my 0.02€
cheers,
stefan