[Python-Dev] PEP 334 - Simple Coroutines via SuspendIteration

[Samuele Pedroni]

James Y Knight wrote:
> 
> On Sep 7, 2004, at 9:48 PM, Clark C. Evans wrote:
> 
>> I've packaged up the idea of a coroutine facility using iterators and an
>> exception, SuspendIteration.
> 
> 
> Very interesting.
> 
>> This proposal assumes that a corresponding iterator written using
>> this class-based method is possible for existing generators.  The
>> challenge seems to be the identification of distinct states within
>> the generator where suspension could occur.
> 
> 
> That is basically impossible. Essentially *every* operation could 
> possibly raise SuspendIteration, because essentially every operation can 
> call an arbitrary python function, and python functions can raise any 
> exception they want. I think you could still make the proposal work in 
> CPython: if I understand its internals properly, it doesn't need to do a 
> transformation to a class iterator, it simply suspends the frame 
> wherever it is. Thus, being able to suspend at any point in the function 
> would not cause an undue performance degradation.

I don't think it is that simple for CPython either, a single bytecode 
can potentially invoke more then just a single builtin or other python 
code, e.g. calling construction can invoke __new__ and __init__,
and then there are all the cases were descriptors are involved with 
their __get__ etc (and __add__,__radd__...). So bytecodes are not the 
right suspension/resumption granularity because you don't want to 
reinvoke things that could have had side-effects.
So you have all the points per bytecode were python code/builtins can be 
invoked or from another POV an exception can be detected.

If I understand the proposal (which is quite vague), like restartable 
syscalls, there is also the matter that whatever raised the 
SuspendIteration should be retried on resumption of the generator, e.g 
calling nested generator next.

So one would have to cherry pick for each bytecode or similar abstract
operations model relevant suspension/resumption points and it would 
still be quite a daunting task to implement this adding the code
for intra bytecode resumption. (Of course this assumes that capturing
the C stack and similar techniques are out of question)

> 
> However, I think it is a deal-breaker for JPython. From the generator 
> PEP: "It's also believed that efficient implementation in Jython 
> requires that the compiler be able to determine potential suspension 
> points at compile-time, and a new keyword makes that easy." If this 
> quote is right about the implementation of Jython (and it seems likely, 
> given the JVM), your proposal makes it impossible to implement 
> generators in Jython.

a hand coded implementation would be a *lot* of work (beyound practical)
for potentially very bad performance and a resulting messy codebase.
One could also encounter resulting code size problems or issues with the 
verifier.