[Python-ideas] Generators are iterators

[Terry Reedy]

On 12/10/2014 4:37 PM, Guido van Rossum wrote:
> On Wed, Dec 10, 2014 at 1:34 PM, Greg Ewing <greg.ewing at canterbury.ac.nz
> <mailto:greg.ewing at canterbury.ac.nz>> wrote:
>
>     Chris Angelico wrote:
>
>         """
>         Under this proposal, generator functions and iterators would be
>         distinct, but related, concepts.  Like the mixing of text and
>         bytes in
>         Python 2, the mixing of generators and iterators has resulted in
>         certain perceived conveniences, but proper separation will make bugs
>         more visible. The distinction is simple: A generator function
>         returns
>         a generator object. The latter is an iterator, having proper
>         __iter__
>         and __next__ methods, while the former has neither and does not
>         follow
>         iterator protocol.
>         """
>
>
>     No, that's still too confused -- all of that was true before as
>     well. Generator functions have never been iterators themselves.
>
>     I think Nick has it nailed -- the difference is that code inside
>     a generator function implementing a __next__ method will behave
>     differently from that inside an ordinary function implementing
>     a __next__ method.
>
>
> That's hardly news though. *Except* for the possibility of raising
> StopIteration to end the iteration, the inside of a generator has always
> been completely different from the inside of a __next__ method
> implementation.

Perhaps some Python equivalent of the C code will make it clearer what 
will and (mostly) will not change.  Correct me if I am wrong, but I 
believe the function __call__ method must do something more or less like 
the following.

class function:
     ...
     def __call__(self, *arg, **kwargs):
         "Setup execution frame and either run it or pass to generator."
         frame = execution_frame(self, args, kwargs)
         # several attributes of self are used to setup the frame
         if no_yield(self.code):
             run(frame)  # until it stops
             return top(frame.stack)
         else:
             return generator(self.code, frame)

The generator must look something like the following.

class generator:
     def __iter__(self):
         return self
     def __init__(self, code, frame):
         self.gi_code = code
         self.gi_frame = frame
         self.gi_running = False
     def __next__(self):
         "Adapt gen-func yield, return to iterator return, raise S.I."
         self.gi_running = True
         run(self.gi_frame)
         if yielded():
             return top(self.gi_frame.stack)
         else:  # returned
             raise StopIteration()
             self.gi_running = False
      ... (close, send, and throw methods, not relevant here)

(I probably have some details of gi_running wrong, and ignored how 
exhausted generator calls are handled.)

Ignoring gi_code, which I believe is only there for introspection (such 
as gi_code.__name__, which I use for error-reporting), the link between 
the generator function and the generator's pre-written __next__ method 
is the suspended generator-function frame, stored as gi_frame.  This is 
what make one generator instance different from another.

Currently, a StopIteration raised during 'run(self.gi_frame)' is passed 
on like any other exception, but treated specially by the generator 
caller.  The change is to wrap the frame execution or do the equivalent 
thereof.

try:
     run(self.gi_frame)
except StopIteration()
     raise RuntimeError

All else remains the same. Do I have this right?

The rationale for the change is the bulk of the PEP.  However, the PEP 
rationale and its acceptance are based on the existing relationship 
between the execution frame of the gf body and the generator.__next__ 
method that translates the 'gf protocol' to the iterator protocol. 
Guido, Nick, and Greg have all pointed to this, but for me, thinking in 
terms of Python equivalents makes it clearer.


-- 
Terry Jan Reedy