[Python-ideas] Generators are iterators

[Terry Reedy]

On 12/12/2014 7:42 AM, Oscar Benjamin wrote:> On 12 December 2014 at 
10:34, Nick Coghlan <ncoghlan at gmail.com> 
wrote:

 >> specific problem deemed worthy of being fixed is that the presence of
 >> "yield" in a function body can implicitly suppress StopIteration
 >> exceptions raised elsewhere in that function body (or in functions it
 >> calls).

This makes no sense to me either.

 > The yield causes the function to become a generator function. The
 > frame for a generator function (like for any other function) will
 > allow uncaught exceptions to propagate to the frame above. The
 > difference between generator functions and other functions is that the
 > code in the body of a generator function is executed when someone
 > calls the generator's __next__ method. Since the caller (the iterator
 > consumer) is expecting StopIteration it is treated as the signalling
 > the end of iteration. The yield suppresses nothing; it is the iterator
 > consumer e.g. the for-loop or the list() function etc. which catches
 > the StopIteration and treats it as termination.
 >
 >> The difference in behaviour between comprehensions and
 >> generator expressions when it comes to embedded function calls that
 >> trigger StopIteration is a special case of that more general
 >> difference.
 >
 > I don't know what you mean by this.

Neither do I

 >> This is a problem unique to generators, it does not affect any other
 >> iterator (since explicit __next__ method implementations do not use
 >> yield).

The parenthetical comment is true.  However, I think part of the problem 
is unique to generator functions, and part is more general -- that loose 
usage of StopIteration creates problems when a .__next__ method calls 
external code.

 > Incorrect. The problem is not unique to generators and the yield is
 > irrelevant.

Nick (and Guido) are looking at the unique part and Oscar is looking at 
he general part.

 > The problem (insofar as it is) is a problem for all
 > iterators since all iterators interact with iterator-consumers and it
 > is the iterator-consumer that catches the StopIteration. Here is an
 > example using map:
 >
 >>>> def func(x):
 > ...     if x < 0:
 > ...         raise StopIteration
 > ...     return x ** 2

This function violates the implicit guideline, which I think should be 
more explicit in the doc, that the only functions that should expose 
StopIteration to the outside world are iterator.__next__ functions and 
the corresponding builtin next().  It is not surprising that this 
violation causes map to misbehave.  However, changing the .__next__ 
methods of map and filter iterators should be a new thread, which I will 
write after posting this.

The g.f. PEP says that generator functions should also follow this 
guideline and not raise StopIteration (but should always return when not 
yielding).

...
 >> The change in the PEP is to change that side effect such that
 >> those exceptions are converted to RuntimeError rather than silently
 >> suppressed - making generator function bodies behave less like
 >> __next__ method implementations.
 >
 > Fine but generator function bodies are __next__ method
 > implementations.

This is about as wrong, or at least as confusing, as saying that the 
function passed to map is a __next__ method implementation.  There is 
only one generator.__next__, just as there is only one map.__next__. 
Both .__next__ methods execute the user function (or function frame) 
passed in to the corresponding .__init__ to calculate the next value to 
return.  So the external functions assist the __next__ methods but are 
not __next__ methods themselves.  In particular, they should not raise 
StopIteration.

-- 
Terry Jan Reedy