[Python-ideas] Generators are iterators

[Terry Reedy]

On 12/11/2014 10:14 AM, Oscar Benjamin wrote:
> On 10 December 2014 at 20:46, Guido van Rossum <guido at python.org> wrote:

> The PEP is still very confused. Primarily the fact is that the PEP is
> attempting to address an issue which affects all iterators but
> proposing a solution which is only about generators. The PEP seems to
> suggest that there is something special about generators in this
> respect when there really isn't. For example:
>
> """
> The interaction of generators and StopIteration is currently somewhat
> surprising, and can conceal obscure bugs. An unexpected exception
> should not result in subtly altered behaviour, but should cause a
> noisy and easily-debugged traceback. Currently, StopIteration can be
> absorbed by the generator construct.
> """
>
> There is no interaction between generators and StopIteration. The
> issue isn't about generators it is about the iterator protocol.
> StopIteration cannot be absorbed by the generator construct.

What is very confusing is the ambiguous use of 'generator' to mean 
either 'generator function' or 'generator class instance'.  Since I do 
not know what either of you mean by 'generator' in each instance above, 
I do not know exactly what either of you meant.

If the PEP and discussants consistency used 'generator' to mean 
"instance of the internal generator class, initialized with a suspended 
instance of a *user-written* generator function body", then I think some 
of the confusion would dissipate.

The PEP is about the conjunction of the following facts (and a few more, 
but I think these are the most salient): a) generator function bodies 
follow a different protocol than iterator __next__ methods, one that 
does not require them to *ever* raise StopIteration; b) if a g.f. body 
does raise StopIteration, it might be a substitute for 'return', but it 
might be a bug -- and apparently bugs do happen in real code; and c) 
generator.__next__ currently trusts that *user-written* g.f.s are never 
buggy.

The PEP proposal is, either literally or in effect, that 
generator.__next should stop trusting StopIteration from a g.f., thereby 
disallowing the sometimes convenient but optional substitution.  The 
justification is that bugs should not pass silently, and now they do.

> I think the PEP would be clearer if it properly acknowledged that the
> problem is a problem for all iterators. The question then is why the
> fix is only targeted at generators and what should be done about the
> same problem that occurs in many other forms. The PEP rationale avoids
> these issues by falsely claiming that generators are special.

Guido today add the following, apparently in response to the above.

+When implementing a regular ``__next__()`` method, the only way to
+indicate the end of the iteration is to raise ``StopIteration``.  So
+catching ``StopIteration`` here and converting it to ``RuntimeError``
+would defeat the purpose.  This is a reminder of the special status of
+generator functions: in a generator function, raising
+``StopIteration`` is redundant since the iteration can be terminated
+by a simple ``return``.

I would put this answer slightly differently.

A __next__ method is supposed to raise StopIteration if *and only if* 
there are no more items to return.  The __next__ method author is 
responsible for fulfilling the contract.  Core developers are 
responsible for generator.__next__; we are not responsible for iterators 
that others write.  Anyone who writes an iterator class whose instances 
are initialized with 3rd party non-iterator code that is executed in 
.__next__, should think about what to do if that code raises StopIteration.

It would be possible for the PEP to recommend that other .__next__ 
methods executing external code might do something similar to what 
generator.__next__ will do.

try:
     execute(self.external_code)
except StopIteration as e:
     raise RuntimeError(<whatever>)

-- 
Terry Jan Reedy