[Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ron Adam]

On 11/22/2014 08:31 AM, Nick Coghlan wrote:
>
> On 22 Nov 2014 02:51, "Antoine Pitrou" <solipsis at pitrou.net
> <mailto:solipsis at pitrou.net>> wrote:
>  >
>  > On Fri, 21 Nov 2014 05:47:58 -0800
>  > Raymond Hettinger <raymond.hettinger at gmail.com
> <mailto:raymond.hettinger at gmail.com>> wrote:
>  > >
>  > > Another issue is that it breaks the way I and others have taught for
> years that generators are a kind of iterator (an object implementing the
> iterator protocol) and that a primary motivation for generators is to
> provide a simpler and more direct way of creating iterators.  However,
> Chris explained that, "This proposal causes a separation of generators and
> iterators, so it's no longer possible to pretend that they're the same
> thing."  That is a major and worrisome conceptual shift.
>  >
>  > I agree with Raymond on this point.
>
> A particularly relevant variant of the idiom is the approach of writing
> "__iter__" directly as a generator, rather than creating a separate custom
> iterator class. In that context, the similarities between the __iter__
> implementation and the corresponding explicit __next__ implementation is a
> beneficial feature.
>
> I'm definitely coming around to the point of view that, even if we wouldn't
> design it the way it currently works given a blank slate, the alternative
> design doesn't provide sufficient benefit to justify the cost of changing
> the behaviour & getting people to retrain their brains.

This all seems more complex than it should be to me.  The way I tend to 
think about it is simply "for" loops in one form or another, catch 
StopIteration.  So if you iterate an iterator manually rather than using it 
as a "for" iterator, you need to catch StopIteration.

If we write a function to act as an iterator, like __next__, we need to 
raise StopIteration from somewhere in it, or let one bubble out from a 
generator if we are manually iterating it on each call... next(g). It's 
possible we may need to do either or both conditionally.  That could mean 
we need to think about refactoring some part of a program, but it doesn't 
mean Python needs to be fixed.

So the lines that split them isn't quite as clear cut as it may seem they 
should be.  That may just be a misplaced ideal.

Any time a StopIteration is raised.. either manually with "raise", or at 
the end of a generator, it should bubble up until a "for loop" iterating 
over that bit of code, catches it, or a try-except catches it, or fail 
loudly.  I think it does do this in normal generators, so I don't see an 
issue with how StopIteration works.


Which gets us back to generator expressions and comprehensions.

Let me know if I got some part of this wrong...   :-)

Comprehensions are used as a convenient way to create an object.  The 
expression parts of the comprehension define the *body* of a loop, so a 
StopIteration raised in it will bubble out.  As it would in any other case 
where it is raised in the body of a loop.

Generator exprssions on the other hand define the *iterator* to be used in 
a for loop. A StopIteration raised in it is caught by the for loop.

So they both work as they are designed, but they look so similar, it looks 
like one is broken.



It looks to me that there are three options...


OPTION 1:

Make comprehensions act more like generator expressions.

It would mean a while loop in the object creation point is converted to a 
for loop. (or something equivalent.)

Then both a comprehension and a generator expressions can be viewed as 
defining iterators, with the same behaviour, rather than comprehensions 
defining the body of the loop, which has the different but valid behaviour 
of StopIteration escaping.

This would make explaining them a *lot* easier as they become the same 
thing used in a different context, rather than two different things used in 
what appears to be similar contexts.


I think this fits with what Guido wants, but does so in a narrower scope, 
only effecting comprehensions.  StopIteration is less likely to leak out.

But it also allows the use of the stop() hack to raise StopIteration in 
comprehensions and terminate them early.  Currently it doesn't work as it 
does in generator expressions.

If the stop() hack works in both comprehensions and generator expressions 
the same way, then maybe we can view it as less of a hack.


OPTION 2:

Make generator expressions more like comprehensions.

This would mean StopIteration would bubble out of generator expressions as 
the person who posted the original topic on python ideas wanted.  And the 
stop hack would no longer work.

Both generator expressions and comprehensions could be viewed as supplying 
the body in a loop.  This is inconsistent with defining generators that act 
as iterators.  So I'm definitely -1 on this option.


OPTION 3:

Document the differences better.




Cheers,
    Ron