[Python-Dev] PEP 479: Change StopIteration handling inside generators

[Terry Reedy]

On 11/22/2014 2:49 PM, Ron Adam wrote:

> On 11/22/2014 08:31 AM, Nick Coghlan wrote:

>> I'm definitely coming around to the point of view that, even if we
>> wouldn't
>> design it the way it currently works given a blank slate, the alternative
>> design doesn't provide sufficient benefit to justify the cost of changing
>> the behaviour & getting people to retrain their brains.

Me too.

> Which gets us back to generator expressions and comprehensions.

 > Comprehensions are used as a convenient way to create an object.  The
> expression parts of the comprehension define the *body* of a loop, so a
> StopIteration raised in it will bubble out.  As it would in any other
> case where it is raised in the body of a loop.
>
> Generator expressions on the other hand define the *iterator* to be used
> in a for loop. A StopIteration raised in it is caught by the for loop.
>
> So they both work as they are designed, but they look so similar, it
> looks like one is broken.

> It looks to me that there are three options...
>
>
> OPTION 1:
>
> Make comprehensions act more like generator expressions.

I was thinking about this also.

> It would mean a while loop in the object creation point is converted to
> a for loop. (or something equivalent.)

The code for [e(i) for i in it] already uses a for loop.  The issue is 
when e(i) raise StopIteration.  The solution to accomplish the above is 
to wrap the for loop in try: ... except StopIteration: pass.

def _list_comp_temp():
     ret = []
     try:
         for i in it:
             ret.append[e(i)]
     except StopIteration:
         pass
     return ret

I believe this would make list(or set)(genexp) == [genexp] (or {genexp}) 
as desired.  In 2.x, there was a second difference, the leaking of 'i'. 
  3.x eliminated one difference, the leaking of the iteration variable, 
but not the other, the leaking of StopIteration.

The document about execution of comprehensions says "the comprehension 
is executed in a separate scope, so names assigned to in the target list 
don’t “leak”. What we would be adding for comprensions (but not genexps) 
is "and StopIteration raised in evaluating the expression" before 
'"leak"'.  We could then document the equality above.

> Then both a comprehension and a generator expressions can be viewed as
> defining iterators,

A comprehension is not an iterator.  The above would make a list or set 
comprehension the same as feeding a genexp to list() or set().

> I think this fits with what Guido wants, but does so in a narrower
> scope, only effecting comprehensions.  StopIteration is less likely to
> leak out.

StopIteration would *not* leak out ever.  This is half of what Guido 
wants, the other half being the issue of obscure bugs with genrators in 
general.

> But it also allows the use of the stop() hack to raise StopIteration in
> comprehensions and terminate them early.

Yep. There is no good way to have next(it) work as intended, including 
in Raymond's example, where it should work, and not let stop() work.  I 
think this falls under our 'consenting adults' principle.

> Currently it doesn't work as it does in generator expressions.
> If the stop() hack works in both comprehensions and generator
> expressions the same way, then maybe we can view it as less of a hack.

-- 
Terry Jan Reedy