Different behaviour in list comps and generator expressions

Wolfgang Maier wolfgang.maier at biologie.uni-freiburg.de
Sat Nov 8 23:42:29 CET 2014


On 08.11.2014 22:31, Wolfgang Maier wrote:
> On 08.11.2014 02:50, Steven D'Aprano wrote:
>> The following list comprehension and generator expression are almost, but
>> not quite, the same:
>>
>> [expr for x in iterable]
>>
>> list(expr for x in iterable)
>>
>>
>> The difference is in the handling of StopIteration raised inside the
>> expr.
>> Generator expressions consume them and halt, while comprehensions allow
>> them to leak out.
>
> This is not the right description of what's happening. It is not the
> generator expression that consumes it, but the list constructor and, of
> course, list can't tell at which level in the inside code StopIteration
> got raised.
> So, yes this had me confused some times, too, but it is really not
> surprising.
>
>> A simple example:
>>
>> iterable = [iter([])]
>> list(next(x) for x in iterable)
>> => returns []
>>
>> But:
>>
>> [next(x) for x in iterable]
>> => raises StopIteration
>>
>
> Yes, but the equivalent to list(next(x) for x in iterable) is:
>
> l = []
> for item in (next(x) for x in iterable):
>      l.append(item)
>
> => returns [] because the for consumes the StopIteration
>
> this, on the other hand raises StopIteration
>
> l = []
> for item in [next(x) for x in iterable]:
>      l.append(item)
>
> because the error gets raised already when for causes iter() to be
> called on the list comprehension.
>
>>
>> Has anyone come across this difference in the wild? Was it a problem?
>> Do you
>> rely on that difference, or is it a nuisance? Has it caused difficulty in
>> debugging code?
>>
>> If you had to keep one behaviour, which would you keep?
>>
>
> The point is: there is no difference in the behavior of comprehensions
> and generator expressions, it is just how you access them:
> for anything that follows the iterator protocol, a comprehension (since
> it is evaluated immediately) raises during the iter() phase, while a
> generator expression (which gets evaluated lazily) raises during the
> next() phase where it gets swallowed.
>
> So in your example you would have to use the silly
>
> list([next(x) for x in iterable])
>
> if you want the error to get raised.
>
> I agree this is all rather non-intuitive, but how would you change it ?
>
>

Ah, I came across the related thread on python-ideas only now and from 
this I can see that, as I expected, you know everything I've written 
above already.

In light of the discussion on the other list:
I did find it annoying occasionally that raising StopIteration inside a 
generator expression conveys a different behavior than elsewhere. It did 
take me quite a while to understand why that is so, but after that it 
did not cause me much of a headache anymore.

I would say that Guido's suggestion of transforming StopIteration raised 
inside a generator into some other error to eliminate ambiguity would 
really help in some situations, but then I'm not sure whether that's 
worth breaking existing code.

Best,
Wolfgang




More information about the Python-list mailing list