[Python-ideas] "Iteration stopping" syntax [Was: Is this PEP-able? for X in ListY while conditionZ:]

[Andrew Barnert]

From: Shane Green <shane at umbrellacode.com>
Sent: Saturday, June 29, 2013 5:10 AM


>Thanks Andrew.  My knee jerk reaction was to strongly prefer option two, which sounds like–if I understood correctly, and I’m not sure I do–it keeps both comprehensions and expressions.  Rereading your points again, I must admit I didn’t see much to justify the knee jerk reaction.

Sorry, it's my fault for conflating two independent choices. Let me refactor things:

Nobody's talking about getting rid of comprehensions. Today, you have the choice to write [comp] instead of list(comp), and that won't change. However, today these differ in that the latter lets you exit early with StopIteration, while the former doesn't. That's what's proposed to change.

Choice 1 is about the language definition. With this change, there is no remaining difference between genexps except for returning a list vs. an iterator. That means we could simplify things by defining the two concepts together (or defining one in terms of the other). I don't see any downside to doing that.

Choice 2 is about the CPython implementation. We can reimplement comprehensions as wrappers around genexps, or we can just add a try…except into comprehensions. The former would simplify the compiler and the interpreter, but at a cost of up to 40% for comprehensions. The latter would leave things no simpler than they are today, but also no slower.

Once put this way, I think the choices are obvious: Simplify the language, don't simplify the implementation.

>I do commonly use list comprehensions precisely *because* of the performance impact, and can think of a few places the 40% would be problematic.

Usually that's a premature optimization. For anything simple enough that the iteration cost isn't swamped by your real work, the performance usually doesn't matter anyway.

But "usually" isn't always, and there definitely are real-world cases where it would hurt.

> Was there a measurable performance difference with approach 2?


Once I realized that the right place to put the try is just outside the loop… that makes it obvious that there is no per-iteration cost, only a constant cost.

If you don't raise an exception through a listcomp, that cost is basically running one more opcode and loading a few more bytes into memory. It adds less than 1% for even a trivial comp that loops 10 times, or for a realistic but still simple comp that loops 3 times.

I'll post actual numbers for local tests and for benchmarks once I get things finished (hopefully Monday).

>On Jun 28, 2013, at 8:16 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

>
>On Jun 28, 2013, at 18:50, Shane Green <shane at umbrellacode.com> wrote:
>>
>>
>>Yes, but it only works for generator expressions and not comprehensions.
>>
>>
>>This is the point if options #1 and 2: make StopIteration work in comps either (1) by redefining comprehensions in terms of genexps or (2) by fiat. 
>>
>>
>>After some research, it turns out that these are equivalent. Replacing any [comprehension] with list(comprehension) is guaranteed by the language (and the CPython implementation) to give you exactly the same value unless (a) something in the comp raises StopIteration, or (b) something in the comp relies on reflective properties (e.g., sys._getframe().f_code.co_flags) that aren't guaranteed anyway.
>>
>>
>>So, other than being 4 characters more verbose and 40% slower, there's already an answer for comprehensions.
>>
>>
>>And if either of those problems is unacceptable, a patch for #1 or #2 is actually pretty easy. 
>>
>>
>>I've got two different proof of concepts: one actually implements the comp as passing the genexp to list, the other just wraps everything after the BUILD_LIST and before the RETURN_VALUE in a the equivalent of try: ... except StopIteration: pass. I need to add some error handling to the C code, and for #2 write sufficient tests that verify that it really does work exactly like #1, but I should have working patches to play with in a couple days.
>>
>>My opinion of that workaround is that it’s also a step backward in terms of readability.  I suspect. 
>>
>>>
>>>if i < 50 else stop() would probably also work, since it throws an exception.  That’s better, IMHO.  
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>On Jun 28, 2013, at 6:38 PM, Andrew Carter <acarter at g.hmc.edu> wrote:
>>>
>>>Digging through the archives (with a quick google search) http://mail.python.org/pipermail/python-ideas/2013-January/019051.html, if you really want an expression it seems you can just do
>>>>
>>>>
>>>>def stop():
>>>>  raise StopIteration
>>>>list(i for i in range(100) if i < 50 or stop())
>>>>
>>>>
>>>>it seems to me that this would provide syntax that doesn't require lambdas.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>On Fri, Jun 28, 2013 at 4:50 PM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>On Fri, Jun 28, 2013 at 7:38 PM, Shane Green <shane at umbrellacode.com> wrote:
>>>>>
>>>>>..
>>>>>>[x until condition for x in l ...] or 
>>>>>>[x for x in l until condition]
>>>>>
>>>>>
>>>>>Just to throw in one more variation:
>>>>>
>>>>>
>>>>>[expr for item in iterable break if condition]
>>>>>
>>>>>
>>>>>(inversion of "if" and "break"reinforces the idea that we are dealing with an expression rather than a statement - compare with "a if cond else b")  
>>>>>_______________________________________________
>>>>>Python-ideas mailing list
>>>>>Python-ideas at python.org
>>>>>http://mail.python.org/mailman/listinfo/python-ideas
>>>
>>_______________________________________________
>>>Python-ideas mailing list
>>>Python-ideas at python.org
>>>http://mail.python.org/mailman/listinfo/python-ideas
>
>
>