[x for x in l if x < 10 else break]? On Jun 25, 2013, at 10:08 AM, Ned Batchelder <ned@nedbatchelder.com> wrote:

You don't have to: use the break statement, that's what it's for. About people teaching students not to use it: the existence of bad teachers teaching silly ideas is not a reason to add syntax to Python.

--Ned.

On 6/25/2013 10:46 AM, jimjhb@aol.com wrote:

...

You shouldn't have to invoke takewhile and a lambda just to break out of for loop.

...

http://docs.python.org/2/library/itertools.html#itertools.takewhile

for item in takewhile(lambda x: x < 5, range(10)): pass

...

...

[People who avoid the 'break' by functionalizing an inner portion of the loop are just kidding themselves and making their own code worse, IMO. Takewhile from itertools also works, but that's clumsy and wordy as well.]

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

So is your suggestion, but it has a different name. If your teacher is this easily fooled, get a new teacher. On Tue, Jun 25, 2013 at 8:03 PM, <jimjhb@aol.com> wrote:

That's a break....

-----Original Message----- From: Shane Green <shane@umbrellacode.com> To: Ned Batchelder <ned@nedbatchelder.com> Cc: python-ideas <python-ideas@python.org> Sent: Tue, Jun 25, 2013 1:13 pm Subject: Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ:

[x for x in l if x < 10 else break]?

On Jun 25, 2013, at 10:08 AM, Ned Batchelder <ned@nedbatchelder.com> wrote:

You don't have to: use the break statement, that's what it's for. About people teaching students not to use it: the existence of bad teachers teaching silly ideas is not a reason to add syntax to Python.

--Ned.

On 6/25/2013 10:46 AM, jimjhb@aol.com wrote:

You shouldn't have to invoke takewhile and a lambda just to break out of for loop.

...

http://docs.python.org/2/library/itertools.html#itertools.takewhile

for item in takewhile(lambda x: x < 5, range(10)): pass

...

[People who avoid the 'break' by functionalizing an inner portion of the loop are just kidding themselves and making their own code worse, IMO. Takewhile from itertools also works, but that's clumsy and wordy as well.]

_______________________________________________ Python-ideas mailing listPython-ideas@python.orghttp://mail.python.org/mailman/listinfo/python-ideas

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

_______________________________________________ Python-ideas mailing listPython-ideas@python.orghttp://mail.python.org/mailman/listinfo/python-ideas

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On 26/06/13 04:12, Stephen J. Turnbull wrote:

Shane Green writes:

...

[x for x in l if x < 10 else break]?

That's currently invalid syntax: break is a statement. I think a while clause (as suggested by David Mertz) would be a more plausible extension of syntax.

I do think extending generator/comprehension syntax is much more plausible than extending for loop syntax (for one thing, "just use break" is not an answer here!)

Comprehensions in Clojure have this feature. http://clojuredocs.org/clojure_core/clojure.core/for ;; :when continues through the collection even if some have the ;; condition evaluate to false, like filter user=> (for [x (range 3 33 2) :when (prime? x)] x) (3 5 7 11 13 17 19 23 29 31) ;; :while stops at the first collection element that evaluates to ;; false, like take-while user=> (for [x (range 3 33 2) :while (prime? x)] x) (3 5 7) (expr for x in seq while cond) is not expandable into a for loop in quite the same way as (expr for x in seq if cond) is: result = [] for x in seq: if cond: result.append(expr) vs result = [] for x in seq: if cond: result.append(expr) else: break but I think it is a natural extension to the generator syntax that fills a real need (or is at least frequently requested), is understandable, and has precedent in at least one other language. But Nick Coghlan has ruled it's not going to happen, although I don't understand what he had against it. -- Steven

On 6/26/2013 12:45 AM, Nick Coghlan wrote:

Comprehensions are currently just syntactic sugar for particular kinds of explicit loop, with a relatively straightforward mechanical translation from the expression form to the statement form. That's an essential property that helps keep Python's expression level and suite level flow control constructs from diverging into two independent languages that happen to share some keywords.

I think there are two points about comprehensions that people miss when they use it as justification to proposed that their personal favorite composition be built in to the language. 1. Comprehensions compose at least three statements, and not just two. 2. Comprehensions embody and implement a basic concept and tool of thought -- defining a class or collection by rule, which is complementary to defining such by roster. The former is probably more common in everyday life and definitely so in law. A statement form of itertools.takewhile is not in the same ballpark. -- Displays are also 'just syntactic sugar for particular kinds of ... loop' -- but with the loop written in the imterpreter implementation language. nums = [1,2,3] is equivalent to nums = [] nums.append(1) nums.append(2) nums.append(3) and in CPython implemented as the internal equivalent of nums = [i for i in <1,2,3>] where <1,2,3> is a sequence of stack values. Tuples are preallocated to their final size and filled in by index.

I would vastly prefer implementing PEP 403 to allow the iterable in a comprehension to be a full generator function

I do not understand what you mean here. PEP 403 is about a new decorator clause, whereas the iterable in comprehension can already be the return from a generator function. -- Terry Jan Reedy

On Wed, Jun 26, 2013 at 3:39 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

I can do this with generators now:

...

...

...

def stop(): ... raise StopIteration ... list(i for i in range(10) if i < 3 or stop()) [0, 1, 2]

Can't the same be allowed in compehensions?

This will only work in generator comprehensions. The reasons are obvious if you think about it. But it isn't too ugly as a technique, IMO. I still like the 'while' clause in generators (notwithstanding the fact it doesn't translate straightforwardly to an "unrolled block"), but this is sort of nice (and maybe I'll start using it):

...

...

dict((i,i*2) for i in range(10) if i < 3 or stop()) {0: 0, 1: 2, 2: 4}

Even though this happens:

...

...

{i:i*2 for i in range(10) if i<3 or stop()} Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <dictcomp> File "<stdin>", line 1, in stop StopIteration

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On 27 June 2013 00:20, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 27 Jun 2013 08:57, "David Mertz" <mertz@gnosis.cx> wrote:

...

On Wed, Jun 26, 2013 at 3:39 PM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:

...

I can do this with generators now:

...

...

...

def stop(): ... raise StopIteration ... list(i for i in range(10) if i < 3 or stop()) [0, 1, 2]

Can't the same be allowed in compehensions?

For reference this was discussed 6 months ago in the thread starting here: http://mail.python.org/pipermail/python-ideas/2013-January/018969.html

...

This will only work in generator comprehensions. The reasons are obvious if you think about it. But it isn't too ugly as a technique, IMO. I still like the 'while' clause in generators (notwithstanding the fact it doesn't translate straightforwardly to an "unrolled block"), but this is sort of nice (and maybe I'll start using it):

...

...

...

dict((i,i*2) for i in range(10) if i < 3 or stop()) {0: 0, 1: 2, 2: 4}

I'm personally not opposed to allowing "else break" after the if clause as a way of terminating iteration early in comprehensions and generator expressions, as it's consistent with the existing translation to an explicit loop:

{i:i*2 for i in range(10) if i<3 else break}

If I remember there was an objection to using "else break" in preference for "else return" as it was deemed (by some) that it wasn't clear which loop the break applied to in the case where there were nested loops. The "else return" version still makes sense when unrolled if the intention is to terminate all the loops.

I'd even be OK with the use of an "else" clause to define alternative entries to use when the condition is false:

{i:i*2 for i in range(10) if i<3 else i:i}

You can use ternary if/else in the expression rather than the if clause: {i:i*2 if i < 3 else i for i in range(10)} I think that's clearer in this case. This always works for list/set comprehensions and for generators. For dict comprehensions it works but it's less good if you need to use it for both key and value, i.e. {i*2: i*3 for i in range(10) if i < 3 else i:i} would become {i*2 if i < 3 else i: i*3 if i < 3 else i for i in range(10)}

A reasonable constraint on the complexity may be to disallow mixing this extended conditional form with the nested loop form.

I would welcome an addition along these lines with that constraint. I rarely use nested loops in comprehensions and often find myself writing simple while loops in situations that would be covered by a comprehension that used "else break". Oscar

From: Ron Adam <ron3200@gmail.com> Sent: Wednesday, June 26, 2013 7:44 PM

The 'if' after the iterator handles the selection of values.  Given that, it makes sense to allow flow control statements after the last if, but not expressions effecting the values.

First, why restrict break to the last if clause? Saying it can't go in a for clause or in the controlled expression I understand, but… currently, there's nothing special about the last clause (or any other clause, except for the first, in genexps only), and it seems odd to change that. But meanwhile, your description made me realize exactly what feels weird about the whole idea. A comprehension doesn't have flow control statements. It has _clauses_. Obviously they're related things; there's a mapping that's rigorously definable and intuitively clear. But they don't have a colon and a suite. (They don't even take the same set of expressions, but that's not important here.) Intuitively, that's because the whole point of a comprehension is that the rest of the comprehension, especially including the core expression, is the suite. Adding a suite breaks that.  Forget that break means the flow control is no longer always downward-local. It also completely turns the meaning of the controlling if statement around. Instead of "if this: the rest of the expression" it's "if this: the suite, else: the rest of the expression". On top of that, the clauses map to nested statements. You can't nest break statements. All of this becomes more obvious if you try to describe things rigorously. Let's rewrite http://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-... to explain the semantics of break. In order to make the issues more obvious, I'm going to ignore continue, use your "only at the end" restriction, and not add a rule saying that the break has to be controlled by an if. (If you want to write [x for subiter in iter for x in subiter break], fine, you get back []—this actually won't be interpretable for a genexp with only one clause, but let that be an error because it translates to an error.) I don't think changing any of those would solve any of the problems, they'd just make things more complicated and harder to see. First, add the new syntax. There are two obvious ways to do it:     comp_if ::=  "if" comp_if_body expression_nocond ["break" | comp_iter] Or:     comp_iter ::= comp_for | comp_if | comp_break     comp_break ::= "break" If you keep the existing semantics, this:     x = [value for value in iter if pred(value) break] … maps to:     x = []     for value in iter:         if pred(value):             break                 x.append(value) Or maybe the append is a sibling of the break rather than a child? Either way, it makes no sense. It has to map to something like this:     x = []     for value in iter:         if pred(value):             break         x.append(value) Which means the clauses are no longer nested, so the simple explanation no longer works. Instead, you need something more convoluted, like: The comprehension consists of a single expression followed by at least one for clause, zero or more for or if clauses, and zero or one break clause. In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and the break clause, if present, a simple statement, and evaluating the expression to produce an element each time the innermost block that does not directly contain a break is entered and a block directly containing a break, if any, is not entered. I think you can make it a little less convoluted by using the concept of exiting a block (which is well-defined, thanks to with statements), but I think that just makes it even less intuitive: The comprehension consists of a single expression followed by at least one for clause, zero or more for or if clauses, and zero or one break clause. In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and the break clause, if present, a simple statement, and evaluating the expression to produce an element each time the innermost block is exited without executing a break. There's no way to explain this simply, because it's no longer simple. As a side note:

    [x for x in range(n) if x<limit pass else continue]

This is already hard to distinguish from a ternary expression, even without the other stuff you bring in later. Compare:

    [x for x in range(n) if n<10 else range(10)]

I don't think that's valid (although I'm not sure without testing). And I'm not sure if it would be as hard for the parser to deal with as it is for a human. But really, if a human can't parse it, it's meaningless.

Let me try to gather together all of the possibilities that have been discussed in this and the two previous threads, plus a couple of obvious ones nobody's mentioned. Unless I'm missing a good idea, or someone can explain why one of these isn't as bad as it seems, I don't like any of them. Some of them are ugly right off the bat. The rest are deceptively appealing until you think them through, but then they're even worse than the obviously bad ones. I'll try to put them in order from least bad to most horrid. (I'm +0.25 on #1, but not until Python 4; -0 on #2; -1 on #3, and it's all downhill from there.) 1. Redefine comprehensions on top of generator expressions instead of defining them in terms of nested blocks.     def stop(): raise StopIteration     x = [value if pred(value) else stop() for value in iterable] This would make the implementation of Python simpler.  It also makes the language conceptually simpler. The subtle differences between [x for x in foo] and list(x for x in foo) are gone. And it's actually a pretty small change to the official semantics. Just replace the last two paragraphs of 6.2.4 with "The comprehension consists of a single expression followed by at least one for clause and zero or more for or if clauses. In this case, the expression and clauses are interpreted as if they were a generator expression, and the elements of the new container are those yielded by that generator expression consumed to completion." (It also makes it easier to fix the messy definition of dict comprehensions, if anyone cares.) Unlike #5, 6, 7, 8, and 10, but like #2, 3, 4, and 9, this only allows you to break out of one for clause, not any. But that's exactly the same as break only being able to break out of one for loop. Nobody complains that Python doesn't have "break 2" or "break :label", right? The real downside is that this is a very radical change. People may sometimes rely on the differences between listcomps and genexps, maybe even without realizing they're doing so. Code that's worked unchanged from 2.0 to 3.3 might break. Also, the obvious implementation would make listcomps about 25-50% slower, and trying to tweak the existing optimized comprehension code to work as-if wrapping a generator (and writing tests for all of the edge cases) sounds like a lot of work. And at best it would still be at least a _little_ slower (handling StopIteration, handling errors in the outer for clause differently from other clauses, etc. aren't free). Still, I could get behind this for Python 4.0. 2. Just require comprehensions to handle StopIteration. The main cost and benefit are the same as #1. However, it makes the language and implementation more complex, rather than simpler. Also, the effects of this less radical change (you can no longer pass StopIteration through a comprehension) seem like they might be harder to explain to people than the more radical one. And, worse, they less radical effects would probably cause more subtle bugs. Which implies a few versions where passing StopIteration through a comprehension triggers a warning. 3. Turn break from a statement into an expression.     x = [value if pred(value) else break for value in iterable] The break expression, when evaluated, would have the exact same effect as executing the statement today (including being an error if it's not nested syntactically under a loop or inside a comprehension). This would be trivial to implement, document, and teach. But I don't think anyone wants an expression that inherently has no value. Python doesn't have any such expressions today. (Yes, you can write a function call that never returns normally—like sys.exit()—but that still has a return, it just never gets there.) Also, today, Python gets a lot of mileage out of separating statements and expressions. You can see the flow at a glance, and there's exactly one side-effect per line. If we're going to lose that, I'd rather get something a lot cooler in exchange. 4. Add a break expression that's only allowed inside a comprehension expression. Basically the same as #3 with less wide-reaching effects… but harder to describe. (How would you write the grammar? Duplicate every expression node: comp_conditional_expression, comp_lambda_form, …? Allow break as an expression syntactically, but make it an error semantically except directly in an expression_stmt or comprehension?) And of course it makes the language less consistent. I'd rather have #3, and just say "don't use break as an expression except in comprehensions" in PEP 8 and in linters. 5. Add a new until statement, and a corresponding until clause to comprehensions.     x = [value for value in iterable until pred(value)] It's intuitively obvious what it means. And it's easy to define:     comp_iter ::= comp_for | comp_if | comp_until     comp_until ::= "until" expression [comp_iter] Then just s/for or if/for, if, or until/g in 6.2.4. Of course you have to define an until statement:     until expression: suite … equivalent to:     if expression: break     else: suit … with syntax:     until_stmt ::= "until" expression ":" suite … and semantics:

until may only occur syntactically nested in a for or while loop, but not nested in a function or class definition within that loop.  If the expression is found to be true, it terminates the nearest enclosing loop; otherwise, the suite is executed.

… and nobody will ever use that statement. Which is a hell of an argument against adding it to the language. Obviously you could use a different new keyword, with the same sense or the opposite. But I don't think anything is any better. A couple of people suggested when, but that's even worse—besides sounding like it means "if" rather than "while" here, it actually _does_ mean "if" in most languages that have it—Clojure, CoffeeScript, Racket, etc. 6. Add the until clause without the statement. Basically the same as #5, but interpreted as-if an until statement existed, which it won't. To me, this is much worse than #5. It adds the same complexity to the language, and makes it inconsistent to boot. To interpret a comprehension, you'll have to map it to a nested block that contains statements that don't exist—you have no experience with, and no way to gain experience with them, and you can't even run the resulting code. 7. Add a "magic" while clause that's basically until with the opposite sense.     x = [value for value in iterable while pred(value)] This reads pretty nicely (at least in trivial comprehensions), it parallels takewhile and friends, and it matches a bunch of other languages (most of the languages where "when" means "if", "while" means this). But it has a completely different meaning from while statements, and in fact a barely-related one. In particular, it's obviously not this:     x = []     for value in iterable:         while pred(value):             x.append(value) What it actually means is:     x = []     for value in iterable:         if pred(value):             x.append(value)        else:            break Imagine trying to teach that to a novice. Or trying to write the formal definition in 6.2.4. This makes the language much more inconsistent than #6.  8. Allow else break in comp_if clauses.     x = [value for value in iterable if pred(value) else break] This one is pretty easy to define rigorously, since it maps to exactly what the while attempt maps to with a slight change to the existing rules. But to me, it makes the code a confusing mess. I'm immediately reading "iterable if pred(value) else break", and that's wrong. Also, while it's pretty easy to convert this to a nested block in your head, it's not as easy to just read out in line, because the rest of the comprehension and the main expression no longer just nest under the end of the clause; instead, they nest under the middle of it Also, an else clause that can't be used for anything but break is very weird. But it doesn't make sense to put anything else there. 9. Add a special comp_break clause.     x = [value for value in iterable if pred(value) break] The syntax is intuitively simple (break clauses nest just like if and for clauses, and the fact that you can't nest anything underneath it is obvious), and also trivial to define (see my last email). But this reverses the sense of the controlling if. In fact, each time I read it, the meaning seems to flip back and forth until I finally get it. And there's no way to make the semantics reasonable if they're defined in terms of nested blocks (as they are today) without lots of special-case language to deal with break. (See my last email.) 10. Allow break in comp_if clauses.     x = [value for value in iterable if pred(value) break] The syntax is almost simple as #9, and it gets you more flexibility (because there's nothing stopping you from putting a break under any if statement, not just the last). However, intuitively this is exactly the same as #9. An if statement with a break means something completely different, and nearly opposite, to one without a break. And trying to define the meaning is even more complicated. ----- Original Message -----

From: Andrew Barnert <abarnert@yahoo.com> To: "ron3200@gmail.com" <ron3200@gmail.com>; "python-ideas@python.org" <python-ideas@python.org> Cc: Sent: Wednesday, June 26, 2013 10:13 PM Subject: Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ:

From: Ron Adam <ron3200@gmail.com>

Sent: Wednesday, June 26, 2013 7:44 PM

...

The 'if' after the iterator handles the selection of values.  Given

...

that, it makes sense to allow flow control statements after the last if, but not expressions effecting the values.

First, why restrict break to the last if clause? Saying it can't go in a for clause or in the controlled expression I understand, but… currently, there's nothing special about the last clause (or any other clause, except for the first, in genexps only), and it seems odd to change that.

But meanwhile, your description made me realize exactly what feels weird about the whole idea.

A comprehension doesn't have flow control statements. It has _clauses_. Obviously they're related things; there's a mapping that's rigorously definable and intuitively clear. But they don't have a colon and a suite. (They don't even take the same set of expressions, but that's not important here.)

Intuitively, that's because the whole point of a comprehension is that the rest of the comprehension, especially including the core expression, is the suite. Adding a suite breaks that. 

Forget that break means the flow control is no longer always downward-local. It also completely turns the meaning of the controlling if statement around. Instead of "if this: the rest of the expression" it's "if this: the suite, else: the rest of the expression".

On top of that, the clauses map to nested statements. You can't nest break statements.

All of this becomes more obvious if you try to describe things rigorously. Let's rewrite http://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-... to explain the semantics of break.

In order to make the issues more obvious, I'm going to ignore continue, use your "only at the end" restriction, and not add a rule saying that the break has to be controlled by an if. (If you want to write [x for subiter in iter for x in subiter break], fine, you get back []—this actually won't be interpretable for a genexp with only one clause, but let that be an error because it translates to an error.) I don't think changing any of those would solve any of the problems, they'd just make things more complicated and harder to see.

First, add the new syntax. There are two obvious ways to do it:

    comp_if ::=  "if" comp_if_body expression_nocond ["break" | comp_iter]

Or:

    comp_iter ::= comp_for | comp_if | comp_break     comp_break ::= "break"

If you keep the existing semantics, this:

    x = [value for value in iter if pred(value) break]

… maps to:

    x = []     for value in iter:         if pred(value):             break                 x.append(value)

Or maybe the append is a sibling of the break rather than a child? Either way, it makes no sense. It has to map to something like this:

    x = []     for value in iter:         if pred(value):             break         x.append(value)

Which means the clauses are no longer nested, so the simple explanation no longer works. Instead, you need something more convoluted, like:

The comprehension consists of a single expression followed by at least one for clause, zero or more for or if clauses, and zero or one break clause. In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and the break clause, if present, a simple statement, and evaluating the expression to produce an element each time the innermost block that does not directly contain a break is entered and a block directly containing a break, if any, is not entered.

I think you can make it a little less convoluted by using the concept of exiting a block (which is well-defined, thanks to with statements), but I think that just makes it even less intuitive:

The comprehension consists of a single expression followed by at least one for clause, zero or more for or if clauses, and zero or one break clause. In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and the break clause, if present, a simple statement, and evaluating the expression to produce an element each time the innermost block is exited without executing a break.

There's no way to explain this simply, because it's no longer simple.

As a side note:

...

    [x for x in range(n) if x<limit pass else continue]

This is already hard to distinguish from a ternary expression, even without the other stuff you bring in later. Compare:

...

    [x for x in range(n) if n<10 else range(10)]

I don't think that's valid (although I'm not sure without testing). And I'm not sure if it would be as hard for the parser to deal with as it is for a human. But really, if a human can't parse it, it's meaningless.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

From: Nick Coghlan <ncoghlan@gmail.com> Sent: Thursday, June 27, 2013 1:43 AM

Someone was asking earlier why I thought PEP 403 was at all relevant to these discussions. It's because it makes it easy to define a one shot generator function to precisely control the iterable in a comprehension. If we assume a world where itertools.takewhile doesn't exist, you could write an equivalent inline:

    @in x = [transform(value) for value in my_takewhile(pred, iterable)]     def my_takewhile(pred, itr):         for x in itr:             if not pred(x):                 break             yield x

I'm still not sure how it's relevant here. Obviously takewhile already exists in the stdlib, and if I wanted stop I'd probably want it frequently enough to define it and reuse it. So, for the current discussion, are you just suggesting using it to break up an overly-verbose or -complex takewhile solution, as an answer to "Yeah, I know about takewhile, but it's too verbose or complex so I don't want to use it"? If so, I don't think it answers that. If someone wants this:     x = [value for value in iterable if value > 0 while value < 10] … and rejects this:     x = [value for value in takewhile(lambda x: x < 10, iterable) if value > 0] … I don't think they'd be happier with this:     @in x = [value for value in takewhile(under10, iterable) if value > 0]     def under10(value):         return value < 10 And as someone who _wouldn't_ reject the takewhile, if I want to break it up, I'd be happier going in the opposite direction:     iterable_to_10 = takewhile(lambda value: value < 10, iterable)     x = [value for value in iterable_to_10 if value > 0] That puts it as a sequence of simple transformations to an iterable, rather than a bunch of simple functions plugged into different places in one big expression. (Although admittedly my stupid toy example, with its total of 1 transformations or functions, isn't the best demonstration of that…) Where your idea would really be useful is, exactly, as you say, "… to get access to full statement syntax, but we only use it in this one place". I don't think the kinds of things people are envisioning putting in while clauses need statement syntax. The only reason statement syntax even really came up is the idea of break and/or a more-statement-like if as an alternative to while. Having a real function wouldn't help with break, because you're not allowed to break across function boundaries. In fact, I think what people really want is the opposite: to write an expression, not a function. That's a big part of the appeal of using comprehensions over map and filter: if you don't already have a ready-made function, no problem (and no lambdas or partials), just use a comprehension and write the expression in-place. People want a similar answer to make takewhile at least as unnecessary as map and filter.

This closely ties to the discussion with subject that starts with “is this PEP-able? [...]” What about [x until condition for x in l ...] or [x for x in l until condition] Condition checked with output values only On Jun 28, 2013, at 3:28 PM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:

27.06.2013 13:49, Andrew Barnert napisał: [...]

...

If someone wants this:

x = [value for value in iterable if value > 0 while value < 10]

… and rejects this:

x = [value for value in takewhile(lambda x: x < 10, iterable) if value > 0]

… I don't think they'd be happier with this:

@in x = [value for value in takewhile(under10, iterable) if value > 0] def under10(value): return value < 10 [...] In fact, I think what people really want is the opposite: to write an expression, not a function. That's a big part of the appeal of using comprehensions over map and filter: if you don't already have a ready-made function, no problem (and no lambdas or partials), just use a comprehension and write the expression in-place. People want a similar answer to make takewhile at least as unnecessary as map and filter.

+1!

Maybe it could be achieved with a new separate generator expression syntax, being orthogonal to the existing generator-expr/comprehension syntaxes?

Maybe such as:

(<name> from <iterable> while <condition>)

E.g.:

g = (line from myfile while line.strip())

...which would be equivalent to:

def _temp(_myfile): iterator = iter(_myfile) line = next(iterator) while line.strip(): yield line line = next(iterator) g = _temp(myfile)

Unlike the existing generator-expr/comprehension syntaxes this syntax is focused on predicate-based iteration stopping (like itertools.takewhile) rather than on any processing of the iterated values.

Obviously, it could be combined with the existing syntaxes, e.g.:

processed = [2 * x for x in (x from seq while x < 100)]

...or probably better (a matter of taste):

items = (x from seq while x < 100) processed = [2 * x for x in items]

Cheers. *j

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Jun 28, 2013, at 16:50, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:

On Fri, Jun 28, 2013 at 7:38 PM, Shane Green <shane@umbrellacode.com> wrote:

...

.. [x until condition for x in l ...] or [x for x in l until condition]

Just to throw in one more variation:

[expr for item in iterable break if condition]

(inversion of "if" and "break"reinforces the idea that we are dealing with an expression rather than a statement - compare with "a if cond else b")

This is pretty much the same as the single-word breakif or ifbreak ideas, but has the advantage of not adding a new keyword. I'm not sure, without looking more carefully, whether the grammar could be ambiguous to the parser. But to a person I think it could be. We've already got two ways an "if" can appear in a comp--an if clause, or a ternary expression. Adding a third seems like it might make it harder to get the meaning just by scanning. As a side note, I think it would really help if we came up with a couple of paradigm examples instead of using content free toy examples. Maybe: [line for line in f break if not line.strip() if not line.startswith("#")] In other words "all comment characters up to the first blank line".

On 27 June 2013 08:28, Andrew Barnert <abarnert@yahoo.com> wrote:

Let me try to gather together all of the possibilities that have been discussed in this and the two previous threads, plus a couple of obvious ones nobody's mentioned.

You've missed out having "else return" in comprehensions. I like this less than a while clause but it was preferred by some as it unrolls perfectly in the case that the intention is to break out of all loops e.g.: [f(x, y) for x in xs for y in ys if g(x, y) else return] becomes _tmplist = [] def _tmpfunc(): for x in xs: for y in ys: if g(x, y): _tmplist.append(f(x, y)) else: return _tmpfunc()

1. Redefine comprehensions on top of generator expressions instead of defining them in terms of nested blocks.

def stop(): raise StopIteration

x = [value if pred(value) else stop() for value in iterable]

I prefer x = [value for value in iterable if pred(value) or stop()] so that the flow control is all on the right hand side of the "in".

This would make the implementation of Python simpler.

It also makes the language conceptually simpler. The subtle differences between [x for x in foo] and list(x for x in foo) are gone.

And it's actually a pretty small change to the official semantics. Just replace the last two paragraphs of 6.2.4 with "The comprehension consists of a single expression followed by at least one for clause and zero or more for or if clauses. In this case, the expression and clauses are interpreted as if they were a generator expression, and the elements of the new container are those yielded by that generator expression consumed to completion." (It also makes it easier to fix the messy definition of dict comprehensions, if anyone cares.)

Unlike #5, 6, 7, 8, and 10, but like #2, 3, 4, and 9, this only allows you to break out of one for clause, not any. But that's exactly the same as break only being able to break out of one for loop. Nobody complains that Python doesn't have "break 2" or "break :label", right?

I'm not sure why you expect that it would only break out of one for clause; I expect it to break out of all of them. That's how it works with generator expressions: Python 2.7.3 (default, Sep 26 2012, 21:51:14) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

list(x + y for x in 'abc' for y in '123') ['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3'] def stop(): raise StopIteration ... list(x + y for x in 'abc' if x == 'a' or stop() for y in '123') ['a1', 'a2', 'a3'] list(x + y for x in 'abc' for y in '123' if y == '1' or stop()) ['a1']

It also works that way if you spell it the way that you did:

...

...

list(x + y if y == '1' else stop() for x in 'abc' for y in '123') ['a1']

2. Just require comprehensions to handle StopIteration.

The main cost and benefit are the same as #1.

However, it makes the language and implementation more complex, rather than simpler.

Also, the effects of this less radical change (you can no longer pass StopIteration through a comprehension) seem like they might be harder to explain to people than the more radical one.

I think that the current behaviour is harder to explain.

And, worse, they less radical effects would probably cause more subtle bugs. Which implies a few versions where passing StopIteration through a comprehension triggers a warning.

I would be happy if it triggered a warning anyway. I can't imagine a reasonable situation where that isn't a bug.

7. Add a "magic" while clause that's basically until with the opposite sense.

x = [value for value in iterable while pred(value)]

This reads pretty nicely (at least in trivial comprehensions), it parallels takewhile and friends, and it matches a bunch of other languages (most of the languages where "when" means "if", "while" means this).

But it has a completely different meaning from while statements, and in fact a barely-related one.

In particular, it's obviously not this:

x = []

for value in iterable: while pred(value): x.append(value)

What it actually means is:

x = []

for value in iterable: if pred(value): x.append(value) else: break

Imagine trying to teach that to a novice.

I can definitely imagine teaching it to a novice. I have taught Python to groups of students who are entirely new to programming and also to groups with prior experience of other languages. I would not teach list comprehensions by unrolling them unless it was a more advanced Python programming course. To explain the list comprehension with while clauses I imagine having the following conversation and interactive session: ''' Okay so a list comprehension is a way of making a new list out of an existing list. Let's say we have a list called numbers_list like

...

...

numbers_list = [1,2,3,4,5,4,3,2,1] numbers_list [1, 2, 3, 4, 5, 4, 3, 2, 1]

Now we want to create a new list called squares_list containing the square of each of the numbers in numbers_list. We can do this very easily with a list comprehension and it looks like

...

...

squares_list = [n ** 2 for n in numbers_list] squares_list [1, 4, 9, 16, 25, 16, 9, 4, 1]

The list comprehension loops through all the numbers in numbers_list and, calling the current number n, computes n squared (n ** 2). As it does this is puts all the n squared numbers into a new list in the same order. We can also add an "if clause" to choose which elements from numbers_list we will use to make the new list. To make a list that is the square of all the numbers from numbers_list that are less than 4 we can do

...

...

[n ** 2 for n in numbers_list if n < 4] [1, 4, 9, 9, 4, 1]

Now the comprehension includes n ** 2 in the new list only if n < 4; otherwise n is ignored and the comprehension moves on to the next number from numbers_list. Also if we want the list comprehension to stop looping over the numbers in numbers_list after, for example, seeing a particular number we can use a "while clause" instead of an "if clause". If we want the comprehension to read numbers from numbers_list only while all of the numbers seen are less than 4 then we could do

...

...

[n ** 2 for n in numbers_list while n < 4] [1, 4, 9]

In this case what happens is that as soon as the comprehension finds the number 4 from numbers_list the while condition isn't true any more so it stops reading numbers from numbers_list. This means it doesn't find the other numbers that are also less than 4 at the end of numbers_list (unlike the if clause). '''

8. Allow else break in comp_if clauses.

x = [value for value in iterable if pred(value) else break]

This one is pretty easy to define rigorously, since it maps to exactly what the while attempt maps to with a slight change to the existing rules.

But to me, it makes the code a confusing mess. I'm immediately reading "iterable if pred(value) else break", and that's wrong.

You wouldn't have that confusion with "else return". Oscar

On 27 June 2013 21:28, Andrew Barnert <abarnert@yahoo.com> wrote:

From: Oscar Benjamin <oscar.j.benjamin@gmail.com>

Sent: Thursday, June 27, 2013 3:16 AM

...

On 27 June 2013 08:28, Andrew Barnert <abarnert@yahoo.com> wrote:

...

Let me try to gather together all of the possibilities that have been discussed in this and the two previous threads, plus a couple of obvious ones nobody's mentioned.

You've missed out having "else return" in comprehensions.

For simplicity, I was just dealing with the "break" solutions, not "continue" and "return" (and "raise" or anything else anyone might want). I should have said that, but I didn't; sorry. So, this goes under case 8.

But it's worth noting that return adds additional mental load on top of break. Comprehensions aren't functions, either conceptually or practically, but it will force you to think of them as functions and carry that cognitive dissonance around as you read them. Comprehensions _are_ loops, so break doesn't have that problem.

FWIW, while I actually agree with you that "else return" doesn't fit because people *think* of comprehensions and generator as loops rather than as nested functions, they *are* defined as following the scoping rules of a nested function and CPython actually implements them that way:

...

...

from dis import dis dis("[x for x in (1, 2, 3)]") 1 0 LOAD_CONST 0 (<code object <listcomp> at 0x7f45710efd00, file "<dis>", line 1>) 3 LOAD_CONST 1 ('<listcomp>') 6 MAKE_FUNCTION 0 9 LOAD_CONST 5 ((1, 2, 3)) 12 GET_ITER 13 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 16 RETURN_VALUE

This is one of the things we cleaned up in Python 3 in order to stop leaking the iteration variable for comprehensions into the surrounding scope - we tried a few other options, but ultimately declaring them to be implicit nested functions proved to be the simplest approach. If you're willing, I'm actually thinking this may be one of those discussions that's worth summarising in a PEP, even if it's just to immediately mark it Rejected. Similar to PEP 315 and a few other PEPs, it can help to have a document that clearly spells out the objective (which I think you summarised nicely as "trying to find a syntactic replacement for itertools.takewhile, just as comprehensions replaced many uses of map and filter"), even if no acceptable solution has been found. That phrasing of the objective also highlights a counter argument I hadn't considered before: if we don't consider takewhile a common enough use case to make it a builtin, why on *earth* are we even discussing the possibility of giving it dedicated syntax? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sorry for the split reply; hit the wrong button. On Jun 27, 2013, at 5:19, Nick Coghlan <ncoghlan@gmail.com> wrote:

If you're willing, I'm actually thinking this may be one of those discussions that's worth summarising in a PEP, even if it's just to immediately mark it Rejected. Similar to PEP 315 and a few other PEPs, it can help to have a document that clearly spells out the objective (which I think you summarised nicely as "trying to find a syntactic replacement for itertools.takewhile, just as comprehensions replaced many uses of map and filter"), even if no acceptable solution has been found.

The ideas are pretty broad-ranging. In particular, #1 got two somewhat supportive responses that had nothing to do with the comprehension termination idea. Do side issues like that need to be discussed first/separately before referencing them in a while clause PEP? Also, we seem pretty far from a consensus on what the actual tradeoffs are for most of the options. For example, is the definition of comps in terms of loops the core idea behind the abstraction, or a minor triviality that's only useful in understanding bad code? Are the differences between comps and genexps a bug or a feature? Finally, how does this connect up to the original idea of this thread, which was to draft a PEP for while clauses in loop statements rather than in comprehensions? Obviously if that existed, it would change the options for comp syntax. Do we need to include that idea in the discussion? (I completely forgot about it while digging up all of the ideas that spun off from it...)

That phrasing of the objective also highlights a counter argument I hadn't considered before: if we don't consider takewhile a common enough use case to make it a builtin, why on *earth* are we even discussing the possibility of giving it dedicated syntax?

Besides early termination for comps, the only good use case I know of is in code that already makes heavy use of itertools, and making one of the functions a builtin wouldn't change anything. And if early termination for comps is a special case, it doesn't seem too unreasonable to consider other ways to handle it. But you're right that "make takewhile a builtin" should probably be considered among the alternatives.

On 27 June 2013 12:28, Andrew Barnert <abarnert@yahoo.com> wrote:

From: Oscar Benjamin <oscar.j.benjamin@gmail.com>

Some of the ideas can only break out of the last loop, some can only break out of all of them, some can break out of any one loop. I don't think it will make a difference that often in comprehensions. A comprehension with two for clauses and an if is already pushing the limits of readability; throwing in a break or return as well just seems like asking for trouble.

So what about Nick's suggestion that a syntax could be limited to cases where there is only one for clause?

...

...

def stop(): raise StopIteration

x = [value if pred(value) else stop() for value in iterable]

I prefer

x = [value for value in iterable if pred(value) or stop()]

so that the flow control is all on the right hand side of the "in".

I suppose this solution allows either. Personally, I think using or for non-trivial flow control is more obscure than helpful. Would you write this?

for value in iterable: if pred(value) or stop(): yield value

No, but I find it readable in the comprehension.

...

...

2. Just require comprehensions to handle StopIteration.

The main cost and benefit are the same as #1.

However, it makes the language and implementation more complex, rather than simpler.

Also, the effects of this less radical change (you can no longer pass StopIteration through a comprehension) seem like they might be harder to explain to people than the more radical one.

I think that the current behaviour is harder to explain.

What's hard to explain about the current behavior? StopIteration passes through comprehensions the same way it does through for loops.

It's hard to explain the difference between list(generator expression) and a list comprehension in this respect as it requires thinking about the mechanics of StopIteration. You can see my previous attempt here: http://mail.python.org/pipermail/python-ideas/2013-January/019051.html

...

...

7. Add a "magic" while clause that's basically until with the opposite sense.

x = [value for value in iterable while pred(value)]

This reads pretty nicely (at least in trivial comprehensions), it parallels takewhile and friends, and it matches a bunch of other languages (most of the languages where "when" means "if", "while" means this).

But it has a completely different meaning from while statements, and in fact a barely-related one. … Imagine trying to teach that to a novice.

I can definitely imagine teaching it to a novice. I have taught Python to groups of students who are entirely new to programming and also to groups with prior experience of other languages. I would not teach list comprehensions by unrolling them unless it was a more advanced Python programming course.

This makes perfect sense today. Going from novice to intermediate understanding of list comprehensions, and many other areas of Python, is almost trivial, and that's part of what makes teaching Python so much easier than teaching, say, C++.

I've seen hundreds of people show up on StackOverflow and similar places completely baffled by a complex list comprehension in some code they've run into. As soon as you show them how to unroll it, they immediately get it. If that were no longer true, how would you get people over that step?

I think that this is really a style problem. The emphasis on unrolling comprehensions makes it seem acceptable to write complex comprehensions that can only be understood after being unrolled. To me, a comprehension is good when you can look at it and immediately know what it does. If it needs to be mentally unrolled before it can be understood then it would probably be better if it was actually unrolled in the source and then everyone can look directly at the unrolled code.

...

...

x = [value for value in iterable if pred(value) else break]

This one is pretty easy to define rigorously, since it maps to exactly what

the while attempt maps to with a slight change to the existing rules.

...

But to me, it makes the code a confusing mess. I'm immediately reading

"iterable if pred(value) else break", and that's wrong.

You wouldn't have that confusion with "else return".

Why not? They're both single-keyword flow-control statements. Why would anyone's brain read else return any differently from else break?

My bad, you're right. I was thinking of the break as an expression because of the way you split it out but it's not a valid expression (just like return). Oscar

On 25 June 2013 18:12, Shane Green <shane@umbrellacode.com> wrote:

[x for x in l if x < 10 else break]?

Humorously, this works: def brk(): raise StopIteration (x if x < 10 else brk() for x in range(100)) list(x if x < 10 else brk() for x in range(100)) Quirkily, but obvious with thought, this does not: [x if x < 10 else brk() for x in range(100)]

MISRA-C 1998 [No gotos, no breaks (except in switches), no continues] MISRA-C 2004 [No gotos, no continues, one break per loop (and also the switches)] MISRA-C 2012 [Some gotos are now allowed!]

C is not python, but this translated to the teaching community and here we are.

Carried forward into python: http://www.csee.umbc.edu/courses/201/spring13/standards.shtml

In practice, many many students are being told to avoid break/continue.

Maybe something from the Python leadership saying breaks are fine? Given their lack of support for gotos it's easy to see how others might feel breaks and continues are bad as well, even in Python.

What more support do we need to show than the fact that the keywords break and continue exists in the language and goto does not? Would some code written by Guido suffice? On his blog http://neopythonic.blogspot.comthere's an example of code he has written with a break without even having to search through the archive.

On 2013-06-26 00:05, Andrew Barnert wrote:

From: "jimjhb@aol.com" <jimjhb@aol.com> Sent: Tuesday, June 25, 2013 11:11 AM

...

Maybe something from the Python leadership saying breaks are fine? Given their lack of support for gotos it's easy to see how others might feel breaks and continues are bad as well, even in Python.

$ grep break cpython/Lib/*py |wc -l 551

$ grep continue cpython/Lib/*py |wc -l 280

These numbers should be a bit more accurate by recursing into packages (and tests, which one may not want, but I'd count them) and only counting actual uses of the keywords in code: [cpython/Lib]$ grinpython.py --without-filename -p '\bbreak\b' | wc -l 740 [cpython/Lib]$ grinpython.py --without-filename -p '\bcontinue\b' | wc -l 607 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

I saw your thread. Thank you for summarizing the result! It looks like the best alternative is itertools.takewhile, but I don't like it.... :( -Jim -----Original Message----- From: Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> To: python-ideas <python-ideas@python.org> Sent: Tue, Jun 25, 2013 4:51 pm Subject: Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ: Hi, I suggested the very same 'while in a comprehension/generator expression' back in January: http://mail.python.org/pipermail/python-ideas/2013-January/018969.html There were many very useful responses suggesting alternative syntax (that I'm using now). The proposal itself was dismissed with the logic that comprehensions can currently be translated directly into explicit for loops, e.g.: [x for x in items if x!=0] equals: result=[] for x in items: if x!=0: result.append(x) This equivalence is considered *very* important and the while statement would break it: [x for x in items while x>0] does *not* translate into: for x in item: while x>0: result.append(x) So, as long as you can't come up with syntax that translates properly, there's no chance of getting it accepted. Best, Wolfgang _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

Ned, I'm sorry. I totally agree. What I meant was that takewhile is the best alternative if you don't use break. MISRA-C 2004 allows one break in a loop, so even MISRA is on board. But their 1998 edict polluted a lot of minds.... -Jim -----Original Message----- From: Ned Batchelder <ned@nedbatchelder.com> To: python-ideas <python-ideas@python.org> Sent: Tue, Jun 25, 2013 5:17 pm Subject: Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ: On 6/25/2013 4:58 PM, jimjhb@aol.com wrote: I saw your thread. Thank you for summarizing the result! It looks like the best alternative is itertools.takewhile, but I don't like it.... :( Ugh! The best alternative is "break." The people advocating not using break are doing so for "readability" reasons. I would be very surprised if they thought takewhile was more readable than a break statement. Slavishly following mindless rules is not the way to write good programs, and it isn't the way to teach people to write good programs. Resist! :) --Ned. -Jim -----Original Message----- From: Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> To: python-ideas <python-ideas@python.org> Sent: Tue, Jun 25, 2013 4:51 pm Subject: Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ: Hi, I suggested the very same 'while in a comprehension/generator expression' back in January: http://mail.python.org/pipermail/python-ideas/2013-January/018969.html There were many very useful responses suggesting alternative syntax (that I'm using now). The proposal itself was dismissed with the logic that comprehensions can currently be translated directly into explicit for loops, e.g.: [x for x in items if x!=0] equals: result=[] for x in items: if x!=0: result.append(x) This equivalence is considered *very* important and the while statement would break it: [x for x in items while x>0] does *not* translate into: for x in item: while x>0: result.append(x) So, as long as you can't come up with syntax that translates properly, there's no chance of getting it accepted. Best, Wolfgang _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On 06/25/2013 04:08 PM, jimjhb@aol.com wrote:

That would bring back my previous notion of fwhile.

fwhile X in ListY and conditionZ:

It was pointed out that the 'and' is ambiguous (though perhaps not fatally problematic) and adding a new keyword is really bad as well.

I guess:

for X in ListY and conditionZ: would just have the ambiguity.

[Or maybe not, as long as the condition is a bool, it's not iterable and thus not confused with list.]

What about allowing break and continue in if/else expressions? In a loop it might look like... break if n>=42 else continue In a comprehension it would be... [ (x if x<10 else break) for x in range(20) ] There has been some interest in adding 'break if condition'. But I think this would be better. Cheers, Ron