Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ:
On Jun 27, 2013, at 5:19, Nick Coghlan <ncoghlan@gmail.com> wrote:
If you're willing, I'm actually thinking this may be one of those discussions that's worth summarising in a PEP, even if it's just to immediately mark it Rejected. Similar to PEP 315 and a few other PEPs, it can help to have a document that clearly spells out the objective (which I think you summarised nicely as "trying to find a syntactic replacement for itertools.takewhile, just as comprehensions replaced many uses of map and filter"), even if no acceptable solution has been found.
The ideas are pretty broad-ranging. In particular, #1 got two somewhat supportive responses that had nothing to do with the comprehension termination idea. Do side issues like that need to be discussed first/separately before
referencing them in a while clause PEP?
Also, we seem pretty far from a consensus on what the actual tradeoffs are for most of the options. For example, is the definition of comps in terms of loops the core idea behind the abstraction, or a minor triviality that's only useful in understanding bad code? Are the differences between comps and genexps a bug or a feature?
Finally, how does this connect up to the original idea of this thread, which was to draft a PEP for while clauses in loop statements rather than in comprehensions? Obviously if that existed, it would change the options for comp syntax. Do we need to include that idea in the discussion? (I completely forgot about it while digging up all of the ideas that spun off from it...)
I don't think while clauses in loop statements are a big gain for the language. There's break and despite all the programming-style discussions going on in this part of the thread, it has been working well for many years and most people find it intuitive to use.
That phrasing of the objective also highlights a counter argument I hadn't considered before: if we don't consider takewhile a common enough use case to make it a builtin, why on *earth* are we even discussing the possibility of giving it dedicated syntax?
Besides early termination for comps, the only good use case I know of is in code that already makes heavy use of itertools, and making one of the functions a builtin wouldn't change anything.
And if early termination for comps is a special case, it doesn't seem too unreasonable to consider other ways to handle it.
But you're right that "make takewhile a builtin" should probably be considered among the alternatives.
But a builtin takewhile would still not come with nicer and easier to read syntax! I guess the use cases are not that rare, it's just that right now people switch to explicit loops when they need early termination because it keeps things readable. I'm encountering this situation quite regularly (reading a small header from large files is the most common example, but there are others). Let me suggest one more solution although it requires a new keyword: introduce *breakif* condition and define its translation as if condition: break . You can now write (x for x in iterable breakif x < 0) and I don't see a way how that could possibly be misread by anyone. Also it would translate unambiguously to the explicit: for x in iterable: breakif x<0 # itself translating to if x<0: break yield x It would work with genexps, comprehensions and explicit loops alike (with very little benefit for the later, though maybe it increases readability even there by making it clear from the start of the line what the purpose of the condition test is). Best, Wolfgang
On Jun 28, 2013, at 1:20, "Wolfgang Maier" <wolfgang.maier@biologie.uni-freiburg.de> wrote:
Let me suggest one more solution although it requires a new keyword: introduce
*breakif* condition
and define its translation as if condition: break .
This is basically the same idea as the until statement (with the opposite truth sense, but all of these are easy to invert)--except that it doesn't introduce a block. This means you can't nest anything underneath it, which means it doesn't work in comprehensions without changing what comprehensions mean. So, it combines the disadvantages of the until solution, because it introduces a new statement that no one will ever use just to provide a meaning for a comp clause that people might, and the various break solutions (other than "make break an expression), because it requires complicating the definition of what comprehensions do. That being said, it clearly reads differently from an until clause or an else break clause intuitively. I think that gives me a glimmer of a better way to organize the choices into a few (not completely independent) ideas organized in a tree, with different options (e.g., choice of keyword or syntax) for most of them. I'll try to write it up later, after letting it stew for a while, but the basics are something like: * No syntax change, just make existing things easier to use (e.g., builtin takewhile) * Breaking expressions (e.g., break as expression) * making comprehensions StopIteration-able (e.g., redefine as list(genexp) * Breaking clauses * that fit nesting rule as-is * by also adding a new statement (until) * by explicitly defining what they map to instead of just "the equivalent statement" (magic while) * that require redefining comprehensions (else break) * by also adding a new statement or modifying an existing one (ifbreak) We'd still need to list the specific versions of each category, because they definitely read differently, especially in simple comprehensions (a simple expression, one for, and the new clause)--which, as people have pointed out, are the most important ones (you can teach novices how to use and read simple comps without explaining the nesting rule, and that should still be true).
You can now write (x for x in iterable breakif x < 0) and I don't see a way how that could possibly be misread by anyone. Also it would translate unambiguously to the explicit:
for x in iterable: breakif x<0 # itself translating to if x<0: break yield x
It would work with genexps, comprehensions and explicit loops alike (with very little benefit for the later, though maybe it increases readability even there by making it clear from the start of the line what the purpose of the condition test is).
On 28 June 2013 18:20, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
Let me suggest one more solution although it requires a new keyword: introduce
*breakif* condition
and define its translation as if condition: break . You can now write (x for x in iterable breakif x < 0) and I don't see a way how that could possibly be misread by anyone. Also it would translate unambiguously to the explicit:
for x in iterable: breakif x<0 # itself translating to if x<0: break yield x
It would work with genexps, comprehensions and explicit loops alike (with very little benefit for the later, though maybe it increases readability even there by making it clear from the start of the line what the purpose of the condition test is).
This (or, more accurately, a slight variant that doesn't need a new keyword) actually sounds quite attractive to me. My rationale for that ties into the just rejected PEP 315, which tried to find an improved "loop and a half" syntax for Python, as well as the ongoing confusion regarding the meaning of the "else" clause on while and for loops. Currently, Python's fully general loop syntax looks like this: while True: # Iteration setup if termination_condition: break # Remained of iteration And the recommended idiom for a search loop looks like this: for x in data: if desired_value(x): break else: raise ValueError("Value not found in {:100!r}".format(data)) Rather than adding a new keyword, we could simply expand the syntax for the existing break statement to be this: break [if <EXPR>] This would simplify the above two standard idioms to the following: while True: # Iteration setup break if termination_condition # Remainder of iteration for x in data: break if desired_value(x) else: raise ValueError("Value not found in {:100!r}".format(data)) A "bare" break would then be equivalent to "break if True". The "else" clause on the loop could then be *explicitly* documented as associated with the "break if <X>" form - the else only executes if the break clause is never true. (That also becomes the justification for only allowing this for break, and not for continue or return: those have no corresponding "else" clause) Once the break statement has been redefined this way, it *then* becomes reasonable to allow the following in comprehensions: data = [x for x in iterable break if x is None] As with other proposals, I would suggest limiting this truncating form to disallow combination with the filtering and nested loop forms (at least initially). The dual use of "if" would make the filtering combination quite hard to read, and the nested loop form would be quite ambiguous as to which loop was being broken. If we start with the syntax restricted, we can relax those restrictions later if we find them too limiting, while if we start off being permissive, backwards compatibility would prevent us from adding restrictions later. I'd be very keen to see this written up as a PEP - it's the first proposal that I feel actually *simplifies* the language in any way (mostly by doing something about those perplexing-to-many else clauses on for and while loops). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 29 June 2013 11:09, Nick Coghlan <ncoghlan@gmail.com> wrote: [talking about "break if <condition>]
I'd be very keen to see this written up as a PEP - it's the first proposal that I feel actually *simplifies* the language in any way (mostly by doing something about those perplexing-to-many else clauses on for and while loops).
+1. I like this syntax, and the rationale behind it. Paul
On Jun 29, 2013, at 3:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
data = [x for x in iterable break if x is None]
+1 I like the syntax really like the reasoning and full story. On Jun 29, 2013, at 3:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 June 2013 18:20, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
Let me suggest one more solution although it requires a new keyword: introduce
*breakif* condition
and define its translation as if condition: break . You can now write (x for x in iterable breakif x < 0) and I don't see a way how that could possibly be misread by anyone. Also it would translate unambiguously to the explicit:
for x in iterable: breakif x<0 # itself translating to if x<0: break yield x
It would work with genexps, comprehensions and explicit loops alike (with very little benefit for the later, though maybe it increases readability even there by making it clear from the start of the line what the purpose of the condition test is).
This (or, more accurately, a slight variant that doesn't need a new keyword) actually sounds quite attractive to me. My rationale for that ties into the just rejected PEP 315, which tried to find an improved "loop and a half" syntax for Python, as well as the ongoing confusion regarding the meaning of the "else" clause on while and for loops.
Currently, Python's fully general loop syntax looks like this:
while True: # Iteration setup if termination_condition: break # Remained of iteration
And the recommended idiom for a search loop looks like this:
for x in data: if desired_value(x): break else: raise ValueError("Value not found in {:100!r}".format(data))
Rather than adding a new keyword, we could simply expand the syntax for the existing break statement to be this:
break [if <EXPR>]
This would simplify the above two standard idioms to the following:
while True: # Iteration setup break if termination_condition # Remainder of iteration
for x in data: break if desired_value(x) else: raise ValueError("Value not found in {:100!r}".format(data))
A "bare" break would then be equivalent to "break if True". The "else" clause on the loop could then be *explicitly* documented as associated with the "break if <X>" form - the else only executes if the break clause is never true. (That also becomes the justification for only allowing this for break, and not for continue or return: those have no corresponding "else" clause)
Once the break statement has been redefined this way, it *then* becomes reasonable to allow the following in comprehensions:
data = [x for x in iterable break if x is None]
As with other proposals, I would suggest limiting this truncating form to disallow combination with the filtering and nested loop forms (at least initially). The dual use of "if" would make the filtering combination quite hard to read, and the nested loop form would be quite ambiguous as to which loop was being broken. If we start with the syntax restricted, we can relax those restrictions later if we find them too limiting, while if we start off being permissive, backwards compatibility would prevent us from adding restrictions later.
I'd be very keen to see this written up as a PEP - it's the first proposal that I feel actually *simplifies* the language in any way (mostly by doing something about those perplexing-to-many else clauses on for and while loops).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 29 June 2013 11:09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Rather than adding a new keyword, we could simply expand the syntax for the existing break statement to be this:
break [if <EXPR>] ... Once the break statement has been redefined this way, it *then* becomes reasonable to allow the following in comprehensions:
data = [x for x in iterable break if x is None]
Almost all of your proposal looks reasonable, but I personally find this quite hard to read; it should be written along the lines of (I'm not proposing this): x for x in iterable; break if x is None if one is to continue having syntax that is pseudo-correct English - a trait I am eager to to keep. In summary, this is hard for me to read because there is no separation of the statements. Because I have not other substantial objections, I'm -0 on this. If you can find a way to "fix" that, I'll be, for all intents and purposes, neutral.
I see your point. I’m wondering if there’s a formatting convention or something that might suffice. Tried a couple but have’t come up with anything neat.
data = [x for x in iterable
break if x is None]
data = [x for x in utterable break if x is None]
I don’t necessarily think this makes the options any more difficult to parse than they were previously; advanced comprehensions can get a bit unwieldy already. With syntax highlighting a color coded “break” keyword separates the generation from the termination. On Jun 29, 2013, at 8:46 AM, Joshua Landau <joshua.landau.ws@gmail.com> wrote:
On 29 June 2013 11:09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Rather than adding a new keyword, we could simply expand the syntax for the existing break statement to be this:
break [if <EXPR>] ... Once the break statement has been redefined this way, it *then* becomes reasonable to allow the following in comprehensions:
data = [x for x in iterable break if x is None]
Almost all of your proposal looks reasonable, but I personally find this quite hard to read; it should be written along the lines of (I'm not proposing this):
x for x in iterable; break if x is None
if one is to continue having syntax that is pseudo-correct English - a trait I am eager to to keep.
In summary, this is hard for me to read because there is no separation of the statements.
Because I have not other substantial objections, I'm -0 on this. If you can find a way to "fix" that, I'll be, for all intents and purposes, neutral. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 29/06/13 20:09, Nick Coghlan wrote:
On 28 June 2013 18:20, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
for x in iterable: breakif x<0 # itself translating to if x<0: break yield x [...] This (or, more accurately, a slight variant that doesn't need a new keyword) actually sounds quite attractive to me. My rationale for that ties into the just rejected PEP 315, which tried to find an improved "loop and a half" syntax for Python, as well as the ongoing confusion regarding the meaning of the "else" clause on while and for loops. [...] Rather than adding a new keyword, we could simply expand the syntax for the existing break statement to be this:
break [if <EXPR>]
This would simplify the above two standard idioms to the following:
while True: # Iteration setup break if termination_condition # Remainder of iteration
It doesn't really *simplify* the standard idiom though. It just saves a line and, trivially, one indent: while True: if condition: break ... And it doesn't even save that if you write it like this: while True: if condition: break ... In fact, I would argue the opposite, it's not *simpler* it is *more complex* because it is a special case for the if keyword: break if condition # allowed continue if condition # maybe allowed? return 'spam' if condition # probably disallowed pass if condition # what's the point? raise Exception if condition # probably disallowed x += 1 if condition # almost certainly disallowed to say nothing of: break if condition else continue Given this suggestion, now people have to learn a third case for "if": - "if condition" starts a new block and must be followed by a colon; - unless it follows an expression, in which case it is the ternary operator and must be followed by "else expression"; - unless it follows a break (or continue? return? raise? ...) in which case it must not be followed by "else".
for x in data: break if desired_value(x) else: raise ValueError("Value not found in {:100!r}".format(data))
A "bare" break would then be equivalent to "break if True". The "else" clause on the loop could then be *explicitly* documented as associated with the "break if <X>" form - the else only executes if the break clause is never true. (That also becomes the justification for only allowing this for break, and not for continue or return: those have no corresponding "else" clause)
It is certainly not correct that the else of while...else and for...else is necessarily associated with the (implied) "if" of "break if condition". It's currently legal to write a for...else loop with no break. Pointless, but legal, so it must remain legal, and you still have an else without an if. Likewise we can do things like this: for x in data: break else: print("data must be empty") for x in data: if condition: return x else: print("condition was never true") I think that it is simply the case that for...else and while...else were misnamed. Great concept, but the keyword doesn't make sense. It isn't an *else* clause in the ordinary English sense of the word, but a *then* clause, and should have been spelled "for...then". But we're stuck with it until Python 4000. Trying to retcon the "else" as being associated with an implied "if" is simply misguided.
Once the break statement has been redefined this way, it *then* becomes reasonable to allow the following in comprehensions:
data = [x for x in iterable break if x is None]
Which reads wrongly for something which is intended to be a single expression. Current comprehensions read like an English expression: [this for x in iterable] "Do this for each x in iterable." [this for x in iterable if condition] "Do this for each x in iterable if condition is true." are both reasonable English-like expressions. Even the proposed (and rejected) form: [this for x in iterable while condition] "Do this for each x in iterable while condition remains true." is an English-like expression. But your proposal isn't: [this for x in iterable break if x is None] "Do this for each x in iterable break if condition is true." The "break if..." clause reads like a second statement. Now I realise that Python is not English and doesn't match English grammar, but still, this proposal doesn't flow off the tongue, it trips and stumbles because it feels like two statements merely shoe-horned into one. Python is, for the most part, an astonishingly beautiful language to read (at least for English speakers) and I'm afraid that "break if condition" in a comprehension is not beautiful. -- Steven
Reminds me of a cartoon in the new yorker: a guy is looking over the shoulder of another who is dressed futuristically, incredulously asking the futuristic guy, “you came back in time just so you could hit ‘reply’ instead of ‘reply all’?” Made me realize my habit of accidentally doing the opposite is far better than being the habit of hitting reply-all :-) On Jun 29, 2013, at 8:56 AM, Steven D'Aprano <steve@pearwood.info> wrote:
It doesn't really *simplify* the standard idiom though. It just saves a line and, trivially, one indent:
while True: if condition: break ...
And it doesn't even save that if you write it like this:
while True: if condition: break ...
This argument has come up before and I don’t really understand or agree with it. It seems to: - gloss over the fact this allows you to do these things in list comprehensions; - argue against the existence of list comprehensions at all, rather than this extension; - and leaves out everything else a list comprehension does. Without this feature list compreshensions aren’t used to describe while loops, period; so you shouldn’t subtract the standard list comprehension contributions from the benefits realized by this change. [line.strip() for line in lines break if not line] compared to: messages = [] while True: line = lines.pop(0) if not line: break messages.push(lines[0].strip()) Of course it would be easier using a for loop, which you’d then be tempted to replace with a compression...
On 30 June 2013 01:56, Steven D'Aprano <steve@pearwood.info> wrote:
In fact, I would argue the opposite, it's not *simpler* it is *more complex* because it is a special case for the if keyword:
break if condition # allowed continue if condition # maybe allowed? return 'spam' if condition # probably disallowed pass if condition # what's the point? raise Exception if condition # probably disallowed x += 1 if condition # almost certainly disallowed
People already have to learn "for/else" and "while/else". Adding "break if" can *only* be justified on the grounds of pairing it up with those two existing else clauses to make appropriate if/else pairs, as "for/break if/else" and "while/break if/else" should actually be easier to learn than the status quo. However, note that I said "I would love to see this idea written up as a PEP", not "I would love to see this idea implemented as part of the language": I'm undecided on that point as yet. It at least has the *potential* to make an existing aspect of the language easier to learn (unlike every other suggestion in any of these threads). Including a plan to deprecate allowing loop else clauses without a corresponding "break" in such a PEP would be a good idea, though. Expanding it to comprehensions is indeed trickier, because the "break" doesn't quite scan correctly. While Joshua said he wasn't proposing it, permitting a semi-colon in the comprehension to separate out a termination clause actually reads pretty well to me (we can avoid ambiguity in the grammar because we don't currently allow ';' anywhere between any kind of bracket): [x for x in iterable; break if x is None] [x for x in data if x; break if x is None] One nice advantage of that notation is that: 1. The statement after the ";" is exactly the statement that would appear in the expanded loop 2. It can be combined unambiguously with a filtering clause 3. It clearly disallows its use with nested loops in the comprehension Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
From: Nick Coghlan <ncoghlan@gmail.com> Sent: Saturday, June 29, 2013 4:51 PM
On 30 June 2013 01:56, Steven D'Aprano <steve@pearwood.info> wrote:
In fact, I would argue the opposite, it's not *simpler* it is *more complex* because it is a special case for the if keyword:
break if condition # allowed continue if condition # maybe allowed? return 'spam' if condition # probably disallowed pass if condition # what's the point? raise Exception if condition # probably disallowed x += 1 if condition # almost certainly disallowed
People already have to learn "for/else" and "while/else". Adding "break if" can *only* be justified on the grounds of pairing it up with those two existing else clauses to make appropriate if/else pairs, as "for/break if/else" and "while/break if/else" should actually be easier to learn than the status quo.
I honestly don't think it would read this way to most people; the else clause would still be confusing. Especially if it continues to means the same thing even without break if, which means as soon as you "learn" the for/break if/else rule you'll learn that it's not really true. Also, if it comes at the cost of making comprehensions harder to learn and understand, I think people use (and see) comprehensions much more often than for/else. Again, this is equivalent to an "until" clause, but worse in three ways: it doesn't scan well, it looks ambiguous (even if it isn't to the parser), and it doesn't have a nested block form to translate to. Compare: [x for x in iterable until x is None] a = [] for x in iterable: until x is None: a.append(x) [x for x in iterable break if x is None] a = [] for x in iterable: break if x is None a.append(x)
While Joshua said he wasn't proposing it, permitting a semi-colon in the comprehension to separate out a termination clause actually reads pretty well to me (we can avoid ambiguity in the grammar because we don't currently allow ';' anywhere between any kind of bracket):
[x for x in iterable; break if x is None] [x for x in data if x; break if x is None]
This seems to be much better. But only because it's fixed to the end of the comprehension; I don't see how it could be extended to work with nested comprehensions. And that brings up a larger point. While there are many languages with break-if/while/until-type clauses in comprehensions, I can't think of any of with such clauses in Python/Haskell-style arbitrary nested comprehensions. In most other languages, you can't nest arbitrary clauses, just loops. Then you can attach a single optional clause of each type to the end of either the whole thing (like Racket) or each loop (like Clojure). In pseudo-Python: [x for row in data, x in row if x until x is None] [x for (row in data), (x in row if x until x is None)] The first of these is clearly weaker in that you can only attach condition clauses to the innermost loop, but usually that's not a problem—most cases where you'd need breaks in two places are probably too complicated to write as comprehensions anyway. And, when it is a problem, you can always explicitly nest and flatten: flatten((x for x in row if x) for row in data until row is None) And it has the advantage over the second one that adding syntax to separate a comprehension-wide clause is much easier than adding syntax to separate per-loop clauses. (I can't think of a way to write the latter that doesn't require extra parentheses.) So, your suggestion turns Python list comprehensions into a hybrid of two styles: Haskell-style ifs mixed with fors, Racket-style per-comprehension break-ifs (and anything else we add in the future). While that sounds bad at first, I'm not sure it really is. I think it would be very easy to explain pedagogically, and it's trivial in grammar terms: comprehension ::= expression comp_for [";" comp_cond] comp_cond ::= comp_breakif comp_breakif ::= "break" "if" expression_nocond All that being said, I still think "until" is better than "break if" even with this syntax.
One nice advantage of that notation is that:
1. The statement after the ";" is exactly the statement that would appear in the expanded loop
But it still breaks the rule that the expression is nested inside the last statement, which means it still makes explaining and documenting comprehensions more difficult. Not _as_ difficult as with a non-nested-block clause that can appear anywhere, but still more difficult than without any such clauses.
2. It can be combined unambiguously with a filtering clause 3. It clearly disallows its use with nested loops in the comprehension
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 30 June 2013 16:43, Andrew Barnert <abarnert@yahoo.com> wrote:
From: Nick Coghlan <ncoghlan@gmail.com>
Sent: Saturday, June 29, 2013 4:51 PM
On 30 June 2013 01:56, Steven D'Aprano <steve@pearwood.info> wrote:
In fact, I would argue the opposite, it's not *simpler* it is *more complex* because it is a special case for the if keyword:
break if condition # allowed continue if condition # maybe allowed? return 'spam' if condition # probably disallowed pass if condition # what's the point? raise Exception if condition # probably disallowed x += 1 if condition # almost certainly disallowed
People already have to learn "for/else" and "while/else". Adding "break if" can *only* be justified on the grounds of pairing it up with those two existing else clauses to make appropriate if/else pairs, as "for/break if/else" and "while/break if/else" should actually be easier to learn than the status quo.
I honestly don't think it would read this way to most people; the else clause would still be confusing. Especially if it continues to means the same thing even without break if, which means as soon as you "learn" the for/break if/else rule you'll learn that it's not really true.
Hence why I said any such PEP should also propose that a dangling "else" on a loop without a break statement should be deprecated and eventually become a syntax error. Without a break, a loop else clause is just pointless indentation of code that could be written in line after the loop instead. Loop else clauses *only* make sense in combination with break, so we should just make that official and enforce it in the compiler. While, we should probably do that regardless, there's no incentive to work on it without some other kind of payoff, like the ability to terminate comprehensions early :)
Also, if it comes at the cost of making comprehensions harder to learn and understand, I think people use (and see) comprehensions much more often than for/else.
While it does have a strong statement/expression dichotomy, Python is still one language, not two. Any proposals that rely on adding new expression-only keywords are dead in the water. PEP 315 has now been explicitly rejected: the official syntax for terminating a loop early is the existing break statement, thus any proposal for terminating a comprehension early must also be based on break if it is to be considered a serious suggestion rather than people just idling passing the time on an internet mailing list (I actually don't mind that last happening a bit here - I think it's an important part of python-ideas serving the purpose it is designed to serve. It's just important to learn the difference between those discussions and the proposals which actually have some hope of surviving the vigorous critique PEPs face on python-dev). Any proposal to allow termination of comprehensions slots into the same design space as PEPs like 403 (the @in pseudo-decorator) and 422 (the __init_class__ hook) - they don't add fundamentally new capabilities to the language the way context managers or generator delegation did, they just propose tidying up a couple of rough edges for things that are already possible, but require structuring code in a slightly awkward way. PEP 409 is an example of an accepted PEP that fits into the same category - it makes it easier to generate clean exception tracebacks when you're deliberately suppressing an inner exception and replacing it with a different one. PEP 3129 (which added class decorators), is another good example of a "clean up" that took an existing concept and adjusted it slightly, rather than adding a fundamental new capability. A proposal to allow early termination of comprehensions has *zero* chance of acceptance as a "major language change" PEP. It simply doesn't have enough to offer in terms of additional expressiveness. PEP 403 (the @in pseudo-decorator) holds the promise of addressing at least some of the many requests over the years for multi-line lambda support, and even *that* is on dubious ground in terms of the additional complexity vs additional expressiveness trade-off. There's nothing wrong with cleanup PEPs, though - they're an important part of refactoring the language design to be a bit more self-consistent. That's why I latched on to the idea of doing something to clean up the known wart that is loop else clauses, and then *expanding* that to offer early termination of comprehensions. It may still get shot down (if Guido doesn't like it, it *will* get shot down), but the "[x for x in y; break if x is None]" variant definitely has a few points in its favour: - the ";" helps it read like English and avoids ambiguity relative to filtering clauses - the "cannot break nested comprehensions" restriction helps limit ambiguity - the statement form adds a new "if" to go with the confusing "else" on loops - it can be paired with deprecation of tolerating else-without-break on loops I think the idea of early termination of comprehensions has a *much* better chance of getting Guido's interest if it helps make the behaviour of else clauses on loops more comprehensible without needing elaborate explanations like http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_el... That still needs a volunteer to make the sales pitch in a PEP and work out how to implement it, though :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
From: Nick Coghlan <ncoghlan@gmail.com> Sent: Sunday, June 30, 2013 12:51 AM
On 30 June 2013 16:43, Andrew Barnert <abarnert@yahoo.com> wrote:
From: Nick Coghlan <ncoghlan@gmail.com>
People already have to learn "for/else" and "while/else". Adding "break if" can *only* be justified on the grounds of pairing it up with those two existing else clauses to make appropriate if/else pairs, as "for/break if/else" and "while/break if/else" should actually be easier to learn than the status quo.
I honestly don't think it would read this way to most people; the else clause would still be confusing. Especially if it continues to means the same thing even without break if, which means as soon as you "learn" the for/break if/else rule you'll learn that it's not really true.
Hence why I said any such PEP should also propose that a dangling "else" on a loop without a break statement should be deprecated and eventually become a syntax error.
Sure, but we already have a way to put a break into a for loop: the break statement. Unless you're also planning to deprecate that, it will still be true that the "for/break if/else" rule is often not true. For example: for i in x: if i > 0: print(i) else: break else: print('Finished') Also, even in your preferred case, this would be the only place in Python where an statement matches another statement at a different indentation level. Is that really going to make Python easier to understand? And finally, as others have pointed out, besides being yet another syntax for if (if statements, ternary expressions, if clauses, and now break if statements and clauses), what it really reads like is perl's postfix if, which is misleading, because you can't do postfix conditionals anywhere else. Killing two birds with one stone is nice, but I don't think this kills either bird.
Also, if it comes at the cost of making comprehensions harder to learn and
understand, I think people use (and see) comprehensions much more often than for/else.
While it does have a strong statement/expression dichotomy, Python is still one language, not two. Any proposals that rely on adding new expression-only keywords are dead in the water.
I think I didn't make my point very clearly. I'm _assuming_ that Python is one language. As a consequence, if some new loop syntax doesn't work well in expressions, it's not worth doing, even if it _does_ work well in statements. Also, the syntax has to follow the regular mapping between clauses and statements—each clause is a nested block statement—or it's not worth doing. I realize that you have a solution for the second point: the semicolon-separated "comp_break_if" clause at the end of a comprehension is clearly different, and therefore it's acceptable that it maps differently. But that means having two mapping rules instead of one, which still makes comprehensions twice as complicated.
PEP 315 has now been explicitly rejected: the official syntax for terminating a loop early is the existing break statement, thus any proposal for terminating a comprehension early must also be based on break if it is to be considered a serious suggestion
And, as I said in my earlier summary, I think that means there is no feasible suggestion. The only option I like is #1 (redefining comps on top of genexprs), and I like it for completely independent reasons. (In fact, even with #1, I still probably wouldn't use stop() in listcomps.) Just as in the language we borrowed listcomps from, takewhile is the right answer. You can stick it around the inner iterator, or one of the sub-iterators in a nested comprehension; there are no possible syntax ambiguities; there's nothing special to learn; etc. The only real problem with takewhile is the same problem as map, filter, reduce, and everything in itertools: Python's lambda syntax sometimes makes it a bit clumsy to turn an expression into a function. And the solution is the same as it is everywhere else in Python: define the function out of line, move the takewhile call itself out of line, find somewhere else to break the expression up somewhere else, use a partial instead of a lambda, or just give up on trying to write an expression and write a statement. (Or push for PEP 403.) It was worth exploring whether there are any obvious options—after all, we did reduce the need for map and filter calls, maybe we could do the same for takewhile—but that doesn't mean we have to pick the least bad if all of them are bad, it just means the answer is no. And I think at this point, your earlier suggestion of drafting a PEP to get all of the bad ideas explicitly rejected is a better use of time than trying to draft a PEP to push for one of them.
On 30 June 2013 21:41, Andrew Barnert <abarnert@yahoo.com> wrote:
It was worth exploring whether there are any obvious options—after all, we did reduce the need for map and filter calls, maybe we could do the same for takewhile—but that doesn't mean we have to pick the least bad if all of them are bad, it just means the answer is no.
And I think at this point, your earlier suggestion of drafting a PEP to get all of the bad ideas explicitly rejected is a better use of time than trying to draft a PEP to push for one of them.
Well said, and well argued. I concede the point :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
PEP 315 has now been explicitly rejected: the official syntax for terminating a loop early is the existing break statement, thus any proposal for terminating a (Nick means terminating a 'for' loop; there is a place to shove a conditional in the 'while' loop syntax.) I think the (informational) PEP should reflect that this is contrary to early notions of structured programming (MISRA-1998) but is in accord with updated notions (MISRA-2004, MISRA-2012). I understand and appreciate
Nick wrote: that many python programmers think this may be silly, but structured programming had a big impact on the computer science committee and some acknowledgement of this is, in my view, warranted. Plus, by citing the later MISRA-2004, it leaves the Python implementation in the free and clear.
I think the idea of early termination of comprehensions has a *much* better chance of getting Guido's interest if it helps make the behaviour of else clauses on loops more comprehensible without needing
It looks like this has happened.
On 30 June 2013 00:51, Nick Coghlan <ncoghlan@gmail.com> wrote:
[x for x in iterable; break if x is None] [x for x in data if x; break if x is None]
One nice advantage of that notation is that:
1. The statement after the ";" is exactly the statement that would appear in the expanded loop 2. It can be combined unambiguously with a filtering clause 3. It clearly disallows its use with nested loops in the comprehension
It has the significant disadvantage that Steven pointed out which is that it doesn't read very well. The most important aspect of a comprehension is its comprehensibility. Consider getting the prime numbers less than 100: primes100 = {p for p in primes(); break if p >= 100} You need to invert the if condition to understand which primes are in the resulting set. With for/while it reads properly and the condition at the right expresses a true property of the elements in the resulting set: primes100 = {p for p in primes() while p < 100} At the moment the obvious way to get the prime numbers less than 100 would be to do something like: from math import ceil, sqrt def isprime(N): return N > 1 and all(N % n for n in range(2, ceil(sqrt(N)))) primes100 = [p for p in range(1, 100) if isprime(p)] However this is a suboptimal algorithm. At the point when we want to determine if the number N is prime we have already found all the primes less than N. We only need to check modulo division against those but this construction doesn't give us an easy way to do that. It's better to have a primes() generator that can keep track of this information: from itertools import count def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen): yield n primes_seen.append(n) This algorithm is actually even poorer as it doesn't stop at sqrt(n). We can fix that with takewhile: from itertools import count, takewhile def primes(): primes_seen = [] for n in count(2): if all(n % p for p in takewhile(lambda p: p**2 < n, primes_seen)): yield n primes_seen.append(n) primes100 = {p for p in takewhile(lambda p: p < 100, primes()} Using for/while this becomes significantly clearer (in my opinion): from itertools import count def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen while p**2 <= n): yield n primes_seen.append(n) primes100 = {p for p in primes() while p < 100} The main objection to for/while seems to be that it doesn't unroll in the same way as current comprehensions. I think that for/while is just as useful for an ordinary for loop as it is for a comprehension. In C you can easily add anything to the termination condition for a loop e.g.: for (i=0; i<N && i && p[i-1]<100; ++i) p[i] = next_prime(); But in Python having multiple termination conditions (or having any termination with an inifinite iterator) means using a break or takewhile/lambda. for n, p in enumerate(primes()): if p > 100: break print('%sth prime is %s' % (n, p)) or perhaps for n, p in enumerate(takewhile(lambda p: p < 100, primes())): print('%sth prime is %s' % (n, p)) or even worse for n, p in enumerate(takewhile((100).__gt__, primes())): print('%sth prime is %s' % (n, p)) I think that it would be better if this could be spelled as for n, p in enumerate(primes()) while p < 100: print('%sth prime is %s' % (n, p)) If that were the case then a for/while comprehension could unroll into a for/while loop just as with current comprehensions: result = [x for y in stuff while z] becomes: result = [] for y in stuff while z: result.append(x) Oscar
Having a bit of a hard time following the status of this as I dropped out for a while, but I’m not really for the semi-colon separator approach. Of all the options already available in list comprehensions, this one actually seems to be one of the most easily because it starts with a keyword and ends at the end. Not that syntax highlighting should be taken into account in general, it’s worth noting the syntax highlighted version really makes this distinction quite clear: One thing I could see doing would be to allow semi-colons anywhere in a list comprehension that’s a boundary between statements of the expanded form. Then they behave just like the optional semi-colons you could put at the end of a line. Sorry if that’s precisely what’s been promoted. . On Jul 1, 2013, at 4:57 AM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 30 June 2013 00:51, Nick Coghlan <ncoghlan@gmail.com> wrote:
[x for x in iterable; break if x is None] [x for x in data if x; break if x is None]
One nice advantage of that notation is that:
1. The statement after the ";" is exactly the statement that would appear in the expanded loop 2. It can be combined unambiguously with a filtering clause 3. It clearly disallows its use with nested loops in the comprehension
It has the significant disadvantage that Steven pointed out which is that it doesn't read very well. The most important aspect of a comprehension is its comprehensibility. Consider getting the prime numbers less than 100:
primes100 = {p for p in primes(); break if p >= 100}
You need to invert the if condition to understand which primes are in the resulting set. With for/while it reads properly and the condition at the right expresses a true property of the elements in the resulting set:
primes100 = {p for p in primes() while p < 100}
At the moment the obvious way to get the prime numbers less than 100 would be to do something like:
from math import ceil, sqrt
def isprime(N): return N > 1 and all(N % n for n in range(2, ceil(sqrt(N))))
primes100 = [p for p in range(1, 100) if isprime(p)]
However this is a suboptimal algorithm. At the point when we want to determine if the number N is prime we have already found all the primes less than N. We only need to check modulo division against those but this construction doesn't give us an easy way to do that.
It's better to have a primes() generator that can keep track of this information:
from itertools import count
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen): yield n primes_seen.append(n)
This algorithm is actually even poorer as it doesn't stop at sqrt(n). We can fix that with takewhile:
from itertools import count, takewhile
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in takewhile(lambda p: p**2 < n, primes_seen)): yield n primes_seen.append(n)
primes100 = {p for p in takewhile(lambda p: p < 100, primes()}
Using for/while this becomes significantly clearer (in my opinion):
from itertools import count
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen while p**2 <= n): yield n primes_seen.append(n)
primes100 = {p for p in primes() while p < 100}
The main objection to for/while seems to be that it doesn't unroll in the same way as current comprehensions. I think that for/while is just as useful for an ordinary for loop as it is for a comprehension. In C you can easily add anything to the termination condition for a loop e.g.:
for (i=0; i<N && i && p[i-1]<100; ++i) p[i] = next_prime();
But in Python having multiple termination conditions (or having any termination with an inifinite iterator) means using a break or takewhile/lambda.
for n, p in enumerate(primes()): if p > 100: break print('%sth prime is %s' % (n, p))
or perhaps
for n, p in enumerate(takewhile(lambda p: p < 100, primes())): print('%sth prime is %s' % (n, p))
or even worse
for n, p in enumerate(takewhile((100).__gt__, primes())): print('%sth prime is %s' % (n, p))
I think that it would be better if this could be spelled as
for n, p in enumerate(primes()) while p < 100: print('%sth prime is %s' % (n, p))
If that were the case then a for/while comprehension could unroll into a for/while loop just as with current comprehensions:
result = [x for y in stuff while z]
becomes:
result = [] for y in stuff while z: result.append(x)
Oscar _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 1 July 2013 16:21, Shane Green <shane@umbrellacode.com> wrote:
Having a bit of a hard time following the status of this as I dropped out
One thing I could see doing would be to allow semi-colons anywhere in a
for a while, but I’m not really for the semi-colon separator approach. Of all the options already available in list comprehensions, this one actually seems to be one of the most easily because it starts with a keyword and ends at the end. Not that syntax highlighting should be taken into account in general, it’s worth noting the syntax highlighted version really makes this distinction quite clear: <IMG> That's somewhat convincing. I'm still not convinced we need new syntax though. \ list comprehension that’s a boundary between statements of the expanded form.
Then they behave just like the optional semi-colons you could put at the
end of a line.
Sorry if that’s precisely what’s been promoted. .
So (I'm just adding examples to your idea): [item for sublist in data for item in sublist; if item] can become [item; for sublist in data; for item in sublist; if item] In a pathological case this is a transform from: [item for sublist in data if sublist for item in sublist if item%3 if item-1] to [item for sublist in data; if sublist; for item in sublist; if item%3; if item-1] But you can always do: [item for sublist in data if sublist for item in sublist if item%3 if item-1] That said, it's not like people *do* those sorts of pathological expressions, is it?
Thank you, you are exactly right about what I meant, and yes, there is the pathological extreme–like using a semicolon to terminate every line in a Python program. The approach exhibits a Pythonnistic consistency, IMHO. The break terminator isn’t a special case, semi colons are optional just like they are for terminating statements, and the semicolons correspond functionally to where they would be had you used them to flatten the expansion onto a single line (without conversion into a comprehension). Lastly, while I think the terminating break will typically be one of the easier components of a comprehension to parse, I have come across some list comprehensions that were challenging to parse visually. If my preferred approach to make them more parsable were semicolons, then applying them at several spots in the might be helpful to achieving that goal. My personal preference is likely to be to break apart the comprehension across lines or, if still shorter than 80 cols, use an extra space or two between sub-statements. On Jul 2, 2013, at 8:05 PM, Joshua Landau <joshua.landau.ws@gmail.com> wrote:
On 1 July 2013 16:21, Shane Green <shane@umbrellacode.com> wrote:
Having a bit of a hard time following the status of this as I dropped out for a while, but I’m not really for the semi-colon separator approach. Of all the options already available in list comprehensions, this one actually seems to be one of the most easily because it starts with a keyword and ends at the end. Not that syntax highlighting should be taken into account in general, it’s worth noting the syntax highlighted version really makes this distinction quite clear:
<IMG>
That's somewhat convincing. I'm still not convinced we need new syntax though. \
One thing I could see doing would be to allow semi-colons anywhere in a list comprehension that’s a boundary between statements of the expanded form.
Then they behave just like the optional semi-colons you could put at the end of a line.
Sorry if that’s precisely what’s been promoted. .
So (I'm just adding examples to your idea):
[item for sublist in data for item in sublist; if item]
can become
[item; for sublist in data; for item in sublist; if item]
In a pathological case this is a transform from:
[item for sublist in data if sublist for item in sublist if item%3 if item-1]
to
[item for sublist in data; if sublist; for item in sublist; if item%3; if item-1]
But you can always do:
[item for sublist in data if sublist for item in sublist if item%3 if item-1]
That said, it's not like people *do* those sorts of pathological expressions, is it?
On Jul 1, 2013, at 4:57, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
This algorithm is actually even poorer as it doesn't stop at sqrt(n). We can fix that with takewhile:
from itertools import count, takewhile
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in takewhile(lambda p: p**2 < n, primes_seen)): yield n primes_seen.append(n)
primes100 = {p for p in takewhile(lambda p: p < 100, primes()}
Using for/while this becomes significantly clearer (in my opinion):
from itertools import count
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen while p**2 <= n): yield n primes_seen.append(n)
primes100 = {p for p in primes() while p < 100}
There are already ways to improve the readability of that line: candidates = takewhile(lambda p: p**2 < n, primes_seen) if all(n % p for p in candidates): Or, better: def candidate(p): return p**2 < n if all(n % p for p in takewhile(candidate, primes_seen)): Yes, this means splitting the line in two just so you can avoid lambda, and it's exactly parallel to the case for if clauses vs. filter. I think the benefit to all of these solutions is pretty firmly established by this point. But, as Nick pointed out earlier, there's probably a reason that filter is a builtin and takewhile is not. It's not _as much_ benefit. And meanwhile, the cost is higher because there's no familiar, pre-existing syntax to borrow. I'll grant that it's entirely possible that the problem is just that I'm much more familiar with while statements than with for...while statements for historical reason, and after a bit of exposure the problem will go away (like ternary statements, which everyone gets pretty quickly, as opposed to for...else, which many stay confused by). But still, we can't expect that python programmers will ever be as familiar with for...while as they are with if (which works the same as in almost every other language, is one of the first constructs every novice is taught, etc.).
On 1 July 2013 12:57, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 30 June 2013 00:51, Nick Coghlan <ncoghlan@gmail.com> wrote:
[x for x in iterable; break if x is None] [x for x in data if x; break if x is None]
One nice advantage of that notation is that:
1. The statement after the ";" is exactly the statement that would appear in the expanded loop 2. It can be combined unambiguously with a filtering clause 3. It clearly disallows its use with nested loops in the comprehension
It has the significant disadvantage that Steven pointed out which is that it doesn't read very well. The most important aspect of a comprehension is its comprehensibility. Consider getting the prime numbers less than 100:
primes100 = {p for p in primes(); break if p >= 100}
You need to invert the if condition to understand which primes are in the resulting set. With for/while it reads properly and the condition at the right expresses a true property of the elements in the resulting set:
primes100 = {p for p in primes() while p < 100}
If you're telling me that "{p for p in primes() while p < 100}" reads better than "{p for p in primes(); break if p >= 100}" I have to disagree strongly. The "break if" form looks beautiful. I know that this involves my not-suggestion, so I might be biased, but that's what I think.
At the moment the obvious way to get the prime numbers less than 100 would be to do something like:
<STUFF> So you have these: {p for p in takewhile(lambda p: p < 100, primes())} set(takewhile(lambda p: p < 100, primes()) {p for p in primes() while p < 100} {p for p in primes(); break if p >= 100} I like them most bottom-to-top, but I don't think new syntax is a cost worth having. If it is, I'm only accepting of the "break if" form, which adds the least grammar. But not that either, because srsly; a semicolon [youtu.be/M94ii6MVilw, contains profanity] in a comprehension?
On Mon, Jul 1, 2013 at 10:25 AM, Joshua Landau <joshua.landau.ws@gmail.com>wrote:
primes100 = {p for p in primes(); break if p >= 100} primes100 = {p for p in primes() while p < 100}
If you're telling me that "{p for p in primes() while p < 100}" reads better than "{p for p in primes(); break if p >= 100}" I have to disagree strongly. The "break if" form looks beautiful.
My own taste is in strong contrast to Joshua's. I think the semi-colon/break-if looks absolutely horrendous and ugly. The 'while' clause looks obvious, beautiful, and intuitive. However, I see the point made by a number of people that the 'while' clause has no straightforward translation into an unrolled loop, and is probably ruled out on that basis. That said, I think the version using Guido's suggestion (and already supported for generator comprehensions) looks fairly nice. I.e. given a support function: def stopif(x): if x: raise StopIteration return True We can express it as: primes100 = set(p for p in primes() if stopif(p >= 100)) Except I'm not sure I like the spelling of 'stopif()', since the repeating the 'if' in the function name reads oddly. I guess I might like this spelling better: primes100 = set(p for p in primes() if still(p < 100)) # Obvious implementation of still() The only change we need is the one Guido has declared as "do it" which is to make other comprehensions act the same as the instantiation with the generator. I.e. this should work: primes100 = {p for p in primes() if still(p < 100)} It doesn't *really* matter whether someone likes my spelling 'still()' or Guido's 'stopif()' better, since either one is trivially implementable by end users if they wish to do so. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 1 July 2013 21:29, David Mertz <mertz@gnosis.cx> wrote:
However, I see the point made by a number of people that the 'while' clause has no straightforward translation into an unrolled loop, and is probably ruled out on that basis.
My thought (in keeping with the title of the thread) is that the comprehension data = [x for y in stuff while z] would unroll as the loop for y in stuff while z: data.append(x) which would also be valid syntax and have the obvious meaning. This is similar to Nick's suggestion that 'break if' be usable in the body of the loop so that data = [x for y in stuff; break if not z] would unroll as for y in stuff: break if not z data.append(y) Having a while clause on for loops is not just good because it saves a couple of lines but because it clearly separates the flow control from the body of the loop (another reason I dislike 'break if'). In other words I find the flow of the loop for p in primes() while p < 100: print(p) easier to understand (immediately) than for p in primes(): if p >= 100: break print(p) These are just trivially small examples. As the body of the loop grows in complexity the readability benefit of moving 'if not z: break' into the top line becomes more significant. You can get the same separation of concerns using takewhile at the expense of a different kind of readability for p in takewhile(lambda p: p < 100, primes()): print(p) However there is another problem with using takewhile in for loops which is that it discards an item from the iterable. Imagine parsing a file such as: csvfile = '''# data.csv # This file begins with an unspecified number of header lines. # Each header line begins with '#'. # I want to keep these lines but need to parse the separately. # The first non-comment line contains the column headers x y z 1 2 3 4 5 6 7 8 9'''.splitlines() You can do csvfile = iter(csvfile) headers = [] for line in csvfile: if not line.startswith('#'): break headers.append(line[1:].strip()) fieldnames = line.split() for line in csvfile: yield {name: int(val) for name, val in zip(fieldnames, line.split())} However if you use takewhile like for line in takewhile(lambda line: line.startswith('#'), csvfile): headers.append(line[1:].split()) then after the loop 'line' holds the last comment line. The discarded column header line is gone and cannot be recovered; takewhile is normally only used when the entire remainder of the iterator is to be discarded. I would propose that for line in csvfile while line.startwith('#'): headers.append(line) would result in 'line' referencing the item that failed the while predicate. Oscar
On 01/07/2013 23:44, Oscar Benjamin wrote:
On 1 July 2013 21:29, David Mertz <mertz@gnosis.cx> wrote:
However, I see the point made by a number of people that the 'while' clause has no straightforward translation into an unrolled loop, and is probably ruled out on that basis.
My thought (in keeping with the title of the thread) is that the comprehension
data = [x for y in stuff while z]
would unroll as the loop
for y in stuff while z: data.append(x)
which would also be valid syntax and have the obvious meaning. This is similar to Nick's suggestion that 'break if' be usable in the body of the loop so that
data = [x for y in stuff; break if not z]
would unroll as
for y in stuff: break if not z data.append(y)
Having a while clause on for loops is not just good because it saves a couple of lines but because it clearly separates the flow control from the body of the loop (another reason I dislike 'break if'). In other words I find the flow of the loop
for p in primes() while p < 100: print(p)
easier to understand (immediately) than
for p in primes(): if p >= 100: break print(p)
These are just trivially small examples. As the body of the loop grows in complexity the readability benefit of moving 'if not z: break' into the top line becomes more significant.
You can get the same separation of concerns using takewhile at the expense of a different kind of readability
for p in takewhile(lambda p: p < 100, primes()): print(p)
However there is another problem with using takewhile in for loops which is that it discards an item from the iterable. Imagine parsing a file such as:
csvfile = '''# data.csv # This file begins with an unspecified number of header lines. # Each header line begins with '#'. # I want to keep these lines but need to parse the separately. # The first non-comment line contains the column headers x y z 1 2 3 4 5 6 7 8 9'''.splitlines()
You can do
csvfile = iter(csvfile) headers = [] for line in csvfile: if not line.startswith('#'): break headers.append(line[1:].strip()) fieldnames = line.split() for line in csvfile: yield {name: int(val) for name, val in zip(fieldnames, line.split())}
However if you use takewhile like
for line in takewhile(lambda line: line.startswith('#'), csvfile): headers.append(line[1:].split())
then after the loop 'line' holds the last comment line. The discarded column header line is gone and cannot be recovered; takewhile is normally only used when the entire remainder of the iterator is to be discarded.
I would propose that
for line in csvfile while line.startwith('#'): headers.append(line)
would result in 'line' referencing the item that failed the while predicate.
So: for item in generator while is_true(item): ... is equivalent to: for item in generator: if not is_true(item): break ... By similar reasoning(?): for item in generator if is_true(item): ... is equivalent to: for item in generator: if not is_true(item): continue ... If we have one, shouldn't we also have the other? If only comprehensions have the 'if' form (IIRC, it has already been rejected for multi-line 'for' loops), then shouldn't only comprehensions have the 'while' form?
On 2 July 2013 00:34, MRAB <python@mrabarnett.plus.com> wrote:
So:
for item in generator while is_true(item): ...
is equivalent to:
for item in generator: if not is_true(item): break ...
By similar reasoning(?):
for item in generator if is_true(item): ...
is equivalent to:
for item in generator: if not is_true(item): continue ...
If we have one, shouldn't we also have the other?
If only comprehensions have the 'if' form (IIRC, it has already been rejected for multi-line 'for' loops), then shouldn't only comprehensions have the 'while' form?
<if> is allowed in comprehensions and loops. I'm only suggesting that for/while wouldn't unroll the same way that for/if does. To me it seems natural that <while> "binds" more tightly to <for> than <if> does. The <if> clause in a comprehension is about per item logic so it belongs in the body of the loop. The proposed <while> clause would affect the flow of the loop globally so it does not. Also I wouldn't propose that for/while is equivalent to for/if/break in the case that there is an else clause. This is one area where takewhile is better than for/if/break since you can do for item in takewhile(lambda: keep_going, iterable): if acceptable(item): break else: raise Error('No acceptable items') instead of for x in iterable: if not keep_going: raise Error('No acceptable items') elif acceptable(item): break else: raise Error('No acceptable items') I would want to be able to write for item in iterable while keep_going: if acceptable(item): break else: raise Error('No acceptable items') and know that either an error was raised or item is bound to an acceptable object. Oscar
2013-07-02 00:44, Oscar Benjamin wrote: [...]
Having a while clause on for loops is not just good because it saves a couple of lines but because it clearly separates the flow control from the body of the loop (another reason I dislike 'break if'). In other words I find the flow of the loop
for p in primes() while p < 100: print(p)
easier to understand (immediately) than
for p in primes(): if p >= 100: break print(p)
+1 Cheers. *j
While this looks attractive to me, and it's definitely better to change statement and comprehension syntax at the same time, this makes the comprehension ambiguous to human parsing. [f(x) for x in list if x > 10] basically can be read as for x in list: if x > 10: f(x) This kind of interpretation becomes critical if you nest more than two levels. But [f(x) for x in list while x < 10] could read either as for x in list while x < 10: f(x) which is how you want it to be read, or (more in line with earlier list comp habits): for x in list: while x < 10: f(x) which would be totally wrong. I don't think this is a very serious problem (certainly not for the interpreter), but it's a stumbling block. On Mon, Jul 1, 2013 at 10:03 PM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
2013-07-02 00:44, Oscar Benjamin wrote: [...]
Having a while clause on for loops is not just good because it saves a
couple of lines but because it clearly separates the flow control from the body of the loop (another reason I dislike 'break if'). In other words I find the flow of the loop
for p in primes() while p < 100: print(p)
easier to understand (immediately) than
for p in primes(): if p >= 100: break print(p)
+1
Cheers. *j
______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
What if the while clause went after the rest of the comprehension, preceded by a comma? [f(x) for x in list, while x < 10] — SpaghettiToastBook On Mon, Jul 1, 2013 at 10:28 PM, Daniel Robinson <gottagetmac@gmail.com> wrote:
While this looks attractive to me, and it's definitely better to change statement and comprehension syntax at the same time, this makes the comprehension ambiguous to human parsing.
[f(x) for x in list if x > 10] basically can be read as
for x in list: if x > 10: f(x)
This kind of interpretation becomes critical if you nest more than two levels. But [f(x) for x in list while x < 10] could read either as
for x in list while x < 10: f(x)
which is how you want it to be read, or (more in line with earlier list comp habits):
for x in list: while x < 10: f(x)
which would be totally wrong.
I don't think this is a very serious problem (certainly not for the interpreter), but it's a stumbling block.
On Mon, Jul 1, 2013 at 10:03 PM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
2013-07-02 00:44, Oscar Benjamin wrote: [...]
Having a while clause on for loops is not just good because it saves a couple of lines but because it clearly separates the flow control from the body of the loop (another reason I dislike 'break if'). In other words I find the flow of the loop
for p in primes() while p < 100: print(p)
easier to understand (immediately) than
for p in primes(): if p >= 100: break print(p)
+1
Cheers. *j
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
In addition to defying the ordinary order of the syntax, that would make this statement totally ambiguous: [f(x), f(y) for x in list1 for y in list2, while x < y] And I definitely don't think commas (other than the f(x), f(y) usage shown above) or semicolons are good ideas. On Mon, Jul 1, 2013 at 10:53 PM, SpaghettiToastBook . < spaghettitoastbook@gmail.com> wrote:
What if the while clause went after the rest of the comprehension, preceded by a comma?
[f(x) for x in list, while x < 10]
— SpaghettiToastBook
On Mon, Jul 1, 2013 at 10:28 PM, Daniel Robinson <gottagetmac@gmail.com> wrote:
While this looks attractive to me, and it's definitely better to change statement and comprehension syntax at the same time, this makes the comprehension ambiguous to human parsing.
[f(x) for x in list if x > 10] basically can be read as
for x in list: if x > 10: f(x)
This kind of interpretation becomes critical if you nest more than two levels. But [f(x) for x in list while x < 10] could read either as
for x in list while x < 10: f(x)
which is how you want it to be read, or (more in line with earlier list comp habits):
for x in list: while x < 10: f(x)
which would be totally wrong.
I don't think this is a very serious problem (certainly not for the interpreter), but it's a stumbling block.
On Mon, Jul 1, 2013 at 10:03 PM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
2013-07-02 00:44, Oscar Benjamin wrote: [...]
Having a while clause on for loops is not just good because it saves a couple of lines but because it clearly separates the flow control from the body of the loop (another reason I dislike 'break if'). In other words I find the flow of the loop
for p in primes() while p < 100: print(p)
easier to understand (immediately) than
for p in primes(): if p >= 100: break print(p)
+1
Cheers. *j
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 07/01/2013 10:27 PM, Daniel Robinson wrote:
In addition to defying the ordinary order of the syntax, that would make this statement totally ambiguous:
[f(x), f(y) for x in list1 for y in list2, while x < y]
And I definitely don't think commas (other than the f(x), f(y) usage shown above) or semicolons are good ideas.
I agree. There is also the problem of how to spell it with both a filter test, and the end test at the same time. You can write it as one or the other with a stop() function, but they don't work independently. [x for x in values if x<50 or stop()] # will work in 3.4 And of course this works now. [x for x in values if x%2==0] To do both is tricky and took me a while to come up with... [x for x in values if stop(x>50) or x%2==0] Where stop(x>50) raises StopIteration if the test is True or False otherwise. Which make the 'or expression' needed even if it's just a 'or True'. It should work, but seems awkward to me. Cheers, Ron
From: David Mertz <mertz@gnosis.cx> Sent: Monday, July 1, 2013 1:29 PM
On Mon, Jul 1, 2013 at 10:25 AM, Joshua Landau <joshua.landau.ws@gmail.com> wrote:
primes100 = {p for p in primes(); break if p >= 100} primes100 = {p for p in primes() while p < 100}
If you're telling me that "{p for p in primes() while p < 100}" reads better than "{p for p in primes(); break if p >= 100}" I have to disagree strongly. The "break if" form looks beautiful.
My own taste is in strong contrast to Joshua's. I think the semi-colon/break-if looks absolutely horrendous and ugly. The 'while' clause looks obvious, beautiful, and intuitive.
It isn't really that the semicolon is inherently ugly, but that it doesn't fit in with today's comprehension syntax. And the problem we're running into here is that, given today's comprehension syntax, there simply _is_ no clear way to extend it any further. Consider that if you try to translate a non-trivial comprehension into English, it's going to have at least commas in it, and likely semicolons. Also, consider that you really never want to nest arbitrary clauses; you just want to nest loops, and attach modifiers to them. So… Haskell-style comprehensions, as we have today, can only be taken so far. But I don't think that's a problem. See http://stupidpythonideas.blogspot.com/2013/07/syntactic-takewhile.html for more details and ramblings.
On 02/07/13 03:25, Joshua Landau wrote:
If you're telling me that "{p for p in primes() while p < 100}" reads better than "{p for p in primes(); break if p >= 100}" I have to disagree strongly. The "break if" form looks beautiful.
Beautiful it may be, but you have something which explicitly looks like *two statements* which actually represents *one expression*. When it comes to mimicry and camouflage, I think that it is wonderful and astonishing when insects look like sticks, plants look like insects, and harmless snakes look like deadly ones. But when it comes to Python code, I think that mimicry is a terrible idea. I am truly not looking forward to fielding questions from confused programmers who extrapolate from the above to multiple statements in a comprehension: {p for p in primes(); print(p); break if p >= 100} and other variations: {x for x in values(); raise ValueError if not condition(x)} And I'm even *less* looking forward to the debates on python-ideas from people who will want to introduce such syntax :-) "Clever" syntax is rarely a good idea. Good syntax should allow the reader to correctly extrapolate to other examples, not mislead them into making errors. -- Steven
Oscar Benjamin wrote:
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen): yield n primes_seen.append(n)
This algorithm is actually even poorer as it doesn't stop at sqrt(n).
Nor should it! When you're only dividing by primes, you can't stop at sqrt(n), you have to divide by *all* the primes less than n. Otherise you could miss a prime factor greater than sqrt(n) whose cofactor is not prime. (Not relevant to the original disussion, I know, but my inner mathematician couldn't restrain himself.) -- Greg
That cofactor would have to have a prime factor less than sqrt(n). On Tuesday, July 2, 2013, Greg Ewing wrote:
Oscar Benjamin wrote:
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen): yield n primes_seen.append(n)
This algorithm is actually even poorer as it doesn't stop at sqrt(n).
Nor should it! When you're only dividing by primes, you can't stop at sqrt(n), you have to divide by *all* the primes less than n. Otherise you could miss a prime factor greater than sqrt(n) whose cofactor is not prime.
(Not relevant to the original disussion, I know, but my inner mathematician couldn't restrain himself.)
-- Greg ______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
Daniel Robinson wrote:
That cofactor would have to have a prime factor less than sqrt(n).
<facepalm> You're right, of course. I feel suitably enfoolished. </facepalm> But there is a slight error in the second implementation: if all(n % p for p in takewhile(lambda p: p**2 < n, primes_seen)): should be if all(n % p for p in takewhile(lambda p: p**2 <= n, primes_seen)): otherwise it thinks that perfect squares are primes. -- Greg
On 3 July 2013 06:52, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Daniel Robinson wrote:
That cofactor would have to have a prime factor less than sqrt(n).
<facepalm> You're right, of course. I feel suitably enfoolished. </facepalm>
It took me a little while to understand what you were getting at. My way of thinking about this is that in every pair of factors either both are equal to sqrt(n) or one is less than sqrt(n) and the other greater.
But there is a slight error in the second implementation:
if all(n % p for p in takewhile(lambda p: p**2 < n, primes_seen)):
should be
if all(n % p for p in takewhile(lambda p: p**2 <= n, primes_seen)):
otherwise it thinks that perfect squares are primes.
Ah, yes. I had it right in the for/while version so I'm going to blame this typo on takewhile/lambda for obfuscating my code. :) Oscar
On 02/07/2013 23:35, Greg Ewing wrote:
Oscar Benjamin wrote:
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen): yield n primes_seen.append(n)
This algorithm is actually even poorer as it doesn't stop at sqrt(n).
Nor should it! When you're only dividing by primes, you can't stop at sqrt(n), you have to divide by *all* the primes less than n. Otherise you could miss a prime factor greater than sqrt(n) whose cofactor is not prime.
(Not relevant to the original disussion, I know, but my inner mathematician couldn't restrain himself.)
But you could at least stop at n/4. Rob Cliffe
On 2 July 2013 23:35, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Oscar Benjamin wrote:
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen): yield n primes_seen.append(n)
This algorithm is actually even poorer as it doesn't stop at sqrt(n).
Nor should it! When you're only dividing by primes, you can't stop at sqrt(n), you have to divide by *all* the primes less than n. Otherise you could miss a prime factor greater than sqrt(n) whose cofactor is not prime.
I'm not convinced. Say you have 7 * 6. That is 7 * 3 * 2, so if 7 has a cofactor of 6 then 2 is a factor. The proof can be generalised.
On Tue, Jul 2, 2013 at 3:35 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
Oscar Benjamin wrote:
def primes(): primes_seen = [] for n in count(2): if all(n % p for p in primes_seen): yield n primes_seen.append(n)
This algorithm is actually even poorer as it doesn't stop at sqrt(n).
Nor should it! When you're only dividing by primes, you can't stop at sqrt(n), you have to divide by *all* the primes less than n. Otherise you could miss a prime factor greater than sqrt(n) whose cofactor is not prime.
(Not relevant to the original disussion, I know, but my inner mathematician couldn't restrain himself.)
That would imply that the cofactor has no prime factors (or else you would have hit them), which would mean it must be prime itself (and therefore you hit the cofactor earilier).
-- Greg
______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
Cool. So it looks like it IS PEP-able. :) I was just thinking that, as keywords go, "break" is pretty unencumbered. It's really just a specific "goto outside of loop". No syntax tied with it, and for all practical purposes, relies on some conditional for its use. So it's not a bad candidate for augmentation. Not to be a pill, but following the same logic, would this mean:
foo = [x for x in range(100) continue if ((x > 5) and (x<95))] print foo [0, 1, 2, 3, 4, 5, 95, 96, 97, 98, 99]
?? (feel free to just slap me regarding this....) - Jim Nick wote:
On 28 June 2013 18:20, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote: Let me suggest one more solution although it requires a new keyword: introduce
*breakif* condition
and define its translation as if condition: break . You can now write (x for x in iterable breakif x < 0) and I don't see a way how that could possibly be misread by anyone. Also it would translate unambiguously to the explicit:
for x in iterable: breakif x<0 # itself translating to if x<0: break yield x
It would work with genexps, comprehensions and explicit loops alike (with very little benefit for the later, though maybe it increases readability even there by making it clear from the start of the line what the purpose of the condition test is).
I'd be very keen to see this written up as a PEP - it's the first
This (or, more accurately, a slight variant that doesn't need a new keyword) actually sounds quite attractive to me. My rationale for that ties into the just rejected PEP 315, which tried to find an improved "loop and a half" syntax for Python, as well as the ongoing confusion regarding the meaning of the "else" clause on while and for loops. Currently, Python's fully general loop syntax looks like this: while True: # Iteration setup if termination_condition: break # Remained of iteration And the recommended idiom for a search loop looks like this: for x in data: if desired_value(x): break else: raise ValueError("Value not found in {:100!r}".format(data)) Rather than adding a new keyword, we could simply expand the syntax for the existing break statement to be this: break [if <EXPR>] This would simplify the above two standard idioms to the following: while True: # Iteration setup break if termination_condition # Remainder of iteration for x in data: break if desired_value(x) else: raise ValueError("Value not found in {:100!r}".format(data)) A "bare" break would then be equivalent to "break if True". The "else" clause on the loop could then be *explicitly* documented as associated with the "break if <X>" form - the else only executes if the break clause is never true. (That also becomes the justification for only allowing this for break, and not for continue or return: those have no corresponding "else" clause) Once the break statement has been redefined this way, it *then* becomes reasonable to allow the following in comprehensions: data = [x for x in iterable break if x is None] As with other proposals, I would suggest limiting this truncating form to disallow combination with the filtering and nested loop forms (at least initially). The dual use of "if" would make the filtering combination quite hard to read, and the nested loop form would be quite ambiguous as to which loop was being broken. If we start with the syntax restricted, we can relax those restrictions later if we find them too limiting, while if we start off being permissive, backwards compatibility would prevent us from adding restrictions later. proposal that I feel actually *simplifies* the language in any way (mostly by doing something about those perplexing-to-many else clauses on for and while loops). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 29 Jun, 2013, at 12:09, Nick Coghlan <ncoghlan@gmail.com> wrote: ...
Rather than adding a new keyword, we could simply expand the syntax for the existing break statement to be this:
break [if <EXPR>]
That saves types two characters as compared with current status-quo ;-)
This would simplify the above two standard idioms to the following:
while True: # Iteration setup break if termination_condition # Remainder of iteration
for x in data: break if desired_value(x) else: raise ValueError("Value not found in {:100!r}".format(data))
A "bare" break would then be equivalent to "break if True". The "else" clause on the loop could then be *explicitly* documented as associated with the "break if <X>" form - the else only executes if the break clause is never true. (That also becomes the justification for only allowing this for break, and not for continue or return: those have no corresponding "else" clause)
Once the break statement has been redefined this way, it *then* becomes reasonable to allow the following in comprehensions:
data = [x for x in iterable break if x is None]
As with other proposals, I would suggest limiting this truncating form to disallow combination with the filtering and nested loop forms (at least initially). The dual use of "if" would make the filtering combination quite hard to read, and the nested loop form would be quite ambiguous as to which loop was being broken. If we start with the syntax restricted, we can relax those restrictions later if we find them too limiting, while if we start off being permissive, backwards compatibility would prevent us from adding restrictions later.
Jikes. I don't like restricting this to loops without filtering and without nesting at all, that makes the language more complex. Has anyone tried to inventory how often this new syntax would be appropriate (with and without the restriction on filtering and nested loops)?
I'd be very keen to see this written up as a PEP - it's the first proposal that I feel actually *simplifies* the language in any way (mostly by doing something about those perplexing-to-many else clauses on for and while loops).
That's very optimistic, I don't think this helps with the for:else: confusion at all. The else part of a for loop already has a straightforward explanation, and that doesn't seem to help at all. An if-component on a 'break' statement has a pretty loose connection to the corresponding 'else', especially when the 'if' part is optional. Ronald
participants (19)
-
Andrew Barnert -
Chris Kaynor -
Daniel Robinson -
David Mertz -
Greg Ewing -
Jan Kaliszewski -
jimjhb＠aol.com -
Joshua Landau -
MRAB -
Nick Coghlan -
Oscar Benjamin -
Paul Moore -
Rob Cliffe -
Ron Adam -
Ronald Oussoren -
Shane Green -
SpaghettiToastBook . -
Steven D'Aprano -
Wolfgang Maier