I know what the regulars among you will be thinking (time machine, high bar for language syntax changes, etc.) so let me start by assuring you that I'm well aware of all of this, that I did research the topic before posting and that this is not the same as a previous suggestion using almost the same subject line. Now here's the proposal: allow an except (or except break) clause to follow for/while loops that will be executed if the loop was terminated by a break statement. The idea is certainly not new. In fact, Nick Coghlan, in his blog post http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_el..., uses it to provide a mental model for the meaning of the else following for/while, but, as far as I'm aware, he never suggested to make it legal Python syntax. Now while it's possible that Nick had a good reason not to do so, I think there would be three advantages to this: - as explained by Nick, the existence of "except break" would strengthen the analogy with try/except/else and help people understand what the existing else clause after a loop is good for. There has been much debate over the else clause in the past, most prominently, a long discussion on this list back in 2009 (I recommend interested people to start with Steven D'Aprano's Summary of it at https://mail.python.org/pipermail/python-ideas/2009-October/006155.html) that shows that for/else is misunderstood by/unknown to many Python programmers. - in some situations for/except/else would make code more readable by bringing logical alternatives closer together and to the same indentation level in the code. Consider a simple example (taken from the docs.python Tutorial: for n in range(2, 10): for x in range(2, n): if n % x == 0: print(n, 'equals', x, '*', n//x) break else: # loop fell through without finding a factor print(n, 'is a prime number') There are two logical outcomes of the inner for loop here - a given number can be either prime or not. However, the two code branches dealing with them end up at different levels of indentation and in different places, one inside and one outside the loop block. This second issue can become much more annoying in more complex code where the loop may contain additional code after the break statement. Now compare this to: for n in range(2, 10): for x in range(2, n): if n % x == 0: break except break: print(n, 'equals', x, '*', n//x) else: # loop fell through without finding a factor print(n, 'is a prime number') IMO, this reflects the logic better. - it could provide an elegant solution for the How to break out of two loops issue. This is another topic that comes up rather regularly (python-list, stackoverflow) and there is again a very good blog post about it, this time from Ned Batchelder at https://nedbatchelder.com/blog/201608/breaking_out_of_two_loops.html. Stealing his example, here's code (at least) a newcomer may come up with before realizing it can't work: s = "a string to examine" for i in range(len(s)): for j in range(i+1, len(s)): if s[i] == s[j]: answer = (i, j) break # How to break twice??? with for/except/else this could be written as: s = "a string to examine" for i in range(len(s)): for j in range(i+1, len(s)): if s[i] == s[j]: break except break: answer = (i, j) break So much for the pros. Of course there are cons, too. The classical one for any syntax change, of course, is: - burden on developers who have to implement and maintain the new syntax. Specifically, this proposal would make parsing/compiling of loops more complicated. Others include: - using except will make people think of exceptions and that may cause new confusion; while that's true, I would argue that, in fact, break and exceptions are rather similar features in that they are gotos in disguise, so except will still be used to catch an interruption in normal control flow. - the new syntax will not help people understand for/else if except is not used; importantly, I'm *not* proposing to disallow the use of for/else without except (if that would ever happen it would be in the *very* distant future) so that would indeed mean that people would encounter for/else, not only in legacy, but also in newly written code. However, I would expect that they would also start seeing for/except increasingly (not least because it solves the "break out of two loops" issue) so they would be nudged towards thinking of the else after for/while more like the else in try/except/else just as Nick proposes it. Interestingly, there has been another proposal on this list several years ago about allowing try/else without except, which I liked at the time and which would have made try/except/]else work exactly as my proposed for/except/else. Here it is: https://mail.python.org/pipermail/python-ideas/2011-November/012875.html - as a result of previous discussions about for/else a section was added to PEP3099 saying: "The else clause in while and for loops will not change semantics, or be removed." However, the proposal here is not to change the else clause semantics, but add an additional except clause. So that's it and while I'm well aware of the slim chances of this getting legal syntax, I would still be happy to get feedback from you :) Best, Wolfgang
On Wed, Mar 01, 2017 at 10:37:17AM +0100, Wolfgang Maier wrote:
Let me see if I understand the proposal in full. You would allow: for i in (1, 2, 3): print(i) if i == 2: break except break: # or just except assert i == 2 print("a break was executed") else: print("never reached") # this is never reached print("for loop is done") as an alternative to something like: broke_out = False for i in (1, 2, 3): print(i) if i == 2: broke_out = True break else: print("never reached") # this is never reached if broke_out: assert i == 2 print("a break was executed") print("for loop is done") I must admit the suggestion seems a little bit neater than having to manage a flag myself, but on the other hand I can't remember the last time I've needed to manage a flag like that. And on the gripping hand, this is even simpler than both alternatives: for i in (1, 2, 3): print(i) if i == 2: assert i == 2 print("a break was executed") break else: print("never reached") # this is never reached print("for loop is done") There are some significant unanswered questions: - Does it matter which order the for...except...else are in? Obviously the for block must come first, but apart from that? - How is this implemented? Currently "break" is a simple unconditional GOTO which jumps past the for block. This will need to change to something significantly more complex. - There are other ways to exit a for-loop than just break. Which of them, if any, will also run the except block? -- Steve
On 01.03.2017 12:56, Steven D'Aprano wrote:
one way to implement this with unconditional GOTOs would be (in pseudocode): LOOP: on break GOTO EXCEPT ELSE: ... GOTO THEN EXCEPT: ... THEN: ... So at the byte-code level (but only there) the order of except and else would be reversed. Was that a reason why you were asking about the order of except and else in my proposal? Anyway, I'm sure there are people much more skilled at compiler programming than me here.
Much snippage; apologies, Wolfgang! On 01/03/17 09:37, Wolfgang Maier wrote:
It reads worse to me, I'm afraid. Moving the "print" disassociates it from the condition that caused it, making it that bit harder to understand. You'd have a more compelling case with a complex loop with multiple breaks all requiring identical processing. However my experience is that such cases are rare, and are usually attempts to do exception handling with out actually using exceptions. I'm not terribly inclined to help people make more work for themselves.
That is a better use case. I must admit I normally handle this sort of thing by putting the loops in a function and returning out of the inner loop. -- Rhodri James *-* Kynesim Ltd
I’d like to see some examples where nested for loops couldn’t easily be avoided in the first place.
Replace the inner loop with a call to any consuming a generator for n in range(2,10): if any(n % x == 0 for x in range(2,n)): print('{} equals {} * {}'.format(n, x, n//x)) else: print('{} is prime'.format(n))
Replace the inner loop with a call to str.find for i, c in enumerate(s): j = s.find(c, i+1) if j >= 0: answer = (i, j) break
On 03/01/2017 01:37 AM, Wolfgang Maier wrote:
Now here's the proposal: allow an except (or except break) clause to follow for/while loops that will be executed if the loop was terminated by a break statement.
I find the proposal interesting. More importantly, the proposal is well written and clear -- thank you! -- ~Ethan~
On 1 March 2017 at 19:37, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
I never really thought about it, as I only use the "else:" clause for search loops where there aren't any side effects in the "break" case (other than the search result being bound to the loop variable), so while I find "except break:" useful as an explanatory tool, I don't have any practical need for it. I think you've made as strong a case for the idea as could reasonably be made :) However, Steven raises a good point that this would complicate the handling of loops in the code generator a fair bit, as it would add up to two additional jump targets in cases wherever the new clause was used. Currently, compiling loops only needs to track the start of the loop (for continue), and the first instruction after the loop (for break). With this change, they'd also need to track: - the start of the "except break" clause (for break when the clause is used) - the start of the "else" clause (for the non-break case when both trailing clauses are present) The design level argument against adding the clause is that it breaks the "one obvious way" principle, as the preferred form for search loops look like this: for item in iterable: if condition(item): break else: # Else clause either raises an exception or sets a default value item = get_default_value() # If we get here, we know "item" is a valid reference operation(item) And you can easily switch the `break` out for a suitable `return` if you move this into a helper function: def find_item_of_interest(iterable): for item in iterable: if condition(item): return item # The early return means we can skip using "else" return get_default_value() Given that basic structure as a foundation, you only switch to the "nested side effect" form if you have to: for item in iterable: if condition(item): operation(item) break else: # Else clause neither raises an exception nor sets a default value condition_was_never_true(iterable) This form is generally less amenable to being extracted into a reusable helper function, since it couples the search loop directly to the operation performed on the bound item, whereas decoupling them gives you a lot more flexibility in the eventual code structure. The proposal in this thread then has the significant downside of only covering the "nested side effect" case: for item in iterable: if condition(item): break except break: operation(item) else: condition_was_never_true(iterable) While being even *less* amenable to being pushed down into a helper function (since converting the "break" to a "return" would bypass the "except break" clause). So while it is cool to see this written up as a concrete proposal (thank you!), I don't think it makes the grade as an actual potential syntax change. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 02.03.2017 06:46, Nick Coghlan wrote:
I think you could get away with only one additional jump target as I showed in my previous reply to Steven. The heavier burden would be on the parser, which would have to distinguish the existing and the two new loop variants (loop with except clause, loop with except and else clause) but, anyway, that's probably not really the point. What weighs heavier, I think, is your design argument.
I'm actually not quite buying this last argument. If you wanted to refactor this to "return" instead of "break", you could simply put the return into the except break block. In many real-world situations with multiple breaks from a loop this could actually make things easier instead of worse. Personally, the "nested side effect" form makes me uncomfortable every time I use it because the side effects on breaking or not breaking the loop don't end up at the same indentation level and not necessarily together. However, I'm gathering from the discussion so far that not too many people are thinking like me about this point, so maybe I should simply adjust my mind-set. All that said, this is a very nice abstract view on things! I really learned quite a bit from this, thank you :) As always though, reality can be expected to be quite a bit more complicated than theory so I decided to check the stdlib for real uses of break. This is quite a tedious task since break is used in many different ways and I couldn't come up with a good automated way of classifying them. So what I did is just go through stdlib code (in reverse alphabetical order) containing the break keyword and put it into categories manually. I only got up to socket.py before losing my enthusiasm, but here's what I found: - overall I looked at 114 code blocks that contain one or more breaks - 84 of these are trivial use cases that simply break out of a while True block or terminate a while/for loop prematurely (no use for any follow-up clause there) - 8 more are causing a side-effect before a single break, and it would be pointless to put this into an except break clause - 3 more cause different, non-redundant side-effects before different breaks from the same loop and, obviously, an except break clause would not help them either => So the vast majority of breaks does *not* need an except break *nor* an else clause, but that's just as expected. Of the remaining 19 non-trivial cases - 9 are variations of your classical search idiom above, i.e., there's an else clause there and nothing more is needed - 6 are variations of your "nested side-effects" form presented above with debatable (see above) benefit from except break - 2 do not use an else clause currently, but have multiple breaks that do partly redundant things that could be combined in a single except break clause - 1 is an example of breaking out of two loops; from sre_parse._parse_sub: [...] # check if all items share a common prefix while True: prefix = None for item in items: if not item: break if prefix is None: prefix = item[0] elif item[0] != prefix: break else: # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) continue # check next one break [...] This could have been written as: [...] # check if all items share a common prefix while True: prefix = None for item in items: if not item: break if prefix is None: prefix = item[0] elif item[0] != prefix: break except break: break # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) [...] - finally, 1 is a complicated break dance to achieve sth that clearly would have been easier with except break; from typing.py: [...] def __subclasscheck__(self, cls): if cls is Any: return True if isinstance(cls, GenericMeta): # For a class C(Generic[T]) where T is co-variant, # C[X] is a subclass of C[Y] iff X is a subclass of Y. origin = self.__origin__ if origin is not None and origin is cls.__origin__: assert len(self.__args__) == len(origin.__parameters__) assert len(cls.__args__) == len(origin.__parameters__) for p_self, p_cls, p_origin in zip(self.__args__, cls.__args__, origin.__parameters__): if isinstance(p_origin, TypeVar): if p_origin.__covariant__: # Covariant -- p_cls must be a subclass of p_self. if not issubclass(p_cls, p_self): break elif p_origin.__contravariant__: # Contravariant. I think it's the opposite. :-) if not issubclass(p_self, p_cls): break else: # Invariant -- p_cls and p_self must equal. if p_self != p_cls: break else: # If the origin's parameter is not a typevar, # insist on invariance. if p_self != p_cls: break else: return True # If we break out of the loop, the superclass gets a chance. if super().__subclasscheck__(cls): return True if self.__extra__ is None or isinstance(cls, GenericMeta): return False return issubclass(cls, self.__extra__) [...] which could be rewritten as: [...] def __subclasscheck__(self, cls): if cls is Any: return True if isinstance(cls, GenericMeta): # For a class C(Generic[T]) where T is co-variant, # C[X] is a subclass of C[Y] iff X is a subclass of Y. origin = self.__origin__ if origin is not None and origin is cls.__origin__: assert len(self.__args__) == len(origin.__parameters__) assert len(cls.__args__) == len(origin.__parameters__) for p_self, p_cls, p_origin in zip(self.__args__, cls.__args__, origin.__parameters__): if isinstance(p_origin, TypeVar): if p_origin.__covariant__: # Covariant -- p_cls must be a subclass of p_self. if not issubclass(p_cls, p_self): break elif p_origin.__contravariant__: # Contravariant. I think it's the opposite. :-) if not issubclass(p_self, p_cls): break else: # Invariant -- p_cls and p_self must equal. if p_self != p_cls: break else: # If the origin's parameter is not a typevar, # insist on invariance. if p_self != p_cls: break except break: # If we break out of the loop, the superclass gets a chance. if super().__subclasscheck__(cls): return True if self.__extra__ is None or isinstance(cls, GenericMeta): return False return issubclass(cls, self.__extra__) return True [...] My summary: I do see use-cases for the except break clause, but, admittedly, they are relatively rare and may be not worth the hassle of introducing new syntax.
On Thu, 2 Mar 2017 at 03:07 Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote: [SNIP]
I wanted to say thanks for taking the time to go through the stdlib and doing such a thorough analysis of the impact of your suggestion! It always helps to have real-world numbers to know whether an idea will be useful (or not).
IOW out of 114 cases, 4 may benefit from an 'except' block? If I'm reading those numbers correctly then ~3.5% of cases would benefit which isn't high enough to add the syntax and related complexity IMO.
On 03/02/2017 07:05 PM, Brett Cannon wrote:
[...]
Hmm, I'm not sure how much sense it makes to express this in percent since the total your comparing to is rather arbitrary. The 114 cases include *any* for/while loop I could find that contains at least a single break. More than 90 of these loops do not use an "else" clause either showing that even this currently supported syntax is used rarely. I found only 19 cases that are complex enough to be candidates for an except clause (17 of these use the else clause). For 9 of these 19 (the ones using the classical search idiom) an except clause would not be applicable, but it could be used in the 10 remaining cases (though all of them could also make use of a flag or could be refactored instead). So depending on what you want to emphasize you could also say that the proposal could affect as much as 10/19 or 52.6% of cases.
On 2 March 2017 at 21:06, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
Fair point - so that would be even with the "single nested side effect" case, but simpler when you had multiple break conditions (and weren't already combined them with "and").
This is why I consider the "search only" form of the loop, where the else clause either sets a default value, or else prevents execution of the code after the loop body (via raise, return, or continue), to be the preferred form: there aren't any meaningful side effects hidden away next to the break statement. If I can't do that, I'm more likely to switch to a classic flag variable that gets checked post-loop execution than I am to push the side effect inside the loop body: search_result = _not_found = object() for item in iterable: if condition(item): search_result = item break if search_result is _not_found: # Handle the "not found" case else: # Handle the "found" case
Thanks for doing that research :)
Those 8 cases could also be reviewed to see whether a flag variable might be clearer than relying on nested side effects or code repetition.
This is a case where a flag variable may be easier to read than loop state manipulations: may_have_common_prefix = True while may_have_common_prefix: prefix = None for item in items: if not item: may_have_common_prefix = False break if prefix is None: prefix = item[0] elif item[0] != prefix: may_have_common_prefix = False break else: # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) Although the whole thing could likely be cleaned up even more via itertools.zip_longest: for first_uncommon_idx, aligned_entries in enumerate(itertools.zip_longest(*items)): if not all_true_and_same(aligned_entries): break else: # Everything was common, so clear all entries first_uncommon_idx = None for item in items: del item[:first_uncommon_idx] (Batching the deletes like that may even be slightly faster than deleting common entries one at a time) Given the following helper function: def all_true_and_same(entries): itr = iter(entries) try: first_entry = next(itr) except StopIteration: return False if not first_entry: return False for entry in itr: if not entry or entry != first_entry: return False return True
I think is another case that is asking for the inner loop to be factored out to a named function, not for reasons of re-use, but for reasons of making the code more readable and self-documenting :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/03/2017 04:36 AM, Nick Coghlan wrote:
[...]
[...]
It's true that using a flag or factoring out redundant code is always a possibility. Having the except clause would clearly not let people do anything they couldn't have done before. On the other hand, the same is true for the else clause - it's only advantage here is that it's existing already - because a single flag could always distinguish between a break having occurred or not: brk = False for item in iterable: if some_condition: brk = True break if brk: do_stuff_upon_breaking_out() else: do_alternative_stuff() is a general pattern that would always work without except *and* else. However, the fact that else exists generates a regrettable asymmetry in that there is direct language support for detecting one outcome, but not the other. Stressing the analogy to try/except/else one more time, it's as if "else" wasn't available for try blocks. You could always use a flag to substitute for it: dealt_with_exception = False try: do_stuff() except: deal_with_exception() dealt_with_exception = True if dealt_with_exception: do_stuff_you_would_do_in_an_else_block() So IMO the real difference here is that the except clause after for would require adding it to the language, while the else clauses are there already. With that we're back at the high bar for adding new syntax :( A somewhat similar case that comes to mind here is PEP 315 -- Enhanced While Loop, which got rejected for two reasons, the first one being pretty much the same as the argument here, i.e., that instead of the proposed do .. while it's always possible to factor out or duplicate a line of code. However, the second reason was that it required the new "do" keyword, something not necessary for the current suggestion.
On 3 March 2017 at 18:47, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
I forget where it came up, but I seem to recall Guido saying that if he were designing Python today, he wouldn't include the "else:" clause on loops, since it inevitably confuses folks the first time they see it. (Hence articles like mine that attempt to link it with try/except/else rather than if/else).
It's worth noting that this asymmetry doesn't necessarily exist in the corresponding C idiom that I assume was the inspiration for the Python equivalent: int data_array_len = sizeof(data_array) / sizeof(data_array[0]); in idx = 0; for (idx = 0; idx < data_array_len; idx++) { if (condition(container[idx])) { break; } } if (idx < data_array_len) { // We found a relevant entry } else { // We didn't find anything } In Python prior to 2.1 (when PEP 234 added the iterator protocol), a similar approach could be used for Python's for loops: num_items = len(container): for idx in range(num_items): if condition(container[idx]): break if num_items and idx < num_items: # We found a relevant entry else: # We didn't find anything However, while my own experience with Python is mainly with 2.2+ (and hence largely after the era where "for i in range(len(container)):" was still common), I've spent a lot of time working with C and the corresponding iterator protocols in C++, and there it is pretty common to move the "entry found" code before the break and then invert the conditional check that appears after the loop: int data_array_len = sizeof(data_array) / sizeof(data_array[0]); int idx = 0; for (idx = 0; idx < data_array_len; idx++) { if (condition(container[idx])) { // We found a relevant entry break; } } if (idx >= data_array_len) { // We didn't find anything } And it's *this* version of the C/C++ idiom that Python's "else:" clause replicates. One key aspect of this particular idiomatic structure is that it retains the same overall shape regardless of whether the inner structure is: if condition(item): # Condition is true, so process the item process(item) break or: if maybe_process_item(item): # Item was processed, so we're done here break Whereas the "post-processing" model can't handle pre-composed helper functions that implement both the conditional check and the item processing, and then report back which branch they took. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Mar 05, 2017 at 01:17:31PM +1000, Nick Coghlan wrote:
Heh, if we exclude all features that confuse people the first time they see it, we'd have to remove threads, Unicode, floating point maths, calls to external processes, anything OS-dependent, metaclasses, classes, ... :-) It took me the longest time to realise that the "else" clause didn't *only* run when the loop sequence is empty. That is, I expected that given: for x in random.choice(["", "a"]): # either empty, or a single item print("run the loop body") else: print("loop sequence is empty") That's because it *seems* to work that way, if you do insufficient testing: for x in []: raise ValueError # dead code is not executed else: print("loop sequence is empty") It is my belief that the problem here is not the else clause itself, but that the name used is misleading. I've seen people other than myself conclude that it means: run the for-loop over the sequence otherwise the sequence is empty, run the ELSE block I've seen people think that it means: set break_seen flag to false run the for-loop if break is executed, set the break_seen flat to true then break if break_seen is false, run the "ELSE NOT BREAK" clause and consequently ask how they can access the break_seen flag for themselves. Presumably they want to write something like: run the for-loop if break_seen is true, do this else (break_seen is false) do that I think that the name "else" here is a "misunderstanding magnet", it leads people to misunderstand the nature of the clause and its implications. For example, I bet that right now there are people reading this and nodding along with me and thinking "maybe we should rename it something more explicit, like "else if no break", completely oblivious to the fact that `break` is NOT the only way to avoid running the `else` clause. I believe that the name should have been "then", not "else". It describes what the code does: run the for-block THEN run the "else" block There's no flag to be tested, and the "else" block simply runs once, after the for-loop, regardless of whether the for-loop runs once or ten times or zero times (empty sequence). To avoid running the "else" ("then") block, you have to exit the entire block of code using: - break - return - raise which will all transfer execution past the end of the for...else (for...then) compound statement. Since I realised that the else block merely runs directly after the for, I've never had any problem with the concept. `break` merely jumps past the for...else block, just as `return` exits the function and `raise` triggers an exception which transfers execution to the surrounding `except` clause. -- Steve
On 3/4/2017 10:17 PM, Nick Coghlan wrote:
(Hence articles like mine that attempt to link it with try/except/else rather than if/else).
The link needs to be done in two separate steps: if-else => while-else => for-else. Step 1. The conceptual difference between an if-clause and a while-clause is that an if-clause ends with an implied 'break' (jump past the rest of the statement to the next statement) while a while-clause ends with an implied 'continue' (jump back to the condition test). In the CPython byte code, this difference is implemented by JUMP_FORWARD versus JUMP_ABSOLUTE (backwards). (The byte code for a while-clause also has two bookkeeping additions, SETUP_LOOP and POP_BLOCK.) In both if-statements and while-statements, the else-clause is executed if and when the condition is false. I wonder if having never programming with with 'jump' or 'goto' makes this harder to understand. I think our doc could better explain how if-statements and while-statements are similar but different. Step 2. A for-loop can be viewed as an initialized while-loop. Kernighan and Ritchie make this explicit in The C Programming Language (p. 56). ''' The for statement for (expr1; expr2; expr3) statement is equivalent to expr1; while (expr2) { statement expr3; } ''' For Python's more specialized for-loop, 'for target-list in iterable: for-suite' can be paraphrased as 'while the iterator yields an object (is not exhausted): do the assignment and suite'. An else-clause, if present, is executed if and when the condition is false, when the iterator is exhausted and raises StopIteration instead of yielding an object. If while-else is understood and the implicit for condition is understood, for-else is pretty straightforward. Equivalent code, with an else-clause, is trickier than in C, without for-else. I think the following is close. I think something like this should be in the doc. _it = iter(iterable) _exhausted = False while not _exhausted: try: _obj = next(_it) except StopIteration: _exhausted = True continue target-list = _obj for-suite else: else-suite Note 1: without the else-clause, _exhausted is not needed. The loop could be 'while True' and the except clause could just be 'break'. Note 2: _obj being assignment compatible with target list is NOT part of the implicit while condition. For example, for a,b in (1,2): print('for') else: print('else') prints nothing except the traceback for TypeError: 'int' object is not iterable. Note 3: C for typically assigns to the loop variable once before the loop and and again at the end of each loop. Python for does the assignment once at the top of the loop. -- Terry Jan Reedy
On 1 March 2017 at 06:37, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
After rethinking over some code I've written in the past, yes, I agree this change could be a nice one. The simple fact that people are commenting that they could chage the code to be an inner function in order to break from nested "for" loops should be a hint this syntax is useful. (I myself have done that with exceptions in some cases). js -><-
On 3/1/17, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
I was thinking bout this analogy: 1. try/else (without except) is SyntaxError. And seems useless. 2. try/break/except is backward compatible: for i in L: try: break except Something: pass except break: # current code has not this so break is applied to for-block 3. for/raise/except (which is natural application of this analogy) could reduce indentation but in my personal view that don't improve readability (but I could be wrong) It could help enhance "break" possibilities so "simplify" double break in nested loops. for broken = False for if condition1(): # I like to "double break" here raise SomeError() if condition2(): break except SomeError: break except break: broken = True 4. for/finally may be useful
On Wed, Mar 01, 2017 at 10:37:17AM +0100, Wolfgang Maier wrote:
Let me see if I understand the proposal in full. You would allow: for i in (1, 2, 3): print(i) if i == 2: break except break: # or just except assert i == 2 print("a break was executed") else: print("never reached") # this is never reached print("for loop is done") as an alternative to something like: broke_out = False for i in (1, 2, 3): print(i) if i == 2: broke_out = True break else: print("never reached") # this is never reached if broke_out: assert i == 2 print("a break was executed") print("for loop is done") I must admit the suggestion seems a little bit neater than having to manage a flag myself, but on the other hand I can't remember the last time I've needed to manage a flag like that. And on the gripping hand, this is even simpler than both alternatives: for i in (1, 2, 3): print(i) if i == 2: assert i == 2 print("a break was executed") break else: print("never reached") # this is never reached print("for loop is done") There are some significant unanswered questions: - Does it matter which order the for...except...else are in? Obviously the for block must come first, but apart from that? - How is this implemented? Currently "break" is a simple unconditional GOTO which jumps past the for block. This will need to change to something significantly more complex. - There are other ways to exit a for-loop than just break. Which of them, if any, will also run the except block? -- Steve
On 01.03.2017 12:56, Steven D'Aprano wrote:
one way to implement this with unconditional GOTOs would be (in pseudocode): LOOP: on break GOTO EXCEPT ELSE: ... GOTO THEN EXCEPT: ... THEN: ... So at the byte-code level (but only there) the order of except and else would be reversed. Was that a reason why you were asking about the order of except and else in my proposal? Anyway, I'm sure there are people much more skilled at compiler programming than me here.
Much snippage; apologies, Wolfgang! On 01/03/17 09:37, Wolfgang Maier wrote:
It reads worse to me, I'm afraid. Moving the "print" disassociates it from the condition that caused it, making it that bit harder to understand. You'd have a more compelling case with a complex loop with multiple breaks all requiring identical processing. However my experience is that such cases are rare, and are usually attempts to do exception handling with out actually using exceptions. I'm not terribly inclined to help people make more work for themselves.
That is a better use case. I must admit I normally handle this sort of thing by putting the loops in a function and returning out of the inner loop. -- Rhodri James *-* Kynesim Ltd
I’d like to see some examples where nested for loops couldn’t easily be avoided in the first place.
Replace the inner loop with a call to any consuming a generator for n in range(2,10): if any(n % x == 0 for x in range(2,n)): print('{} equals {} * {}'.format(n, x, n//x)) else: print('{} is prime'.format(n))
Replace the inner loop with a call to str.find for i, c in enumerate(s): j = s.find(c, i+1) if j >= 0: answer = (i, j) break
On 03/01/2017 01:37 AM, Wolfgang Maier wrote:
Now here's the proposal: allow an except (or except break) clause to follow for/while loops that will be executed if the loop was terminated by a break statement.
I find the proposal interesting. More importantly, the proposal is well written and clear -- thank you! -- ~Ethan~
On 1 March 2017 at 19:37, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
I never really thought about it, as I only use the "else:" clause for search loops where there aren't any side effects in the "break" case (other than the search result being bound to the loop variable), so while I find "except break:" useful as an explanatory tool, I don't have any practical need for it. I think you've made as strong a case for the idea as could reasonably be made :) However, Steven raises a good point that this would complicate the handling of loops in the code generator a fair bit, as it would add up to two additional jump targets in cases wherever the new clause was used. Currently, compiling loops only needs to track the start of the loop (for continue), and the first instruction after the loop (for break). With this change, they'd also need to track: - the start of the "except break" clause (for break when the clause is used) - the start of the "else" clause (for the non-break case when both trailing clauses are present) The design level argument against adding the clause is that it breaks the "one obvious way" principle, as the preferred form for search loops look like this: for item in iterable: if condition(item): break else: # Else clause either raises an exception or sets a default value item = get_default_value() # If we get here, we know "item" is a valid reference operation(item) And you can easily switch the `break` out for a suitable `return` if you move this into a helper function: def find_item_of_interest(iterable): for item in iterable: if condition(item): return item # The early return means we can skip using "else" return get_default_value() Given that basic structure as a foundation, you only switch to the "nested side effect" form if you have to: for item in iterable: if condition(item): operation(item) break else: # Else clause neither raises an exception nor sets a default value condition_was_never_true(iterable) This form is generally less amenable to being extracted into a reusable helper function, since it couples the search loop directly to the operation performed on the bound item, whereas decoupling them gives you a lot more flexibility in the eventual code structure. The proposal in this thread then has the significant downside of only covering the "nested side effect" case: for item in iterable: if condition(item): break except break: operation(item) else: condition_was_never_true(iterable) While being even *less* amenable to being pushed down into a helper function (since converting the "break" to a "return" would bypass the "except break" clause). So while it is cool to see this written up as a concrete proposal (thank you!), I don't think it makes the grade as an actual potential syntax change. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 02.03.2017 06:46, Nick Coghlan wrote:
I think you could get away with only one additional jump target as I showed in my previous reply to Steven. The heavier burden would be on the parser, which would have to distinguish the existing and the two new loop variants (loop with except clause, loop with except and else clause) but, anyway, that's probably not really the point. What weighs heavier, I think, is your design argument.
I'm actually not quite buying this last argument. If you wanted to refactor this to "return" instead of "break", you could simply put the return into the except break block. In many real-world situations with multiple breaks from a loop this could actually make things easier instead of worse. Personally, the "nested side effect" form makes me uncomfortable every time I use it because the side effects on breaking or not breaking the loop don't end up at the same indentation level and not necessarily together. However, I'm gathering from the discussion so far that not too many people are thinking like me about this point, so maybe I should simply adjust my mind-set. All that said, this is a very nice abstract view on things! I really learned quite a bit from this, thank you :) As always though, reality can be expected to be quite a bit more complicated than theory so I decided to check the stdlib for real uses of break. This is quite a tedious task since break is used in many different ways and I couldn't come up with a good automated way of classifying them. So what I did is just go through stdlib code (in reverse alphabetical order) containing the break keyword and put it into categories manually. I only got up to socket.py before losing my enthusiasm, but here's what I found: - overall I looked at 114 code blocks that contain one or more breaks - 84 of these are trivial use cases that simply break out of a while True block or terminate a while/for loop prematurely (no use for any follow-up clause there) - 8 more are causing a side-effect before a single break, and it would be pointless to put this into an except break clause - 3 more cause different, non-redundant side-effects before different breaks from the same loop and, obviously, an except break clause would not help them either => So the vast majority of breaks does *not* need an except break *nor* an else clause, but that's just as expected. Of the remaining 19 non-trivial cases - 9 are variations of your classical search idiom above, i.e., there's an else clause there and nothing more is needed - 6 are variations of your "nested side-effects" form presented above with debatable (see above) benefit from except break - 2 do not use an else clause currently, but have multiple breaks that do partly redundant things that could be combined in a single except break clause - 1 is an example of breaking out of two loops; from sre_parse._parse_sub: [...] # check if all items share a common prefix while True: prefix = None for item in items: if not item: break if prefix is None: prefix = item[0] elif item[0] != prefix: break else: # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) continue # check next one break [...] This could have been written as: [...] # check if all items share a common prefix while True: prefix = None for item in items: if not item: break if prefix is None: prefix = item[0] elif item[0] != prefix: break except break: break # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) [...] - finally, 1 is a complicated break dance to achieve sth that clearly would have been easier with except break; from typing.py: [...] def __subclasscheck__(self, cls): if cls is Any: return True if isinstance(cls, GenericMeta): # For a class C(Generic[T]) where T is co-variant, # C[X] is a subclass of C[Y] iff X is a subclass of Y. origin = self.__origin__ if origin is not None and origin is cls.__origin__: assert len(self.__args__) == len(origin.__parameters__) assert len(cls.__args__) == len(origin.__parameters__) for p_self, p_cls, p_origin in zip(self.__args__, cls.__args__, origin.__parameters__): if isinstance(p_origin, TypeVar): if p_origin.__covariant__: # Covariant -- p_cls must be a subclass of p_self. if not issubclass(p_cls, p_self): break elif p_origin.__contravariant__: # Contravariant. I think it's the opposite. :-) if not issubclass(p_self, p_cls): break else: # Invariant -- p_cls and p_self must equal. if p_self != p_cls: break else: # If the origin's parameter is not a typevar, # insist on invariance. if p_self != p_cls: break else: return True # If we break out of the loop, the superclass gets a chance. if super().__subclasscheck__(cls): return True if self.__extra__ is None or isinstance(cls, GenericMeta): return False return issubclass(cls, self.__extra__) [...] which could be rewritten as: [...] def __subclasscheck__(self, cls): if cls is Any: return True if isinstance(cls, GenericMeta): # For a class C(Generic[T]) where T is co-variant, # C[X] is a subclass of C[Y] iff X is a subclass of Y. origin = self.__origin__ if origin is not None and origin is cls.__origin__: assert len(self.__args__) == len(origin.__parameters__) assert len(cls.__args__) == len(origin.__parameters__) for p_self, p_cls, p_origin in zip(self.__args__, cls.__args__, origin.__parameters__): if isinstance(p_origin, TypeVar): if p_origin.__covariant__: # Covariant -- p_cls must be a subclass of p_self. if not issubclass(p_cls, p_self): break elif p_origin.__contravariant__: # Contravariant. I think it's the opposite. :-) if not issubclass(p_self, p_cls): break else: # Invariant -- p_cls and p_self must equal. if p_self != p_cls: break else: # If the origin's parameter is not a typevar, # insist on invariance. if p_self != p_cls: break except break: # If we break out of the loop, the superclass gets a chance. if super().__subclasscheck__(cls): return True if self.__extra__ is None or isinstance(cls, GenericMeta): return False return issubclass(cls, self.__extra__) return True [...] My summary: I do see use-cases for the except break clause, but, admittedly, they are relatively rare and may be not worth the hassle of introducing new syntax.
On Thu, 2 Mar 2017 at 03:07 Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote: [SNIP]
I wanted to say thanks for taking the time to go through the stdlib and doing such a thorough analysis of the impact of your suggestion! It always helps to have real-world numbers to know whether an idea will be useful (or not).
IOW out of 114 cases, 4 may benefit from an 'except' block? If I'm reading those numbers correctly then ~3.5% of cases would benefit which isn't high enough to add the syntax and related complexity IMO.
On 03/02/2017 07:05 PM, Brett Cannon wrote:
[...]
Hmm, I'm not sure how much sense it makes to express this in percent since the total your comparing to is rather arbitrary. The 114 cases include *any* for/while loop I could find that contains at least a single break. More than 90 of these loops do not use an "else" clause either showing that even this currently supported syntax is used rarely. I found only 19 cases that are complex enough to be candidates for an except clause (17 of these use the else clause). For 9 of these 19 (the ones using the classical search idiom) an except clause would not be applicable, but it could be used in the 10 remaining cases (though all of them could also make use of a flag or could be refactored instead). So depending on what you want to emphasize you could also say that the proposal could affect as much as 10/19 or 52.6% of cases.
On 2 March 2017 at 21:06, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
Fair point - so that would be even with the "single nested side effect" case, but simpler when you had multiple break conditions (and weren't already combined them with "and").
This is why I consider the "search only" form of the loop, where the else clause either sets a default value, or else prevents execution of the code after the loop body (via raise, return, or continue), to be the preferred form: there aren't any meaningful side effects hidden away next to the break statement. If I can't do that, I'm more likely to switch to a classic flag variable that gets checked post-loop execution than I am to push the side effect inside the loop body: search_result = _not_found = object() for item in iterable: if condition(item): search_result = item break if search_result is _not_found: # Handle the "not found" case else: # Handle the "found" case
Thanks for doing that research :)
Those 8 cases could also be reviewed to see whether a flag variable might be clearer than relying on nested side effects or code repetition.
This is a case where a flag variable may be easier to read than loop state manipulations: may_have_common_prefix = True while may_have_common_prefix: prefix = None for item in items: if not item: may_have_common_prefix = False break if prefix is None: prefix = item[0] elif item[0] != prefix: may_have_common_prefix = False break else: # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) Although the whole thing could likely be cleaned up even more via itertools.zip_longest: for first_uncommon_idx, aligned_entries in enumerate(itertools.zip_longest(*items)): if not all_true_and_same(aligned_entries): break else: # Everything was common, so clear all entries first_uncommon_idx = None for item in items: del item[:first_uncommon_idx] (Batching the deletes like that may even be slightly faster than deleting common entries one at a time) Given the following helper function: def all_true_and_same(entries): itr = iter(entries) try: first_entry = next(itr) except StopIteration: return False if not first_entry: return False for entry in itr: if not entry or entry != first_entry: return False return True
I think is another case that is asking for the inner loop to be factored out to a named function, not for reasons of re-use, but for reasons of making the code more readable and self-documenting :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/03/2017 04:36 AM, Nick Coghlan wrote:
[...]
[...]
It's true that using a flag or factoring out redundant code is always a possibility. Having the except clause would clearly not let people do anything they couldn't have done before. On the other hand, the same is true for the else clause - it's only advantage here is that it's existing already - because a single flag could always distinguish between a break having occurred or not: brk = False for item in iterable: if some_condition: brk = True break if brk: do_stuff_upon_breaking_out() else: do_alternative_stuff() is a general pattern that would always work without except *and* else. However, the fact that else exists generates a regrettable asymmetry in that there is direct language support for detecting one outcome, but not the other. Stressing the analogy to try/except/else one more time, it's as if "else" wasn't available for try blocks. You could always use a flag to substitute for it: dealt_with_exception = False try: do_stuff() except: deal_with_exception() dealt_with_exception = True if dealt_with_exception: do_stuff_you_would_do_in_an_else_block() So IMO the real difference here is that the except clause after for would require adding it to the language, while the else clauses are there already. With that we're back at the high bar for adding new syntax :( A somewhat similar case that comes to mind here is PEP 315 -- Enhanced While Loop, which got rejected for two reasons, the first one being pretty much the same as the argument here, i.e., that instead of the proposed do .. while it's always possible to factor out or duplicate a line of code. However, the second reason was that it required the new "do" keyword, something not necessary for the current suggestion.
On 3 March 2017 at 18:47, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
I forget where it came up, but I seem to recall Guido saying that if he were designing Python today, he wouldn't include the "else:" clause on loops, since it inevitably confuses folks the first time they see it. (Hence articles like mine that attempt to link it with try/except/else rather than if/else).
It's worth noting that this asymmetry doesn't necessarily exist in the corresponding C idiom that I assume was the inspiration for the Python equivalent: int data_array_len = sizeof(data_array) / sizeof(data_array[0]); in idx = 0; for (idx = 0; idx < data_array_len; idx++) { if (condition(container[idx])) { break; } } if (idx < data_array_len) { // We found a relevant entry } else { // We didn't find anything } In Python prior to 2.1 (when PEP 234 added the iterator protocol), a similar approach could be used for Python's for loops: num_items = len(container): for idx in range(num_items): if condition(container[idx]): break if num_items and idx < num_items: # We found a relevant entry else: # We didn't find anything However, while my own experience with Python is mainly with 2.2+ (and hence largely after the era where "for i in range(len(container)):" was still common), I've spent a lot of time working with C and the corresponding iterator protocols in C++, and there it is pretty common to move the "entry found" code before the break and then invert the conditional check that appears after the loop: int data_array_len = sizeof(data_array) / sizeof(data_array[0]); int idx = 0; for (idx = 0; idx < data_array_len; idx++) { if (condition(container[idx])) { // We found a relevant entry break; } } if (idx >= data_array_len) { // We didn't find anything } And it's *this* version of the C/C++ idiom that Python's "else:" clause replicates. One key aspect of this particular idiomatic structure is that it retains the same overall shape regardless of whether the inner structure is: if condition(item): # Condition is true, so process the item process(item) break or: if maybe_process_item(item): # Item was processed, so we're done here break Whereas the "post-processing" model can't handle pre-composed helper functions that implement both the conditional check and the item processing, and then report back which branch they took. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Mar 05, 2017 at 01:17:31PM +1000, Nick Coghlan wrote:
Heh, if we exclude all features that confuse people the first time they see it, we'd have to remove threads, Unicode, floating point maths, calls to external processes, anything OS-dependent, metaclasses, classes, ... :-) It took me the longest time to realise that the "else" clause didn't *only* run when the loop sequence is empty. That is, I expected that given: for x in random.choice(["", "a"]): # either empty, or a single item print("run the loop body") else: print("loop sequence is empty") That's because it *seems* to work that way, if you do insufficient testing: for x in []: raise ValueError # dead code is not executed else: print("loop sequence is empty") It is my belief that the problem here is not the else clause itself, but that the name used is misleading. I've seen people other than myself conclude that it means: run the for-loop over the sequence otherwise the sequence is empty, run the ELSE block I've seen people think that it means: set break_seen flag to false run the for-loop if break is executed, set the break_seen flat to true then break if break_seen is false, run the "ELSE NOT BREAK" clause and consequently ask how they can access the break_seen flag for themselves. Presumably they want to write something like: run the for-loop if break_seen is true, do this else (break_seen is false) do that I think that the name "else" here is a "misunderstanding magnet", it leads people to misunderstand the nature of the clause and its implications. For example, I bet that right now there are people reading this and nodding along with me and thinking "maybe we should rename it something more explicit, like "else if no break", completely oblivious to the fact that `break` is NOT the only way to avoid running the `else` clause. I believe that the name should have been "then", not "else". It describes what the code does: run the for-block THEN run the "else" block There's no flag to be tested, and the "else" block simply runs once, after the for-loop, regardless of whether the for-loop runs once or ten times or zero times (empty sequence). To avoid running the "else" ("then") block, you have to exit the entire block of code using: - break - return - raise which will all transfer execution past the end of the for...else (for...then) compound statement. Since I realised that the else block merely runs directly after the for, I've never had any problem with the concept. `break` merely jumps past the for...else block, just as `return` exits the function and `raise` triggers an exception which transfers execution to the surrounding `except` clause. -- Steve
On 3/4/2017 10:17 PM, Nick Coghlan wrote:
(Hence articles like mine that attempt to link it with try/except/else rather than if/else).
The link needs to be done in two separate steps: if-else => while-else => for-else. Step 1. The conceptual difference between an if-clause and a while-clause is that an if-clause ends with an implied 'break' (jump past the rest of the statement to the next statement) while a while-clause ends with an implied 'continue' (jump back to the condition test). In the CPython byte code, this difference is implemented by JUMP_FORWARD versus JUMP_ABSOLUTE (backwards). (The byte code for a while-clause also has two bookkeeping additions, SETUP_LOOP and POP_BLOCK.) In both if-statements and while-statements, the else-clause is executed if and when the condition is false. I wonder if having never programming with with 'jump' or 'goto' makes this harder to understand. I think our doc could better explain how if-statements and while-statements are similar but different. Step 2. A for-loop can be viewed as an initialized while-loop. Kernighan and Ritchie make this explicit in The C Programming Language (p. 56). ''' The for statement for (expr1; expr2; expr3) statement is equivalent to expr1; while (expr2) { statement expr3; } ''' For Python's more specialized for-loop, 'for target-list in iterable: for-suite' can be paraphrased as 'while the iterator yields an object (is not exhausted): do the assignment and suite'. An else-clause, if present, is executed if and when the condition is false, when the iterator is exhausted and raises StopIteration instead of yielding an object. If while-else is understood and the implicit for condition is understood, for-else is pretty straightforward. Equivalent code, with an else-clause, is trickier than in C, without for-else. I think the following is close. I think something like this should be in the doc. _it = iter(iterable) _exhausted = False while not _exhausted: try: _obj = next(_it) except StopIteration: _exhausted = True continue target-list = _obj for-suite else: else-suite Note 1: without the else-clause, _exhausted is not needed. The loop could be 'while True' and the except clause could just be 'break'. Note 2: _obj being assignment compatible with target list is NOT part of the implicit while condition. For example, for a,b in (1,2): print('for') else: print('else') prints nothing except the traceback for TypeError: 'int' object is not iterable. Note 3: C for typically assigns to the loop variable once before the loop and and again at the end of each loop. Python for does the assignment once at the top of the loop. -- Terry Jan Reedy
On 1 March 2017 at 06:37, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
After rethinking over some code I've written in the past, yes, I agree this change could be a nice one. The simple fact that people are commenting that they could chage the code to be an inner function in order to break from nested "for" loops should be a hint this syntax is useful. (I myself have done that with exceptions in some cases). js -><-
On 3/1/17, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
I was thinking bout this analogy: 1. try/else (without except) is SyntaxError. And seems useless. 2. try/break/except is backward compatible: for i in L: try: break except Something: pass except break: # current code has not this so break is applied to for-block 3. for/raise/except (which is natural application of this analogy) could reduce indentation but in my personal view that don't improve readability (but I could be wrong) It could help enhance "break" possibilities so "simplify" double break in nested loops. for broken = False for if condition1(): # I like to "double break" here raise SomeError() if condition2(): break except SomeError: break except break: broken = True 4. for/finally may be useful
participants (11)
-
Brett Cannon -
Clint Hepner -
Ethan Furman -
Joao S. O. Bueno -
Nick Coghlan -
Pavol Lisy -
Rhodri James -
Steven D'Aprano -
Sven R. Kunze -
Terry Reedy -
Wolfgang Maier