On 02.03.2017 06:46, Nick Coghlan wrote:

On 1 March 2017 at 19:37, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de <mailto:wolfgang.maier@biologie.uni-freiburg.de>> wrote:

Now here's the proposal: allow an except (or except break) clause to follow for/while loops that will be executed if the loop was terminated by a break statement.

Now while it's possible that Nick had a good reason not to do so,

I never really thought about it, as I only use the "else:" clause for search loops where there aren't any side effects in the "break" case (other than the search result being bound to the loop variable), so while I find "except break:" useful as an explanatory tool, I don't have any practical need for it.

I think you've made as strong a case for the idea as could reasonably be made :)

However, Steven raises a good point that this would complicate the handling of loops in the code generator a fair bit, as it would add up to two additional jump targets in cases wherever the new clause was used.

Currently, compiling loops only needs to track the start of the loop (for continue), and the first instruction after the loop (for break). With this change, they'd also need to track:

- the start of the "except break" clause (for break when the clause is used) - the start of the "else" clause (for the non-break case when both trailing clauses are present)

I think you could get away with only one additional jump target as I showed in my previous reply to Steven. The heavier burden would be on the parser, which would have to distinguish the existing and the two new loop variants (loop with except clause, loop with except and else clause) but, anyway, that's probably not really the point. What weighs heavier, I think, is your design argument.

The design level argument against adding the clause is that it breaks the "one obvious way" principle, as the preferred form for search loops look like this:

for item in iterable: if condition(item): break else: # Else clause either raises an exception or sets a default value item = get_default_value()

# If we get here, we know "item" is a valid reference operation(item)

And you can easily switch the `break` out for a suitable `return` if you move this into a helper function:

def find_item_of_interest(iterable): for item in iterable: if condition(item): return item # The early return means we can skip using "else" return get_default_value()

Given that basic structure as a foundation, you only switch to the "nested side effect" form if you have to:

for item in iterable: if condition(item): operation(item) break else: # Else clause neither raises an exception nor sets a default value condition_was_never_true(iterable)

This form is generally less amenable to being extracted into a reusable helper function, since it couples the search loop directly to the operation performed on the bound item, whereas decoupling them gives you a lot more flexibility in the eventual code structure.

The proposal in this thread then has the significant downside of only covering the "nested side effect" case:

for item in iterable: if condition(item): break except break: operation(item) else: condition_was_never_true(iterable)

While being even *less* amenable to being pushed down into a helper function (since converting the "break" to a "return" would bypass the "except break" clause).

I'm actually not quite buying this last argument. If you wanted to refactor this to "return" instead of "break", you could simply put the return into the except break block. In many real-world situations with multiple breaks from a loop this could actually make things easier instead of worse. Personally, the "nested side effect" form makes me uncomfortable every time I use it because the side effects on breaking or not breaking the loop don't end up at the same indentation level and not necessarily together. However, I'm gathering from the discussion so far that not too many people are thinking like me about this point, so maybe I should simply adjust my mind-set. All that said, this is a very nice abstract view on things! I really learned quite a bit from this, thank you :) As always though, reality can be expected to be quite a bit more complicated than theory so I decided to check the stdlib for real uses of break. This is quite a tedious task since break is used in many different ways and I couldn't come up with a good automated way of classifying them. So what I did is just go through stdlib code (in reverse alphabetical order) containing the break keyword and put it into categories manually. I only got up to socket.py before losing my enthusiasm, but here's what I found: - overall I looked at 114 code blocks that contain one or more breaks - 84 of these are trivial use cases that simply break out of a while True block or terminate a while/for loop prematurely (no use for any follow-up clause there) - 8 more are causing a side-effect before a single break, and it would be pointless to put this into an except break clause - 3 more cause different, non-redundant side-effects before different breaks from the same loop and, obviously, an except break clause would not help them either => So the vast majority of breaks does *not* need an except break *nor* an else clause, but that's just as expected. Of the remaining 19 non-trivial cases - 9 are variations of your classical search idiom above, i.e., there's an else clause there and nothing more is needed - 6 are variations of your "nested side-effects" form presented above with debatable (see above) benefit from except break - 2 do not use an else clause currently, but have multiple breaks that do partly redundant things that could be combined in a single except break clause - 1 is an example of breaking out of two loops; from sre_parse._parse_sub: [...] # check if all items share a common prefix while True: prefix = None for item in items: if not item: break if prefix is None: prefix = item[0] elif item[0] != prefix: break else: # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) continue # check next one break [...] This could have been written as: [...] # check if all items share a common prefix while True: prefix = None for item in items: if not item: break if prefix is None: prefix = item[0] elif item[0] != prefix: break except break: break # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix) [...] - finally, 1 is a complicated break dance to achieve sth that clearly would have been easier with except break; from typing.py: [...] def __subclasscheck__(self, cls): if cls is Any: return True if isinstance(cls, GenericMeta): # For a class C(Generic[T]) where T is co-variant, # C[X] is a subclass of C[Y] iff X is a subclass of Y. origin = self.__origin__ if origin is not None and origin is cls.__origin__: assert len(self.__args__) == len(origin.__parameters__) assert len(cls.__args__) == len(origin.__parameters__) for p_self, p_cls, p_origin in zip(self.__args__, cls.__args__, origin.__parameters__): if isinstance(p_origin, TypeVar): if p_origin.__covariant__: # Covariant -- p_cls must be a subclass of p_self. if not issubclass(p_cls, p_self): break elif p_origin.__contravariant__: # Contravariant. I think it's the opposite. :-) if not issubclass(p_self, p_cls): break else: # Invariant -- p_cls and p_self must equal. if p_self != p_cls: break else: # If the origin's parameter is not a typevar, # insist on invariance. if p_self != p_cls: break else: return True # If we break out of the loop, the superclass gets a chance. if super().__subclasscheck__(cls): return True if self.__extra__ is None or isinstance(cls, GenericMeta): return False return issubclass(cls, self.__extra__) [...] which could be rewritten as: [...] def __subclasscheck__(self, cls): if cls is Any: return True if isinstance(cls, GenericMeta): # For a class C(Generic[T]) where T is co-variant, # C[X] is a subclass of C[Y] iff X is a subclass of Y. origin = self.__origin__ if origin is not None and origin is cls.__origin__: assert len(self.__args__) == len(origin.__parameters__) assert len(cls.__args__) == len(origin.__parameters__) for p_self, p_cls, p_origin in zip(self.__args__, cls.__args__, origin.__parameters__): if isinstance(p_origin, TypeVar): if p_origin.__covariant__: # Covariant -- p_cls must be a subclass of p_self. if not issubclass(p_cls, p_self): break elif p_origin.__contravariant__: # Contravariant. I think it's the opposite. :-) if not issubclass(p_self, p_cls): break else: # Invariant -- p_cls and p_self must equal. if p_self != p_cls: break else: # If the origin's parameter is not a typevar, # insist on invariance. if p_self != p_cls: break except break: # If we break out of the loop, the superclass gets a chance. if super().__subclasscheck__(cls): return True if self.__extra__ is None or isinstance(cls, GenericMeta): return False return issubclass(cls, self.__extra__) return True [...] My summary: I do see use-cases for the except break clause, but, admittedly, they are relatively rare and may be not worth the hassle of introducing new syntax.

On 03/02/2017 07:05 PM, Brett Cannon wrote:

- overall I looked at 114 code blocks that contain one or more breaks

I wanted to say thanks for taking the time to go through the stdlib and doing such a thorough analysis of the impact of your suggestion! It always helps to have real-world numbers to know whether an idea will be useful (or not).

- 84 of these are trivial use cases that simply break out of a while True block or terminate a while/for loop prematurely (no use for any follow-up clause there)

- 8 more are causing a side-effect before a single break, and it would be pointless to put this into an except break clause

- 3 more cause different, non-redundant side-effects before different breaks from the same loop and, obviously, an except break clause would not help them either

=> So the vast majority of breaks does *not* need an except break *nor* an else clause, but that's just as expected.

Of the remaining 19 non-trivial cases

- 9 are variations of your classical search idiom above, i.e., there's an else clause there and nothing more is needed

- 6 are variations of your "nested side-effects" form presented above with debatable (see above) benefit from except break

- 2 do not use an else clause currently, but have multiple breaks that do partly redundant things that could be combined in a single except break clause

- 1 is an example of breaking out of two loops; from sre_parse._parse_sub:

[...]

- finally, 1 is a complicated break dance to achieve sth that clearly would have been easier with except break; from typing.py:

My summary: I do see use-cases for the except break clause, but, admittedly, they are relatively rare and may be not worth the hassle of introducing new syntax.

IOW out of 114 cases, 4 may benefit from an 'except' block? If I'm reading those numbers correctly then ~3.5% of cases would benefit which isn't high enough to add the syntax and related complexity IMO.

Hmm, I'm not sure how much sense it makes to express this in percent since the total your comparing to is rather arbitrary. The 114 cases include *any* for/while loop I could find that contains at least a single break. More than 90 of these loops do not use an "else" clause either showing that even this currently supported syntax is used rarely. I found only 19 cases that are complex enough to be candidates for an except clause (17 of these use the else clause). For 9 of these 19 (the ones using the classical search idiom) an except clause would not be applicable, but it could be used in the 10 remaining cases (though all of them could also make use of a flag or could be refactored instead). So depending on what you want to emphasize you could also say that the proposal could affect as much as 10/19 or 52.6% of cases.

On 03/03/2017 04:36 AM, Nick Coghlan wrote:

On 2 March 2017 at 21:06, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de <mailto:wolfgang.maier@biologie.uni-freiburg.de>> wrote:

...

- overall I looked at 114 code blocks that contain one or more breaks

Thanks for doing that research :)

...

Of the remaining 19 non-trivial cases

- 9 are variations of your classical search idiom above, i.e., there's an else clause there and nothing more is needed

- 6 are variations of your "nested side-effects" form presented above with debatable (see above) benefit from except break

- 2 do not use an else clause currently, but have multiple breaks that do partly redundant things that could be combined in a single except break clause

Those 8 cases could also be reviewed to see whether a flag variable might be clearer than relying on nested side effects or code repetition.

[...]

This is a case where a flag variable may be easier to read than loop state manipulations:

may_have_common_prefix = True while may_have_common_prefix: prefix = None for item in items: if not item: may_have_common_prefix = False break if prefix is None: prefix = item[0] elif item[0] != prefix: may_have_common_prefix = False break else: # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix)

Although the whole thing could likely be cleaned up even more via itertools.zip_longest:

for first_uncommon_idx, aligned_entries in enumerate(itertools.zip_longest(*items)): if not all_true_and_same(aligned_entries): break else: # Everything was common, so clear all entries first_uncommon_idx = None for item in items: del item[:first_uncommon_idx]

(Batching the deletes like that may even be slightly faster than deleting common entries one at a time)

Given the following helper function:

def all_true_and_same(entries): itr = iter(entries) try: first_entry = next(itr) except StopIteration: return False if not first_entry: return False for entry in itr: if not entry or entry != first_entry: return False return True

...

- finally, 1 is a complicated break dance to achieve sth that clearly would have been easier with except break; from typing.py:

[...]

I think is another case that is asking for the inner loop to be factored out to a named function, not for reasons of re-use, but for reasons of making the code more readable and self-documenting :)

It's true that using a flag or factoring out redundant code is always a possibility. Having the except clause would clearly not let people do anything they couldn't have done before. On the other hand, the same is true for the else clause - it's only advantage here is that it's existing already - because a single flag could always distinguish between a break having occurred or not: brk = False for item in iterable: if some_condition: brk = True break if brk: do_stuff_upon_breaking_out() else: do_alternative_stuff() is a general pattern that would always work without except *and* else. However, the fact that else exists generates a regrettable asymmetry in that there is direct language support for detecting one outcome, but not the other. Stressing the analogy to try/except/else one more time, it's as if "else" wasn't available for try blocks. You could always use a flag to substitute for it: dealt_with_exception = False try: do_stuff() except: deal_with_exception() dealt_with_exception = True if dealt_with_exception: do_stuff_you_would_do_in_an_else_block() So IMO the real difference here is that the except clause after for would require adding it to the language, while the else clauses are there already. With that we're back at the high bar for adding new syntax :( A somewhat similar case that comes to mind here is PEP 315 -- Enhanced While Loop, which got rejected for two reasons, the first one being pretty much the same as the argument here, i.e., that instead of the proposed do .. while it's always possible to factor out or duplicate a line of code. However, the second reason was that it required the new "do" keyword, something not necessary for the current suggestion.

On 3 March 2017 at 18:47, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:

On 03/03/2017 04:36 AM, Nick Coghlan wrote:

...

On 2 March 2017 at 21:06, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de <mailto:wolfgang.maier@biologie.uni-freiburg.de>> wrote:

- overall I looked at 114 code blocks that contain one or more breaks

...

Thanks for doing that research :)

Of the remaining 19 non-trivial cases

...

- 9 are variations of your classical search idiom above, i.e., there's an else clause there and nothing more is needed

- 6 are variations of your "nested side-effects" form presented above with debatable (see above) benefit from except break

- 2 do not use an else clause currently, but have multiple breaks that do partly redundant things that could be combined in a single except break clause

Those 8 cases could also be reviewed to see whether a flag variable might be clearer than relying on nested side effects or code repetition.

[...]

...

This is a case where a flag variable may be easier to read than loop state manipulations:

may_have_common_prefix = True while may_have_common_prefix: prefix = None for item in items: if not item: may_have_common_prefix = False break if prefix is None: prefix = item[0] elif item[0] != prefix: may_have_common_prefix = False break else: # all subitems start with a common "prefix". # move it out of the branch for item in items: del item[0] subpatternappend(prefix)

Although the whole thing could likely be cleaned up even more via itertools.zip_longest:

for first_uncommon_idx, aligned_entries in enumerate(itertools.zip_longest(*items)): if not all_true_and_same(aligned_entries): break else: # Everything was common, so clear all entries first_uncommon_idx = None for item in items: del item[:first_uncommon_idx]

(Batching the deletes like that may even be slightly faster than deleting common entries one at a time)

Given the following helper function:

def all_true_and_same(entries): itr = iter(entries) try: first_entry = next(itr) except StopIteration: return False if not first_entry: return False for entry in itr: if not entry or entry != first_entry: return False return True

- finally, 1 is a complicated break dance to achieve sth that

...

clearly would have been easier with except break; from typing.py:

[...]

...

I think is another case that is asking for the inner loop to be factored out to a named function, not for reasons of re-use, but for reasons of making the code more readable and self-documenting :)

It's true that using a flag or factoring out redundant code is always a possibility. Having the except clause would clearly not let people do anything they couldn't have done before. On the other hand, the same is true for the else clause - it's only advantage here is that it's existing already

I forget where it came up, but I seem to recall Guido saying that if he were designing Python today, he wouldn't include the "else:" clause on loops, since it inevitably confuses folks the first time they see it. (Hence articles like mine that attempt to link it with try/except/else rather than if/else).

- because a single flag could always distinguish between a break having occurred or not:

brk = False for item in iterable: if some_condition: brk = True break if brk: do_stuff_upon_breaking_out() else: do_alternative_stuff()

is a general pattern that would always work without except *and* else.

However, the fact that else exists generates a regrettable asymmetry in that there is direct language support for detecting one outcome, but not the other.

It's worth noting that this asymmetry doesn't necessarily exist in the corresponding C idiom that I assume was the inspiration for the Python equivalent: int data_array_len = sizeof(data_array) / sizeof(data_array[0]); in idx = 0; for (idx = 0; idx < data_array_len; idx++) { if (condition(container[idx])) { break; } } if (idx < data_array_len) { // We found a relevant entry } else { // We didn't find anything } In Python prior to 2.1 (when PEP 234 added the iterator protocol), a similar approach could be used for Python's for loops: num_items = len(container): for idx in range(num_items): if condition(container[idx]): break if num_items and idx < num_items: # We found a relevant entry else: # We didn't find anything However, while my own experience with Python is mainly with 2.2+ (and hence largely after the era where "for i in range(len(container)):" was still common), I've spent a lot of time working with C and the corresponding iterator protocols in C++, and there it is pretty common to move the "entry found" code before the break and then invert the conditional check that appears after the loop: int data_array_len = sizeof(data_array) / sizeof(data_array[0]); int idx = 0; for (idx = 0; idx < data_array_len; idx++) { if (condition(container[idx])) { // We found a relevant entry break; } } if (idx >= data_array_len) { // We didn't find anything } And it's *this* version of the C/C++ idiom that Python's "else:" clause replicates. One key aspect of this particular idiomatic structure is that it retains the same overall shape regardless of whether the inner structure is: if condition(item): # Condition is true, so process the item process(item) break or: if maybe_process_item(item): # Item was processed, so we're done here break Whereas the "post-processing" model can't handle pre-composed helper functions that implement both the conditional check and the item processing, and then report back which branch they took. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 3/4/2017 10:17 PM, Nick Coghlan wrote:

I forget where it came up, but I seem to recall Guido saying that if he were designing Python today, he wouldn't include the "else:" clause on loops, since it inevitably confuses folks the first time they see it.

(Hence articles like mine that attempt to link it with try/except/else rather than if/else).

The link needs to be done in two separate steps: if-else => while-else => for-else. Step 1. The conceptual difference between an if-clause and a while-clause is that an if-clause ends with an implied 'break' (jump past the rest of the statement to the next statement) while a while-clause ends with an implied 'continue' (jump back to the condition test). In the CPython byte code, this difference is implemented by JUMP_FORWARD versus JUMP_ABSOLUTE (backwards). (The byte code for a while-clause also has two bookkeeping additions, SETUP_LOOP and POP_BLOCK.) In both if-statements and while-statements, the else-clause is executed if and when the condition is false. I wonder if having never programming with with 'jump' or 'goto' makes this harder to understand. I think our doc could better explain how if-statements and while-statements are similar but different. Step 2. A for-loop can be viewed as an initialized while-loop. Kernighan and Ritchie make this explicit in The C Programming Language (p. 56). ''' The for statement for (expr1; expr2; expr3) statement is equivalent to expr1; while (expr2) { statement expr3; } ''' For Python's more specialized for-loop, 'for target-list in iterable: for-suite' can be paraphrased as 'while the iterator yields an object (is not exhausted): do the assignment and suite'. An else-clause, if present, is executed if and when the condition is false, when the iterator is exhausted and raises StopIteration instead of yielding an object. If while-else is understood and the implicit for condition is understood, for-else is pretty straightforward. Equivalent code, with an else-clause, is trickier than in C, without for-else. I think the following is close. I think something like this should be in the doc. _it = iter(iterable) _exhausted = False while not _exhausted: try: _obj = next(_it) except StopIteration: _exhausted = True continue target-list = _obj for-suite else: else-suite Note 1: without the else-clause, _exhausted is not needed. The loop could be 'while True' and the except clause could just be 'break'. Note 2: _obj being assignment compatible with target list is NOT part of the implicit while condition. For example, for a,b in (1,2): print('for') else: print('else') prints nothing except the traceback for TypeError: 'int' object is not iterable. Note 3: C for typically assigns to the loop variable once before the loop and and again at the end of each loop. Python for does the assignment once at the top of the loop. -- Terry Jan Reedy