Retrying EAFP without DRY
Just running another half-baked "easy" idea up to see what others think of it. One of the cases where EAFP is clumsy is if the "forgiveness" portion is actually retrying what failed after fixing things. I just ran into this in real code, and the resulting discussion suggested this change. Eiffel's exception-handling mechanism is built assuming that's the *standard* way to do things. Seems to work ok, but if I had to choose one of the two, I'll take try/except. So here's the proposal: A single new keyword/clause, "retry". It has the same syntax as an "except" clause, can be used anywhere "except" can be used, and can be intermingled with them in the same try statement. There's probably a better syntax, but this is easy to describe. The behavior change from except is that instead of exiting the "try" statement when the "retry" clause ends, it restarts the try clause. In python, this code: try: # try block except: # except block retry: # retry block else: # else block finally: # finally block Would provide the same behavior as this code: while True: try: # try block except: # except block retry: # retry block else: # else block finally: # finally block break Where except & retry can be repeated, include exceptions to check against, and intermixed with each other. The break would propagate up to the else block and any except blocks if there was no finally. If there's no else, the break from it winds up at the end of the try. The use case, as mentioned, is avoiding doing EAFP without repeating the code in the exception handler. I.e., the choices without retry are things like (LBYL): if not isinstance(x, my_class): x = fixup(x) mangle(x) other_stuff(x) or EAFP: try: mangle(x) except: x = fixup(x) mangle(x) other_stuff(x) or: while True: try: mangle(x) break except: x = fixup(x) other_stuff(x) with: try: mangle(x) retry: e = fixup(x) other_stuff(x) My gut reaction is cool, but not clear it's worth a new keyword. So maybe there's a tweak to the except clause that would have the same effect, but I don't see anything obvious. <mike
On Fri, Jan 20, 2012 at 4:36 PM, Mike Meyer <mwm@mired.org> wrote:
Can you write this in terms of current Python code? I don't understand exactly when a retry block would be executed. In Eiffel, retry is a statement, not a clause. Analogous to that would be: try: # block except: # block if condition: retry The equivalent of this in current Python is while True: try: # block except: # block if condition: continue # retry break FWIW, if this turns out to be a good idea, an alternate spelling that would not require a new keyword is 'try again'. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
Benjamin Peterson wrote:
"again" could be a not-actually-a-keyword keyword like "as" used to be. [steve@sylar ~]$ python2.3 Python 2.3.7 (#1, Aug 12 2010, 00:17:29) [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Not that I support that, but it is possible. -- Steven
On Sat, Jan 21, 2012 at 11:40 AM, Bruce Leban <bruce@leapyear.org> wrote:
And, indeed, I'd consider "while True" to be a fairly idiomatic way to write "keep trying this until it works (and we exit the loop via break) or we give up (and raise an exception)" without repeating ourselves. Alternatively, I'd push the entire try/except block into a closure and call that from the loop until it reported it had worked. A nested generator could achieve much the same thing. I don't think there needs to be "one obvious way" to do this, since it's a rare need and the existing tools (while loops with break/continue, a loop with an appropriate flag, a loop plus a closure, a nested generator, etc) all provide ways to implement it reasonably cleanly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, 2012-01-20 at 16:36 -0800, Mike Meyer wrote:
This looks like it would be very tricky to get right from a python programming stand point. Remember that python is full of one time iterators that won't repeat properly on a second try. Also many function calls mutate the arguments. So it's possible that a data structure could be changed on the first try, and give different results on a second try in an unexpected way. Cheers, Ron
On Fri, Jan 20, 2012 at 4:36 PM, Mike Meyer <mwm@mired.org> wrote:
I would be very interested if you give at least the general gist of the part of your code in question. Real-world use cases are always useful data.
Right, that has evil code duplication.
Personally, I would put the `break` in a separate `else` clause.
Your mention duplication as a gripe, yet there's no duplication in this version. I take then it that you just dislike this idiom?
Do I misunderstand your proposal, or are you missing an "except: retry" here? If you aren't, then how is infinite-retrying avoided? I would think that in many (most?) cases, we'd like to cap the number of reattempts. Currently, this looks like: for _ in range(5): try: buy_off_ebay(item, price) except OutbidError: price = highest_bid_for(item) * 1.05 # continue else: break # Some fiddling can be had over where to put the `break` # and whether to include an explicit `continue`. With what I understand `retry` would be: # Not even bothering with the capping try: buy_off_ebay(item, price) except OutbidError: retry retry: price = highest_bid_for(item) * 1.05 # I'd have to maintain the count myself?! # That's easily going to squander the 1 line this saved. I grant you that this has saved 1 level of indentation, whatever that's worth (personally, I don't put enough premium on it in this case to justify new syntax). It's also slightly more explicit (once you know what `retry` does, of course). And, in more general cases, a very few lines can be saved. But when one gets to the more general, complicated cases, it seems to me that you're better off just writing and making use of your own try_N_times(make_bid, N) [or whatever your "keep retrying" heuristic is] function, which argues against adding new syntax for this.
My view on Python's choice of control structures is that it provides a near-minimal, simple (so long as you overlook the obscure and seldom-used `else` clause of `for` & `while`) set of them that you can then combine to achieve fancier behavior (e.g. while + try + break = retry). For example, there's no do-while (instead: while True + if-break); there's no unless/until (instead: while/if + not); there's no C-style `for` (instead: for+range(), or a custom while); there's no `redo` (instead: nested while True + if-break/continue); and thus, no `retry` (instead: nested try within a loop). This keeps the core language small, aesthetically clean, uniform, and easy-to-learn. If one understands `if`, `while`, `for`, `try`, `break`, and `continue` (and `not`, but that's merely a boolean operator, as opposed to a control structure), then one can easily work out from scratch what the idioms do, and one learns to recognize them with just a bit of practice/experience. Also, it's easy to tweak the idioms when your control flow changes or becomes more complicated, thus allowing for flexibility. On the other hand, some clarity/explicitness is definitely lost. There's a design trade-off here; I don't think Python has tended towards the side that would add `retry`. Cheers, Chris -- http://rebertia.com
Mike Meyer wrote:
The behavior change from except is that instead of exiting the "try" statement when the "retry" clause ends, it restarts the try
This sounds like retry should be a flow-control statement, like continue or break, not a block. E.g.: try: something() except ValueError: if condition: retry else: raise "retry" will jump back to the start of the try block -- a limited form of GOTO, with all the pros and cons of this.
Um, you're trying to explain the behaviour of retry here, but your Python code includes a retry. This is one case where recursion is NOT your friend.
None of your explanation is clear to me. Under what circumstances will the retry block be executed? E.g. given: try: len(None) except TypeError: print("spam spam spam") retry: print("a") what will happen? How about these? #1 try: len(None) retry: print("a") except TypeError: print("spam spam spam") #2 try: len(None) except TypeError: print("spam spam spam") retry: print("a") except ValueError: print("this never gets called") retry: print("b") #3 try: len(None) except TypeError: print("spam spam spam") retry: print("a") retry: print("b")
I think that the idiom of a while or for loop is easy enough to read and write: for _ in range(5): # retry five times try: do_something(x) except SpamError: x = fix_up(x) else: break else: raise HamError("tried 5 times, giving up now") I just wish that break and continue could be written outside of a loop, so you can factor out common code: def do_the_thing(x): try: do_something(x) except SpamError: x = fix_up(x) else: break def try_repeatedly(n, func): for _ in range(n): func() else: raise HamError('tried %d times, giving up now" % n) try_repeatedly(5, do_the_thing) -- Steven
On Fri, Jan 20, 2012 at 11:47 PM, Steven D'Aprano <steve@pearwood.info> wrote: <snip>
You also seem to have some shenanigans going on with `x`.
Easily accomplished: def do_the_thing(x): try: do_something(x) except SpamError: fix_up(x) return False else: return True def try_repeatedly(n, func, arg): for _ in range(n): if func(arg): break else: raise HamError('tried %d times, giving up now" % n) try_repeatedly(5, do_the_thing, y) Cheers, Chris
On Jan 20, 2012, at 10:57 PM, Steven D'Aprano wrote:
I'd make the place I factor out code different: def try_repeatedly(times, exception, callback, args, kwargs): for _ in range(times): try: return callback(*args, **kwargs) except exception: continue break error_message = "Tried {} times but could not complete callback {}" error_message = error_message.format(times, repr(callback)) raise Exception(error_message) try_repeatedly(5, SpamError, do_something, [x])
Steven D'Aprano writes:
Mike Meyer wrote:
"retry" will jump back to the start of the try block
This doesn't sound very intuitive to me.
-- a limited form of GOTO, with all the pros and cons of this.
All control flow is a limited form of GOTO. But in this case, retry can only go to one place, about as unlike GOTO as I can imagine. It certainly doesn't have any cons of GOTO that "while" doesn't have!
Agreed. I find this much easier to understand than the "retry" form. It seems to me that rather than a new keyword "retry", I'd want to use the existing "continue" here as with other looping constructs. The problem is that this would interfere with other uses of continue, not to mention that it would be inconsistent with break. While this thought experiment doesn't prove anything, it makes me wonder if the idea is really coherent (at least in the context of Python today).
I really don't like the idea of using a nonlocal construct as part of ordinary control flow. I guess I could get used to this particular idiom, but it's not like the explicit loop is terribly burdensome, to write or to read.
Stephen J. Turnbull wrote:
No, not very. You might not guess what it does, but it's not that hard to learn: "retry jumps back to the beginning of the try block".
The main con that I can think of is that it isn't clear that you're looping until you reach the retry. try: ... except Spam: ... except Ham: ... else: if flag: ... else: retry # Ah, we're in a loop! I don't hate this, but nor am I going to champion it. If someone else wants to champion it, I'd vote +0. It just feels more natural than making retry a block clause (Mike's proposal), but I'm not really convinced it's necessary. [...]
continue can't work, because it introduces ambiguity if you have a try inside a loop, or a loop inside a try. We could arbitrarily declare that try-continue wins over loop-continue, or visa versa, but whatever decision we make, it will leave half the would-be users unhappy. Besides, there's a neat symmetry: you try something, then you re-try it. (Believe it or not, I've only just noticed this.) -- Steven
On 21 January 2012 07:47, Steven D'Aprano <steve@pearwood.info> wrote:
That's the way I would interpret a "retry" statement - and it's nothing like the OP's proposal as far as I can see (where retry introduces a suite). I'd be -1 on a retry keyword that worked any way other than this (simply because this is "clearly" the most obvious interpretation). Whether a retry statement is worth having at all, though, is something I'm not sure of - I'd like to see some real-world use cases first. I've never encountered the need for it myself. And I can't honestly imagine why the while True...try...break" idiom would ever not be sufficient. Paul.
On Sat, 21 Jan 2012 11:46:19 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
Yes, it's not much like what I proposed. But it does solve the same problem, and in a much better way than I proposed. Because of that, I'm not going to try and fix the error in the OP of not translating the proposal to proper Python :-(.
I saw a number of request for "real world uses cases". I thought I covered that in the OP. This ideas was *prompted* by a real world use case where we wanted to wrap an exception in our own private exception before handling it. Because the code that would raise the exception - other exceptions missing the attributes we added to ours - was the code we want to run after handling them, we're left with LBYL or DRY or wrapping a loop around the code when it wasn't really a loop. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Sun, Jan 22, 2012 at 1:30 PM, Mike Meyer <mwm@mired.org> wrote:
The thing is, until this message, you never described any concrete details about your specific use-case, other than that you had one. Now you have, but in prose (which is regrettably imprecise) rather than code; I'm still not entirely sure what your use case looks like. As best I can parse it, a nested try-except sounds like it would work. People like use-cases because they can reveal motivations or subtleties that often turn out to be significant, and they can more easily critique them than completely abstract/general cases. Cheers, Chris
On Sun, 22 Jan 2012 13:52:52 -0800 Chris Rebert <pyideas@rebertia.com> wrote:
I can't give you the actual code (without having to deal with lawyers) because it comes from a client's proprietary product. I could make up code, but that's no better than the pseudo-code I gave in the OP.
As best I can parse it, a nested try-except sounds like it would work.
I don't see how. Here's the pseudo-code example from the OP that most resembles the code we finally committed, doing LBYL instead of EAFP: if not isinstance(x, my_class): x = fixup(x) mangle(x) other_stuff(x) That doesn't show you where the exception would happen though, so here (also from the OP) is the version that violates DRY: try: mangle(x) except: x = fixup(x) mangle(x) other_stuff(x) The only thing the prose deception added to the above is that x is known to be an exception, and we wanted to make sure it had an appropriate attribute. The fact that it's an exception isn't really relevant, but that we needed it to have a specific attribute means those two might be better written as: if not isinstance(x, my_class): x = my_class(fixup(x)) x.mangle() other_stuff(x) and try: x.mangle() except: x = my_class(fixup(x)) x.mangle() other_stuff(x) If you've got a nested try/except solution that does EAFP and DRY, I'd be interested in seeing it. The OP included some loop-based variants as well.
Which is why I provided the best I could under the circumstances and why I'm one of the first to ask for them when they aren't present. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Nick Coghlan writes:
That's a question of point of view. If "thing" is thought of as a "try" (an operation that might fail), yes, he wants to do the same thing a nondeterministic number of times, and in general, more than once: it's a loop. If "thing" is thought of as a "block" (which isn't a try), then the "things" done with and without exception are different: it's not a loop, it's a conditional. Looking at his example "superficially" (I intend no deprecation by that word, just a point of view in looking at the code), I have some sympathy for Mike's claim that "it's not a loop and it violates DRY." I "know what he means" (and I bet you do, too!) However, when I try to define that, even informally, I arrive at the paragraph above, and I end up coming down on the side that you can't consistently claim that "it's not a loop" *and* claim that "it violates DRY", in some deeper sense.
On Mon, Jan 23, 2012 at 2:34 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Exactly - in a very real sense, "retry once" is just a special case of a more general looping pattern: you execute the body either once or twice, perhaps with some massaging of state between the two attempts. One way to write it is: for attempt in (1, 2): # This iterable controls your number of attempts try: # Do whatever except ExpectedException: x = prevent_expected_exception(x) continue # continue == retry break # break == success (just like a search loop) else: raise Exception("Operation failed (made %d attempts)" % attempt) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 23 Jan 2012 15:07:01 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
I think that highlights why it's not really a loop. While "retry once" is a general case of a looping pattern, it's *also* just a special case of a more general exception handling pattern. Because it fits under both special cases, it can be wrapped in a loop. But *any* sequence of instructions can be wrapped in a loop. All you need is a program counter (python is missing the tools to express it really cleanly, so I'm pretending): for pc in range(sys.maxsize): switch pc: 0: first_block 1: second_block 2, 3: third_block 4, 6: fourth_block 5: fifth_block 7: sixth_block if not done: pc -= 1 else: break pc += 1 So what of this is a real loop? The third block is in a two-count loop. The sixth block is in a while not done loop. The fourth block corresponds to the case here - you want to rerun one bit of code with another in the middle. You really don't want a general-purpose loop. So having to write one doesn't express the intent correctly. Of course, you can write a general-purpose loop this way. And there are probably cases where such a loop is the intent. Which lowers the value of a "retry" statement. <mike
On Mon, 2012-01-23 at 13:34 +0900, Stephen J. Turnbull wrote:
You also have to consider the size and scope of the block. Anything more than one or two simple instructions will be difficult to achieve in a clean way. In the case of only one or to instruction, a try block works just fine. For larger scopes, I think it would require some form of "UNDO_BLOCK" opcode, so that the visited frame states can be restored before a retry attempt is done. It would break if any c code functions visits any frame that is outside the frames the byte code is executing in. I'd also be concerned about memory usage and speed, because it could lead to some very inefficient routines. Cheers, Ron
On Mon, Jan 23, 2012 at 01:34:44PM +0900, Stephen J. Turnbull wrote:
I'm afraid I don't understand this. What's a block (that isn't a try) here, and how does it differ from a thing, and what makes it a conditional? If all you are saying is that the caller may want to abstract out "repeat this thing until it succeeds (or fails permanently)" into a single operation without explicitly writing a for or while loop, that's what functions are for. map(func, seq) doesn't cease to be a loop just because you don't write it as an explicit loop. If you mean something different from this, I have no idea what you mean.
I don't. To me, "retry this thing repeatedly" is fundamentally a loop.
If it violates DRY, then by definition you must be repeating yourself. If you repeat yourself, then the obvious way to avoid repeating yourself is to place the repeated code inside a loop. Why is this a problem? What is especially confusing is that the proposed syntax is *defined* as looping back to the start of the try block, like a GOTO. If you draw the program flowchart of the construct, it loops backwards. Mike even stated that his proposal was exactly identical in behaviour to a try inside a while True loop. If he wants to argue that saving one indent level is so important that it's worth new syntax, that's one thing, but I am perplexed at claims that something which is identical to a loop isn't a loop. -- Steven
On Tue, 24 Jan 2012 11:00:53 +1100 Steven D'Aprano <steve@pearwood.info> wrote:
I don't. To me, "retry this thing repeatedly" is fundamentally a loop.
What's being abstracted out isn't "retry this thing repeatedly". it's "I want to retry this thing after tweaking things if it fails." In particular, different ways of failure might require different tweaks before the retry, second failures would be handled differently from first failures, etc. <mike
On Mon, Jan 23, 2012 at 8:06 PM, Mike Meyer <mwm@mired.org> wrote:
So it doesn't seem like a loop because you hope to do it only once? Or because you're thinking of the thing-retried as the candidate loop, when that is just a constant function, and actual the loop is a loop over things-to-try-first (a loop over functions, rather than data)? -jJ
Jim Jewett writes:
So it doesn't seem like a loop because you hope to do it only once?
With s/hope/expect/, that hits the nail on the head. I don't think syntax can express that cleanly, but I can't help thinking it's of good thing if somebody like Mike tries to find a way. He might succeed!
On Tue, Jan 24, 2012 at 1:18 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Special casing "looping with at most two iterations" and "looping where the body is a single try statement" both seem like very poor ideas. OK, so some people apparently take issue with having to map "retry" to "loop", but how does that even come close to justifying making *everyone* learn a third looping construct? We can't even get consensus that PEP 315's generalised while loops (which allow you to write loop-and-a-half constructs without using break) would be a net win for the language over the existing idiom. I'll note that under PEP 315, the problem discussed in this thread could be handled as: do ... while retry: # '...' stands in for the suite below retry = False try: # attempt consistent operation except ExpectedException: # set up for next attempt based on result of current attempt # even the list of expected exceptions can be made dynamic! retry = True Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
"Stephen J. Turnbull" <stephen@xemacs.org> wrote:
Right. Doing it more than once is an exceptional case. And is more exceptional every time through. That using an LBYL idiom, or simply repeating yourself, means you don't need a loop says to me that it isn't really a loop. Which is why using loop to write it feels wrong. That I translated into a loop was because the only tools Python has for this is a loop. That no more makes it a loop than map being a function call makes it not a loop. I'd say that using retry to write what really is a loop (ie - you can't know a maximum number of iterations at compile time) would constitute abuse. A very attractive abuse, at that. Which certainly counts against out.
He might succeed!
Or someone else will figure it out after I've correctly described the problem. -- Sent from my Android tablet with K-9 Mail. Please excuse my brevity.
On Tue, Jan 24, 2012 at 2:44 PM, Mike Meyer <mwm@mired.org> wrote:
That using an LBYL idiom, or simply repeating yourself, means you don't need a loop says to me that it isn't really a loop. Which is why using loop to write it feels wrong.
A retry is just a specific kind of loop that executes 1 or 2 times (or perhaps more if you're allowed to trigger the retry more than once). You don't actually need loops in general, since you can use recursion or repetition instead, so saying "look, i can rewrite it without the loop, so it's not really a loop!" doesn't mean all that much in an objective sense. The argument that it might be worth having dedicated syntax for a loop that runs 1 or 2 times is rather unconvincing when we don't even have dedicated syntax for a loop that runs 1 or more times (see PEP 315). We've survived this long with two variants of a loop that runs 0 or more times and using break as appropriate to handle all the other cases (e.g. while+break provides loop-and-a-half semantics and I've written for loops with an unconditional break at the end to get "0 or 1" iteration). Making the case that we need another looping construct is a fairly tall order (even PEP 315 doesn't suggest a completely new construct - it only proposes a generalisation of the existing while loops). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer <mwm@mired.org> wrote:
The argument isn't that we need a new syntax for a small set of loops, it's that the only ways to implement retrying after an exception leave a code smell.
Uh, saying "retrying is fundamentally a looping operation" is not a code smell. How do you plan to implement retry if not as a looping construct under the hood? No matter how many times you assert otherwise, "go back to this earlier point in the code and resume execution from there" is pretty much the *definition* of a loop. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 23, 2012, at 9:00 PM, Nick Coghlan wrote:
I do think there's something code smelly about a retry--if it didn't work the first time, why should it work the second time after you give it a whack? Either whacking is good and you should do it in advance or it's bad and you should do something more sophisticated--but I don't see how creating language level support for retrying would remove the smell. Accessing deeply.nested.methods.and_.properties is a code smell too, even though it has language level support from Python. (You could imagine it being otherwise, if Python insisted that each attribute access get its own line, but that wouldn't remove the smell either.) The whole point of a "smell" is that it's not directly bad, but it's a sign that maybe you were thinking of something wrong at a different level, so it's time to re-architect a little.
On Tue, Jan 24, 2012 at 5:19 PM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
I don't see how creating language level support for retrying would remove the smell.
I didn't even think of that aspect - very good point :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2012/1/24 Carl M. Johnson <cmjohnson.mailinglist@gmail.com>:
When could you need to retry a second time in a legitimate way? Mhhh, because you are trying to acquire a DHCP lease, you made a DHCPDISCOVER, and you have to retry on the same udp connexion you opened as long as you have no response, or you dont dont have a timeout, therefore you need to count the retry/time elapsed. Following the metaphor of the DHCP lease, where you need to retry with the same context than the try, you'd then claim that you may also need DHCPLEASE handling (a low level auth in DHCP) and retry all the way down from the DHCPDISCOVER if your IP is not accepted after a DHCPNAK. And then you discover that these problems of error handling with keeping context would best be handled in an evenemential fashion rather than with adding a weired hack in sequential programming. This case, and others are best treated with twisted or tornado, but do you really want to import tornado or twisted when you need a rough finite state automata ? (I don't have the answer) -- Regards, Jul
Carl M. Johnson wrote:
"Whacking" is not necessarily bad or difficult, and is not necessarily a code smell. There are many common situations where "try again" is natural and expected. E.g. in response to a busy signal, you should wait a little while before retrying: delay = 5 # Initial delay between attempts in seconds. for _ in range(MAX_ATTEMPTS): try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503: # Service Unavailable. time.sleep(delay) delay *= 2 # Exponential back-off. else: raise else: break This could be written without the for-loop using a hypothetical retry statement, but it doesn't really gain us much: delay = 2 # Initial delay between attempts in seconds. count = 0 try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503 and count < MAX_ATTEMPTS: # Service Unavailable time.sleep(delay) delay *= 2 # Exponential back-off count += 1 retry else: raise Although you save one indent level, you don't save any lines of code, and it costs you the effort of handling the book-keeping that a for-loop would give you for free. I don't count this as a win. -- Steven
On 24 January 2012 11:19, Steven D'Aprano <steve@pearwood.info> wrote:
Having read that, I agree that the "retry" doesn't buy you much. But I do think there's a pattern in there that it would be nice to be able to abstract out, namely the exponential back-off. The pattern of "try X, if Y happens, then do Z and wait, then retry" is probably common enough in certain types of application that it would be nice to encapsulate it in a library routine. (I haven't needed it myself, admittedly...) I can't quite see how to do it, though - I thought about clever uses of with statements, or abstract classes with defined callback methods you could use, but couldn't find anything obvious. (The big issue being that "Y" is that a particular exception is raised in this case, but may be something else in general). A construct that let end users abstract this type of pattern would probably be a far bigger win than a retry statement. (And it may be that it has the benefit of already existing, I just couldn't see it :-)) Paul.
On Tue, Jan 24, 2012 at 9:36 PM, Paul Moore <p.f.moore@gmail.com> wrote:
You just need to move the pause inside the iterator: def backoff(attempts, first_delay, scale=2): delay = first_delay for attempt in range(1, attempts+1): yield attempt time.sleep(delay) delay *= 2 for __ in backoff(MAX_ATTEMPTS, 5): try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503: # Service Unavailable. continue raise break You can also design smarter versions where the object yielded is mutable, making it easy to pass state back into the iterator. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 24, 2012 at 1:01 PM, Chris Rebert <pyideas@rebertia.com> wrote:
It can't be.
It really should be, though. What do to on a fixable or temporary failure is pretty clearly an execution context.
with RetryStrategy() as keepgoing: while keepgoing(): ... The catch is that keepgoing() is usually either simple enough to inline (even without the with statement) or sufficiently complicated that it needs to modify something the rest of the suite sees. Passing an object just to hold these changes adds its own code smell. -jJ
On Mon, 23 Jan 2012 21:19:08 -1000 "Carl M. Johnson" <cmjohnson.mailinglist@gmail.com> wrote:
That's the LBYL way of expressing the code. You know, the one that manages to express the "loop" without either repeating any code or using an actual loop. Except applying the fix without checking to see if it's needed is most likely a bug. You might want to argue that LBYL isn't a code smell. I've fixed enough bugs caused by it to disagree, but at this point it's a style argument. <mike
Mike Meyer wrote:
By this reasoning, "for i in range(n)" is a code smell, because n might happen to be 1. You can't know that the loop will run once until you actually try. The point of the retry idiom is that it could run twice, thrice, four times, ... up to whatever limit you impose (if any!). Even if you expect that 999 times out of a thousand it will only run once, you write it in a loop because you want to cover the 1 remaining time where it will run multiple times. That's not a code smell, it is the obvious way to write an operation that may need to be retried.
There's no limit to the bizarre and obfuscatory code that a clever enough, or foolish enough, coder can write. But putting something that needs to run one OR MORE times inside a loop is neither bizarre nor obfuscatory. That's what loops are for. -- Steven
On Wed, 25 Jan 2012 05:37:07 +1100 Steven D'Aprano <steve@pearwood.info> wrote:
Not quite, because n might also happen to *not* be 1. You could even run it no times, if n were 0. Nothing wrong with any of that.
You can't know that the loop will run once until you actually try.
Which is not a code smell. However, if you can tell by reading the code that it will only run once (or never run), like this one: for i in range(1): Then it's a code smell! <mike
On Wed, Jan 25, 2012 at 4:45 AM, Mike Meyer <mwm@mired.org> wrote:
I agree specifically in regards to range() with a literal argument, but it really depends on the iterator. Using break to force a single iteration can be nicer than calling next() and catching StopIteration. For example, code like the following makes it easy to special case the first entry in an iterator: walk_iter = os.walk(topdir) for dirpath, files, subdirs in walk_iter: # Special case the top level directory break else: raise RuntimeError("No dir entry for {!r}".format(topdir)) for dirpath, files, subdirs in walk_iter: # Process the subdirectories Python has very powerful looping constructs already - we don't need more just because some folks have been trained to think that break and continue are evil and haven't considered whether or not these prejudices inherited from C and C++ are still relevant in Python. Like early returns, break and continue are potentially dangerous in C and C++ because having multiple exit points from a scope increases the chance of leaking memory (or some other resource). By contrast, garbage collection and context managers mean that making appropriate use of early returns, break and continue is quite easy and safe in Python (and often clearer than the alternatives that try to avoid them). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 25 Jan 2012 10:04:37 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
And we've now gotten to the level of picking apart real examples. I don't think a loop that never loops is a code smell because I think break or continue are evil (the opposite, in fact; I'd like a multi-level break/continue, but that's for another topic). I think it's a code smell because there's a good chance it can be refactored into straight-line code that would be better in a number of ways. For instance, I would write your example like so: try: walk_iter = os.walk(topdir) dirpath, files, subdirs = next(wi) # Special case the top level directory except StopIteration: raise RuntimeError("No dir entry for {!r}".format(topdir)) # other exception handling here, as appropriate. for dirpath, files, subdirs in walk_iter: # Process the subdirectories Exactly what winds up in the try block will vary depending on circumstances. Putting only the invocation of next in it would duplicate your code. os.walk isn't documented as returning exceptions, and ignores problems with listdir, so adding it seems to be a nop. Handling the special case for the top level directory with this try seems like a win, because all the exceptions from dealing with the top level directory get grouped together. It might be useful to handle exceptions from the subdir processing as well, but the choice is easy to make with this version. In any case, it's easier to change this version as needed to deal with exceptions than the original version. The other issue may be just me - I expect exiting a loop from the bottom to be a normal flow path. So the for version reads to me like raising RuntimeError is normal, not exceptional.
Likewise, the try statement is very powerful. That break/continue may or may not be evil is immaterial. The point is to make the try statement more powerful. So long as you incorrectly see this as "just another looping construct", your conclusions will be flawed. Not necessarily wrong, just flawed. It can also be used to fix LBYL and DRY code smells. <mike
On Wed, Jan 25, 2012 at 10:52 AM, Mike Meyer <mwm@mired.org> wrote:
It *is* just another looping construct: "retry" = "go back to the start of the try block". It's really just proposing a weird way to spell while+continue. Exception handling and repeating a section of code are *not* the same thing - it doesn't make sense to bundle the two into a single mega-construct. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 24, 2012 at 8:19 PM, Mike Meyer <mwm@mired.org> wrote:
It seems to me that combining looping and exception handling in the same construct makes code harder to read. When you come across any looping construct, you know immediately that the following block could occur any number of times, depending on the line where the construct starts. In the case of try/except/retry, it may come as a surprise that the block can actually repeat. Also, you would probably, in most cases, need to add additional logic before the "retry" to ensure that you don't loop forever if the "fixup" you apply does not prevent the exception from happening again. This logic, instead of being tied to the loop itself, now ends up well hidden in one or many possible exception handlers.
On Tue, 24 Jan 2012 21:53:49 -0500 Vince <vince.vinet@gmail.com> wrote:
Except the block in the try *doesn't* repeat. That block is only executed once. It may be *started* multiple times if there are exceptional conditions, but once it finishes executing, it's over. That's sort of the point. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Mike Meyer writes:
The point is to make the try statement more powerful.
Not on python-ideas, it isn't. Here, the point is to make Python more expressive (including clarity in "expressive"). That may mean making some statements less expressive (in the extreme, eliminating them entirely, as in the case of "print").
So long as you incorrectly see this as "just another looping construct", your conclusions will be flawed.
That's unfair. Nobody said "just". If anything, you were trying in earlier posts to maintain the opposite extreme ("not a looping construct").
On Wed, 25 Jan 2012 10:28:06 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
I don't know that nobody said "just". What got said by Nick - repeatedly - was "We don't need another looping construct." I don't think it's reasonable to dismiss it without considering how it works in the other roles for which it's suitable.
Didn't we just have a discussion about the ambiguity of English, and how amorphism isn't pythonic? The point of the *proposal* is to make the try statement more powerful. The point of the *list* is to make Python more expressive. The point of the *thread* is to see if the former does the latter. Given that intro, I think the conclusion is "not this proposal", with opinions ranging from "we don't need it at all" to "Might be nice, but costs to much for the expected use cases." In thinking about it, I think it's to special purpose. There are at least four interesting variants of "retry": 1) start the block from the top; 2) start the block somewhere internally (and where?); 3&4) Same two, disabling the except handler that ran the retry. It's not clear there's even a sane way to define #2. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On 1/24/2012 7:52 PM, Mike Meyer wrote:
I would too, but ...
I disagree with turning 'try' into a specialized loop construct by adding a version of continue, which is a version of goto. We currently have a general overt loop (while), the covert equivalent (tail recursion), and a 'specialized' loop (for) that covers the majority of loop needs, leaving the generalized loop (while) to cover everything else. A 'try' statement in itself it a glorified label statement that also sets up the context for exception statements. Exception statement are conditional statements where the condition is an exception state. A marked program position plus a conditional jump is what makes a loop. -- Terry Jan Reedy
On Tue, Jan 24, 2012 at 12:18 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
... I've written for loops with an unconditional break at the end to get "0 or 1" iteration.
Ouch ... That seems even worse than the C do {...} while (0); idiom. Why didn't you just use an "if"? (Or at least a try ... except StopIteration) -jJ
On 24 January 2012 03:18, Stephen J. Turnbull <stephen@xemacs.org> wrote:
If Python was like Haskell or Lisp, where you can define your own control structures, and this was a proposal to add a new control structure to the stdlib, I'd be fine with it. It *is* a somewhat unusual case, and from some people's viewpoint, certainly, the construct is not really a loop. So having "from exception_handling import retry" would be plausible. But Python doesn't have user defined control structures, by design, and indeed takes a strongly minimalist position on adding control structures - we don't even have do...while (yet), as Nick has pointed out. So in that context, I believe that this case is not sufficiently special, and the "not a loop" viewpoint is not sufficiently universal, to warrant a language change. Certainly, I wouldn't argue that this should be part of the language before do...while (which is more generally useful) gets in. Paul.
On Mon, Jan 23, 2012 at 10:18 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Jim Jewett writes:
> So it doesn't seem like a loop because you hope to do it only once?
With s/hope/expect/, that hits the nail on the head.
Ah... I have often wanted a clean way to indicate "This branch can happen, but isn't normal." (Well, besides a comment that might be seen as condescending if it is *obviously* an edge case.) The retry proposal is just specializing that for when the weird branch includes a loop. Right now, the best I can do is hope that the special case (and possibly the normal case, if it is repeated) can be factored out, so that I can write if not doit(args): if not doit(tweak1(args)): if not doit(tweak2(args)): raise ReallyCant(args) or if oddcase(args): handle_oddcase(args) else: # Alternatively, make this suite much longer, so that it is # "obviously" the main point of the function. return _real_function(args) That said, I've wanted unusual-case annotation more when I thought the compiler might use the information. Without compiler support, I'm not sure how much takeup there would be for the resulting documentation-only construct. -jJ
On Tue, Jan 24, 2012 at 11:06 AM, Mike Meyer <mwm@mired.org> wrote:
But that's just normal loop-and-a-half behaviour, where the first half of the loop is consistent, but the second half depends on the results of the first half (including whether or not an exception is thrown). while attempts_remaining(): try: # attempt consistent operation except ExpectedException: # set up for next attempt based on result of current attempt # even the list of expected exceptions can be made dynamic! else: break # success! else: # All attempts failed! I'm not sure what it is about having an upper limit of 2 iterations, or having the loop exit criteria be "didn't throw an exception" rather than an ordinary conditional expression, that makes you feel like this construct isn't just an ordinary loop-and-a-half (and best thought about that way). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jan 20, 2012 at 4:36 PM, Mike Meyer <mwm@mired.org> wrote:
Can you write this in terms of current Python code? I don't understand exactly when a retry block would be executed. In Eiffel, retry is a statement, not a clause. Analogous to that would be: try: # block except: # block if condition: retry The equivalent of this in current Python is while True: try: # block except: # block if condition: continue # retry break FWIW, if this turns out to be a good idea, an alternate spelling that would not require a new keyword is 'try again'. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
Benjamin Peterson wrote:
"again" could be a not-actually-a-keyword keyword like "as" used to be. [steve@sylar ~]$ python2.3 Python 2.3.7 (#1, Aug 12 2010, 00:17:29) [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Not that I support that, but it is possible. -- Steven
On Sat, Jan 21, 2012 at 11:40 AM, Bruce Leban <bruce@leapyear.org> wrote:
And, indeed, I'd consider "while True" to be a fairly idiomatic way to write "keep trying this until it works (and we exit the loop via break) or we give up (and raise an exception)" without repeating ourselves. Alternatively, I'd push the entire try/except block into a closure and call that from the loop until it reported it had worked. A nested generator could achieve much the same thing. I don't think there needs to be "one obvious way" to do this, since it's a rare need and the existing tools (while loops with break/continue, a loop with an appropriate flag, a loop plus a closure, a nested generator, etc) all provide ways to implement it reasonably cleanly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, 2012-01-20 at 16:36 -0800, Mike Meyer wrote:
This looks like it would be very tricky to get right from a python programming stand point. Remember that python is full of one time iterators that won't repeat properly on a second try. Also many function calls mutate the arguments. So it's possible that a data structure could be changed on the first try, and give different results on a second try in an unexpected way. Cheers, Ron
On Fri, Jan 20, 2012 at 4:36 PM, Mike Meyer <mwm@mired.org> wrote:
I would be very interested if you give at least the general gist of the part of your code in question. Real-world use cases are always useful data.
Right, that has evil code duplication.
Personally, I would put the `break` in a separate `else` clause.
Your mention duplication as a gripe, yet there's no duplication in this version. I take then it that you just dislike this idiom?
Do I misunderstand your proposal, or are you missing an "except: retry" here? If you aren't, then how is infinite-retrying avoided? I would think that in many (most?) cases, we'd like to cap the number of reattempts. Currently, this looks like: for _ in range(5): try: buy_off_ebay(item, price) except OutbidError: price = highest_bid_for(item) * 1.05 # continue else: break # Some fiddling can be had over where to put the `break` # and whether to include an explicit `continue`. With what I understand `retry` would be: # Not even bothering with the capping try: buy_off_ebay(item, price) except OutbidError: retry retry: price = highest_bid_for(item) * 1.05 # I'd have to maintain the count myself?! # That's easily going to squander the 1 line this saved. I grant you that this has saved 1 level of indentation, whatever that's worth (personally, I don't put enough premium on it in this case to justify new syntax). It's also slightly more explicit (once you know what `retry` does, of course). And, in more general cases, a very few lines can be saved. But when one gets to the more general, complicated cases, it seems to me that you're better off just writing and making use of your own try_N_times(make_bid, N) [or whatever your "keep retrying" heuristic is] function, which argues against adding new syntax for this.
My view on Python's choice of control structures is that it provides a near-minimal, simple (so long as you overlook the obscure and seldom-used `else` clause of `for` & `while`) set of them that you can then combine to achieve fancier behavior (e.g. while + try + break = retry). For example, there's no do-while (instead: while True + if-break); there's no unless/until (instead: while/if + not); there's no C-style `for` (instead: for+range(), or a custom while); there's no `redo` (instead: nested while True + if-break/continue); and thus, no `retry` (instead: nested try within a loop). This keeps the core language small, aesthetically clean, uniform, and easy-to-learn. If one understands `if`, `while`, `for`, `try`, `break`, and `continue` (and `not`, but that's merely a boolean operator, as opposed to a control structure), then one can easily work out from scratch what the idioms do, and one learns to recognize them with just a bit of practice/experience. Also, it's easy to tweak the idioms when your control flow changes or becomes more complicated, thus allowing for flexibility. On the other hand, some clarity/explicitness is definitely lost. There's a design trade-off here; I don't think Python has tended towards the side that would add `retry`. Cheers, Chris -- http://rebertia.com
Mike Meyer wrote:
The behavior change from except is that instead of exiting the "try" statement when the "retry" clause ends, it restarts the try
This sounds like retry should be a flow-control statement, like continue or break, not a block. E.g.: try: something() except ValueError: if condition: retry else: raise "retry" will jump back to the start of the try block -- a limited form of GOTO, with all the pros and cons of this.
Um, you're trying to explain the behaviour of retry here, but your Python code includes a retry. This is one case where recursion is NOT your friend.
None of your explanation is clear to me. Under what circumstances will the retry block be executed? E.g. given: try: len(None) except TypeError: print("spam spam spam") retry: print("a") what will happen? How about these? #1 try: len(None) retry: print("a") except TypeError: print("spam spam spam") #2 try: len(None) except TypeError: print("spam spam spam") retry: print("a") except ValueError: print("this never gets called") retry: print("b") #3 try: len(None) except TypeError: print("spam spam spam") retry: print("a") retry: print("b")
I think that the idiom of a while or for loop is easy enough to read and write: for _ in range(5): # retry five times try: do_something(x) except SpamError: x = fix_up(x) else: break else: raise HamError("tried 5 times, giving up now") I just wish that break and continue could be written outside of a loop, so you can factor out common code: def do_the_thing(x): try: do_something(x) except SpamError: x = fix_up(x) else: break def try_repeatedly(n, func): for _ in range(n): func() else: raise HamError('tried %d times, giving up now" % n) try_repeatedly(5, do_the_thing) -- Steven
On Fri, Jan 20, 2012 at 11:47 PM, Steven D'Aprano <steve@pearwood.info> wrote: <snip>
You also seem to have some shenanigans going on with `x`.
Easily accomplished: def do_the_thing(x): try: do_something(x) except SpamError: fix_up(x) return False else: return True def try_repeatedly(n, func, arg): for _ in range(n): if func(arg): break else: raise HamError('tried %d times, giving up now" % n) try_repeatedly(5, do_the_thing, y) Cheers, Chris
On Jan 20, 2012, at 10:57 PM, Steven D'Aprano wrote:
I'd make the place I factor out code different: def try_repeatedly(times, exception, callback, args, kwargs): for _ in range(times): try: return callback(*args, **kwargs) except exception: continue break error_message = "Tried {} times but could not complete callback {}" error_message = error_message.format(times, repr(callback)) raise Exception(error_message) try_repeatedly(5, SpamError, do_something, [x])
Steven D'Aprano writes:
Mike Meyer wrote:
"retry" will jump back to the start of the try block
This doesn't sound very intuitive to me.
-- a limited form of GOTO, with all the pros and cons of this.
All control flow is a limited form of GOTO. But in this case, retry can only go to one place, about as unlike GOTO as I can imagine. It certainly doesn't have any cons of GOTO that "while" doesn't have!
Agreed. I find this much easier to understand than the "retry" form. It seems to me that rather than a new keyword "retry", I'd want to use the existing "continue" here as with other looping constructs. The problem is that this would interfere with other uses of continue, not to mention that it would be inconsistent with break. While this thought experiment doesn't prove anything, it makes me wonder if the idea is really coherent (at least in the context of Python today).
I really don't like the idea of using a nonlocal construct as part of ordinary control flow. I guess I could get used to this particular idiom, but it's not like the explicit loop is terribly burdensome, to write or to read.
Stephen J. Turnbull wrote:
No, not very. You might not guess what it does, but it's not that hard to learn: "retry jumps back to the beginning of the try block".
The main con that I can think of is that it isn't clear that you're looping until you reach the retry. try: ... except Spam: ... except Ham: ... else: if flag: ... else: retry # Ah, we're in a loop! I don't hate this, but nor am I going to champion it. If someone else wants to champion it, I'd vote +0. It just feels more natural than making retry a block clause (Mike's proposal), but I'm not really convinced it's necessary. [...]
continue can't work, because it introduces ambiguity if you have a try inside a loop, or a loop inside a try. We could arbitrarily declare that try-continue wins over loop-continue, or visa versa, but whatever decision we make, it will leave half the would-be users unhappy. Besides, there's a neat symmetry: you try something, then you re-try it. (Believe it or not, I've only just noticed this.) -- Steven
On 21 January 2012 07:47, Steven D'Aprano <steve@pearwood.info> wrote:
That's the way I would interpret a "retry" statement - and it's nothing like the OP's proposal as far as I can see (where retry introduces a suite). I'd be -1 on a retry keyword that worked any way other than this (simply because this is "clearly" the most obvious interpretation). Whether a retry statement is worth having at all, though, is something I'm not sure of - I'd like to see some real-world use cases first. I've never encountered the need for it myself. And I can't honestly imagine why the while True...try...break" idiom would ever not be sufficient. Paul.
On Sat, 21 Jan 2012 11:46:19 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
Yes, it's not much like what I proposed. But it does solve the same problem, and in a much better way than I proposed. Because of that, I'm not going to try and fix the error in the OP of not translating the proposal to proper Python :-(.
I saw a number of request for "real world uses cases". I thought I covered that in the OP. This ideas was *prompted* by a real world use case where we wanted to wrap an exception in our own private exception before handling it. Because the code that would raise the exception - other exceptions missing the attributes we added to ours - was the code we want to run after handling them, we're left with LBYL or DRY or wrapping a loop around the code when it wasn't really a loop. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Sun, Jan 22, 2012 at 1:30 PM, Mike Meyer <mwm@mired.org> wrote:
The thing is, until this message, you never described any concrete details about your specific use-case, other than that you had one. Now you have, but in prose (which is regrettably imprecise) rather than code; I'm still not entirely sure what your use case looks like. As best I can parse it, a nested try-except sounds like it would work. People like use-cases because they can reveal motivations or subtleties that often turn out to be significant, and they can more easily critique them than completely abstract/general cases. Cheers, Chris
On Sun, 22 Jan 2012 13:52:52 -0800 Chris Rebert <pyideas@rebertia.com> wrote:
I can't give you the actual code (without having to deal with lawyers) because it comes from a client's proprietary product. I could make up code, but that's no better than the pseudo-code I gave in the OP.
As best I can parse it, a nested try-except sounds like it would work.
I don't see how. Here's the pseudo-code example from the OP that most resembles the code we finally committed, doing LBYL instead of EAFP: if not isinstance(x, my_class): x = fixup(x) mangle(x) other_stuff(x) That doesn't show you where the exception would happen though, so here (also from the OP) is the version that violates DRY: try: mangle(x) except: x = fixup(x) mangle(x) other_stuff(x) The only thing the prose deception added to the above is that x is known to be an exception, and we wanted to make sure it had an appropriate attribute. The fact that it's an exception isn't really relevant, but that we needed it to have a specific attribute means those two might be better written as: if not isinstance(x, my_class): x = my_class(fixup(x)) x.mangle() other_stuff(x) and try: x.mangle() except: x = my_class(fixup(x)) x.mangle() other_stuff(x) If you've got a nested try/except solution that does EAFP and DRY, I'd be interested in seeing it. The OP included some loop-based variants as well.
Which is why I provided the best I could under the circumstances and why I'm one of the first to ask for them when they aren't present. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Nick Coghlan writes:
That's a question of point of view. If "thing" is thought of as a "try" (an operation that might fail), yes, he wants to do the same thing a nondeterministic number of times, and in general, more than once: it's a loop. If "thing" is thought of as a "block" (which isn't a try), then the "things" done with and without exception are different: it's not a loop, it's a conditional. Looking at his example "superficially" (I intend no deprecation by that word, just a point of view in looking at the code), I have some sympathy for Mike's claim that "it's not a loop and it violates DRY." I "know what he means" (and I bet you do, too!) However, when I try to define that, even informally, I arrive at the paragraph above, and I end up coming down on the side that you can't consistently claim that "it's not a loop" *and* claim that "it violates DRY", in some deeper sense.
On Mon, Jan 23, 2012 at 2:34 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Exactly - in a very real sense, "retry once" is just a special case of a more general looping pattern: you execute the body either once or twice, perhaps with some massaging of state between the two attempts. One way to write it is: for attempt in (1, 2): # This iterable controls your number of attempts try: # Do whatever except ExpectedException: x = prevent_expected_exception(x) continue # continue == retry break # break == success (just like a search loop) else: raise Exception("Operation failed (made %d attempts)" % attempt) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 23 Jan 2012 15:07:01 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
I think that highlights why it's not really a loop. While "retry once" is a general case of a looping pattern, it's *also* just a special case of a more general exception handling pattern. Because it fits under both special cases, it can be wrapped in a loop. But *any* sequence of instructions can be wrapped in a loop. All you need is a program counter (python is missing the tools to express it really cleanly, so I'm pretending): for pc in range(sys.maxsize): switch pc: 0: first_block 1: second_block 2, 3: third_block 4, 6: fourth_block 5: fifth_block 7: sixth_block if not done: pc -= 1 else: break pc += 1 So what of this is a real loop? The third block is in a two-count loop. The sixth block is in a while not done loop. The fourth block corresponds to the case here - you want to rerun one bit of code with another in the middle. You really don't want a general-purpose loop. So having to write one doesn't express the intent correctly. Of course, you can write a general-purpose loop this way. And there are probably cases where such a loop is the intent. Which lowers the value of a "retry" statement. <mike
On Mon, 2012-01-23 at 13:34 +0900, Stephen J. Turnbull wrote:
You also have to consider the size and scope of the block. Anything more than one or two simple instructions will be difficult to achieve in a clean way. In the case of only one or to instruction, a try block works just fine. For larger scopes, I think it would require some form of "UNDO_BLOCK" opcode, so that the visited frame states can be restored before a retry attempt is done. It would break if any c code functions visits any frame that is outside the frames the byte code is executing in. I'd also be concerned about memory usage and speed, because it could lead to some very inefficient routines. Cheers, Ron
On Mon, Jan 23, 2012 at 01:34:44PM +0900, Stephen J. Turnbull wrote:
I'm afraid I don't understand this. What's a block (that isn't a try) here, and how does it differ from a thing, and what makes it a conditional? If all you are saying is that the caller may want to abstract out "repeat this thing until it succeeds (or fails permanently)" into a single operation without explicitly writing a for or while loop, that's what functions are for. map(func, seq) doesn't cease to be a loop just because you don't write it as an explicit loop. If you mean something different from this, I have no idea what you mean.
I don't. To me, "retry this thing repeatedly" is fundamentally a loop.
If it violates DRY, then by definition you must be repeating yourself. If you repeat yourself, then the obvious way to avoid repeating yourself is to place the repeated code inside a loop. Why is this a problem? What is especially confusing is that the proposed syntax is *defined* as looping back to the start of the try block, like a GOTO. If you draw the program flowchart of the construct, it loops backwards. Mike even stated that his proposal was exactly identical in behaviour to a try inside a while True loop. If he wants to argue that saving one indent level is so important that it's worth new syntax, that's one thing, but I am perplexed at claims that something which is identical to a loop isn't a loop. -- Steven
On Tue, 24 Jan 2012 11:00:53 +1100 Steven D'Aprano <steve@pearwood.info> wrote:
I don't. To me, "retry this thing repeatedly" is fundamentally a loop.
What's being abstracted out isn't "retry this thing repeatedly". it's "I want to retry this thing after tweaking things if it fails." In particular, different ways of failure might require different tweaks before the retry, second failures would be handled differently from first failures, etc. <mike
On Mon, Jan 23, 2012 at 8:06 PM, Mike Meyer <mwm@mired.org> wrote:
So it doesn't seem like a loop because you hope to do it only once? Or because you're thinking of the thing-retried as the candidate loop, when that is just a constant function, and actual the loop is a loop over things-to-try-first (a loop over functions, rather than data)? -jJ
Jim Jewett writes:
So it doesn't seem like a loop because you hope to do it only once?
With s/hope/expect/, that hits the nail on the head. I don't think syntax can express that cleanly, but I can't help thinking it's of good thing if somebody like Mike tries to find a way. He might succeed!
On Tue, Jan 24, 2012 at 1:18 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Special casing "looping with at most two iterations" and "looping where the body is a single try statement" both seem like very poor ideas. OK, so some people apparently take issue with having to map "retry" to "loop", but how does that even come close to justifying making *everyone* learn a third looping construct? We can't even get consensus that PEP 315's generalised while loops (which allow you to write loop-and-a-half constructs without using break) would be a net win for the language over the existing idiom. I'll note that under PEP 315, the problem discussed in this thread could be handled as: do ... while retry: # '...' stands in for the suite below retry = False try: # attempt consistent operation except ExpectedException: # set up for next attempt based on result of current attempt # even the list of expected exceptions can be made dynamic! retry = True Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
"Stephen J. Turnbull" <stephen@xemacs.org> wrote:
Right. Doing it more than once is an exceptional case. And is more exceptional every time through. That using an LBYL idiom, or simply repeating yourself, means you don't need a loop says to me that it isn't really a loop. Which is why using loop to write it feels wrong. That I translated into a loop was because the only tools Python has for this is a loop. That no more makes it a loop than map being a function call makes it not a loop. I'd say that using retry to write what really is a loop (ie - you can't know a maximum number of iterations at compile time) would constitute abuse. A very attractive abuse, at that. Which certainly counts against out.
He might succeed!
Or someone else will figure it out after I've correctly described the problem. -- Sent from my Android tablet with K-9 Mail. Please excuse my brevity.
On Tue, Jan 24, 2012 at 2:44 PM, Mike Meyer <mwm@mired.org> wrote:
That using an LBYL idiom, or simply repeating yourself, means you don't need a loop says to me that it isn't really a loop. Which is why using loop to write it feels wrong.
A retry is just a specific kind of loop that executes 1 or 2 times (or perhaps more if you're allowed to trigger the retry more than once). You don't actually need loops in general, since you can use recursion or repetition instead, so saying "look, i can rewrite it without the loop, so it's not really a loop!" doesn't mean all that much in an objective sense. The argument that it might be worth having dedicated syntax for a loop that runs 1 or 2 times is rather unconvincing when we don't even have dedicated syntax for a loop that runs 1 or more times (see PEP 315). We've survived this long with two variants of a loop that runs 0 or more times and using break as appropriate to handle all the other cases (e.g. while+break provides loop-and-a-half semantics and I've written for loops with an unconditional break at the end to get "0 or 1" iteration). Making the case that we need another looping construct is a fairly tall order (even PEP 315 doesn't suggest a completely new construct - it only proposes a generalisation of the existing while loops). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan@gmail.com> wrote:
The argument isn't that we need a new syntax for a small set of loops, it's that the only ways to implement retrying after an exception leave a code smell. -- Sent from my Android tablet with K-9 Mail. Please excuse my brevity.
On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer <mwm@mired.org> wrote:
The argument isn't that we need a new syntax for a small set of loops, it's that the only ways to implement retrying after an exception leave a code smell.
Uh, saying "retrying is fundamentally a looping operation" is not a code smell. How do you plan to implement retry if not as a looping construct under the hood? No matter how many times you assert otherwise, "go back to this earlier point in the code and resume execution from there" is pretty much the *definition* of a loop. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 23, 2012, at 9:00 PM, Nick Coghlan wrote:
I do think there's something code smelly about a retry--if it didn't work the first time, why should it work the second time after you give it a whack? Either whacking is good and you should do it in advance or it's bad and you should do something more sophisticated--but I don't see how creating language level support for retrying would remove the smell. Accessing deeply.nested.methods.and_.properties is a code smell too, even though it has language level support from Python. (You could imagine it being otherwise, if Python insisted that each attribute access get its own line, but that wouldn't remove the smell either.) The whole point of a "smell" is that it's not directly bad, but it's a sign that maybe you were thinking of something wrong at a different level, so it's time to re-architect a little.
On Tue, Jan 24, 2012 at 5:19 PM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
I don't see how creating language level support for retrying would remove the smell.
I didn't even think of that aspect - very good point :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2012/1/24 Carl M. Johnson <cmjohnson.mailinglist@gmail.com>:
When could you need to retry a second time in a legitimate way? Mhhh, because you are trying to acquire a DHCP lease, you made a DHCPDISCOVER, and you have to retry on the same udp connexion you opened as long as you have no response, or you dont dont have a timeout, therefore you need to count the retry/time elapsed. Following the metaphor of the DHCP lease, where you need to retry with the same context than the try, you'd then claim that you may also need DHCPLEASE handling (a low level auth in DHCP) and retry all the way down from the DHCPDISCOVER if your IP is not accepted after a DHCPNAK. And then you discover that these problems of error handling with keeping context would best be handled in an evenemential fashion rather than with adding a weired hack in sequential programming. This case, and others are best treated with twisted or tornado, but do you really want to import tornado or twisted when you need a rough finite state automata ? (I don't have the answer) -- Regards, Jul
Carl M. Johnson wrote:
"Whacking" is not necessarily bad or difficult, and is not necessarily a code smell. There are many common situations where "try again" is natural and expected. E.g. in response to a busy signal, you should wait a little while before retrying: delay = 5 # Initial delay between attempts in seconds. for _ in range(MAX_ATTEMPTS): try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503: # Service Unavailable. time.sleep(delay) delay *= 2 # Exponential back-off. else: raise else: break This could be written without the for-loop using a hypothetical retry statement, but it doesn't really gain us much: delay = 2 # Initial delay between attempts in seconds. count = 0 try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503 and count < MAX_ATTEMPTS: # Service Unavailable time.sleep(delay) delay *= 2 # Exponential back-off count += 1 retry else: raise Although you save one indent level, you don't save any lines of code, and it costs you the effort of handling the book-keeping that a for-loop would give you for free. I don't count this as a win. -- Steven
On 24 January 2012 11:19, Steven D'Aprano <steve@pearwood.info> wrote:
Having read that, I agree that the "retry" doesn't buy you much. But I do think there's a pattern in there that it would be nice to be able to abstract out, namely the exponential back-off. The pattern of "try X, if Y happens, then do Z and wait, then retry" is probably common enough in certain types of application that it would be nice to encapsulate it in a library routine. (I haven't needed it myself, admittedly...) I can't quite see how to do it, though - I thought about clever uses of with statements, or abstract classes with defined callback methods you could use, but couldn't find anything obvious. (The big issue being that "Y" is that a particular exception is raised in this case, but may be something else in general). A construct that let end users abstract this type of pattern would probably be a far bigger win than a retry statement. (And it may be that it has the benefit of already existing, I just couldn't see it :-)) Paul.
On Tue, Jan 24, 2012 at 9:36 PM, Paul Moore <p.f.moore@gmail.com> wrote:
You just need to move the pause inside the iterator: def backoff(attempts, first_delay, scale=2): delay = first_delay for attempt in range(1, attempts+1): yield attempt time.sleep(delay) delay *= 2 for __ in backoff(MAX_ATTEMPTS, 5): try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503: # Service Unavailable. continue raise break You can also design smarter versions where the object yielded is mutable, making it easy to pass state back into the iterator. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 24, 2012 at 1:01 PM, Chris Rebert <pyideas@rebertia.com> wrote:
It can't be.
It really should be, though. What do to on a fixable or temporary failure is pretty clearly an execution context.
with RetryStrategy() as keepgoing: while keepgoing(): ... The catch is that keepgoing() is usually either simple enough to inline (even without the with statement) or sufficiently complicated that it needs to modify something the rest of the suite sees. Passing an object just to hold these changes adds its own code smell. -jJ
On Mon, 23 Jan 2012 21:19:08 -1000 "Carl M. Johnson" <cmjohnson.mailinglist@gmail.com> wrote:
That's the LBYL way of expressing the code. You know, the one that manages to express the "loop" without either repeating any code or using an actual loop. Except applying the fix without checking to see if it's needed is most likely a bug. You might want to argue that LBYL isn't a code smell. I've fixed enough bugs caused by it to disagree, but at this point it's a style argument. <mike
Mike Meyer wrote:
By this reasoning, "for i in range(n)" is a code smell, because n might happen to be 1. You can't know that the loop will run once until you actually try. The point of the retry idiom is that it could run twice, thrice, four times, ... up to whatever limit you impose (if any!). Even if you expect that 999 times out of a thousand it will only run once, you write it in a loop because you want to cover the 1 remaining time where it will run multiple times. That's not a code smell, it is the obvious way to write an operation that may need to be retried.
There's no limit to the bizarre and obfuscatory code that a clever enough, or foolish enough, coder can write. But putting something that needs to run one OR MORE times inside a loop is neither bizarre nor obfuscatory. That's what loops are for. -- Steven
On Wed, 25 Jan 2012 05:37:07 +1100 Steven D'Aprano <steve@pearwood.info> wrote:
Not quite, because n might also happen to *not* be 1. You could even run it no times, if n were 0. Nothing wrong with any of that.
You can't know that the loop will run once until you actually try.
Which is not a code smell. However, if you can tell by reading the code that it will only run once (or never run), like this one: for i in range(1): Then it's a code smell! <mike
On Wed, Jan 25, 2012 at 4:45 AM, Mike Meyer <mwm@mired.org> wrote:
I agree specifically in regards to range() with a literal argument, but it really depends on the iterator. Using break to force a single iteration can be nicer than calling next() and catching StopIteration. For example, code like the following makes it easy to special case the first entry in an iterator: walk_iter = os.walk(topdir) for dirpath, files, subdirs in walk_iter: # Special case the top level directory break else: raise RuntimeError("No dir entry for {!r}".format(topdir)) for dirpath, files, subdirs in walk_iter: # Process the subdirectories Python has very powerful looping constructs already - we don't need more just because some folks have been trained to think that break and continue are evil and haven't considered whether or not these prejudices inherited from C and C++ are still relevant in Python. Like early returns, break and continue are potentially dangerous in C and C++ because having multiple exit points from a scope increases the chance of leaking memory (or some other resource). By contrast, garbage collection and context managers mean that making appropriate use of early returns, break and continue is quite easy and safe in Python (and often clearer than the alternatives that try to avoid them). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 25 Jan 2012 10:04:37 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
And we've now gotten to the level of picking apart real examples. I don't think a loop that never loops is a code smell because I think break or continue are evil (the opposite, in fact; I'd like a multi-level break/continue, but that's for another topic). I think it's a code smell because there's a good chance it can be refactored into straight-line code that would be better in a number of ways. For instance, I would write your example like so: try: walk_iter = os.walk(topdir) dirpath, files, subdirs = next(wi) # Special case the top level directory except StopIteration: raise RuntimeError("No dir entry for {!r}".format(topdir)) # other exception handling here, as appropriate. for dirpath, files, subdirs in walk_iter: # Process the subdirectories Exactly what winds up in the try block will vary depending on circumstances. Putting only the invocation of next in it would duplicate your code. os.walk isn't documented as returning exceptions, and ignores problems with listdir, so adding it seems to be a nop. Handling the special case for the top level directory with this try seems like a win, because all the exceptions from dealing with the top level directory get grouped together. It might be useful to handle exceptions from the subdir processing as well, but the choice is easy to make with this version. In any case, it's easier to change this version as needed to deal with exceptions than the original version. The other issue may be just me - I expect exiting a loop from the bottom to be a normal flow path. So the for version reads to me like raising RuntimeError is normal, not exceptional.
Likewise, the try statement is very powerful. That break/continue may or may not be evil is immaterial. The point is to make the try statement more powerful. So long as you incorrectly see this as "just another looping construct", your conclusions will be flawed. Not necessarily wrong, just flawed. It can also be used to fix LBYL and DRY code smells. <mike
On Wed, Jan 25, 2012 at 10:52 AM, Mike Meyer <mwm@mired.org> wrote:
It *is* just another looping construct: "retry" = "go back to the start of the try block". It's really just proposing a weird way to spell while+continue. Exception handling and repeating a section of code are *not* the same thing - it doesn't make sense to bundle the two into a single mega-construct. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 24, 2012 at 8:19 PM, Mike Meyer <mwm@mired.org> wrote:
It seems to me that combining looping and exception handling in the same construct makes code harder to read. When you come across any looping construct, you know immediately that the following block could occur any number of times, depending on the line where the construct starts. In the case of try/except/retry, it may come as a surprise that the block can actually repeat. Also, you would probably, in most cases, need to add additional logic before the "retry" to ensure that you don't loop forever if the "fixup" you apply does not prevent the exception from happening again. This logic, instead of being tied to the loop itself, now ends up well hidden in one or many possible exception handlers.
On Tue, 24 Jan 2012 21:53:49 -0500 Vince <vince.vinet@gmail.com> wrote:
Except the block in the try *doesn't* repeat. That block is only executed once. It may be *started* multiple times if there are exceptional conditions, but once it finishes executing, it's over. That's sort of the point. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Mike Meyer writes:
The point is to make the try statement more powerful.
Not on python-ideas, it isn't. Here, the point is to make Python more expressive (including clarity in "expressive"). That may mean making some statements less expressive (in the extreme, eliminating them entirely, as in the case of "print").
So long as you incorrectly see this as "just another looping construct", your conclusions will be flawed.
That's unfair. Nobody said "just". If anything, you were trying in earlier posts to maintain the opposite extreme ("not a looping construct").
On Wed, 25 Jan 2012 10:28:06 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
I don't know that nobody said "just". What got said by Nick - repeatedly - was "We don't need another looping construct." I don't think it's reasonable to dismiss it without considering how it works in the other roles for which it's suitable.
Didn't we just have a discussion about the ambiguity of English, and how amorphism isn't pythonic? The point of the *proposal* is to make the try statement more powerful. The point of the *list* is to make Python more expressive. The point of the *thread* is to see if the former does the latter. Given that intro, I think the conclusion is "not this proposal", with opinions ranging from "we don't need it at all" to "Might be nice, but costs to much for the expected use cases." In thinking about it, I think it's to special purpose. There are at least four interesting variants of "retry": 1) start the block from the top; 2) start the block somewhere internally (and where?); 3&4) Same two, disabling the except handler that ran the retry. It's not clear there's even a sane way to define #2. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On 1/24/2012 7:52 PM, Mike Meyer wrote:
I would too, but ...
I disagree with turning 'try' into a specialized loop construct by adding a version of continue, which is a version of goto. We currently have a general overt loop (while), the covert equivalent (tail recursion), and a 'specialized' loop (for) that covers the majority of loop needs, leaving the generalized loop (while) to cover everything else. A 'try' statement in itself it a glorified label statement that also sets up the context for exception statements. Exception statement are conditional statements where the condition is an exception state. A marked program position plus a conditional jump is what makes a loop. -- Terry Jan Reedy
On Tue, Jan 24, 2012 at 12:18 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
... I've written for loops with an unconditional break at the end to get "0 or 1" iteration.
Ouch ... That seems even worse than the C do {...} while (0); idiom. Why didn't you just use an "if"? (Or at least a try ... except StopIteration) -jJ
On 24 January 2012 03:18, Stephen J. Turnbull <stephen@xemacs.org> wrote:
If Python was like Haskell or Lisp, where you can define your own control structures, and this was a proposal to add a new control structure to the stdlib, I'd be fine with it. It *is* a somewhat unusual case, and from some people's viewpoint, certainly, the construct is not really a loop. So having "from exception_handling import retry" would be plausible. But Python doesn't have user defined control structures, by design, and indeed takes a strongly minimalist position on adding control structures - we don't even have do...while (yet), as Nick has pointed out. So in that context, I believe that this case is not sufficiently special, and the "not a loop" viewpoint is not sufficiently universal, to warrant a language change. Certainly, I wouldn't argue that this should be part of the language before do...while (which is more generally useful) gets in. Paul.
On Mon, Jan 23, 2012 at 10:18 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Jim Jewett writes:
> So it doesn't seem like a loop because you hope to do it only once?
With s/hope/expect/, that hits the nail on the head.
Ah... I have often wanted a clean way to indicate "This branch can happen, but isn't normal." (Well, besides a comment that might be seen as condescending if it is *obviously* an edge case.) The retry proposal is just specializing that for when the weird branch includes a loop. Right now, the best I can do is hope that the special case (and possibly the normal case, if it is repeated) can be factored out, so that I can write if not doit(args): if not doit(tweak1(args)): if not doit(tweak2(args)): raise ReallyCant(args) or if oddcase(args): handle_oddcase(args) else: # Alternatively, make this suite much longer, so that it is # "obviously" the main point of the function. return _real_function(args) That said, I've wanted unusual-case annotation more when I thought the compiler might use the information. Without compiler support, I'm not sure how much takeup there would be for the resulting documentation-only construct. -jJ
On Tue, Jan 24, 2012 at 11:06 AM, Mike Meyer <mwm@mired.org> wrote:
But that's just normal loop-and-a-half behaviour, where the first half of the loop is consistent, but the second half depends on the results of the first half (including whether or not an exception is thrown). while attempts_remaining(): try: # attempt consistent operation except ExpectedException: # set up for next attempt based on result of current attempt # even the list of expected exceptions can be made dynamic! else: break # success! else: # All attempts failed! I'm not sure what it is about having an upper limit of 2 iterations, or having the loop exit criteria be "didn't throw an exception" rather than an ordinary conditional expression, that makes you feel like this construct isn't just an ordinary loop-and-a-half (and best thought about that way). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (17)
-
Benjamin Peterson -
Bruce Leban -
Carl M. Johnson -
Chris Rebert -
Ethan Furman -
Guido van Rossum -
Jakob Bowyer -
Jim Jewett -
julien tayon -
Mike Meyer -
Nick Coghlan -
Paul Moore -
Ron Adam -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy -
Vince