retry many times decorator

[Steven D'Aprano]

On Thu, 28 Jun 2012 17:26:36 +0100, andrea crotti wrote:

>> Returning a boolean isn't very Pythonic. It would be better, IMHO, if
>> it could swallow a specified exception (or specified exceptions?)
>> raised when an attempt failed, up to the maximum permitted number of
>> attempts. If the final attempt fails, propagate the exception. --
>> http://mail.python.org/mailman/listinfo/python-list
> 
> 
> Yes right I also didn't like it..  Now it become something as below, so
> I capture every possible exception (since it must be generic) 

I disagree. If you make a coding error in your function, why do you think 
it is useful to retry that buggy code over and over again? It's never 
going to get less buggy unless you see the exception and fix the bug.

For any operation that you want to retry, identify the *temporary* 
errors, catch them, and retry the request. *Permanent* errors should 
immediately fail, without retrying. *Unexpected* errors should not be 
caught, since they probably represent a bug in your code.

E.g. when trying to read a HTTP resource, there is no point retrying a 
400 Bad Request repeatedly, and it is rather anti-social to keep 
hammering a website when it's already told you that your syntax is wrong. 
Likewise, if you get 401, there is no point in retrying with the exact 
same request -- you need a *different* request, this time using 
authentication. A 403 is a permanent error and there's no point in 
retrying. A 404 might be temporary (although it probably isn't). A 503 or 
504 is likely to be temporary. And so forth.

http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

And if you get an unexpected AttributeError: 'NoneType' object has no 
attribute 'read' (as I just did today, in one of my scripts), that's a 
bug that should be fixed.

Almost any other resource you might try is likely to have similar 
responses: some errors will indicate temporary failures, others will be 
permanent, and some will indicate a bug in your code. I think it is rude 
and pointless to repeatedly retry the same failed request if it cannot 
possibly succeed.

I also find that exponential backoff for the delay is best. E.g. wait 1 
second after the first failed attempt, 2 seconds after the second, 4 
seconds after the third, 8 seconds after the fourth, etc.


-- 
Steven