Mailman 3 Improving Catching Exceptions - Python-ideas

newer
JavaScript-Style Object Creation...

Improving Catching Exceptions

Sven R. Kunze

22 Jun 2017 22 Jun '17

8:30 p.m.

Hi folks, just one note I'd like to dump here. We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough. There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479 2) nested ImportError 3) nested AttributeError 1) is clear. 2) usually can be dealt with by applying the following pattern: try: import user except ImportError: import sys if sys.exc_info()[2].tb_next: raise Chris showed how to deal with 3). Catching nested exception is not what people want many times. Am I the only one getting the impression that there's a common theme here? Regards, Sven

Show replies by date

Steven D'Aprano

22 Jun 22 Jun

8:55 p.m.

On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:

...

Hi folks,

just one note I'd like to dump here.

We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough? (2) Is that list growing? (3) You seem to be implying that "catch narrow exceptions" is bad advice and we should catch Exception instead. How does that help?

...

There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479

StopIteration and generators have been around a long time, since Python 2.2 I think, so this is not new. To the extent this was a problem, it is fixed now.

...

2) nested ImportError 3) nested AttributeError

Both of those have been around since Python 1.x days, so not new either. If the list is growing, can you give some more recent examples?

...

1) is clear. 2) usually can be dealt with by applying the following pattern:

try: import user except ImportError: import sys if sys.exc_info()[2].tb_next: raise

I've never needed to write something like that for ImportError. It seems like an anti-pattern to me: sometimes it will silently swallow the exception, and now `user` will remain undefined, a landmine waiting to explode (with NameError) in your code.

...

Chris showed how to deal with 3). Catching nested exception is not what people want many times.

Isn't it? Why not? Can you explain further?

...

Am I the only one getting the impression that there's a common theme here?

I don't know what common theme you see. I can't see one. Do you actually have a proposal? -- Steve

Cameron Simpson

11:29 p.m.

On 23Jun2017 06:55, Steven D'Aprano wrote:

...

On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:

...
We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes. Consider: try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ... Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5]. Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts. One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function. Something equivalent to: try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ... There doesn't seem to be a concise way to write that. It might not even be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback. I can imagine wanting to write something like this: try: foo(bah[5]) except shallow IndexError as e: ... deduce that there is no bah[5] ... Note that one can then deduce the missing bah[5] instead of inferring it. Obviously the actual syntax above is a nonstarter, but something that succinct and direct would be very handy. The nested exception issue actually bites me regularly, almost always with properties. The property system appears designed to allow one to make "conditional" properties, which appear to exist only in some circumstances. I wrote one of them just the other day, along the lines of: @property def target(self): if len(self.targets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') However, more commonly I end up hiding coding errors with @property, particularly nasty when the coding error is deep in some nested call. Here is a nondeep example based on the above: @property def target(self): if len(self.targgets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Here I have misspelt ".targets" as ".targgets". And quietly the .target property is simply missing, and a caller may then infer, incorrectly, things about the number of targets. What I, as the coder, actually wanted was for the errant .targgets reference to trigger something different from Attribute error, something akin to a NameError. (Obviously it _is_ a missing attribute and that is what AttributeError is for, but within a property that is ... unhelpful.) This is so common that I actually keep around a special hack: def prop(func): ''' The builtin @property decorator lets internal AttributeErrors escape. While that can support properties that appear to exist conditionally, in practice this is almost never what I want, and it masks deeper errors. Hence this wrapper for @property that transmutes internal AttributeErrors into RuntimeErrors. ''' def wrapper(*a, **kw): try: return func(*a, **kw) except AttributeError as e: e2 = RuntimeError("inner function %s raised %s" % (func, e)) if sys.version_info[0] >= 3: try: eval('raise e2 from e', globals(), locals()) except: # FIXME: why does this raise a SyntaxError? raise e else: raise e2 return property(wrapper) and often define properties like this: from cs.py.func import prop ....... @prop def target(self): if len(self.targgets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Same shape, better semantics from a debugging point of view. This is just one example where "nested" exceptions can be a misleading behavioural symptom.

...

...
Chris showed how to deal with 3). Catching nested exception is not what people want many times.

Isn't it? Why not? Can you explain further?

I hope this real world example shows why the scenario is real, and that my discussion shows why for me at least it would be handy to _easily_ catch the "shallow" exception only. Cheers, Cameron Simpson

Andy Dirnberger

11:47 p.m.

...

On Jun 22, 2017, at 7:29 PM, Cameron Simpson wrote:

...
On 23Jun2017 06:55, Steven D'Aprano wrote:

...
On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote: We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes.

Consider:

try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5].

Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts.

One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function.

Something equivalent to:

try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ...

There doesn't seem to be a concise way to write that. It might not even be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback.

How about something like this? try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

...

Cheers, Cameron Simpson _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Cameron Simpson

23 Jun 23 Jun

1:02 a.m.

On 22Jun2017 19:47, Andy Dirnberger wrote:

...

...
On Jun 22, 2017, at 7:29 PM, Cameron Simpson wrote: try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5].

Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts. [...]

How about something like this?

try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

That is the kind of refactor to which I alluded in the paragraph above. Doing that a lot tends to obscure the core logic of the code, hence the desire for something more succinct requiring less internal code fiddling. Cheers, Cameron Simpson

Sven R. Kunze

2:20 p.m.

On 23.06.2017 03:02, Cameron Simpson wrote:

...

...
How about something like this?

try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

That is the kind of refactor to which I alluded in the paragraph above. Doing that a lot tends to obscure the core logic of the code, hence the desire for something more succinct requiring less internal code fiddling.

And depending on how complex bha.__getitem__ is, it can raise IndexError unintentionally as well. So, rewriting the outer code doesn't even help then. :-( Regards, Sven

Paul Moore

2:59 p.m.

On 23 June 2017 at 15:20, Sven R. Kunze wrote:

...

On 23.06.2017 03:02, Cameron Simpson wrote:

How about something like this?

try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

That is the kind of refactor to which I alluded in the paragraph above. Doing that a lot tends to obscure the core logic of the code, hence the desire for something more succinct requiring less internal code fiddling.

And depending on how complex bha.__getitem__ is, it can raise IndexError unintentionally as well. So, rewriting the outer code doesn't even help then. :-(

At this point, it becomes unclear to me what constitutes an "intentional" IndexError, as opposed to an "unintentional" one, at least in any sense that can actually be implemented. I appreciate that you want IndexError to mean "there is no 5th element in bah". But if bah has a __getitem__ that raises IndexError for any reason other than that, then the __getitem__ implementation has a bug. And while it might be nice to be able to continue working properly even when the code you're executing has bugs, I think it's a bit optimistic to hope for :-) On the other hand, I do see the point that insisting on finer and finer grained exception handling ultimately ends up with unreadable code. But it's not a problem I'd expect to see much in real life code (where code is either not written that defensively, because either there's context that allows the coder to make assumptions that objects will behave reasonably sanely, or the code gets refactored to put the exception handling in a function, or something like that). Paul

Cameron Simpson

10:38 p.m.

On 23Jun2017 15:59, Paul Moore wrote:

...

On 23 June 2017 at 15:20, Sven R. Kunze wrote:

...
On 23.06.2017 03:02, Cameron Simpson wrote: How about something like this?

try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

That is the kind of refactor to which I alluded in the paragraph above. Doing that a lot tends to obscure the core logic of the code, hence the desire for something more succinct requiring less internal code fiddling.

And depending on how complex bha.__getitem__ is, it can raise IndexError unintentionally as well. So, rewriting the outer code doesn't even help then. :-(

At this point, it becomes unclear to me what constitutes an "intentional" IndexError, as opposed to an "unintentional" one, at least in any sense that can actually be implemented.

While I agree that in object with its own __getitem__ would look "deep", what I was actually suggesting as a possibility was a "shallow" except catch, not some magic "intentional" semantic. A shallow catch would effectively need to mean "the exceptions uppermost traceback frame referers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods. [...snip...]

...

On the other hand, I do see the point that insisting on finer and finer grained exception handling ultimately ends up with unreadable code. But it's not a problem I'd expect to see much in real life code (where code is either not written that defensively, because either there's context that allows the coder to make assumptions that objects will behave reasonably sanely, or the code gets refactored to put the exception handling in a function, or something like that).

Sure, there are many circumstances where a succinct "shallow catch" might not be useful. But there are also plenty of circumstances where one would like just this flavour of precision. Cheers, Cameron Simpson

Greg Ewing

24 Jun 24 Jun

1:14 a.m.

Cameron Simpson wrote:

...

A shallow catch would effectively need to mean "the exceptions uppermost traceback frame referers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods.

I think it would be a very bad idea to have a language construct that only works for built-in types. It would be far too unpredictable and fragile. -- Greg

Stephan Houben

23 Jun 23 Jun

10:23 a.m.

Hi Andy, What you propose is essentially the "try .. catch .. in" construct as described for Standard ML in: https://pdfs.semanticscholar.org/b24a/60f84b296482769bb6752feeb3d93ba6aee8.p... Something similar for Clojure is at: https://github.com/rufoa/try-let So clearly this is something more people have struggled with. The paper above goes into deep detail on the practical and (proof-)theoretical advantages of such a construct. Stephan 2017-06-23 1:47 GMT+02:00 Andy Dirnberger :

...

...
On Jun 22, 2017, at 7:29 PM, Cameron Simpson wrote:

...
On 23Jun2017 06:55, Steven D'Aprano wrote:

...
On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote: We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes.

Consider:

try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5].

Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts.

One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function.

Something equivalent to:

try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ...

There doesn't seem to be a concise way to write that. It might not even be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback.

How about something like this?

try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

...
Cheers, Cameron Simpson _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Andy Dirnberger

3:09 p.m.

Hi Stephan, On Fri, Jun 23, 2017 at 6:23 AM, Stephan Houben wrote:

...

Hi Andy,

What you propose is essentially the "try .. catch .. in" construct as described for Standard ML in:

It's not really a proposal. It's existing syntax. I was suggesting a way to implement the example that would catch an IndexError raised by accessing elements in bah but not those raised by foo.

...

https://pdfs.semanticscholar.org/b24a/60f84b296482769bb6752feeb3d93b a6aee8.pdf

Something similar for Clojure is at: https://github.com/rufoa/try-let

So clearly this is something more people have struggled with. The paper above goes into deep detail on the practical and (proof-)theoretical advantages of such a construct.

Stephan

Andy

...

2017-06-23 1:47 GMT+02:00 Andy Dirnberger :

...
...
On Jun 22, 2017, at 7:29 PM, Cameron Simpson wrote:

...
On 23Jun2017 06:55, Steven D'Aprano wrote:

...
On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote: We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to

...
...
...
...
where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes.

Consider:

try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5].

Naturally one can rearrange this code to call foo() outside that

...
...
One can easily want, instead, some kind of "shallow except", which

would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in

grow try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts. the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function.

...
...
Something equivalent to:

try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ...

There doesn't seem to be a concise way to write that. It might not even

be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback.

How about something like this?

try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

...
Cheers, Cameron Simpson _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Stephan Houben

6:30 p.m.

2017-06-23 17:09 GMT+02:00 Andy Dirnberger :

...

It's not really a proposal. It's existing syntax.

Wow! I have been using Python since 1.5.2 and I never knew this. This is not Guido's famous time machine in action, by any chance? Guess there's some code to refactor using this construct now... Stephan 2017-06-23 17:09 GMT+02:00 Andy Dirnberger :

...

Hi Stephan,

On Fri, Jun 23, 2017 at 6:23 AM, Stephan Houben wrote:

...
Hi Andy,

What you propose is essentially the "try .. catch .. in" construct as described for Standard ML in:

It's not really a proposal. It's existing syntax. I was suggesting a way to implement the example that would catch an IndexError raised by accessing elements in bah but not those raised by foo.

...
https://pdfs.semanticscholar.org/b24a/60f84b296482769bb6752feeb3d93ba6aee8.p...

Something similar for Clojure is at: https://github.com/rufoa/try-let

So clearly this is something more people have struggled with. The paper above goes into deep detail on the practical and (proof-)theoretical advantages of such a construct.

Stephan

Andy

...
2017-06-23 1:47 GMT+02:00 Andy Dirnberger :

...
...
On Jun 22, 2017, at 7:29 PM, Cameron Simpson wrote:

...
On 23Jun2017 06:55, Steven D'Aprano wrote:

...
On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote: We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes.

Consider:

try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5].

Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts.

One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function.

Something equivalent to:

try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ...

There doesn't seem to be a concise way to write that. It might not even be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback.

How about something like this?

try: val = bah[5] except IndexError: # handle your expected exception here else: foo(val)

...
Cheers, Cameron Simpson _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Cameron Simpson

10:43 p.m.

On 23Jun2017 20:30, Stephan Houben wrote:

...

2017-06-23 17:09 GMT+02:00 Andy Dirnberger :

...
It's not really a proposal. It's existing syntax.

Wow! I have been using Python since 1.5.2 and I never knew this. This is not Guido's famous time machine in action, by any chance? Guess there's some code to refactor using this construct now...

Alas, no. It is existing syntax in Standard ML, not in Python. Cheers, Cameron Simpson

Greg Ewing

24 Jun 24 Jun

1:21 a.m.

Cameron Simpson wrote:

...

Alas, no. It is existing syntax in Standard ML, not in Python.

But Python doesn't need it, because try-except-else covers the same thing. -- Greg

Nick Coghlan

23 Jun 23 Jun

1:48 a.m.

On 23 June 2017 at 09:29, Cameron Simpson wrote:

...

This is so common that I actually keep around a special hack:

def prop(func): ''' The builtin @property decorator lets internal AttributeErrors escape. While that can support properties that appear to exist conditionally, in practice this is almost never what I want, and it masks deeper errors. Hence this wrapper for @property that transmutes internal AttributeErrors into RuntimeErrors. ''' def wrapper(*a, **kw): try: return func(*a, **kw) except AttributeError as e: e2 = RuntimeError("inner function %s raised %s" % (func, e)) if sys.version_info[0] >= 3: try: eval('raise e2 from e', globals(), locals()) except: # FIXME: why does this raise a SyntaxError? raise e else: raise e2 return property(wrapper)

Slight tangent, but I do sometimes wonder if adding a decorator factory like the following to functools might be useful: def raise_when_returned(return_exc): def decorator(f): @wraps(f) def wrapper(*args, **kwds): try: result = f(*args, **kwds) except selective_exc as unexpected_exc: msg = "inner function {} raised {}".format(f, unexpected_exc) raise RuntimeError(msg) from unexpected_exc if isinstance(result, return_exc): raise result return result It's essentially a generalisation of PEP 479 to arbitrary exception types, since it lets you mark a particular exception type as being communicated back to the wrapper via the return channel rather than as a regular exception: def with_traceback(exc): try: raise exc except BaseException as caught_exc: return caught_exc @property @raise_when_returned(AttributeError) def target(self): if len(self.targets) == 1: return self.targets[0] return with_traceback(AttributeError('only exists when this has exactly one target')) The part I don't like about that approach is the fact that you need to mess about with the exception internals to get a halfway decent traceback on the AttributeError. The main alternative would be to add a "convert_exception" context manager in contextlib, so you could write the example property as: @property def target(self): with convert_exception(AttributeError): if len(self.targets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Where "convert_exception" would be something like: def convert_exception(exc_type): """Prevents the given exception type from escaping a region of code by converting it to RuntimeError""" if not issubclass(exc_type, Exception): raise TypeError("Only Exception subclasses can be flagged as unexpected") try: yield except exc_type as unexpected_exc: new_exc = RuntimeError("Unexpected exception") raise new_exc from unexpected_exc The API for this could potentially be made more flexible to allow easy substition of lookup errors with attribute errors and vice-versa (e.g. via additional keyword-only parameters) To bring the tangent back closer to Sven's original point, there are probably also some parts of the import system (such as executing the body of a found module) where the case can be made that we should be converting ImportError to RuntimeError, rather than letting the ImportError escape (with essentially the same rationale as PEP 479). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Cameron Simpson

11 p.m.

On 23Jun2017 11:48, Nick Coghlan wrote:

...

On 23 June 2017 at 09:29, Cameron Simpson wrote:

...
This is so common that I actually keep around a special hack:

def prop(func): ''' The builtin @property decorator lets internal AttributeErrors escape. While that can support properties that appear to exist conditionally, in practice this is almost never what I want, and it masks deeper errors. Hence this wrapper for @property that transmutes internal AttributeErrors into RuntimeErrors. ''' def wrapper(*a, **kw): try: return func(*a, **kw) except AttributeError as e: e2 = RuntimeError("inner function %s raised %s" % (func, e)) if sys.version_info[0] >= 3: try: eval('raise e2 from e', globals(), locals()) except: # FIXME: why does this raise a SyntaxError? raise e else: raise e2 return property(wrapper)

Slight tangent, but I do sometimes wonder if adding a decorator factory like the following to functools might be useful:

def raise_when_returned(return_exc): def decorator(f): @wraps(f) def wrapper(*args, **kwds): try: result = f(*args, **kwds) except selective_exc as unexpected_exc: msg = "inner function {} raised {}".format(f, unexpected_exc) raise RuntimeError(msg) from unexpected_exc if isinstance(result, return_exc): raise result return result

It's essentially a generalisation of PEP 479 to arbitrary exception types, since it lets you mark a particular exception type as being communicated back to the wrapper via the return channel rather than as a regular exception:

def with_traceback(exc): try: raise exc except BaseException as caught_exc: return caught_exc

@property @raise_when_returned(AttributeError) def target(self): if len(self.targets) == 1: return self.targets[0] return with_traceback(AttributeError('only exists when this has exactly one target'))

Funnily enough I have an @transmute decorator which serves just this purpose. It doesn't see as much use as I might imagine, but that is partially because my function predates "raise ... from", which meant that it loses the stack trace from the transmuted exception, impeding debugging. I need to revisit it with that in mind. So yes, your proposed decorator has supporting real world use cases in my world. Cheers, Cameron Simpson

MRAB

1:55 a.m.

On 2017-06-23 00:29, Cameron Simpson wrote:

...

On 23Jun2017 06:55, Steven D'Aprano wrote:

...
On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:

...
We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes.

Consider:

try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5].

Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts.

One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function.

Something equivalent to:

try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ...

There doesn't seem to be a concise way to write that. It might not even be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback.

[snip] Increment a counter on every function call and record it on the exception, perhaps? If the exception's call count == the current call count, it was raised in this function.

Steven D'Aprano

7:02 p.m.

On Fri, Jun 23, 2017 at 09:29:23AM +1000, Cameron Simpson wrote:

...

On 23Jun2017 06:55, Steven D'Aprano wrote:

...
On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:

...
We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes.

But why teach it to newbies? Sven explicitly mentions teaching beginners. If we are talking about advanced features for experts, that's one thing, but it's another if we're talking about Python 101 taught to beginners and newbies. Do we really need to be teaching beginners how to deal with circular imports beyond "don't do it"?

...

Consider:

try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5].

Indeed -- if both foo and bah[5] can raise IndexError when the coder believes that only bah[5] can, then the above code is simply buggy. On the other hand, if the author is correct that foo cannot raise IndexError, then the code as given is fine.

...

Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts.

Sadly, that is often the nature of real code, as opposed to "toy" or textbook code that demonstrates an algorithm as cleanly as possible. It's been said that for every line of code in the textbook, the function needs ten lines in production.

...

One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function.

I think the concept of a "shallow exception" is ill-defined, and to the degree that it is defined, it is *dangerous*: a bug magnet waiting to strike. What do you mean by "directly raised from the surface code"? Why is bah[5] "surface code" but foo(x) is not? But call a function (or method). But worse, it seems that the idea of "shallow" or "deep" depends on *implementation details* of where the exception comes from. For example, changing from a recursive function to a while loop might change the exception from "50 function calls deep" to "1 function deep". What makes bah[5] "shallow"? For all you know, it calls a chain of a dozen __getitem__ methods, due to inheritance or proxying, before the exception is actually raised. Or it might call just a single __getitem__ method, but the method's implementation puts the error checking into a helper method: def __getitem__(self, n): self._validate(n) # may raise IndexError ... How many function calls are shallow, versus deep? This concept of a shallow exception is, it seems to me, a bug magnet. It is superficially attractive, but then you realise that: try: spam[5] except shallow IndexError: ... will behave differently depending on how spam is implemented, even if the interface (raises IndexError) is identical. It seems to me that this concept is trying to let us substitute some sort of undefined but mechanical measurement of "shallowness" for actually understanding what our code does. I don't think this can work. It would be awesome if there was some way for our language to Do What We Mean instead of What We Say. And then we can grow a money tree, and have a magic plum-pudding that stays the same size no matter how many slices we eat, and electricity so cheap the power company pays you to use it... *wink*

...

The nested exception issue actually bites me regularly, almost always with properties. [...] However, more commonly I end up hiding coding errors with @property, particularly nasty when the coding error is deep in some nested call. Here is a nondeep example based on the above:

@property def target(self): if len(self.targgets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target')

The obvious solution to this is to learn to spell correctly :-) Actually, a linter probably would have picked up that typo. But I do see that the issue if more than just typos. [...]

...

try: eval('raise e2 from e', globals(), locals()) except: # FIXME: why does this raise a SyntaxError?

Because "raise" is a statement, not an expression. You need exec(). -- Steve

Cameron Simpson

10:56 p.m.

On 24Jun2017 05:02, Steven D'Aprano wrote:

...

On Fri, Jun 23, 2017 at 09:29:23AM +1000, Cameron Simpson wrote:

...
On 23Jun2017 06:55, Steven D'Aprano wrote:

...
On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:

...
We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough.

(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes.

But why teach it to newbies? Sven explicitly mentions teaching beginners. If we are talking about advanced features for experts, that's one thing, but it's another if we're talking about Python 101 taught to beginners and newbies.

It depends. Explaining that exceptions from called code can be mishandled by a naive try/except is something newbies need to learn to avoid common pitfalls with exceptions, and a real world situation that must be kept in mind when acting on caught exceptions.

...

Do we really need to be teaching beginners how to deal with circular imports beyond "don't do it"?

Sven's example is with import. The situation is more general. [... snip basic example of simple code where IndexError can arise from multiple causes ...]

...

...
Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts.

Sadly, that is often the nature of real code, as opposed to "toy" or textbook code that demonstrates an algorithm as cleanly as possible. It's been said that for every line of code in the textbook, the function needs ten lines in production.

But not always so. And the reason for many language constructs and idioms is explicitly to combat what would otherwise need 10 lines of code (or 100 to do it "right" with corner cases), obscuring the core task and reducing readability/maintainability. So "Sadly, that is often the nature of real code" is not itself an argument against this idea.

...

...
One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function.

I think the concept of a "shallow exception" is ill-defined, and to the degree that it is defined, it is *dangerous*: a bug magnet waiting to strike.

What do you mean by "directly raised from the surface code"? Why is bah[5] "surface code" but foo(x) is not? But call a function (or method). [...]

I've replied to Paul Moore and suggested this definition as implementable and often useful: A shallow catch would effectively need to mean "the exception's uppermost traceback frame refers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods. The target here is not perform magic but to have a useful tool to identify exceptions that arise fairly directly from the adjacent clode and not what it calls. Without writing cumbersome and fragile boilerplate to dig into an exception's traceback. Cheers, Cameron Simpson

MRAB

11:37 p.m.

On 2017-06-23 23:56, Cameron Simpson wrote:

...

On 24Jun2017 05:02, Steven D'Aprano wrote: [snip]

...

...
I think the concept of a "shallow exception" is ill-defined, and to the degree that it is defined, it is *dangerous*: a bug magnet waiting to strike.

What do you mean by "directly raised from the surface code"? Why is bah[5] "surface code" but foo(x) is not? But call a function (or method). [...]

I've replied to Paul Moore and suggested this definition as implementable and often useful:

A shallow catch would effectively need to mean "the exception's uppermost traceback frame refers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods.

The target here is not perform magic but to have a useful tool to identify exceptions that arise fairly directly from the adjacent clode and not what it calls. Without writing cumbersome and fragile boilerplate to dig into an exception's traceback.

I think a "shallow exception" would be one that's part of a defined API, as distinct from one that is an artifact of the implementation, a leak in the abstraction. It's like when "raise ... from None" was introduced to help in those cases where you want to replace an exception that's a detail of the (current) internal implementation with one that's intended for the user.

Sven R. Kunze

26 Jun 26 Jun

4:29 p.m.

On 24.06.2017 01:37, MRAB wrote:

...

I think a "shallow exception" would be one that's part of a defined API, as distinct from one that is an artifact of the implementation, a leak in the abstraction.

I like the "shallow exception" idea most. It's simple and it covers most if not all issues. You also hit the nail with pointing to leaking abstractions.

...

It's like when "raise ... from None" was introduced to help in those cases where you want to replace an exception that's a detail of the (current) internal implementation with one that's intended for the user.

Regards, Sven PS: This "shallow exception" proposal could help e.g. Django improving their template system. Here's it's the other way round: the exception handling is done by Django and depending on the exception it will fall back to a different attribute access method. I can remember us implementing such a method which accidentally raised a caught exception which we then never saw. Debugging this was a mess and took a quite some time.

Nick Coghlan

27 Jun 27 Jun

11:41 a.m.

On 27 June 2017 at 02:29, Sven R. Kunze wrote:

...

On 24.06.2017 01:37, MRAB wrote:

...
I think a "shallow exception" would be one that's part of a defined API, as distinct from one that is an artifact of the implementation, a leak in the abstraction.

I like the "shallow exception" idea most. It's simple and it covers most if not all issues. You also hit the nail with pointing to leaking abstractions.

The shallow exception notion breaks a fairly fundamental refactoring principle in Python: you should be able to replace an arbitrary expression with a subfunction or subgenerator that produces the same result without any of the surrounding code being able to tell the difference. By contrast, Steven's exception_guard recipe just takes the existing "raise X from Y" feature, and makes it available as a context manager and function decorator. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sven R. Kunze

7:49 p.m.

On 27.06.2017 13:41, Nick Coghlan wrote:

...

The shallow exception notion breaks a fairly fundamental refactoring principle in Python: you should be able to replace an arbitrary expression with a subfunction or subgenerator that produces the same result without any of the surrounding code being able to tell the difference.

I would agree with you here but this "refactoring principle in Python" doesn't work for control flow. Just look at "return", "break", "continue" etc. Exceptions are another way of handling control flow. So, this doesn't apply here IMO.

...

By contrast, Steven's exception_guard recipe just takes the existing "raise X from Y" feature, and makes it available as a context manager and function decorator.

I don't see how this helps differentiating shallow and nested exceptions such as: try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib Regards, Sven PS: this has nothing to do with cyclic imports. It can be a misconfiguration of the system which fails nested imports. In those cases, we fallback silently.

Chris Angelico

8:03 p.m.

On Wed, Jun 28, 2017 at 5:49 AM, Sven R. Kunze wrote:

...

On 27.06.2017 13:41, Nick Coghlan wrote:

...
The shallow exception notion breaks a fairly fundamental refactoring principle in Python: you should be able to replace an arbitrary expression with a subfunction or subgenerator that produces the same result without any of the surrounding code being able to tell the difference.

I would agree with you here but this "refactoring principle in Python" doesn't work for control flow.

Just look at "return", "break", "continue" etc. Exceptions are another way of handling control flow. So, this doesn't apply here IMO.

The ability to safely refactor control flow is part of why 'yield from' exists, and why PEP 479 changed how StopIteration bubbles. Local control flow is hard to refactor, but exceptions are global control flow, and most certainly CAN be refactored safely. ChrisA

Nick Coghlan

28 Jun 28 Jun

2:25 a.m.

On 28 June 2017 at 06:03, Chris Angelico wrote:

...

On Wed, Jun 28, 2017 at 5:49 AM, Sven R. Kunze wrote:

...
I would agree with you here but this "refactoring principle in Python" doesn't work for control flow.

Just look at "return", "break", "continue" etc. Exceptions are another way of handling control flow. So, this doesn't apply here IMO.

The ability to safely refactor control flow is part of why 'yield from' exists, and why PEP 479 changed how StopIteration bubbles. Local control flow is hard to refactor, but exceptions are global control flow, and most certainly CAN be refactored safely.

And PEP 479 establishes a precedent for how we handle the cases where we decide we *don't* want a particular exception type to propagate normally: create a boundary on the stack that converts the otherwise ambiguous exception type to RuntimeError. While generator functions now do that implicitly for StopIteration, and "raise X from Y" lets people write suitable exception handlers themselves, we don't offer an easy way to do it with a context manager (with statement as stack boundary), asynchronous context manager (async with statement as stack boundary), or a function decorator (execution frame as stack boundary). So while I prefer "contextlib.convert_exception" as the name (rather than the "exception_guard" Steven used in his recipe), I'd definitely be open to a bugs.python.org RFE and a PR against contextlib to add such a construct to Python 3.7. We'd have a few specific API details to work out (e.g. whether or not to accept arbitrary conversion functions as conversion targets in addition to accepting exception types and iterables of exception types, whether or not to allow "None" as the conversion target to get the same behaviour as "contextlib.suppress"), but I'm already sold on the general concept. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Chris Angelico

3:16 a.m.

On Wed, Jun 28, 2017 at 12:25 PM, Nick Coghlan wrote:

...

While generator functions now do that implicitly for StopIteration, and "raise X from Y" lets people write suitable exception handlers themselves, we don't offer an easy way to do it with a context manager (with statement as stack boundary), asynchronous context manager (async with statement as stack boundary), or a function decorator (execution frame as stack boundary).

So while I prefer "contextlib.convert_exception" as the name (rather than the "exception_guard" Steven used in his recipe), I'd definitely be open to a bugs.python.org RFE and a PR against contextlib to add such a construct to Python 3.7.

We'd have a few specific API details to work out (e.g. whether or not to accept arbitrary conversion functions as conversion targets in addition to accepting exception types and iterables of exception types, whether or not to allow "None" as the conversion target to get the same behaviour as "contextlib.suppress"), but I'm already sold on the general concept.

I agree broadly, but I'm sure there'll be the usual ton of bikeshedding about the details. The idea behind this decorator, AIUI, is a declaration that "a FooException coming out of here is a bug", and if I were looking for that, I'd look for something about the function leaking an exception, or preventing exceptions. So maybe convert_exception will work, but definitely have a docs reference from contextlib.suppress to this ("if exceptions of this type would indicate code bugs, consider convert_exception instead"). In my testing, I've called it "no_leak" or some variant thereon, though that's a shorthand that wouldn't suit the stdlib. ChrisA

Nick Coghlan

6 a.m.

On 28 June 2017 at 13:16, Chris Angelico wrote:

...

On Wed, Jun 28, 2017 at 12:25 PM, Nick Coghlan wrote:

...
While generator functions now do that implicitly for StopIteration, and "raise X from Y" lets people write suitable exception handlers themselves, we don't offer an easy way to do it with a context manager (with statement as stack boundary), asynchronous context manager (async with statement as stack boundary), or a function decorator (execution frame as stack boundary).

So while I prefer "contextlib.convert_exception" as the name (rather than the "exception_guard" Steven used in his recipe), I'd definitely be open to a bugs.python.org RFE and a PR against contextlib to add such a construct to Python 3.7.

We'd have a few specific API details to work out (e.g. whether or not to accept arbitrary conversion functions as conversion targets in addition to accepting exception types and iterables of exception types, whether or not to allow "None" as the conversion target to get the same behaviour as "contextlib.suppress"), but I'm already sold on the general concept.

I agree broadly, but I'm sure there'll be the usual ton of bikeshedding about the details. The idea behind this decorator, AIUI, is a declaration that "a FooException coming out of here is a bug", and if I were looking for that, I'd look for something about the function leaking an exception, or preventing exceptions. So maybe convert_exception will work, but definitely have a docs reference from contextlib.suppress to this ("if exceptions of this type would indicate code bugs, consider convert_exception instead"). In my testing, I've called it "no_leak" or some variant thereon, though that's a shorthand that wouldn't suit the stdlib.

Right, and I'd like us to keep in mind the KeyError -> AttributeError (and vice-versa) use case as well. Similar to ExitStack, it would be appropriate to make some additions to the "recipes" section in the docs that covered things like "Keep AttributeError from being suppressed in a property implementation". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sven R. Kunze

11:48 a.m.

On 28.06.2017 08:00, Nick Coghlan wrote:

...

Right, and I'd like us to keep in mind the KeyError -> AttributeError (and vice-versa) use case as well. Similar to ExitStack, it would be appropriate to make some additions to the "recipes" section in the docs that covered things like "Keep AttributeError from being suppressed in a property implementation".

As it was snipped away, let me ask again: I don't see how this helps differentiating shallow and nested exceptions such as: try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib At least in my tests, exception_guard works this way and I don't see any improvements to current behavior. Moreover, I am somewhat skeptical that using this recipe will really improve the situation. It's a lot of code where users don't have any stdlib support. I furthermore doubt that all Python coders will now wrap their properties using the guard. So, using these properties we will have almost no improvement. I still don't see it as the responsibility of coder of the property to guard against anything. Nobody is forced to catch exceptions when using a property. If that's the "best" outcome, I will stick to https://stackoverflow.com/questions/20459166/how-to-catch-an-importerror-non... because 1) Google finds it for me and 2) we don't have to maintain 100 lines of code ourself. Regards, Sven

Nick Coghlan

1:26 p.m.

On 28 June 2017 at 21:48, Sven R. Kunze wrote:

...

As it was snipped away, let me ask again:

I don't see how this helps differentiating shallow and nested exceptions such as:

try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib

...

At least in my tests, exception_guard works this way and I don't see any improvements to current behavior. Moreover, I am somewhat skeptical that using this recipe will really improve the situation. It's a lot of code where users don't have any stdlib support. I furthermore doubt that all Python coders will now wrap their properties using the guard. So, using these properties we will have almost no improvement. I still don't see it as the responsibility of coder of the property to guard against anything. Nobody is forced to catch exceptions when using a property.

Honestly, if folks are trying to write complex Python code without using at least "pylint -E" as a static check for typos in attribute names (regardless of whether those lines get executed or not), then inadvertently hiding AttributeError in property and __getattr__ implementations is likely to be the least of their problems. So pylint's structural checks, type analysis tools like MyPy, or more advanced IDEs like PyCharm are typically going to be a better option for folks wanting to guard against bugs in their *own* code than adding defensive code purely as a cross-check on their own work. The cases I'm interested in are the ones where you're either developing some kind of framework and you need to code that framework defensively to guard against unexpected failures in the components you're executing (e.g. exec_module() in the PEP 451 import protocol), or else you're needing to adapt between two different kinds of exception reporting protocol (e.g. KeyError to AttributeError and vice-versa). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nathaniel Smith

7:14 p.m.

On Jun 28, 2017 6:26 AM, "Nick Coghlan" wrote: On 28 June 2017 at 21:48, Sven R. Kunze wrote:

...

As it was snipped away, let me ask again:

I don't see how this helps differentiating shallow and nested exceptions such as:

try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib

There are two answers to that: 1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about 2. There's a reasonable case to be made that importlib should include an ImportError -> RuntimeError conversion around the call to loader.exec_module (in the same spirit as PEP 479). That way, in: try: import myspeciallib except ImportError: import fallbacks.MySpecialLib as myspeciallib any caught ImportError would relate to "myspeciallib", while uncaught ImportErrors arising from *executing* "myspeciallib" will be converted to RuntimeError, with the original ImportError as their __cause__. So it would make sense to file an RFE against 3.7 proposing that behavioural change (we couldn't reasonably do anything like that with the old `load_module()` importer API, as raising ImportError was how that API signalled "I can't load that". We don't have that problem with `exec_module()`). What about modules that want to raise ImportError to indicate that they aren't available on the current system, perhaps because some of their dependencies are missing? For example, 'import ssl' should raise an ImportError if 'ssl.py' is present but '_ssl.so' is missing; the existence of '_ssl.so' is an internal implementation detail. And perhaps 'import trio.ssl' should raise ImportError if 'ssl' is missing. (Historically not all systems have openssl available, so this is a common situation where existing libraries contain ImportError guards.) With PEP 479 there was a different and better way to generate a StopIteration if you wanted one (just 'return'). Here I'm afraid existing projects might actually be relying on the implicit exception leakage in significant numbers :-/ More generally, my impression is that this is one of the reasons why exceptions have fallen out of favor in more recent languages. They're certainly workable, and python's certainly not going to change now, but they do have these awkward aspects that weren't as clear 20 years ago and that now we have to live with. -n

Sven R. Kunze

10:22 p.m.

On 28.06.2017 21:14, Nathaniel Smith wrote:

...

With PEP 479 there was a different and better way to generate a StopIteration if you wanted one (just 'return'). Here I'm afraid existing projects might actually be relying on the implicit exception leakage in significant numbers :-/

My concern as well.

...

More generally, my impression is that this is one of the reasons why exceptions have fallen out of favor in more recent languages. They're certainly workable, and python's certainly not going to change now, but they do have these awkward aspects that weren't as clear 20 years ago and that now we have to live with.

I am quite satisfied with the ability of exceptions to expose bugs as fast and clear as possible. I still think we can improve on the catching side a little bit to narrow down the relevant exceptions. Other than that, I would be interested to hear what system you have in mind. What alternative (borrowed from more recent languages) can you imagine? Checking return values like C or golang? No ability to catch them at all? How to handle bugs in the context of UI applications where a crash in front of the user should be avoided? Regards, Sven

Nick Coghlan

29 Jun 29 Jun

2:57 a.m.

On 29 June 2017 at 05:14, Nathaniel Smith wrote:

...

What about modules that want to raise ImportError to indicate that they aren't available on the current system, perhaps because some of their dependencies are missing? For example, 'import ssl' should raise an ImportError if 'ssl.py' is present but '_ssl.so' is missing; the existence of '_ssl.so' is an internal implementation detail. And perhaps 'import trio.ssl' should raise ImportError if 'ssl' is missing. (Historically not all systems have openssl available, so this is a common situation where existing libraries contain ImportError guards.)

With PEP 479 there was a different and better way to generate a StopIteration if you wanted one (just 'return'). Here I'm afraid existing projects might actually be relying on the implicit exception leakage in significant numbers :-/

Hence "it may be worth filing an RFE so we can discuss the implications", rather than "we should definitely do it". The kinds of cases you cite will already fail for import guards that check for "exc.name == 'the_module_being_imported'" though, so I'd be OK with requiring modules that actually wanted that behaviour to do: try: import other_module except ImportError as exc: raise ImportError("other_module is missing", name=__name__, path=__file__) from exc The guard around exec_module() would then look something like: try: loader.exec_module(mod) except ImportError as exc: if exc.name == mod.__name__: raise msg = f"Failed to import '{exc.name}' from '{mod.__name__}'" raise RuntimeError(msg) from exc Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sven R. Kunze

28 Jun 28 Jun

10:10 p.m.

On 28.06.2017 15:26, Nick Coghlan wrote:

...

1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about

I see, didn't know that one. I gave it a try and it's not 100% the behavior I have expected, but one could workaround if the valid package structure is known. Not sure for certain. "from foo.bar.baz import abc" can yield to "exc.name" being one of "foo", "foo.bar" or "foo.bar.baz". Not perfect but sort of doable.

...

2. There's a reasonable case to be made that importlib should include an ImportError -> RuntimeError conversion around the call to loader.exec_module (in the same spirit as PEP 479). That way, in:

try: import myspeciallib except ImportError: import fallbacks.MySpecialLib as myspeciallib

any caught ImportError would relate to "myspeciallib", while uncaught ImportErrors arising from *executing* "myspeciallib" will be converted to RuntimeError, with the original ImportError as their __cause__.

Generally changing the behavior for ImportError doesn't sound like it would work for all projects out there. For fallback imports, I am on your side, that's a real use case which can be solved by changing the behavior of ImportErrors. But for all imports? I don't know if that's good idea.

...

[People should use tools, guard against bugs and try to avoid mistakes.]

Sure, but I don't see who this can help, if I use third-party code. The cases, which I described in the original post, were simple cases, where we catch too many exception. So, I don't even have the chance to see the error, to file a bug report, to issue a pull request, etc. etc.

...

The cases I'm interested in are the ones where you're either developing some kind of framework and you need to code that framework defensively to guard against unexpected failures in the components you're executing (e.g. exec_module() in the PEP 451 import protocol), or else you're needing to adapt between two different kinds of exception reporting protocol (e.g. KeyError to AttributeError and vice-versa).

I am unsure what you mean by those abstract words "framework" and "components". But let me state it in different words: there are *raisers* and *catchers* which do the respective thing with exceptions. If you control the code on both sides, things are easy to change. Pre-condition: you know the bug in the first place, which is hard when you catch too much. If you control the raiser only, it doesn't help to say: "don't make mistakes, configure systems right, code better, etc." People will make mistakes, systems will be misconfigured, linters don't find everything, etc. If you control the catcher only, you definitely want to narrow down the amount of caught exceptions as far as possible. This was the original intend of this thread IIRC. This way you help to discover bugs in raising code. Addition benefit, you catching code reacts only to the right exceptions. One word about frameworks here. Django, for instance, is on both sides. The template engine is mostly on the catchers side, whereas the database layer is on the raisers side. I get the feeling that the solutions presented here are way too complicated and error-prone. My opinion on this topic still is that catching exceptions is not mandatory. Nobody is forced to do it and it's even better to let exceptions bubble up to visualize bugs. If one still needs to catch them, he should only catch those he really, really needs to catch and nothing more. If this cannot be improved sensibly, well, so be it. Although I still don't find the argument presented against "catching shallow exception" a little bit too abstract compared to the practical benefit. Maybe, there's a better solution, maybe not. Regards, Sven

Steven D'Aprano

3:51 p.m.

On Wed, Jun 28, 2017 at 12:25:12PM +1000, Nick Coghlan wrote: [...]

...

So while I prefer "contextlib.convert_exception" as the name (rather than the "exception_guard" Steven used in his recipe), I'd definitely be open to a bugs.python.org RFE and a PR against contextlib to add such a construct to Python 3.7.

http://bugs.python.org/issue30792 -- Steve

Greg Ewing

24 Jun 24 Jun

1:02 a.m.

Cameron Simpson wrote:

...

try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ...

One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code;

The problem I see with that is how to define what counts as "surface code". If the __getitem__ method of bah is written in Python, I don't see how you could tell that an IndexError raised by it should be caught, but one raised by foo() shouldn't. In any case, this doesn't address the issue raised by the OP, which in this example is that if the implementation of bah.__getitem__ calls something else that raises an IndexError, there's no easy way to distinguish that from one raised by bah.__getitem__ itself. I don't see any way to solve that by messing around with different try-except constructs. It can only be addressed from within bah.__getitem__ itself, by having it catch any incidental IndexErrors and turn them into a different exception. -- Greg

Steven D'Aprano

10:03 a.m.

On Sat, Jun 24, 2017 at 01:02:55PM +1200, Greg Ewing wrote:

...

In any case, this doesn't address the issue raised by the OP, which in this example is that if the implementation of bah.__getitem__ calls something else that raises an IndexError, there's no easy way to distinguish that from one raised by bah.__getitem__ itself.

I'm not convinced that's a meaningful distinction to make in general. Consider the difference between these two classes: class X: def __getitem__(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError ... class Y: def __getitem__(self, n): self._validate(n) ... def _validate(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError The difference is a mere difference of refactoring. Why should one of them be treated as "bah.__getitem__ raises itself" versus "bah.__getitem__ calls something which raises"? That's just an implementation detail. I think we're over-generalizing this problem. There's two actual issues here, and we shouldn't conflate them as the same problem: (1) People write buggy code based on invalid assumptions of what can and can't raise. E.g.: try: foo(baz[5]) except IndexError: ... # assume baz[5] failed (but maybe foo can raise too?) (2) There's a *specific* problem with property where a bug in your getter or setter that raises AttributeError will be masked, appearing as if the property itself doesn't exist. In the case of (1), there's nothing Python the language can do to fix that. The solution is to write better code. Question your assumptions. Think carefully about your pre-conditions and post-conditions and invariants. Plan ahead. Read the documentation of foo before assuming it won't raise. In other words, be a better programmer. If only it were that easy :-( (Aside: I've been thinking for a long time that design by contract is a very useful feature to have. It should be possibly to set a contract that states that this function won't raise a certain exception, and if it does, treat it as a runtime error. But this is still at a very early point in my thinking.) Python libraries rarely give a definitive list of what exceptions functions can raise, so unless you wrote it yourself and know exactly what it can and cannot do, defensive coding suggests that you assume any function call might raise any exception at all: try: item = baz[5] except IndexError: ... # assume baz[5] failed else: foo(item) Can we fix that? Well, maybe we should re-consider the rejection of PEP 463 (exception-catching expressions). https://www.python.org/dev/peps/pep-0463/ Maybe we need a better way to assert that a certain function won't raise a particular exception: try: item = bah[5] without IndexError: foo(item) except IndexError: ... # assume baz[5] failed (But how is that different from try...except...else?) In the case of (2), the masking of bugs inside property getters if they happen to raise AttributeError, I think the std lib can help with that. Perhaps a context manager or decorator (or both) that converts one exception to another? @property @bounce_exception(AttributeError, RuntimeError) def spam(self): ... Now spam.__get__ cannot raise AttributeError, if it does, it will be converted to RuntimeError. If you need finer control over the code that is guarded use the context manager form: @property def spam(self): with bounce_exception(AttributeError, RuntimeError): # guarded if condition: ... # not guarded raise AttributeError('property spam doesn't exist yet') -- Steve

Greg Ewing

12:31 p.m.

Steven D'Aprano wrote:

...

class X: def __getitem__(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError ...

class Y: def __getitem__(self, n): self._validate(n) ... def _validate(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError

Why should one of them be treated as "bah.__getitem__ raises itself" versus "bah.__getitem__ calls something which raises"?

They shouldn't be treated differently -- they're both legitimate ways for __getitem__ to signal that the item doesn't exist. What *should* be treated differently is if an IndexError occurs incidentally from something else that __getitem__ does. In other words, Y's __getitem__ should be written something like def __getitem__(self, n): self.validate(n) # If we get an IndexError from here on, it's a bug try: # work out the result and return it except IndexError as e: raise RuntimeError from e

...

I think we're over-generalizing this problem. There's two actual issues here, and we shouldn't conflate them as the same problem:

(1) People write buggy code based on invalid assumptions of what can and can't raise. E.g.:

(2) There's a *specific* problem with property where a bug in your getter or setter that raises AttributeError will be masked, appearing as if the property itself doesn't exist.

Agreed. Case 1 can usually be handled by rewriting the code so as to make the scope of exception catching as narrow as possible. Case 2 needs to be addressed within the method concerned on a case-by-case basis. If there's a general principle there, it's something like this: If you're writing a method that uses an exception as part of it's protocol, you should catch any incidental occurrences of the same exception and reraise it as a different exception. I don't think there's anything more the Python language could do to help with either of those.

...

(Aside: I've been thinking for a long time that design by contract is a very useful feature to have. It should be possibly to set a contract that states that this function won't raise a certain exception, and if it does, treat it as a runtime error. But this is still at a very early point in my thinking.)

That sounds dangerously similar to Java's checked exceptions, which has turned out to be a huge nuisance and not very helpful.

...

Maybe we need a better way to assert that a certain function won't raise a particular exception:

try: item = bah[5] without IndexError: foo(item) except IndexError: ... # assume baz[5] failed

(But how is that different from try...except...else?)

It's no different, if I understand what it's supposed to mean correctly.

...

@property @bounce_exception(AttributeError, RuntimeError) def spam(self): ...

In the case of property getters, it seems to me you're almost always going to want that functionality, so maybe it should be incorporated into the property decorator itself. The only cases where you wouldn't want it would be if your property dynamically figures out whether it exists or not, and in those rare cases you would just have to write your own descriptor class. -- Greg

Nick Coghlan

1:45 p.m.

On 24 June 2017 at 22:31, Greg Ewing wrote:

...

Steven D'Aprano wrote:

...
I think we're over-generalizing this problem. There's two actual issues here, and we shouldn't conflate them as the same problem:

(1) People write buggy code based on invalid assumptions of what can and can't raise. E.g.:

(2) There's a *specific* problem with property where a bug in your getter or setter that raises AttributeError will be masked, appearing as if the property itself doesn't exist.

Agreed.

Case 1 can usually be handled by rewriting the code so as to make the scope of exception catching as narrow as possible.

Case 2 needs to be addressed within the method concerned on a case-by-case basis. If there's a general principle there, it's something like this: If you're writing a method that uses an exception as part of it's protocol, you should catch any incidental occurrences of the same exception and reraise it as a different exception.

I don't think there's anything more the Python language could do to help with either of those.

While I used to think that, I'm no longer sure it's true, as it seems to me that a `contextlib.convert_exception` context manager could help with both of them. (That's technically the standard library helping, rather than the language per se, but it's still taking a currently obscure implementation pattern and making it more obvious by giving it a name) So if we assume that existed, and converted the given exception to RuntimeError (the same way PEP 479 does for StopIteration), we'd be able to write magic methods and properties in the following style: def __getitem__(self, n): self.validate(n) with contextlib.convert_exception(IndexError): # If we get an IndexError in here, it's a bug return self._getitem(n) That idiom then works the same way regardless of how far away your code is from the exception handler you're trying to bypass - you could just as easily put it inside the `self._getitem(n)` helper method instead. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Steven D'Aprano

25 Jun 25 Jun

5:21 p.m.

On Sat, Jun 24, 2017 at 11:45:25PM +1000, Nick Coghlan wrote:

...

While I used to think that, I'm no longer sure it's true, as it seems to me that a `contextlib.convert_exception` context manager could help with both of them.

Here is a recipe for such a context manager which is also useable as a decorator: https://code.activestate.com/recipes/580808-guard-against-an-exception-in-th... or just https://code.activestate.com/recipes/580808 It should work with Python 2.6 through 3.6 and later. try: with exception_guard(ZeroDivisionError): 1/0 # raises ZeroDivisionError except RuntimeError: print ('ZeroDivisionError replaced by RuntimeError') -- Steve

Jan Kaliszewski

8 Jul 8 Jul

12:57 a.m.

2017-06-25 Greg Ewing dixit:

...

...
(2) There's a *specific* problem with property where a bug in your getter or setter that raises AttributeError will be masked, appearing as if the property itself doesn't exist. [...] Case 2 needs to be addressed within the method concerned on a case-by-case basis. If there's a general principle there, it's something like this: If you're writing a method that uses an exception as part of it's protocol, you should catch any incidental occurrences of the same exception and reraise it as a different exception.

I don't think there's anything more the Python language could do to help with either of those.

In "case 2", maybe some auto-zeroing flag (or even a decrementing counter?) could be useful? Please, consider the following draft: class ExtAttributeError(AttributeError): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._propagate_beyond = True # of course this should be a class and it should support not only # getters but also setters and deleters -- but you get the idea... def new_property(func): def wrapper(self): try: return func(self) except AttributeError as exc: if getattr(exc, '_propagate_beyond', False): exc._propagate_beyond = False raise raise RuntimeError( f'Unexpected {exc.__class__.__name__}') from exc return wrapper Then we could have: class Spam: @new_property def target(self): if len(self.targgets) == 1: return self.targets[0] raise ExtAttributeError( 'only exists when this has exactly one target') Cheers. *j

Jan Kaliszewski

5:15 p.m.

New subject: Improving Catching Exceptions [ERRATUM]

2017-07-08 Jan Kaliszewski dixit:

...

return wrapper

Should be: return property(wrapper)

Rob Cliffe

26 Jun 26 Jun

12:06 a.m.

On 24/06/2017 11:03, Steven D'Aprano wrote:

...

On Sat, Jun 24, 2017 at 01:02:55PM +1200, Greg Ewing wrote:

...
In any case, this doesn't address the issue raised by the OP, which in this example is that if the implementation of bah.__getitem__ calls something else that raises an IndexError, there's no easy way to distinguish that from one raised by bah.__getitem__ itself. I'm not convinced that's a meaningful distinction to make in general. Consider the difference between these two classes:

class X: def __getitem__(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError ...

class Y: def __getitem__(self, n): self._validate(n) ... def _validate(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError

The difference is a mere difference of refactoring. Why should one of them be treated as "bah.__getitem__ raises itself" versus "bah.__getitem__ calls something which raises"? That's just an implementation detail.

I think we're over-generalizing this problem. There's two actual issues here, and we shouldn't conflate them as the same problem:

(1) People write buggy code based on invalid assumptions of what can and can't raise. E.g.:

try: foo(baz[5]) except IndexError: ... # assume baz[5] failed (but maybe foo can raise too?)

(2) There's a *specific* problem with property where a bug in your getter or setter that raises AttributeError will be masked, appearing as if the property itself doesn't exist.

In the case of (1), there's nothing Python the language can do to fix that. The solution is to write better code. Question your assumptions. Think carefully about your pre-conditions and post-conditions and invariants. Plan ahead. Read the documentation of foo before assuming it won't raise. In other words, be a better programmer.

If only it were that easy :-(

(Aside: I've been thinking for a long time that design by contract is a very useful feature to have. It should be possibly to set a contract that states that this function won't raise a certain exception, and if it does, treat it as a runtime error. But this is still at a very early point in my thinking.)

Python libraries rarely give a definitive list of what exceptions functions can raise, so unless you wrote it yourself and know exactly what it can and cannot do, defensive coding suggests that you assume any function call might raise any exception at all:

try: item = baz[5] except IndexError: ... # assume baz[5] failed else: foo(item)

Can we fix that? Well, maybe we should re-consider the rejection of PEP 463 (exception-catching expressions).

https://www.python.org/dev/peps/pep-0463/ I'm all in favour of that :-) but I don't see how it helps in this example:

try: item = (baz[5] except IndexError: SomeSentinelValue) if item == SomeSentinelValue: ... # assume baz[5] failed else: foo(item) is clunkier than the original version. Or am I missing something? Only if the normal and exceptional cases could be handled the same way would it help: foo(baz[5] except IndexError: 0) Rob Cliffe

...

Maybe we need a better way to assert that a certain function won't raise a particular exception:

try: item = bah[5] without IndexError: foo(item) except IndexError: ... # assume baz[5] failed

(But how is that different from try...except...else?)

In the case of (2), the masking of bugs inside property getters if they happen to raise AttributeError, I think the std lib can help with that. Perhaps a context manager or decorator (or both) that converts one exception to another?

@property @bounce_exception(AttributeError, RuntimeError) def spam(self): ...

Now spam.__get__ cannot raise AttributeError, if it does, it will be converted to RuntimeError. If you need finer control over the code that is guarded use the context manager form:

@property def spam(self): with bounce_exception(AttributeError, RuntimeError): # guarded if condition: ... # not guarded raise AttributeError('property spam doesn't exist yet')

2477

Age (days ago)

2493

Last active (days ago)

List overview

Download

41 comments

13 participants

participants (13)

Andy Dirnberger
Cameron Simpson
Chris Angelico
Greg Ewing
Jan Kaliszewski
MRAB
Nathaniel Smith
Nick Coghlan
Paul Moore
Rob Cliffe
Stephan Houben
Steven D'Aprano
Sven R. Kunze

Improving Catching Exceptions

Cameron Simpson

Cameron Simpson

Cameron Simpson

Stephan Houben

Stephan Houben

Cameron Simpson

Cameron Simpson

Cameron Simpson

Jan Kaliszewski

Jan Kaliszewski

tags

participants (13)