Improving Catching Exceptions

Hi folks, just one note I'd like to dump here. We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough. There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479 2) nested ImportError 3) nested AttributeError 1) is clear. 2) usually can be dealt with by applying the following pattern: try: import user except ImportError: import sys if sys.exc_info()[2].tb_next: raise Chris showed how to deal with 3). Catching nested exception is not what people want many times. Am I the only one getting the impression that there's a common theme here? Regards, Sven

On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:
(1) Under what circumstances is it not enough? (2) Is that list growing? (3) You seem to be implying that "catch narrow exceptions" is bad advice and we should catch Exception instead. How does that help?
There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479
StopIteration and generators have been around a long time, since Python 2.2 I think, so this is not new. To the extent this was a problem, it is fixed now.
2) nested ImportError 3) nested AttributeError
Both of those have been around since Python 1.x days, so not new either. If the list is growing, can you give some more recent examples?
I've never needed to write something like that for ImportError. It seems like an anti-pattern to me: sometimes it will silently swallow the exception, and now `user` will remain undefined, a landmine waiting to explode (with NameError) in your code.
Chris showed how to deal with 3). Catching nested exception is not what people want many times.
Isn't it? Why not? Can you explain further?
Am I the only one getting the impression that there's a common theme here?
I don't know what common theme you see. I can't see one. Do you actually have a proposal? -- Steve

On 23Jun2017 06:55, Steven D'Aprano <steve@pearwood.info> wrote:
I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes. Consider: try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ... Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5]. Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts. One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function. Something equivalent to: try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ... There doesn't seem to be a concise way to write that. It might not even be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback. I can imagine wanting to write something like this: try: foo(bah[5]) except shallow IndexError as e: ... deduce that there is no bah[5] ... Note that one can then deduce the missing bah[5] instead of inferring it. Obviously the actual syntax above is a nonstarter, but something that succinct and direct would be very handy. The nested exception issue actually bites me regularly, almost always with properties. The property system appears designed to allow one to make "conditional" properties, which appear to exist only in some circumstances. I wrote one of them just the other day, along the lines of: @property def target(self): if len(self.targets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') However, more commonly I end up hiding coding errors with @property, particularly nasty when the coding error is deep in some nested call. Here is a nondeep example based on the above: @property def target(self): if len(self.targgets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Here I have misspelt ".targets" as ".targgets". And quietly the .target property is simply missing, and a caller may then infer, incorrectly, things about the number of targets. What I, as the coder, actually wanted was for the errant .targgets reference to trigger something different from Attribute error, something akin to a NameError. (Obviously it _is_ a missing attribute and that is what AttributeError is for, but within a property that is ... unhelpful.) This is so common that I actually keep around a special hack: def prop(func): ''' The builtin @property decorator lets internal AttributeErrors escape. While that can support properties that appear to exist conditionally, in practice this is almost never what I want, and it masks deeper errors. Hence this wrapper for @property that transmutes internal AttributeErrors into RuntimeErrors. ''' def wrapper(*a, **kw): try: return func(*a, **kw) except AttributeError as e: e2 = RuntimeError("inner function %s raised %s" % (func, e)) if sys.version_info[0] >= 3: try: eval('raise e2 from e', globals(), locals()) except: # FIXME: why does this raise a SyntaxError? raise e else: raise e2 return property(wrapper) and often define properties like this: from cs.py.func import prop ....... @prop def target(self): if len(self.targgets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Same shape, better semantics from a debugging point of view. This is just one example where "nested" exceptions can be a misleading behavioural symptom.
I hope this real world example shows why the scenario is real, and that my discussion shows why for me at least it would be handy to _easily_ catch the "shallow" exception only. Cheers, Cameron Simpson <cs@zip.com.au>

On 22Jun2017 19:47, Andy Dirnberger <dirn@dirnonline.com> wrote:
That is the kind of refactor to which I alluded in the paragraph above. Doing that a lot tends to obscure the core logic of the code, hence the desire for something more succinct requiring less internal code fiddling. Cheers, Cameron Simpson <cs@zip.com.au>

On 23 June 2017 at 15:20, Sven R. Kunze <srkunze@mail.de> wrote:
At this point, it becomes unclear to me what constitutes an "intentional" IndexError, as opposed to an "unintentional" one, at least in any sense that can actually be implemented. I appreciate that you want IndexError to mean "there is no 5th element in bah". But if bah has a __getitem__ that raises IndexError for any reason other than that, then the __getitem__ implementation has a bug. And while it might be nice to be able to continue working properly even when the code you're executing has bugs, I think it's a bit optimistic to hope for :-) On the other hand, I do see the point that insisting on finer and finer grained exception handling ultimately ends up with unreadable code. But it's not a problem I'd expect to see much in real life code (where code is either not written that defensively, because either there's context that allows the coder to make assumptions that objects will behave reasonably sanely, or the code gets refactored to put the exception handling in a function, or something like that). Paul

On 23Jun2017 15:59, Paul Moore <p.f.moore@gmail.com> wrote:
While I agree that in object with its own __getitem__ would look "deep", what I was actually suggesting as a possibility was a "shallow" except catch, not some magic "intentional" semantic. A shallow catch would effectively need to mean "the exceptions uppermost traceback frame referers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods. [...snip...]
Sure, there are many circumstances where a succinct "shallow catch" might not be useful. But there are also plenty of circumstances where one would like just this flavour of precision. Cheers, Cameron Simpson <cs@zip.com.au>

Hi Andy, What you propose is essentially the "try .. catch .. in" construct as described for Standard ML in: https://pdfs.semanticscholar.org/b24a/60f84b296482769bb6752feeb3d93ba6aee8.p... Something similar for Clojure is at: https://github.com/rufoa/try-let So clearly this is something more people have struggled with. The paper above goes into deep detail on the practical and (proof-)theoretical advantages of such a construct. Stephan 2017-06-23 1:47 GMT+02:00 Andy Dirnberger <dirn@dirnonline.com>:

Hi Stephan, On Fri, Jun 23, 2017 at 6:23 AM, Stephan Houben <stephanh42@gmail.com> wrote:
It's not really a proposal. It's existing syntax. I was suggesting a way to implement the example that would catch an IndexError raised by accessing elements in bah but not those raised by foo.
Andy

2017-06-23 17:09 GMT+02:00 Andy Dirnberger <dirn@dirnonline.com>:
It's not really a proposal. It's existing syntax.
Wow! I have been using Python since 1.5.2 and I never knew this. This is not Guido's famous time machine in action, by any chance? Guess there's some code to refactor using this construct now... Stephan 2017-06-23 17:09 GMT+02:00 Andy Dirnberger <dirn@dirnonline.com>:

On 23 June 2017 at 09:29, Cameron Simpson <cs@zip.com.au> wrote:
Slight tangent, but I do sometimes wonder if adding a decorator factory like the following to functools might be useful: def raise_when_returned(return_exc): def decorator(f): @wraps(f) def wrapper(*args, **kwds): try: result = f(*args, **kwds) except selective_exc as unexpected_exc: msg = "inner function {} raised {}".format(f, unexpected_exc) raise RuntimeError(msg) from unexpected_exc if isinstance(result, return_exc): raise result return result It's essentially a generalisation of PEP 479 to arbitrary exception types, since it lets you mark a particular exception type as being communicated back to the wrapper via the return channel rather than as a regular exception: def with_traceback(exc): try: raise exc except BaseException as caught_exc: return caught_exc @property @raise_when_returned(AttributeError) def target(self): if len(self.targets) == 1: return self.targets[0] return with_traceback(AttributeError('only exists when this has exactly one target')) The part I don't like about that approach is the fact that you need to mess about with the exception internals to get a halfway decent traceback on the AttributeError. The main alternative would be to add a "convert_exception" context manager in contextlib, so you could write the example property as: @property def target(self): with convert_exception(AttributeError): if len(self.targets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Where "convert_exception" would be something like: def convert_exception(exc_type): """Prevents the given exception type from escaping a region of code by converting it to RuntimeError""" if not issubclass(exc_type, Exception): raise TypeError("Only Exception subclasses can be flagged as unexpected") try: yield except exc_type as unexpected_exc: new_exc = RuntimeError("Unexpected exception") raise new_exc from unexpected_exc The API for this could potentially be made more flexible to allow easy substition of lookup errors with attribute errors and vice-versa (e.g. via additional keyword-only parameters) To bring the tangent back closer to Sven's original point, there are probably also some parts of the import system (such as executing the body of a found module) where the case can be made that we should be converting ImportError to RuntimeError, rather than letting the ImportError escape (with essentially the same rationale as PEP 479). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 23Jun2017 11:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
Funnily enough I have an @transmute decorator which serves just this purpose. It doesn't see as much use as I might imagine, but that is partially because my function predates "raise ... from", which meant that it loses the stack trace from the transmuted exception, impeding debugging. I need to revisit it with that in mind. So yes, your proposed decorator has supporting real world use cases in my world. Cheers, Cameron Simpson <cs@zip.com.au>

On Fri, Jun 23, 2017 at 09:29:23AM +1000, Cameron Simpson wrote:
But why teach it to newbies? Sven explicitly mentions teaching beginners. If we are talking about advanced features for experts, that's one thing, but it's another if we're talking about Python 101 taught to beginners and newbies. Do we really need to be teaching beginners how to deal with circular imports beyond "don't do it"?
Indeed -- if both foo and bah[5] can raise IndexError when the coder believes that only bah[5] can, then the above code is simply buggy. On the other hand, if the author is correct that foo cannot raise IndexError, then the code as given is fine.
Sadly, that is often the nature of real code, as opposed to "toy" or textbook code that demonstrates an algorithm as cleanly as possible. It's been said that for every line of code in the textbook, the function needs ten lines in production.
I think the concept of a "shallow exception" is ill-defined, and to the degree that it is defined, it is *dangerous*: a bug magnet waiting to strike. What do you mean by "directly raised from the surface code"? Why is bah[5] "surface code" but foo(x) is not? But call a function (or method). But worse, it seems that the idea of "shallow" or "deep" depends on *implementation details* of where the exception comes from. For example, changing from a recursive function to a while loop might change the exception from "50 function calls deep" to "1 function deep". What makes bah[5] "shallow"? For all you know, it calls a chain of a dozen __getitem__ methods, due to inheritance or proxying, before the exception is actually raised. Or it might call just a single __getitem__ method, but the method's implementation puts the error checking into a helper method: def __getitem__(self, n): self._validate(n) # may raise IndexError ... How many function calls are shallow, versus deep? This concept of a shallow exception is, it seems to me, a bug magnet. It is superficially attractive, but then you realise that: try: spam[5] except shallow IndexError: ... will behave differently depending on how spam is implemented, even if the interface (raises IndexError) is identical. It seems to me that this concept is trying to let us substitute some sort of undefined but mechanical measurement of "shallowness" for actually understanding what our code does. I don't think this can work. It would be awesome if there was some way for our language to Do What We Mean instead of What We Say. And then we can grow a money tree, and have a magic plum-pudding that stays the same size no matter how many slices we eat, and electricity so cheap the power company pays you to use it... *wink*
The obvious solution to this is to learn to spell correctly :-) Actually, a linter probably would have picked up that typo. But I do see that the issue if more than just typos. [...]
Because "raise" is a statement, not an expression. You need exec(). -- Steve

On 24Jun2017 05:02, Steven D'Aprano <steve@pearwood.info> wrote:
It depends. Explaining that exceptions from called code can be mishandled by a naive try/except is something newbies need to learn to avoid common pitfalls with exceptions, and a real world situation that must be kept in mind when acting on caught exceptions.
Do we really need to be teaching beginners how to deal with circular imports beyond "don't do it"?
Sven's example is with import. The situation is more general. [... snip basic example of simple code where IndexError can arise from multiple causes ...]
But not always so. And the reason for many language constructs and idioms is explicitly to combat what would otherwise need 10 lines of code (or 100 to do it "right" with corner cases), obscuring the core task and reducing readability/maintainability. So "Sadly, that is often the nature of real code" is not itself an argument against this idea.
I've replied to Paul Moore and suggested this definition as implementable and often useful: A shallow catch would effectively need to mean "the exception's uppermost traceback frame refers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods. The target here is not perform magic but to have a useful tool to identify exceptions that arise fairly directly from the adjacent clode and not what it calls. Without writing cumbersome and fragile boilerplate to dig into an exception's traceback. Cheers, Cameron Simpson <cs@zip.com.au>

On 2017-06-23 23:56, Cameron Simpson wrote:
On 24Jun2017 05:02, Steven D'Aprano <steve@pearwood.info> wrote: [snip]
I think a "shallow exception" would be one that's part of a defined API, as distinct from one that is an artifact of the implementation, a leak in the abstraction. It's like when "raise ... from None" was introduced to help in those cases where you want to replace an exception that's a detail of the (current) internal implementation with one that's intended for the user.

On 24.06.2017 01:37, MRAB wrote:
I like the "shallow exception" idea most. It's simple and it covers most if not all issues. You also hit the nail with pointing to leaking abstractions.
Regards, Sven PS: This "shallow exception" proposal could help e.g. Django improving their template system. Here's it's the other way round: the exception handling is done by Django and depending on the exception it will fall back to a different attribute access method. I can remember us implementing such a method which accidentally raised a caught exception which we then never saw. Debugging this was a mess and took a quite some time.

On 27 June 2017 at 02:29, Sven R. Kunze <srkunze@mail.de> wrote:
The shallow exception notion breaks a fairly fundamental refactoring principle in Python: you should be able to replace an arbitrary expression with a subfunction or subgenerator that produces the same result without any of the surrounding code being able to tell the difference. By contrast, Steven's exception_guard recipe just takes the existing "raise X from Y" feature, and makes it available as a context manager and function decorator. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27.06.2017 13:41, Nick Coghlan wrote:
I would agree with you here but this "refactoring principle in Python" doesn't work for control flow. Just look at "return", "break", "continue" etc. Exceptions are another way of handling control flow. So, this doesn't apply here IMO.
I don't see how this helps differentiating shallow and nested exceptions such as: try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib Regards, Sven PS: this has nothing to do with cyclic imports. It can be a misconfiguration of the system which fails nested imports. In those cases, we fallback silently.

On Wed, Jun 28, 2017 at 5:49 AM, Sven R. Kunze <srkunze@mail.de> wrote:
The ability to safely refactor control flow is part of why 'yield from' exists, and why PEP 479 changed how StopIteration bubbles. Local control flow is hard to refactor, but exceptions are global control flow, and most certainly CAN be refactored safely. ChrisA

On 28 June 2017 at 06:03, Chris Angelico <rosuav@gmail.com> wrote:
And PEP 479 establishes a precedent for how we handle the cases where we decide we *don't* want a particular exception type to propagate normally: create a boundary on the stack that converts the otherwise ambiguous exception type to RuntimeError. While generator functions now do that implicitly for StopIteration, and "raise X from Y" lets people write suitable exception handlers themselves, we don't offer an easy way to do it with a context manager (with statement as stack boundary), asynchronous context manager (async with statement as stack boundary), or a function decorator (execution frame as stack boundary). So while I prefer "contextlib.convert_exception" as the name (rather than the "exception_guard" Steven used in his recipe), I'd definitely be open to a bugs.python.org RFE and a PR against contextlib to add such a construct to Python 3.7. We'd have a few specific API details to work out (e.g. whether or not to accept arbitrary conversion functions as conversion targets in addition to accepting exception types and iterables of exception types, whether or not to allow "None" as the conversion target to get the same behaviour as "contextlib.suppress"), but I'm already sold on the general concept. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Jun 28, 2017 at 12:25 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree broadly, but I'm sure there'll be the usual ton of bikeshedding about the details. The idea behind this decorator, AIUI, is a declaration that "a FooException coming out of here is a bug", and if I were looking for that, I'd look for something about the function leaking an exception, or preventing exceptions. So maybe convert_exception will work, but definitely have a docs reference from contextlib.suppress to this ("if exceptions of this type would indicate code bugs, consider convert_exception instead"). In my testing, I've called it "no_leak" or some variant thereon, though that's a shorthand that wouldn't suit the stdlib. ChrisA

On 28 June 2017 at 13:16, Chris Angelico <rosuav@gmail.com> wrote:
Right, and I'd like us to keep in mind the KeyError -> AttributeError (and vice-versa) use case as well. Similar to ExitStack, it would be appropriate to make some additions to the "recipes" section in the docs that covered things like "Keep AttributeError from being suppressed in a property implementation". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28.06.2017 08:00, Nick Coghlan wrote:
As it was snipped away, let me ask again: I don't see how this helps differentiating shallow and nested exceptions such as: try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib At least in my tests, exception_guard works this way and I don't see any improvements to current behavior. Moreover, I am somewhat skeptical that using this recipe will really improve the situation. It's a lot of code where users don't have any stdlib support. I furthermore doubt that all Python coders will now wrap their properties using the guard. So, using these properties we will have almost no improvement. I still don't see it as the responsibility of coder of the property to guard against anything. Nobody is forced to catch exceptions when using a property. If that's the "best" outcome, I will stick to https://stackoverflow.com/questions/20459166/how-to-catch-an-importerror-non... because 1) Google finds it for me and 2) we don't have to maintain 100 lines of code ourself. Regards, Sven

On 28 June 2017 at 21:48, Sven R. Kunze <srkunze@mail.de> wrote:
There are two answers to that: 1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about 2. There's a reasonable case to be made that importlib should include an ImportError -> RuntimeError conversion around the call to loader.exec_module (in the same spirit as PEP 479). That way, in: try: import myspeciallib except ImportError: import fallbacks.MySpecialLib as myspeciallib any caught ImportError would relate to "myspeciallib", while uncaught ImportErrors arising from *executing* "myspeciallib" will be converted to RuntimeError, with the original ImportError as their __cause__. So it would make sense to file an RFE against 3.7 proposing that behavioural change (we couldn't reasonably do anything like that with the old `load_module()` importer API, as raising ImportError was how that API signalled "I can't load that". We don't have that problem with `exec_module()`).
Honestly, if folks are trying to write complex Python code without using at least "pylint -E" as a static check for typos in attribute names (regardless of whether those lines get executed or not), then inadvertently hiding AttributeError in property and __getattr__ implementations is likely to be the least of their problems. So pylint's structural checks, type analysis tools like MyPy, or more advanced IDEs like PyCharm are typically going to be a better option for folks wanting to guard against bugs in their *own* code than adding defensive code purely as a cross-check on their own work. The cases I'm interested in are the ones where you're either developing some kind of framework and you need to code that framework defensively to guard against unexpected failures in the components you're executing (e.g. exec_module() in the PEP 451 import protocol), or else you're needing to adapt between two different kinds of exception reporting protocol (e.g. KeyError to AttributeError and vice-versa). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jun 28, 2017 6:26 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote: On 28 June 2017 at 21:48, Sven R. Kunze <srkunze@mail.de> wrote:
There are two answers to that: 1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about 2. There's a reasonable case to be made that importlib should include an ImportError -> RuntimeError conversion around the call to loader.exec_module (in the same spirit as PEP 479). That way, in: try: import myspeciallib except ImportError: import fallbacks.MySpecialLib as myspeciallib any caught ImportError would relate to "myspeciallib", while uncaught ImportErrors arising from *executing* "myspeciallib" will be converted to RuntimeError, with the original ImportError as their __cause__. So it would make sense to file an RFE against 3.7 proposing that behavioural change (we couldn't reasonably do anything like that with the old `load_module()` importer API, as raising ImportError was how that API signalled "I can't load that". We don't have that problem with `exec_module()`). What about modules that want to raise ImportError to indicate that they aren't available on the current system, perhaps because some of their dependencies are missing? For example, 'import ssl' should raise an ImportError if 'ssl.py' is present but '_ssl.so' is missing; the existence of '_ssl.so' is an internal implementation detail. And perhaps 'import trio.ssl' should raise ImportError if 'ssl' is missing. (Historically not all systems have openssl available, so this is a common situation where existing libraries contain ImportError guards.) With PEP 479 there was a different and better way to generate a StopIteration if you wanted one (just 'return'). Here I'm afraid existing projects might actually be relying on the implicit exception leakage in significant numbers :-/ More generally, my impression is that this is one of the reasons why exceptions have fallen out of favor in more recent languages. They're certainly workable, and python's certainly not going to change now, but they do have these awkward aspects that weren't as clear 20 years ago and that now we have to live with. -n

On 28.06.2017 21:14, Nathaniel Smith wrote:
My concern as well.
I am quite satisfied with the ability of exceptions to expose bugs as fast and clear as possible. I still think we can improve on the catching side a little bit to narrow down the relevant exceptions. Other than that, I would be interested to hear what system you have in mind. What alternative (borrowed from more recent languages) can you imagine? Checking return values like C or golang? No ability to catch them at all? How to handle bugs in the context of UI applications where a crash in front of the user should be avoided? Regards, Sven

On 29 June 2017 at 05:14, Nathaniel Smith <njs@pobox.com> wrote:
Hence "it may be worth filing an RFE so we can discuss the implications", rather than "we should definitely do it". The kinds of cases you cite will already fail for import guards that check for "exc.name == 'the_module_being_imported'" though, so I'd be OK with requiring modules that actually wanted that behaviour to do: try: import other_module except ImportError as exc: raise ImportError("other_module is missing", name=__name__, path=__file__) from exc The guard around exec_module() would then look something like: try: loader.exec_module(mod) except ImportError as exc: if exc.name == mod.__name__: raise msg = f"Failed to import '{exc.name}' from '{mod.__name__}'" raise RuntimeError(msg) from exc Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28.06.2017 15:26, Nick Coghlan wrote:
1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about
I see, didn't know that one. I gave it a try and it's not 100% the behavior I have expected, but one could workaround if the valid package structure is known. Not sure for certain. "from foo.bar.baz import abc" can yield to "exc.name" being one of "foo", "foo.bar" or "foo.bar.baz". Not perfect but sort of doable.
Generally changing the behavior for ImportError doesn't sound like it would work for all projects out there. For fallback imports, I am on your side, that's a real use case which can be solved by changing the behavior of ImportErrors. But for all imports? I don't know if that's good idea.
[People should use tools, guard against bugs and try to avoid mistakes.]
Sure, but I don't see who this can help, if I use third-party code. The cases, which I described in the original post, were simple cases, where we catch too many exception. So, I don't even have the chance to see the error, to file a bug report, to issue a pull request, etc. etc.
I am unsure what you mean by those abstract words "framework" and "components". But let me state it in different words: there are *raisers* and *catchers* which do the respective thing with exceptions. If you control the code on both sides, things are easy to change. Pre-condition: you know the bug in the first place, which is hard when you catch too much. If you control the raiser only, it doesn't help to say: "don't make mistakes, configure systems right, code better, etc." People will make mistakes, systems will be misconfigured, linters don't find everything, etc. If you control the catcher only, you definitely want to narrow down the amount of caught exceptions as far as possible. This was the original intend of this thread IIRC. This way you help to discover bugs in raising code. Addition benefit, you catching code reacts only to the right exceptions. One word about frameworks here. Django, for instance, is on both sides. The template engine is mostly on the catchers side, whereas the database layer is on the raisers side. I get the feeling that the solutions presented here are way too complicated and error-prone. My opinion on this topic still is that catching exceptions is not mandatory. Nobody is forced to do it and it's even better to let exceptions bubble up to visualize bugs. If one still needs to catch them, he should only catch those he really, really needs to catch and nothing more. If this cannot be improved sensibly, well, so be it. Although I still don't find the argument presented against "catching shallow exception" a little bit too abstract compared to the practical benefit. Maybe, there's a better solution, maybe not. Regards, Sven

On Wed, Jun 28, 2017 at 12:25:12PM +1000, Nick Coghlan wrote: [...]
http://bugs.python.org/issue30792 -- Steve

Cameron Simpson wrote:
The problem I see with that is how to define what counts as "surface code". If the __getitem__ method of bah is written in Python, I don't see how you could tell that an IndexError raised by it should be caught, but one raised by foo() shouldn't. In any case, this doesn't address the issue raised by the OP, which in this example is that if the implementation of bah.__getitem__ calls something else that raises an IndexError, there's no easy way to distinguish that from one raised by bah.__getitem__ itself. I don't see any way to solve that by messing around with different try-except constructs. It can only be addressed from within bah.__getitem__ itself, by having it catch any incidental IndexErrors and turn them into a different exception. -- Greg

On Sat, Jun 24, 2017 at 01:02:55PM +1200, Greg Ewing wrote:
I'm not convinced that's a meaningful distinction to make in general. Consider the difference between these two classes: class X: def __getitem__(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError ... class Y: def __getitem__(self, n): self._validate(n) ... def _validate(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError The difference is a mere difference of refactoring. Why should one of them be treated as "bah.__getitem__ raises itself" versus "bah.__getitem__ calls something which raises"? That's just an implementation detail. I think we're over-generalizing this problem. There's two actual issues here, and we shouldn't conflate them as the same problem: (1) People write buggy code based on invalid assumptions of what can and can't raise. E.g.: try: foo(baz[5]) except IndexError: ... # assume baz[5] failed (but maybe foo can raise too?) (2) There's a *specific* problem with property where a bug in your getter or setter that raises AttributeError will be masked, appearing as if the property itself doesn't exist. In the case of (1), there's nothing Python the language can do to fix that. The solution is to write better code. Question your assumptions. Think carefully about your pre-conditions and post-conditions and invariants. Plan ahead. Read the documentation of foo before assuming it won't raise. In other words, be a better programmer. If only it were that easy :-( (Aside: I've been thinking for a long time that design by contract is a very useful feature to have. It should be possibly to set a contract that states that this function won't raise a certain exception, and if it does, treat it as a runtime error. But this is still at a very early point in my thinking.) Python libraries rarely give a definitive list of what exceptions functions can raise, so unless you wrote it yourself and know exactly what it can and cannot do, defensive coding suggests that you assume any function call might raise any exception at all: try: item = baz[5] except IndexError: ... # assume baz[5] failed else: foo(item) Can we fix that? Well, maybe we should re-consider the rejection of PEP 463 (exception-catching expressions). https://www.python.org/dev/peps/pep-0463/ Maybe we need a better way to assert that a certain function won't raise a particular exception: try: item = bah[5] without IndexError: foo(item) except IndexError: ... # assume baz[5] failed (But how is that different from try...except...else?) In the case of (2), the masking of bugs inside property getters if they happen to raise AttributeError, I think the std lib can help with that. Perhaps a context manager or decorator (or both) that converts one exception to another? @property @bounce_exception(AttributeError, RuntimeError) def spam(self): ... Now spam.__get__ cannot raise AttributeError, if it does, it will be converted to RuntimeError. If you need finer control over the code that is guarded use the context manager form: @property def spam(self): with bounce_exception(AttributeError, RuntimeError): # guarded if condition: ... # not guarded raise AttributeError('property spam doesn't exist yet') -- Steve

Steven D'Aprano wrote:
They shouldn't be treated differently -- they're both legitimate ways for __getitem__ to signal that the item doesn't exist. What *should* be treated differently is if an IndexError occurs incidentally from something else that __getitem__ does. In other words, Y's __getitem__ should be written something like def __getitem__(self, n): self.validate(n) # If we get an IndexError from here on, it's a bug try: # work out the result and return it except IndexError as e: raise RuntimeError from e
Agreed. Case 1 can usually be handled by rewriting the code so as to make the scope of exception catching as narrow as possible. Case 2 needs to be addressed within the method concerned on a case-by-case basis. If there's a general principle there, it's something like this: If you're writing a method that uses an exception as part of it's protocol, you should catch any incidental occurrences of the same exception and reraise it as a different exception. I don't think there's anything more the Python language could do to help with either of those.
That sounds dangerously similar to Java's checked exceptions, which has turned out to be a huge nuisance and not very helpful.
It's no different, if I understand what it's supposed to mean correctly.
In the case of property getters, it seems to me you're almost always going to want that functionality, so maybe it should be incorporated into the property decorator itself. The only cases where you wouldn't want it would be if your property dynamically figures out whether it exists or not, and in those rare cases you would just have to write your own descriptor class. -- Greg

On 24 June 2017 at 22:31, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
While I used to think that, I'm no longer sure it's true, as it seems to me that a `contextlib.convert_exception` context manager could help with both of them. (That's technically the standard library helping, rather than the language per se, but it's still taking a currently obscure implementation pattern and making it more obvious by giving it a name) So if we assume that existed, and converted the given exception to RuntimeError (the same way PEP 479 does for StopIteration), we'd be able to write magic methods and properties in the following style: def __getitem__(self, n): self.validate(n) with contextlib.convert_exception(IndexError): # If we get an IndexError in here, it's a bug return self._getitem(n) That idiom then works the same way regardless of how far away your code is from the exception handler you're trying to bypass - you could just as easily put it inside the `self._getitem(n)` helper method instead. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Jun 24, 2017 at 11:45:25PM +1000, Nick Coghlan wrote:
Here is a recipe for such a context manager which is also useable as a decorator: https://code.activestate.com/recipes/580808-guard-against-an-exception-in-th... or just https://code.activestate.com/recipes/580808 It should work with Python 2.6 through 3.6 and later. try: with exception_guard(ZeroDivisionError): 1/0 # raises ZeroDivisionError except RuntimeError: print ('ZeroDivisionError replaced by RuntimeError') -- Steve

2017-06-25 Greg Ewing <greg.ewing@canterbury.ac.nz> dixit:
In "case 2", maybe some auto-zeroing flag (or even a decrementing counter?) could be useful? Please, consider the following draft: class ExtAttributeError(AttributeError): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._propagate_beyond = True # of course this should be a class and it should support not only # getters but also setters and deleters -- but you get the idea... def new_property(func): def wrapper(self): try: return func(self) except AttributeError as exc: if getattr(exc, '_propagate_beyond', False): exc._propagate_beyond = False raise raise RuntimeError( f'Unexpected {exc.__class__.__name__}') from exc return wrapper Then we could have: class Spam: @new_property def target(self): if len(self.targgets) == 1: return self.targets[0] raise ExtAttributeError( 'only exists when this has exactly one target') Cheers. *j

On 24/06/2017 11:03, Steven D'Aprano wrote:
try: item = (baz[5] except IndexError: SomeSentinelValue) if item == SomeSentinelValue: ... # assume baz[5] failed else: foo(item) is clunkier than the original version. Or am I missing something? Only if the normal and exceptional cases could be handled the same way would it help: foo(baz[5] except IndexError: 0) Rob Cliffe

On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:
(1) Under what circumstances is it not enough? (2) Is that list growing? (3) You seem to be implying that "catch narrow exceptions" is bad advice and we should catch Exception instead. How does that help?
There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479
StopIteration and generators have been around a long time, since Python 2.2 I think, so this is not new. To the extent this was a problem, it is fixed now.
2) nested ImportError 3) nested AttributeError
Both of those have been around since Python 1.x days, so not new either. If the list is growing, can you give some more recent examples?
I've never needed to write something like that for ImportError. It seems like an anti-pattern to me: sometimes it will silently swallow the exception, and now `user` will remain undefined, a landmine waiting to explode (with NameError) in your code.
Chris showed how to deal with 3). Catching nested exception is not what people want many times.
Isn't it? Why not? Can you explain further?
Am I the only one getting the impression that there's a common theme here?
I don't know what common theme you see. I can't see one. Do you actually have a proposal? -- Steve

On 23Jun2017 06:55, Steven D'Aprano <steve@pearwood.info> wrote:
I believe that he means that it isn't precise enough. In particular, "nested exceptions" to me, from his use cases, means exceptions thrown from within functions called at the top level. I want this control too sometimes. Consider: try: foo(bah[5]) except IndexError as e: ... infer that there is no bah[5] ... Of course, it is possible that bah[5] existed and that foo() raised an IndexError of its own. One might intend some sane handling of a missing bah[5] but instead silently conceal the IndexError from foo() by mishandling it as a missing bah[5]. Naturally one can rearrange this code to call foo() outside that try/except, but that degree of control often leads to quite fiddly looking code with the core flow obscured by many tiny try/excepts. One can easily want, instead, some kind of "shallow except", which would catch exceptions only if they were directly raised from the surface code; such a construct would catch the IndexError from a missing bah[5] in the example above, but _not_ catch an IndexError raised from deeper code such within the foo() function. Something equivalent to: try: foo(bah[5]) except IndexError as e: if e.__traceback__ not directly from the try..except lines: raise ... infer that there is no bah[5] ... There doesn't seem to be a concise way to write that. It might not even be feasible at all, as one doesn't have a way to identify the line(s) within the try/except in a form that one can recognise in a traceback. I can imagine wanting to write something like this: try: foo(bah[5]) except shallow IndexError as e: ... deduce that there is no bah[5] ... Note that one can then deduce the missing bah[5] instead of inferring it. Obviously the actual syntax above is a nonstarter, but something that succinct and direct would be very handy. The nested exception issue actually bites me regularly, almost always with properties. The property system appears designed to allow one to make "conditional" properties, which appear to exist only in some circumstances. I wrote one of them just the other day, along the lines of: @property def target(self): if len(self.targets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') However, more commonly I end up hiding coding errors with @property, particularly nasty when the coding error is deep in some nested call. Here is a nondeep example based on the above: @property def target(self): if len(self.targgets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Here I have misspelt ".targets" as ".targgets". And quietly the .target property is simply missing, and a caller may then infer, incorrectly, things about the number of targets. What I, as the coder, actually wanted was for the errant .targgets reference to trigger something different from Attribute error, something akin to a NameError. (Obviously it _is_ a missing attribute and that is what AttributeError is for, but within a property that is ... unhelpful.) This is so common that I actually keep around a special hack: def prop(func): ''' The builtin @property decorator lets internal AttributeErrors escape. While that can support properties that appear to exist conditionally, in practice this is almost never what I want, and it masks deeper errors. Hence this wrapper for @property that transmutes internal AttributeErrors into RuntimeErrors. ''' def wrapper(*a, **kw): try: return func(*a, **kw) except AttributeError as e: e2 = RuntimeError("inner function %s raised %s" % (func, e)) if sys.version_info[0] >= 3: try: eval('raise e2 from e', globals(), locals()) except: # FIXME: why does this raise a SyntaxError? raise e else: raise e2 return property(wrapper) and often define properties like this: from cs.py.func import prop ....... @prop def target(self): if len(self.targgets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Same shape, better semantics from a debugging point of view. This is just one example where "nested" exceptions can be a misleading behavioural symptom.
I hope this real world example shows why the scenario is real, and that my discussion shows why for me at least it would be handy to _easily_ catch the "shallow" exception only. Cheers, Cameron Simpson <cs@zip.com.au>

On 22Jun2017 19:47, Andy Dirnberger <dirn@dirnonline.com> wrote:
That is the kind of refactor to which I alluded in the paragraph above. Doing that a lot tends to obscure the core logic of the code, hence the desire for something more succinct requiring less internal code fiddling. Cheers, Cameron Simpson <cs@zip.com.au>

On 23 June 2017 at 15:20, Sven R. Kunze <srkunze@mail.de> wrote:
At this point, it becomes unclear to me what constitutes an "intentional" IndexError, as opposed to an "unintentional" one, at least in any sense that can actually be implemented. I appreciate that you want IndexError to mean "there is no 5th element in bah". But if bah has a __getitem__ that raises IndexError for any reason other than that, then the __getitem__ implementation has a bug. And while it might be nice to be able to continue working properly even when the code you're executing has bugs, I think it's a bit optimistic to hope for :-) On the other hand, I do see the point that insisting on finer and finer grained exception handling ultimately ends up with unreadable code. But it's not a problem I'd expect to see much in real life code (where code is either not written that defensively, because either there's context that allows the coder to make assumptions that objects will behave reasonably sanely, or the code gets refactored to put the exception handling in a function, or something like that). Paul

On 23Jun2017 15:59, Paul Moore <p.f.moore@gmail.com> wrote:
While I agree that in object with its own __getitem__ would look "deep", what I was actually suggesting as a possibility was a "shallow" except catch, not some magic "intentional" semantic. A shallow catch would effectively need to mean "the exceptions uppermost traceback frame referers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods. [...snip...]
Sure, there are many circumstances where a succinct "shallow catch" might not be useful. But there are also plenty of circumstances where one would like just this flavour of precision. Cheers, Cameron Simpson <cs@zip.com.au>

Hi Andy, What you propose is essentially the "try .. catch .. in" construct as described for Standard ML in: https://pdfs.semanticscholar.org/b24a/60f84b296482769bb6752feeb3d93ba6aee8.p... Something similar for Clojure is at: https://github.com/rufoa/try-let So clearly this is something more people have struggled with. The paper above goes into deep detail on the practical and (proof-)theoretical advantages of such a construct. Stephan 2017-06-23 1:47 GMT+02:00 Andy Dirnberger <dirn@dirnonline.com>:

Hi Stephan, On Fri, Jun 23, 2017 at 6:23 AM, Stephan Houben <stephanh42@gmail.com> wrote:
It's not really a proposal. It's existing syntax. I was suggesting a way to implement the example that would catch an IndexError raised by accessing elements in bah but not those raised by foo.
Andy

2017-06-23 17:09 GMT+02:00 Andy Dirnberger <dirn@dirnonline.com>:
It's not really a proposal. It's existing syntax.
Wow! I have been using Python since 1.5.2 and I never knew this. This is not Guido's famous time machine in action, by any chance? Guess there's some code to refactor using this construct now... Stephan 2017-06-23 17:09 GMT+02:00 Andy Dirnberger <dirn@dirnonline.com>:

On 23 June 2017 at 09:29, Cameron Simpson <cs@zip.com.au> wrote:
Slight tangent, but I do sometimes wonder if adding a decorator factory like the following to functools might be useful: def raise_when_returned(return_exc): def decorator(f): @wraps(f) def wrapper(*args, **kwds): try: result = f(*args, **kwds) except selective_exc as unexpected_exc: msg = "inner function {} raised {}".format(f, unexpected_exc) raise RuntimeError(msg) from unexpected_exc if isinstance(result, return_exc): raise result return result It's essentially a generalisation of PEP 479 to arbitrary exception types, since it lets you mark a particular exception type as being communicated back to the wrapper via the return channel rather than as a regular exception: def with_traceback(exc): try: raise exc except BaseException as caught_exc: return caught_exc @property @raise_when_returned(AttributeError) def target(self): if len(self.targets) == 1: return self.targets[0] return with_traceback(AttributeError('only exists when this has exactly one target')) The part I don't like about that approach is the fact that you need to mess about with the exception internals to get a halfway decent traceback on the AttributeError. The main alternative would be to add a "convert_exception" context manager in contextlib, so you could write the example property as: @property def target(self): with convert_exception(AttributeError): if len(self.targets) == 1: return self.targets[0] raise AttributeError('only exists when this has exactly one target') Where "convert_exception" would be something like: def convert_exception(exc_type): """Prevents the given exception type from escaping a region of code by converting it to RuntimeError""" if not issubclass(exc_type, Exception): raise TypeError("Only Exception subclasses can be flagged as unexpected") try: yield except exc_type as unexpected_exc: new_exc = RuntimeError("Unexpected exception") raise new_exc from unexpected_exc The API for this could potentially be made more flexible to allow easy substition of lookup errors with attribute errors and vice-versa (e.g. via additional keyword-only parameters) To bring the tangent back closer to Sven's original point, there are probably also some parts of the import system (such as executing the body of a found module) where the case can be made that we should be converting ImportError to RuntimeError, rather than letting the ImportError escape (with essentially the same rationale as PEP 479). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 23Jun2017 11:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
Funnily enough I have an @transmute decorator which serves just this purpose. It doesn't see as much use as I might imagine, but that is partially because my function predates "raise ... from", which meant that it loses the stack trace from the transmuted exception, impeding debugging. I need to revisit it with that in mind. So yes, your proposed decorator has supporting real world use cases in my world. Cheers, Cameron Simpson <cs@zip.com.au>

On Fri, Jun 23, 2017 at 09:29:23AM +1000, Cameron Simpson wrote:
But why teach it to newbies? Sven explicitly mentions teaching beginners. If we are talking about advanced features for experts, that's one thing, but it's another if we're talking about Python 101 taught to beginners and newbies. Do we really need to be teaching beginners how to deal with circular imports beyond "don't do it"?
Indeed -- if both foo and bah[5] can raise IndexError when the coder believes that only bah[5] can, then the above code is simply buggy. On the other hand, if the author is correct that foo cannot raise IndexError, then the code as given is fine.
Sadly, that is often the nature of real code, as opposed to "toy" or textbook code that demonstrates an algorithm as cleanly as possible. It's been said that for every line of code in the textbook, the function needs ten lines in production.
I think the concept of a "shallow exception" is ill-defined, and to the degree that it is defined, it is *dangerous*: a bug magnet waiting to strike. What do you mean by "directly raised from the surface code"? Why is bah[5] "surface code" but foo(x) is not? But call a function (or method). But worse, it seems that the idea of "shallow" or "deep" depends on *implementation details* of where the exception comes from. For example, changing from a recursive function to a while loop might change the exception from "50 function calls deep" to "1 function deep". What makes bah[5] "shallow"? For all you know, it calls a chain of a dozen __getitem__ methods, due to inheritance or proxying, before the exception is actually raised. Or it might call just a single __getitem__ method, but the method's implementation puts the error checking into a helper method: def __getitem__(self, n): self._validate(n) # may raise IndexError ... How many function calls are shallow, versus deep? This concept of a shallow exception is, it seems to me, a bug magnet. It is superficially attractive, but then you realise that: try: spam[5] except shallow IndexError: ... will behave differently depending on how spam is implemented, even if the interface (raises IndexError) is identical. It seems to me that this concept is trying to let us substitute some sort of undefined but mechanical measurement of "shallowness" for actually understanding what our code does. I don't think this can work. It would be awesome if there was some way for our language to Do What We Mean instead of What We Say. And then we can grow a money tree, and have a magic plum-pudding that stays the same size no matter how many slices we eat, and electricity so cheap the power company pays you to use it... *wink*
The obvious solution to this is to learn to spell correctly :-) Actually, a linter probably would have picked up that typo. But I do see that the issue if more than just typos. [...]
Because "raise" is a statement, not an expression. You need exec(). -- Steve

On 24Jun2017 05:02, Steven D'Aprano <steve@pearwood.info> wrote:
It depends. Explaining that exceptions from called code can be mishandled by a naive try/except is something newbies need to learn to avoid common pitfalls with exceptions, and a real world situation that must be kept in mind when acting on caught exceptions.
Do we really need to be teaching beginners how to deal with circular imports beyond "don't do it"?
Sven's example is with import. The situation is more general. [... snip basic example of simple code where IndexError can arise from multiple causes ...]
But not always so. And the reason for many language constructs and idioms is explicitly to combat what would otherwise need 10 lines of code (or 100 to do it "right" with corner cases), obscuring the core task and reducing readability/maintainability. So "Sadly, that is often the nature of real code" is not itself an argument against this idea.
I've replied to Paul Moore and suggested this definition as implementable and often useful: A shallow catch would effectively need to mean "the exception's uppermost traceback frame refers to one of the program lines in the try/except suite". Which would work well for lists and other builtin types. And might be insufficient for a duck-type with python-coded dunder methods. The target here is not perform magic but to have a useful tool to identify exceptions that arise fairly directly from the adjacent clode and not what it calls. Without writing cumbersome and fragile boilerplate to dig into an exception's traceback. Cheers, Cameron Simpson <cs@zip.com.au>

On 2017-06-23 23:56, Cameron Simpson wrote:
On 24Jun2017 05:02, Steven D'Aprano <steve@pearwood.info> wrote: [snip]
I think a "shallow exception" would be one that's part of a defined API, as distinct from one that is an artifact of the implementation, a leak in the abstraction. It's like when "raise ... from None" was introduced to help in those cases where you want to replace an exception that's a detail of the (current) internal implementation with one that's intended for the user.

On 24.06.2017 01:37, MRAB wrote:
I like the "shallow exception" idea most. It's simple and it covers most if not all issues. You also hit the nail with pointing to leaking abstractions.
Regards, Sven PS: This "shallow exception" proposal could help e.g. Django improving their template system. Here's it's the other way round: the exception handling is done by Django and depending on the exception it will fall back to a different attribute access method. I can remember us implementing such a method which accidentally raised a caught exception which we then never saw. Debugging this was a mess and took a quite some time.

On 27 June 2017 at 02:29, Sven R. Kunze <srkunze@mail.de> wrote:
The shallow exception notion breaks a fairly fundamental refactoring principle in Python: you should be able to replace an arbitrary expression with a subfunction or subgenerator that produces the same result without any of the surrounding code being able to tell the difference. By contrast, Steven's exception_guard recipe just takes the existing "raise X from Y" feature, and makes it available as a context manager and function decorator. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27.06.2017 13:41, Nick Coghlan wrote:
I would agree with you here but this "refactoring principle in Python" doesn't work for control flow. Just look at "return", "break", "continue" etc. Exceptions are another way of handling control flow. So, this doesn't apply here IMO.
I don't see how this helps differentiating shallow and nested exceptions such as: try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib Regards, Sven PS: this has nothing to do with cyclic imports. It can be a misconfiguration of the system which fails nested imports. In those cases, we fallback silently.

On Wed, Jun 28, 2017 at 5:49 AM, Sven R. Kunze <srkunze@mail.de> wrote:
The ability to safely refactor control flow is part of why 'yield from' exists, and why PEP 479 changed how StopIteration bubbles. Local control flow is hard to refactor, but exceptions are global control flow, and most certainly CAN be refactored safely. ChrisA

On 28 June 2017 at 06:03, Chris Angelico <rosuav@gmail.com> wrote:
And PEP 479 establishes a precedent for how we handle the cases where we decide we *don't* want a particular exception type to propagate normally: create a boundary on the stack that converts the otherwise ambiguous exception type to RuntimeError. While generator functions now do that implicitly for StopIteration, and "raise X from Y" lets people write suitable exception handlers themselves, we don't offer an easy way to do it with a context manager (with statement as stack boundary), asynchronous context manager (async with statement as stack boundary), or a function decorator (execution frame as stack boundary). So while I prefer "contextlib.convert_exception" as the name (rather than the "exception_guard" Steven used in his recipe), I'd definitely be open to a bugs.python.org RFE and a PR against contextlib to add such a construct to Python 3.7. We'd have a few specific API details to work out (e.g. whether or not to accept arbitrary conversion functions as conversion targets in addition to accepting exception types and iterables of exception types, whether or not to allow "None" as the conversion target to get the same behaviour as "contextlib.suppress"), but I'm already sold on the general concept. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Jun 28, 2017 at 12:25 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree broadly, but I'm sure there'll be the usual ton of bikeshedding about the details. The idea behind this decorator, AIUI, is a declaration that "a FooException coming out of here is a bug", and if I were looking for that, I'd look for something about the function leaking an exception, or preventing exceptions. So maybe convert_exception will work, but definitely have a docs reference from contextlib.suppress to this ("if exceptions of this type would indicate code bugs, consider convert_exception instead"). In my testing, I've called it "no_leak" or some variant thereon, though that's a shorthand that wouldn't suit the stdlib. ChrisA

On 28 June 2017 at 13:16, Chris Angelico <rosuav@gmail.com> wrote:
Right, and I'd like us to keep in mind the KeyError -> AttributeError (and vice-versa) use case as well. Similar to ExitStack, it would be appropriate to make some additions to the "recipes" section in the docs that covered things like "Keep AttributeError from being suppressed in a property implementation". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28.06.2017 08:00, Nick Coghlan wrote:
As it was snipped away, let me ask again: I don't see how this helps differentiating shallow and nested exceptions such as: try: with exception_guard(ImportError): import myspeciallib except RuntimeError: # catches shallow and nested ones import fallbacks.MySpecialLib as myspeciallib At least in my tests, exception_guard works this way and I don't see any improvements to current behavior. Moreover, I am somewhat skeptical that using this recipe will really improve the situation. It's a lot of code where users don't have any stdlib support. I furthermore doubt that all Python coders will now wrap their properties using the guard. So, using these properties we will have almost no improvement. I still don't see it as the responsibility of coder of the property to guard against anything. Nobody is forced to catch exceptions when using a property. If that's the "best" outcome, I will stick to https://stackoverflow.com/questions/20459166/how-to-catch-an-importerror-non... because 1) Google finds it for me and 2) we don't have to maintain 100 lines of code ourself. Regards, Sven

On 28 June 2017 at 21:48, Sven R. Kunze <srkunze@mail.de> wrote:
There are two answers to that: 1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about 2. There's a reasonable case to be made that importlib should include an ImportError -> RuntimeError conversion around the call to loader.exec_module (in the same spirit as PEP 479). That way, in: try: import myspeciallib except ImportError: import fallbacks.MySpecialLib as myspeciallib any caught ImportError would relate to "myspeciallib", while uncaught ImportErrors arising from *executing* "myspeciallib" will be converted to RuntimeError, with the original ImportError as their __cause__. So it would make sense to file an RFE against 3.7 proposing that behavioural change (we couldn't reasonably do anything like that with the old `load_module()` importer API, as raising ImportError was how that API signalled "I can't load that". We don't have that problem with `exec_module()`).
Honestly, if folks are trying to write complex Python code without using at least "pylint -E" as a static check for typos in attribute names (regardless of whether those lines get executed or not), then inadvertently hiding AttributeError in property and __getattr__ implementations is likely to be the least of their problems. So pylint's structural checks, type analysis tools like MyPy, or more advanced IDEs like PyCharm are typically going to be a better option for folks wanting to guard against bugs in their *own* code than adding defensive code purely as a cross-check on their own work. The cases I'm interested in are the ones where you're either developing some kind of framework and you need to code that framework defensively to guard against unexpected failures in the components you're executing (e.g. exec_module() in the PEP 451 import protocol), or else you're needing to adapt between two different kinds of exception reporting protocol (e.g. KeyError to AttributeError and vice-versa). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jun 28, 2017 6:26 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote: On 28 June 2017 at 21:48, Sven R. Kunze <srkunze@mail.de> wrote:
There are two answers to that: 1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about 2. There's a reasonable case to be made that importlib should include an ImportError -> RuntimeError conversion around the call to loader.exec_module (in the same spirit as PEP 479). That way, in: try: import myspeciallib except ImportError: import fallbacks.MySpecialLib as myspeciallib any caught ImportError would relate to "myspeciallib", while uncaught ImportErrors arising from *executing* "myspeciallib" will be converted to RuntimeError, with the original ImportError as their __cause__. So it would make sense to file an RFE against 3.7 proposing that behavioural change (we couldn't reasonably do anything like that with the old `load_module()` importer API, as raising ImportError was how that API signalled "I can't load that". We don't have that problem with `exec_module()`). What about modules that want to raise ImportError to indicate that they aren't available on the current system, perhaps because some of their dependencies are missing? For example, 'import ssl' should raise an ImportError if 'ssl.py' is present but '_ssl.so' is missing; the existence of '_ssl.so' is an internal implementation detail. And perhaps 'import trio.ssl' should raise ImportError if 'ssl' is missing. (Historically not all systems have openssl available, so this is a common situation where existing libraries contain ImportError guards.) With PEP 479 there was a different and better way to generate a StopIteration if you wanted one (just 'return'). Here I'm afraid existing projects might actually be relying on the implicit exception leakage in significant numbers :-/ More generally, my impression is that this is one of the reasons why exceptions have fallen out of favor in more recent languages. They're certainly workable, and python's certainly not going to change now, but they do have these awkward aspects that weren't as clear 20 years ago and that now we have to live with. -n

On 28.06.2017 21:14, Nathaniel Smith wrote:
My concern as well.
I am quite satisfied with the ability of exceptions to expose bugs as fast and clear as possible. I still think we can improve on the catching side a little bit to narrow down the relevant exceptions. Other than that, I would be interested to hear what system you have in mind. What alternative (borrowed from more recent languages) can you imagine? Checking return values like C or golang? No ability to catch them at all? How to handle bugs in the context of UI applications where a crash in front of the user should be avoided? Regards, Sven

On 29 June 2017 at 05:14, Nathaniel Smith <njs@pobox.com> wrote:
Hence "it may be worth filing an RFE so we can discuss the implications", rather than "we should definitely do it". The kinds of cases you cite will already fail for import guards that check for "exc.name == 'the_module_being_imported'" though, so I'd be OK with requiring modules that actually wanted that behaviour to do: try: import other_module except ImportError as exc: raise ImportError("other_module is missing", name=__name__, path=__file__) from exc The guard around exec_module() would then look something like: try: loader.exec_module(mod) except ImportError as exc: if exc.name == mod.__name__: raise msg = f"Failed to import '{exc.name}' from '{mod.__name__}'" raise RuntimeError(msg) from exc Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28.06.2017 15:26, Nick Coghlan wrote:
1. In 3.3+ you can just catch ImportError, check "exc.name", and re-raise if it's not for the module you care about
I see, didn't know that one. I gave it a try and it's not 100% the behavior I have expected, but one could workaround if the valid package structure is known. Not sure for certain. "from foo.bar.baz import abc" can yield to "exc.name" being one of "foo", "foo.bar" or "foo.bar.baz". Not perfect but sort of doable.
Generally changing the behavior for ImportError doesn't sound like it would work for all projects out there. For fallback imports, I am on your side, that's a real use case which can be solved by changing the behavior of ImportErrors. But for all imports? I don't know if that's good idea.
[People should use tools, guard against bugs and try to avoid mistakes.]
Sure, but I don't see who this can help, if I use third-party code. The cases, which I described in the original post, were simple cases, where we catch too many exception. So, I don't even have the chance to see the error, to file a bug report, to issue a pull request, etc. etc.
I am unsure what you mean by those abstract words "framework" and "components". But let me state it in different words: there are *raisers* and *catchers* which do the respective thing with exceptions. If you control the code on both sides, things are easy to change. Pre-condition: you know the bug in the first place, which is hard when you catch too much. If you control the raiser only, it doesn't help to say: "don't make mistakes, configure systems right, code better, etc." People will make mistakes, systems will be misconfigured, linters don't find everything, etc. If you control the catcher only, you definitely want to narrow down the amount of caught exceptions as far as possible. This was the original intend of this thread IIRC. This way you help to discover bugs in raising code. Addition benefit, you catching code reacts only to the right exceptions. One word about frameworks here. Django, for instance, is on both sides. The template engine is mostly on the catchers side, whereas the database layer is on the raisers side. I get the feeling that the solutions presented here are way too complicated and error-prone. My opinion on this topic still is that catching exceptions is not mandatory. Nobody is forced to do it and it's even better to let exceptions bubble up to visualize bugs. If one still needs to catch them, he should only catch those he really, really needs to catch and nothing more. If this cannot be improved sensibly, well, so be it. Although I still don't find the argument presented against "catching shallow exception" a little bit too abstract compared to the practical benefit. Maybe, there's a better solution, maybe not. Regards, Sven

On Wed, Jun 28, 2017 at 12:25:12PM +1000, Nick Coghlan wrote: [...]
http://bugs.python.org/issue30792 -- Steve

Cameron Simpson wrote:
The problem I see with that is how to define what counts as "surface code". If the __getitem__ method of bah is written in Python, I don't see how you could tell that an IndexError raised by it should be caught, but one raised by foo() shouldn't. In any case, this doesn't address the issue raised by the OP, which in this example is that if the implementation of bah.__getitem__ calls something else that raises an IndexError, there's no easy way to distinguish that from one raised by bah.__getitem__ itself. I don't see any way to solve that by messing around with different try-except constructs. It can only be addressed from within bah.__getitem__ itself, by having it catch any incidental IndexErrors and turn them into a different exception. -- Greg

On Sat, Jun 24, 2017 at 01:02:55PM +1200, Greg Ewing wrote:
I'm not convinced that's a meaningful distinction to make in general. Consider the difference between these two classes: class X: def __getitem__(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError ... class Y: def __getitem__(self, n): self._validate(n) ... def _validate(self, n): if n < 0: n += len(self) if not 0 <= n < len(self): raise IndexError The difference is a mere difference of refactoring. Why should one of them be treated as "bah.__getitem__ raises itself" versus "bah.__getitem__ calls something which raises"? That's just an implementation detail. I think we're over-generalizing this problem. There's two actual issues here, and we shouldn't conflate them as the same problem: (1) People write buggy code based on invalid assumptions of what can and can't raise. E.g.: try: foo(baz[5]) except IndexError: ... # assume baz[5] failed (but maybe foo can raise too?) (2) There's a *specific* problem with property where a bug in your getter or setter that raises AttributeError will be masked, appearing as if the property itself doesn't exist. In the case of (1), there's nothing Python the language can do to fix that. The solution is to write better code. Question your assumptions. Think carefully about your pre-conditions and post-conditions and invariants. Plan ahead. Read the documentation of foo before assuming it won't raise. In other words, be a better programmer. If only it were that easy :-( (Aside: I've been thinking for a long time that design by contract is a very useful feature to have. It should be possibly to set a contract that states that this function won't raise a certain exception, and if it does, treat it as a runtime error. But this is still at a very early point in my thinking.) Python libraries rarely give a definitive list of what exceptions functions can raise, so unless you wrote it yourself and know exactly what it can and cannot do, defensive coding suggests that you assume any function call might raise any exception at all: try: item = baz[5] except IndexError: ... # assume baz[5] failed else: foo(item) Can we fix that? Well, maybe we should re-consider the rejection of PEP 463 (exception-catching expressions). https://www.python.org/dev/peps/pep-0463/ Maybe we need a better way to assert that a certain function won't raise a particular exception: try: item = bah[5] without IndexError: foo(item) except IndexError: ... # assume baz[5] failed (But how is that different from try...except...else?) In the case of (2), the masking of bugs inside property getters if they happen to raise AttributeError, I think the std lib can help with that. Perhaps a context manager or decorator (or both) that converts one exception to another? @property @bounce_exception(AttributeError, RuntimeError) def spam(self): ... Now spam.__get__ cannot raise AttributeError, if it does, it will be converted to RuntimeError. If you need finer control over the code that is guarded use the context manager form: @property def spam(self): with bounce_exception(AttributeError, RuntimeError): # guarded if condition: ... # not guarded raise AttributeError('property spam doesn't exist yet') -- Steve

Steven D'Aprano wrote:
They shouldn't be treated differently -- they're both legitimate ways for __getitem__ to signal that the item doesn't exist. What *should* be treated differently is if an IndexError occurs incidentally from something else that __getitem__ does. In other words, Y's __getitem__ should be written something like def __getitem__(self, n): self.validate(n) # If we get an IndexError from here on, it's a bug try: # work out the result and return it except IndexError as e: raise RuntimeError from e
Agreed. Case 1 can usually be handled by rewriting the code so as to make the scope of exception catching as narrow as possible. Case 2 needs to be addressed within the method concerned on a case-by-case basis. If there's a general principle there, it's something like this: If you're writing a method that uses an exception as part of it's protocol, you should catch any incidental occurrences of the same exception and reraise it as a different exception. I don't think there's anything more the Python language could do to help with either of those.
That sounds dangerously similar to Java's checked exceptions, which has turned out to be a huge nuisance and not very helpful.
It's no different, if I understand what it's supposed to mean correctly.
In the case of property getters, it seems to me you're almost always going to want that functionality, so maybe it should be incorporated into the property decorator itself. The only cases where you wouldn't want it would be if your property dynamically figures out whether it exists or not, and in those rare cases you would just have to write your own descriptor class. -- Greg
participants (13)
-
Andy Dirnberger
-
Cameron Simpson
-
Chris Angelico
-
Greg Ewing
-
Jan Kaliszewski
-
MRAB
-
Nathaniel Smith
-
Nick Coghlan
-
Paul Moore
-
Rob Cliffe
-
Stephan Houben
-
Steven D'Aprano
-
Sven R. Kunze