So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout It'd be nice if there was a way to... make this easier to deal with. Perhaps something where, something like this: def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout becomes: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() code_we_assume_is_safe() if args.something and some_condition: let_exception_through = True raise ExceptionWeCareAbout except ExceptionWeCareAbout as exc: if let_exception_through: raise else: raise RuntimeError from exc (or equivalent) and something like: def foo() with Bar: try: baz() except Bar: raise becomes: def foo(): try: try: baz() except Bar: allow_exception = True raise except Bar as exc: if allow_exception: raise else: raise RuntimeError from exc (thus allowing explicit exception propagation) Additionally, note that it would only apply to the raise itself - something like `raise may_raise_the_same_error()` would desugar as if: exception = may_raise_the_same_error() allow_exception = True raise exception Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc. It does, however, encourage better exception hygiene. Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted. We know something like this would've saved us a lot of trouble, so we're sharing the idea in the hopes it can, in the future, help others.
On Thu, Sep 30, 2021 at 11:03 AM Soni L. <fakedme+py@gmail.com> wrote:
So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions:
def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout
It'd be nice if there was a way to... make this easier to deal with. Perhaps something where, something like this:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout
becomes:
def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() code_we_assume_is_safe() if args.something and some_condition: let_exception_through = True raise ExceptionWeCareAbout except ExceptionWeCareAbout as exc: if let_exception_through: raise else: raise RuntimeError from exc
(or equivalent)
and something like:
def foo() with Bar: try: baz() except Bar: raise
becomes:
def foo(): try: try: baz() except Bar: allow_exception = True raise except Bar as exc: if allow_exception: raise else: raise RuntimeError from exc
(thus allowing explicit exception propagation)
Additionally, note that it would only apply to the raise itself - something like `raise may_raise_the_same_error()` would desugar as if:
exception = may_raise_the_same_error() allow_exception = True raise exception
Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc. It does, however, encourage better exception hygiene. Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted. We know something like this would've saved us a lot of trouble, so we're sharing the idea in the hopes it can, in the future, help others.
Can you give a realistic example of how this works, and how you could accidentally leak the exact same exception that you'd be intentionally raising? It looks to me like you possibly should be splitting the function into the recursive part and the setup part, where only the setup is capable of raising that exception. I've seen a LOT of bad Python code caused by an assumption that "recursion" must always mean "calling the external API". A much better model, in a lot of cases, is: def _somefunc_recursive(x, y, z): ... _somefunc_recursive(x, y + 1, z - 1) def somefunc(x, y, z=4): ... _somefunc_recursive(x, y, z) return someresult ChrisA
On 2021-09-29 10:09 p.m., Chris Angelico wrote:
On Thu, Sep 30, 2021 at 11:03 AM Soni L. <fakedme+py@gmail.com> wrote:
So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions:
def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout
It'd be nice if there was a way to... make this easier to deal with. Perhaps something where, something like this:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout
becomes:
def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() code_we_assume_is_safe() if args.something and some_condition: let_exception_through = True raise ExceptionWeCareAbout except ExceptionWeCareAbout as exc: if let_exception_through: raise else: raise RuntimeError from exc
(or equivalent)
and something like:
def foo() with Bar: try: baz() except Bar: raise
becomes:
def foo(): try: try: baz() except Bar: allow_exception = True raise except Bar as exc: if allow_exception: raise else: raise RuntimeError from exc
(thus allowing explicit exception propagation)
Additionally, note that it would only apply to the raise itself - something like `raise may_raise_the_same_error()` would desugar as if:
exception = may_raise_the_same_error() allow_exception = True raise exception
Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc. It does, however, encourage better exception hygiene. Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted. We know something like this would've saved us a lot of trouble, so we're sharing the idea in the hopes it can, in the future, help others.
Can you give a realistic example of how this works, and how you could accidentally leak the exact same exception that you'd be intentionally raising? It looks to me like you possibly should be splitting the function into the recursive part and the setup part, where only the setup is capable of raising that exception.
I've seen a LOT of bad Python code caused by an assumption that "recursion" must always mean "calling the external API". A much better model, in a lot of cases, is:
def _somefunc_recursive(x, y, z): ... _somefunc_recursive(x, y + 1, z - 1)
def somefunc(x, y, z=4): ... _somefunc_recursive(x, y, z) return someresult
def get_property_value(self, prop): """Returns the value associated with the given property. If duplicated, an earlier value should override a later value. Args: prop (DataProperty): The property. Returns: The value associated with the given property. Raises: PropertyError: If the property is not supported by this data source. LookupError: If the property is supported, but isn't available. ValueError: If the property doesn't have exactly one value. """ iterator = self.get_property_values(prop) try: # note: unpacking ret, = iterator except LookupError as exc: raise RuntimeError from exc # don't accidentally swallow bugs in the iterator return ret (real code. spot all the exception-related time-bombs! the class this is in... well, one of its subclasses wraps other instances of this class. it gets Fun when you add up all the ways downstream code can be wrong and interact badly with this. in this case, the main concern is about swallowing ValueError, altho no harm would be caused by also wrapping unexpected PropertyErrors into RuntimeErrors. anyway, this thing is a bit of a maintenance nightmare, but this idea would help a lot.) we believe this is how we'd write it with this idea: def get_property_value(self, prop) with ValueError, LookupError, PropertyError: """Returns the value associated with the given property. If duplicated, an earlier value should override a later value. Args: prop (DataProperty): The property. Returns: The value associated with the given property. Raises: PropertyError: If the property is not supported by this data source. LookupError: If the property is supported, but isn't available. ValueError: If the property doesn't have exactly one value. """ try: iterator = self.get_property_values(prop) except PropertyError, LookupError: raise # note: unpacking try: ret = next(iterator) except StopIteration as exc: raise ValueError from exc try: next(iterator) raise ValueError except StopIteration: pass return ret we strongly believe this would fix the relevant time-bombs :)
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XDLYGF... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Sep 29, 2021 at 10:01:52PM -0300, Soni L. wrote:
So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions:
Is that the Royal We or are you actually speaking on behalf of other people?
def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout
I don't see an obvious recursive call there, although I suppose it could be buried in some.user_code(). It isn't clear to me why it matters that this could be recursive, or why you raise RuntimeError. By the way, you may not need the "raise...from exc" syntax. The difference between: except NameError: raise RuntimeError and except NameError as exc: raise RuntimeError from exc is *extremely* minimal. In a practical sense, the main difference is that the first traceback will say: NameError traceback During handling of the above exception, another exception occurred RuntimeError traceback and the second will say: NameError traceback The above exception was the direct cause of the following exception RuntimeError traceback So it might not matter that much to you. Anyway, that's just a minor aside.
It'd be nice if there was a way to... make this easier to deal with.
It isn't clear to me what part of the above isn't already easy to deal with. Maybe you need a more realistic, runnable, example that demonstrates why this is an actual problem? And maybe the actual problem is not exception handling, but a sub-optimal design for your algorithm. (More comments to follow.) -- Steve
In Soni's original code snippet, there is a clear separation of code that is inside the try block from code that is outside the try block:
def a_potentially_recursive_function(some, args): try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() # and more code following...
This is good exception hygiene. Only some.user_code is guarded by the try block. If it turns out that code_we_assume_is_safe is not actually safe, and fails with an exception, it won't be caught by the try block and you will know about it. In your re-written syntactic sugar, you have: # new syntax
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() # and more code following...
which becomes:
def a_potentially_recursive_function(some, args): try: some.user_code() code_we_assume_is_safe() # and more code following... except ExceptionWeCareAbout as exc:
which is the opposite of good exception hygiene. Too much code, including the wrong code, is guarded by the try block, which means that the compiler has to *guess your meaning* and set a flag to decide whether to re-raise the exception or run the except block. Its not clear how the compiler guesses that. Is it only because you have an explicit `raise ExceptionWeCareAbout` in the code? What if the exception is not explicit? # explicitly raising ExceptionWeCareAbout raise ExceptionWeCareAbout # not explicit raise some_exception_object_we_prepared_earlier raise prepare_exception(*args) # returns an ExceptionWeCareAbout instance verify_or_raise(condition) # raises ExceptionWeCareAbout
Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc.
So that's two points against it.
It does, however, encourage better exception hygiene.
Except it doesn't, it makes it worse. So that's three points against it.
Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted.
Or based on your examples, increase the number of swallowed exceptions, and be much harder to refactor code safely. -- Steve
On 2021-09-29 11:46 p.m., Steven D'Aprano wrote:
In Soni's original code snippet, there is a clear separation of code that is inside the try block from code that is outside the try block:
def a_potentially_recursive_function(some, args): try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() # and more code following...
This is good exception hygiene.
Only some.user_code is guarded by the try block. If it turns out that code_we_assume_is_safe is not actually safe, and fails with an exception, it won't be caught by the try block and you will know about it.
Except no, because ExceptionWeCareAbout is part of the public API. If it's not actually safe, it might raise an ExceptionWeCareAbout... ... which gets promptly swallowed by the API consumer, who thinks it was intentional. But any ExceptionWeCareAbout in user code, because it's explicitly guarded against/wrapped, doesn't get silently swallowed the same way. [no comments on the rest of your points because they're all based on this core misunderstanding.]
In your re-written syntactic sugar, you have:
# new syntax
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() # and more code following...
which becomes:
def a_potentially_recursive_function(some, args): try: some.user_code() code_we_assume_is_safe() # and more code following... except ExceptionWeCareAbout as exc:
which is the opposite of good exception hygiene. Too much code, including the wrong code, is guarded by the try block, which means that the compiler has to *guess your meaning* and set a flag to decide whether to re-raise the exception or run the except block.
Its not clear how the compiler guesses that. Is it only because you have an explicit `raise ExceptionWeCareAbout` in the code? What if the exception is not explicit?
# explicitly raising ExceptionWeCareAbout raise ExceptionWeCareAbout
# not explicit raise some_exception_object_we_prepared_earlier
raise prepare_exception(*args) # returns an ExceptionWeCareAbout instance
verify_or_raise(condition) # raises ExceptionWeCareAbout
Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc.
So that's two points against it.
It does, however, encourage better exception hygiene.
Except it doesn't, it makes it worse. So that's three points against it.
Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted.
Or based on your examples, increase the number of swallowed exceptions, and be much harder to refactor code safely.
On Thu, Sep 30, 2021 at 12:03:37AM -0300, Soni L. wrote:
Only some.user_code is guarded by the try block. If it turns out that code_we_assume_is_safe is not actually safe, and fails with an exception, it won't be caught by the try block and you will know about it.
Except no, because ExceptionWeCareAbout is part of the public API. [...]
You have not convinced me that I have misunderstood the proposal. As I said, some better, runnable code might help. But for the sake of the argument, suppose I have misunderstood and your analysis is correct.. You have just demonstrated that your proposed syntax hurts readability. In your original function, it is easy to recognise potentially poor exception hygiene at a glance: "Too much stuff inside a try block == potential bad hygiene" With your proposed syntactic sugar, there is no visible try block, and it is exceedingly unclear which parts of the function body are protected by an implicit try block, and which parts will have the exception caught and turned into RuntimeError, and which parts will have the exception caught and re-raised.
[no comments on the rest of your points because they're all based on this core misunderstanding.]
The rest of my post is more important. My comments asking how the compiler is supposed to know which part of the code needs to be guarded with a "re-raise the exception" flag still apply, regardless of whether I have misunderstood your API or not. Your syntax has: def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout # Line (A) How does the compiler know that *only* ExceptionWeCareAbout originating in Line (A) should be re-raised, and any other location turned into RuntimeError? What if I factor out those last two lines and make it: def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() check_condition_or_raise(args.something, some_condition) How does the compiler decide to re-raise exceptions originating in the last line but not the first two? What if I use a pre-prepared exception instance, or an alias, or both? BadThing = ExceptionWeCareAbout ERROR = BadThing("a thing happened") def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ERROR Your proposal doesn't make it clear how the compiler decides which parts of the body should allow the exception through and which should re-raise. -- Steve
On 2021-09-30 4:15 a.m., Steven D'Aprano wrote:
On Thu, Sep 30, 2021 at 12:03:37AM -0300, Soni L. wrote:
Only some.user_code is guarded by the try block. If it turns out that code_we_assume_is_safe is not actually safe, and fails with an exception, it won't be caught by the try block and you will know about it.
Except no, because ExceptionWeCareAbout is part of the public API. [...]
You have not convinced me that I have misunderstood the proposal. As I said, some better, runnable code might help. But for the sake of the argument, suppose I have misunderstood and your analysis is correct..
You have just demonstrated that your proposed syntax hurts readability. In your original function, it is easy to recognise potentially poor exception hygiene at a glance:
"Too much stuff inside a try block == potential bad hygiene"
With your proposed syntactic sugar, there is no visible try block, and it is exceedingly unclear which parts of the function body are protected by an implicit try block, and which parts will have the exception caught and turned into RuntimeError, and which parts will have the exception caught and re-raised.
You misnderstand exception hygiene. It isn't about "do the least stuff in try blocks", but about "don't shadow unrelated exceptions into your public API". For example, generators don't allow you to manually raise StopIteration anymore:
next((next(iter([])) for x in [1, 2, 3])) Traceback (most recent call last): File "<stdin>", line 1, in <genexpr> StopIteration
The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: generator raised StopIteration This is a (limited) form of exception hygiene. Can we generalize it? Can we do better about it? This effectively means all generators *are* wrapped in a try/except, so your point about "too much stuff inside a try block" goes directly against accepted practice and even existing python features as they're implemented.
[no comments on the rest of your points because they're all based on this core misunderstanding.]
The rest of my post is more important.
My comments asking how the compiler is supposed to know which part of the code needs to be guarded with a "re-raise the exception" flag still apply, regardless of whether I have misunderstood your API or not.
Your syntax has:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout # Line (A)
How does the compiler know that *only* ExceptionWeCareAbout originating in Line (A) should be re-raised, and any other location turned into RuntimeError?
Same way Rust decides whether to propagate or unwrap a Result: you *must* tell the compiler.
What if I factor out those last two lines and make it:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() check_condition_or_raise(args.something, some_condition)
How does the compiler decide to re-raise exceptions originating in the last line but not the first two?
In this case, it explicitly doesn't. You explicitly told it the last line doesn't raise any exceptions that contribute to your API's exception surface. You *must* use try: check_condition_or_raise(args.something, some_condition) except ExceptionWeCareAbout: raise (Verbosity can be improved if this feature gets widely used, but it's beside the point.)
What if I use a pre-prepared exception instance, or an alias, or both?
BadThing = ExceptionWeCareAbout
ERROR = BadThing("a thing happened")
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ERROR
Your proposal doesn't make it clear how the compiler decides which parts of the body should allow the exception through and which should re-raise.
This works fine because any explicit raise will always poke through the generated try/except.
On Thu, Sep 30, 2021 at 8:43 PM Soni L. <fakedme+py@gmail.com> wrote:
You misnderstand exception hygiene. It isn't about "do the least stuff in try blocks", but about "don't shadow unrelated exceptions into your public API".
For example, generators don't allow you to manually raise StopIteration anymore:
next((next(iter([])) for x in [1, 2, 3])) Traceback (most recent call last): File "<stdin>", line 1, in <genexpr> StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: generator raised StopIteration
This is a (limited) form of exception hygiene. Can we generalize it? Can we do better about it? This effectively means all generators *are* wrapped in a try/except, so your point about "too much stuff inside a try block" goes directly against accepted practice and even existing python features as they're implemented.
The reason for this is that StopException is nothing more than an *implementation detail* of generators. Look at this code: where is StopException? def gen(): yield 5 x = (yield 7) yield 9 if x: return 11 yield 1 return 3 The code doesn't raise StopException other than because that's the way that iterables are implemented. As a function, it simply does its work, with yield points and the ability to return a value. That's why a leaking StopException can and should be turned into RuntimeError. But what you're talking about doesn't have this clear distinction, other than in *your own definitions*. You have deemed that, in some areas, a certain exception should be turned into a RuntimeError; but in other areas, it shouldn't. To me, that sounds like a job for a context manager, not a function-level declaration.
My comments asking how the compiler is supposed to know which part of the code needs to be guarded with a "re-raise the exception" flag still apply, regardless of whether I have misunderstood your API or not.
Your syntax has:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout # Line (A)
How does the compiler know that *only* ExceptionWeCareAbout originating in Line (A) should be re-raised, and any other location turned into RuntimeError?
Same way Rust decides whether to propagate or unwrap a Result: you *must* tell the compiler.
Please elaborate. We can already write this: def foo(): with fail_on_exception(ExceptionWeCareAbout): some.user_code() if some_condition: raise ExceptionWeCareAbout Does that count as telling the compiler? If not, what is it you're trying to do, and how is the compiler supposed to know which ones to permit and which to wrap in RuntimeError?
What if I factor out those last two lines and make it:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() check_condition_or_raise(args.something, some_condition)
How does the compiler decide to re-raise exceptions originating in the last line but not the first two?
In this case, it explicitly doesn't. You explicitly told it the last line doesn't raise any exceptions that contribute to your API's exception surface.
You *must* use try: check_condition_or_raise(args.something, some_condition) except ExceptionWeCareAbout: raise
(Verbosity can be improved if this feature gets widely used, but it's beside the point.)
Ewww eww ewww. I have seen horrific Java code that exists solely to satisfy arbitrary function exception declarations. It does not improve the code.
This works fine because any explicit raise will always poke through the generated try/except.
So what you're saying is that the raise statement will always raise an exception, but that any exception raised from any other function won't. Okay. So you basically want exceptions to... not be exceptions. You want to use exceptions as if they're return values. Why not just use return values? ChrisA
On 2021-09-30 10:08 a.m., Chris Angelico wrote:
On Thu, Sep 30, 2021 at 8:43 PM Soni L. <fakedme+py@gmail.com> wrote:
You misnderstand exception hygiene. It isn't about "do the least stuff in try blocks", but about "don't shadow unrelated exceptions into your public API".
For example, generators don't allow you to manually raise StopIteration anymore:
next((next(iter([])) for x in [1, 2, 3])) Traceback (most recent call last): File "<stdin>", line 1, in <genexpr> StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: generator raised StopIteration
This is a (limited) form of exception hygiene. Can we generalize it? Can we do better about it? This effectively means all generators *are* wrapped in a try/except, so your point about "too much stuff inside a try block" goes directly against accepted practice and even existing python features as they're implemented.
The reason for this is that StopException is nothing more than an *implementation detail* of generators. Look at this code: where is StopException?
def gen(): yield 5 x = (yield 7) yield 9 if x: return 11 yield 1 return 3
The code doesn't raise StopException other than because that's the way that iterables are implemented. As a function, it simply does its work, with yield points and the ability to return a value. That's why a leaking StopException can and should be turned into RuntimeError.
But what you're talking about doesn't have this clear distinction, other than in *your own definitions*. You have deemed that, in some areas, a certain exception should be turned into a RuntimeError; but in other areas, it shouldn't. To me, that sounds like a job for a context manager, not a function-level declaration.
But generators *are* iterators. By definition. In fact this had to be a breaking change *because there was code in the wild that relied on it*! Imagine if that code could be changed to be: def gen() with StopIteration: try: yield next(foo) except StopIteration: raise and have the StopIteration propagate as a StopIteration instead of RuntimeError! (altho supporting this *specific* use-case would probably be painful given that this is mostly a purely syntactic transformation.)
My comments asking how the compiler is supposed to know which part of the code needs to be guarded with a "re-raise the exception" flag still apply, regardless of whether I have misunderstood your API or not.
Your syntax has:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout # Line (A)
How does the compiler know that *only* ExceptionWeCareAbout originating in Line (A) should be re-raised, and any other location turned into RuntimeError?
Same way Rust decides whether to propagate or unwrap a Result: you *must* tell the compiler.
Please elaborate. We can already write this:
def foo(): with fail_on_exception(ExceptionWeCareAbout): some.user_code() if some_condition: raise ExceptionWeCareAbout
Does that count as telling the compiler? If not, what is it you're trying to do, and how is the compiler supposed to know which ones to permit and which to wrap in RuntimeError?
With a source transformation, really. that is: def foo() with exceptions: something raise ... always transforms into: def foo(): set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError = False try: something set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError = True raise ... except exceptions as exc: if set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError: raise else: raise RuntimeError from exc that is: the "with exceptions" becomes "except exceptions", and every "raise" gains an "set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError = True" immediately before it (mostly - kinda glossing over the whole "the expression of the raise doesn't get to, itself, raise its own exceptions", but anyway). It gets clearer/etc if you have a more complex function that isn't a tiny wrapper. A tiny wrapper with 3 different exceptional exit conditions is inherently gonna look a little busy, but a larger wrapper with only one or two would actually look clearer! For example this: (real code) def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: first = next(iterator) except StopIteration: return (x for x in ()) except abdl.exceptions.ValidationError as exc: raise LookupError from exc except LookupError as exc: raise RuntimeError from exc # don't accidentally swallow bugs in the iterator return itertools.chain([first], iterator) vs: def get_property_values(self, prop) with PropertyError, LookupError: try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: first = next(iterator) except StopIteration: return (x for x in ()) except abdl.exceptions.ValidationError as exc: raise LookupError from exc return itertools.chain([first], iterator) (arguably the call to get_supported_properties should also be moved outside the try, but that actually doesn't change that a whole line got removed!) In this case, not only does it clean stuff up, it also solves potential maintainability issues. Without this feature, this would need a bunch more blocks to get the correct exception hygiene.
What if I factor out those last two lines and make it:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() check_condition_or_raise(args.something, some_condition)
How does the compiler decide to re-raise exceptions originating in the last line but not the first two?
In this case, it explicitly doesn't. You explicitly told it the last line doesn't raise any exceptions that contribute to your API's exception surface.
You *must* use try: check_condition_or_raise(args.something, some_condition) except ExceptionWeCareAbout: raise
(Verbosity can be improved if this feature gets widely used, but it's beside the point.)
Ewww eww ewww. I have seen horrific Java code that exists solely to satisfy arbitrary function exception declarations. It does not improve the code.
This is explicitly NOT checked exceptions. Do not mistake these. If anything this is the *direct opposite* (direct antithesis?) of checked exceptions.
This works fine because any explicit raise will always poke through the generated try/except.
So what you're saying is that the raise statement will always raise an exception, but that any exception raised from any other function won't. Okay. So you basically want exceptions to... not be exceptions. You want to use exceptions as if they're return values.
Why not just use return values?
Because they're "unpythonth". Yes they're a thing in Rust and Rust is the inspiration for this idea (and a good part of the reason we're rewriting our code in Rust) but we do think it's possible to have a pythonic solution to a pythonic problem. We've seen how many exceptions are accidentally swallowed by python web frameworks and how much of a debugging nightmare it can make. That's why we write code that guards against unexpected exceptions. Like the (real code) we pasted here above. (It's saved us plenty of trouble *in practice*, so this is a real issue.)
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5CPTIF... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Oct 1, 2021 at 12:03 AM Soni L. <fakedme+py@gmail.com> wrote:
But what you're talking about doesn't have this clear distinction, other than in *your own definitions*. You have deemed that, in some areas, a certain exception should be turned into a RuntimeError; but in other areas, it shouldn't. To me, that sounds like a job for a context manager, not a function-level declaration.
But generators *are* iterators. By definition. In fact this had to be a breaking change *because there was code in the wild that relied on it*!
There was code in the wild that assumed it. Every change breaks someone's workflow.
Imagine if that code could be changed to be:
def gen() with StopIteration: try: yield next(foo) except StopIteration: raise
and have the StopIteration propagate as a StopIteration instead of RuntimeError! (altho supporting this *specific* use-case would probably be painful given that this is mostly a purely syntactic transformation.)
Not an improvement. Most definitely not an improvement. The whole point of generator functions is that they early-terminate by returning, same as every other function does. def gen(): try: yield next(foo) except StopIteration: pass Or "return" instead of "pass" if you have other code after that and you want to not continue. StopIteration is part of the API of next(), but it is not part of the API of gen(), which simply has "yield" and "return"; thus the "except StopIteration" is part of handling the call to next().
With a source transformation, really. that is:
def foo() with exceptions: something raise ...
always transforms into:
def foo(): set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError = False try: something set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError = True raise ... except exceptions as exc: if set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError: raise else: raise RuntimeError from exc
that is: the "with exceptions" becomes "except exceptions", and every "raise" gains an "set_to_True_to_pass_through_instead_of_wrapping_in_RuntimeError = True" immediately before it (mostly - kinda glossing over the whole "the expression of the raise doesn't get to, itself, raise its own exceptions", but anyway).
How would this be handled? def foo() with RuntimeError: try: foo() except RuntimeError as e: raise What about this? def foo() with KeyError: things = {} try: return things["foo"] except KeyError: return things["bar"] Or this? def foo() with KeyError: things = {} try: things["foo"] finally: things["bar"] I'm not nitpicking your code transformation here, btw; I'm asking what your intuition is about this kind of code. If "def ... with Exception:" is supposed to suppress exceptions, what happens if they get caught?
For example this: (real code)
def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: first = next(iterator) except StopIteration: return (x for x in ()) except abdl.exceptions.ValidationError as exc: raise LookupError from exc except LookupError as exc: raise RuntimeError from exc # don't accidentally swallow bugs in the iterator return itertools.chain([first], iterator)
There's a lot of exception transformation happening here, and I don't understand, without context, the full purpose of it all. But the fact that you're raising LookupError directly seems a tad odd (it's a parent class for IndexError and KeyError), and it seems like a custom exception type would solve all of this. I *think* what's happening here is that you transform ValidationError into LookupError, but only if it happens on the very first yielded result?? Then how about: class PropertyNotFoundError(LookupError): pass def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: yield next(iterator) except StopIteration: pass except abdl.exceptions.ValidationError as exc: raise PropertyNotFoundError from exc yield from iterator If the factory raises some other LookupError, then that's not a PropertyNotFoundError. And if it *does* raise PropertyNotFoundError, then that is clearly deliberate, and it's a signal that the property is, in fact, not found. Your code, as currently written (and including the rewrite), is *still* capable of leaking a faulty exception - just as long as the factory returns one good value first. So I'm not sure what you actually gain by this transformation.
In this case, not only does it clean stuff up, it also solves potential maintainability issues. Without this feature, this would need a bunch more blocks to get the correct exception hygiene.
I'm still not sure what you mean by "correct exception hygiene". What you seem to be saying is "transform these exceptions into these others, if they occur in these places", which as mentioned is a really good job for a context manager (or a plain try/except).
So what you're saying is that the raise statement will always raise an exception, but that any exception raised from any other function won't. Okay. So you basically want exceptions to... not be exceptions. You want to use exceptions as if they're return values.
Why not just use return values?
Because they're "unpythonth". Yes they're a thing in Rust and Rust is the inspiration for this idea (and a good part of the reason we're rewriting our code in Rust) but we do think it's possible to have a pythonic solution to a pythonic problem. We've seen how many exceptions are accidentally swallowed by python web frameworks and how much of a debugging nightmare it can make. That's why we write code that guards against unexpected exceptions. Like the (real code) we pasted here above. (It's saved us plenty of trouble *in practice*, so this is a real issue.)
I don't understand. You're using return value functionality and behaviour, but you feel obliged to use exceptions for some reason, and so you want exceptions to stop bubbling. But if you want the error condition to not bubble, don't use bubbling exception handling. There's nothing wrong with using return values in Python. Can't you use a sentinel object (the way NotImplemented can be returned from certain dunder functions) to do this job? It seems FAR cleaner than warping your API around the need to use exceptions. ChrisA
On 2021-09-30 11:23 a.m., Chris Angelico wrote:
For example this: (real code)
def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: first = next(iterator) except StopIteration: return (x for x in ()) except abdl.exceptions.ValidationError as exc: raise LookupError from exc except LookupError as exc: raise RuntimeError from exc # don't accidentally swallow bugs in the iterator return itertools.chain([first], iterator)
There's a lot of exception transformation happening here, and I don't understand, without context, the full purpose of it all. But the fact that you're raising LookupError directly seems a tad odd (it's a parent class for IndexError and KeyError), and it seems like a custom exception type would solve all of this. I *think* what's happening here is that you transform ValidationError into LookupError, but only if it happens on the very first yielded result?? Then how about:
class PropertyNotFoundError(LookupError): pass
def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: yield next(iterator) except StopIteration: pass except abdl.exceptions.ValidationError as exc: raise PropertyNotFoundError from exc yield from iterator
If the factory raises some other LookupError, then that's not a PropertyNotFoundError. And if it *does* raise PropertyNotFoundError, then that is clearly deliberate, and it's a signal that the property is, in fact, not found.
Your code, as currently written (and including the rewrite), is *still* capable of leaking a faulty exception - just as long as the factory returns one good value first. So I'm not sure what you actually gain by this transformation.
Okay let us stop you right there. This generator is not a function. Generators don't do anything until iterated. There's a HUGE semantic difference there. You're making assumptions that you could easily invalidate yourself if you paid attention to the original code. Don't do that. try: iterator = foo.get_property_values(prop) except PropertyError, etc: pass # handle it for thing in iterator: pass # exceptions in the iteration actually get propagated because they mean something went wrong and the program should terminate. in the real code, this includes spurious ValidationError etc, because those are *never* supposed to happen.
On Fri, Oct 1, 2021 at 1:10 AM Soni L. <fakedme+py@gmail.com> wrote:
On 2021-09-30 11:23 a.m., Chris Angelico wrote:
For example this: (real code)
def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: first = next(iterator) except StopIteration: return (x for x in ()) except abdl.exceptions.ValidationError as exc: raise LookupError from exc except LookupError as exc: raise RuntimeError from exc # don't accidentally swallow bugs in the iterator return itertools.chain([first], iterator)
There's a lot of exception transformation happening here, and I don't understand, without context, the full purpose of it all. But the fact that you're raising LookupError directly seems a tad odd (it's a parent class for IndexError and KeyError), and it seems like a custom exception type would solve all of this. I *think* what's happening here is that you transform ValidationError into LookupError, but only if it happens on the very first yielded result?? Then how about:
class PropertyNotFoundError(LookupError): pass
def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: yield next(iterator) except StopIteration: pass except abdl.exceptions.ValidationError as exc: raise PropertyNotFoundError from exc yield from iterator
If the factory raises some other LookupError, then that's not a PropertyNotFoundError. And if it *does* raise PropertyNotFoundError, then that is clearly deliberate, and it's a signal that the property is, in fact, not found.
Your code, as currently written (and including the rewrite), is *still* capable of leaking a faulty exception - just as long as the factory returns one good value first. So I'm not sure what you actually gain by this transformation.
Okay let us stop you right there. This generator is not a function.
Generators don't do anything until iterated.
There's a HUGE semantic difference there. You're making assumptions that you could easily invalidate yourself if you paid attention to the original code. Don't do that.
try: iterator = foo.get_property_values(prop) except PropertyError, etc: pass # handle it for thing in iterator: pass # exceptions in the iteration actually get propagated because they mean something went wrong and the program should terminate. in the real code, this includes spurious ValidationError etc, because those are *never* supposed to happen.
Congratulations, you showed me some extremely complicated code, and then got miffed when I messed up something in trying to simplify it. Well done, you successfully tripped me up in my attempt to make code better. Can we stick to the point, please? You still haven't shown why (a) a context manager, nor (b) a custom exception, cannot solve this problem far better than this extremely magical syntax. ChrisA
On 2021-09-30 2:04 p.m., Chris Angelico wrote:
On Fri, Oct 1, 2021 at 1:10 AM Soni L. <fakedme+py@gmail.com> wrote:
On 2021-09-30 11:23 a.m., Chris Angelico wrote:
For example this: (real code)
def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: first = next(iterator) except StopIteration: return (x for x in ()) except abdl.exceptions.ValidationError as exc: raise LookupError from exc except LookupError as exc: raise RuntimeError from exc # don't accidentally swallow bugs in the iterator return itertools.chain([first], iterator)
There's a lot of exception transformation happening here, and I don't understand, without context, the full purpose of it all. But the fact that you're raising LookupError directly seems a tad odd (it's a parent class for IndexError and KeyError), and it seems like a custom exception type would solve all of this. I *think* what's happening here is that you transform ValidationError into LookupError, but only if it happens on the very first yielded result?? Then how about:
class PropertyNotFoundError(LookupError): pass
def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: yield next(iterator) except StopIteration: pass except abdl.exceptions.ValidationError as exc: raise PropertyNotFoundError from exc yield from iterator
If the factory raises some other LookupError, then that's not a PropertyNotFoundError. And if it *does* raise PropertyNotFoundError, then that is clearly deliberate, and it's a signal that the property is, in fact, not found.
Your code, as currently written (and including the rewrite), is *still* capable of leaking a faulty exception - just as long as the factory returns one good value first. So I'm not sure what you actually gain by this transformation.
Okay let us stop you right there. This generator is not a function.
Generators don't do anything until iterated.
There's a HUGE semantic difference there. You're making assumptions that you could easily invalidate yourself if you paid attention to the original code. Don't do that.
try: iterator = foo.get_property_values(prop) except PropertyError, etc: pass # handle it for thing in iterator: pass # exceptions in the iteration actually get propagated because they mean something went wrong and the program should terminate. in the real code, this includes spurious ValidationError etc, because those are *never* supposed to happen.
Congratulations, you showed me some extremely complicated code, and then got miffed when I messed up something in trying to simplify it. Well done, you successfully tripped me up in my attempt to make code better.
Fair enough, we went too far there, sorry.
Can we stick to the point, please? You still haven't shown why (a) a context manager, nor (b) a custom exception, cannot solve this problem far better than this extremely magical syntax.
This syntax is far from magical. We're not sure how the context manager would work here, but, a custom exception doesn't work, because you still have the... whatever you'd call this issue: https://mail.python.org/pipermail/python-ideas/2017-June/046109.html (one of Steven's links also linked to this and it's fairly relevant.) The imports issue sadly wouldn't be solved with this feature, but attributes would: def __getattr__(self, name) with AttributeError: stuff and things if the thing: return some thing raise AttributeError This also applies anywhere else you'd have something wrapping another thing and they happen to use the same exceptions in slightly distinct ways. Yes, it's most noticeable with StopIteration (which got tweaked with the change to generators), imports (because the modules you depend on generally also depend on other modules), and attributes (because you might do a foo.typo in your __getattr__ and it's not particularly uncommon to dynamically probe attributes). But it does happen in other cases too. (KeyError is an interesting one, especially when making a wrapper over a dict/subclassing dict/etc.) In one of the examples, we showed this function: https://github.com/ganarchy/GAnarchy/blob/993a8ca85db1564e64550276d61d972342... In reality, we really should also be catching any PropertyError in the unpacking, just in case someone happened to implement get_property_values incorrectly and it waited until iteration to raise, because that's a case where a bug would be swallowed. And the only reason we didn't bother to guard against ValueError during iteration is simply this: property kinds have arity, some are 1-ary and some are n-ary. it's a programming bug to call get_property_value on an n-ary property, so we decided to assume nobody's gonna actually catch those ValueError. This is, in practice, an extremely dangerous assumption, especially in the wider ecosystem, because, as it's a documented part of the API, ppl do assume the implementation is strictly correct about it. But it would also be wrong to not make it a documented part of the API.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YY6T5J... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Oct 1, 2021 at 6:12 AM Soni L. <fakedme+py@gmail.com> wrote:
Congratulations, you showed me some extremely complicated code, and then got miffed when I messed up something in trying to simplify it. Well done, you successfully tripped me up in my attempt to make code better.
Fair enough, we went too far there, sorry.
Apology accepted.
Can we stick to the point, please? You still haven't shown why (a) a context manager, nor (b) a custom exception, cannot solve this problem far better than this extremely magical syntax.
This syntax is far from magical. We're not sure how the context manager would work here, but, a custom exception doesn't work, because you still have the... whatever you'd call this issue: https://mail.python.org/pipermail/python-ideas/2017-June/046109.html (one of Steven's links also linked to this and it's fairly relevant.)
I don't think ImportError is actually a problem; if you're catching ImportError to cope with the fact that some module can't be loaded, wouldn't it be *correct* to catch the fact that a dependency of it failed to load? The exception states the exact failing module, so if you need to distinguish, you can. As mentioned, PEP 479 isn't really about "nested StopIteration", it's about the interaction between generator functions and iterators. They're using different protocols that happen to collide on an implementation detail. The trouble with the "nested AttributeError" problem is that it's indistinguishable from a genuinely refactored example. So that kinda is what you're talking about, and I'll focus on this one from here.
The imports issue sadly wouldn't be solved with this feature, but attributes would:
def __getattr__(self, name) with AttributeError: stuff and things if the thing: return some thing raise AttributeError
This also applies anywhere else you'd have something wrapping another thing and they happen to use the same exceptions in slightly distinct ways.
Right. And this IS a reasonable consideration. But most getattr methods are pretty simple, and would be best solved by something like: def __getattr__(self, name): with dontraise(AttributeError): stuff and things if the thing: return some thing raise AttributeError
Yes, it's most noticeable with StopIteration (which got tweaked with the change to generators), imports (because the modules you depend on generally also depend on other modules), and attributes (because you might do a foo.typo in your __getattr__ and it's not particularly uncommon to dynamically probe attributes). But it does happen in other cases too. (KeyError is an interesting one, especially when making a wrapper over a dict/subclassing dict/etc.)
In one of the examples, we showed this function: https://github.com/ganarchy/GAnarchy/blob/993a8ca85db1564e64550276d61d972342...
In reality, we really should also be catching any PropertyError in the unpacking, just in case someone happened to implement get_property_values incorrectly and it waited until iteration to raise, because that's a case where a bug would be swallowed. And the only reason we didn't bother to guard against ValueError during iteration is simply this: property kinds have arity, some are 1-ary and some are n-ary. it's a programming bug to call get_property_value on an n-ary property, so we decided to assume nobody's gonna actually catch those ValueError. This is, in practice, an extremely dangerous assumption, especially in the wider ecosystem, because, as it's a documented part of the API, ppl do assume the implementation is strictly correct about it. But it would also be wrong to not make it a documented part of the API.
I still think that your use of LookupError is confusing the issue somewhat, partly because it's a base class rather than something that's ever raised per se. If you were using a custom exception type, how likely is it that that exception would be raised during the unpacking? Under what situations would this happen, and might it actually be an intended way for that property value to say "actually, heh... I don't exist"? Because the logical way to spell that would be "raise PropertyNotFoundError" (or whatever you use instead of LookupError). IOW, bubbling is exactly correct. You still haven't explained how that isn't the case. ChrisA
On 2021-09-30 6:08 p.m., Chris Angelico wrote:
I still think that your use of LookupError is confusing the issue somewhat, partly because it's a base class rather than something that's ever raised per se.
If you were using a custom exception type, how likely is it that that exception would be raised during the unpacking? Under what situations would this happen, and might it actually be an intended way for that property value to say "actually, heh... I don't exist"? Because the logical way to spell that would be "raise PropertyNotFoundError" (or whatever you use instead of LookupError). IOW, bubbling is exactly correct. You still haven't explained how that isn't the case.
Bubbling is correct, but it doesn't deny that explicit is better than implicit. Rust's `?` operator (generally referred to as "try" by the devs) has a lot that can be learned from, here. (Not to say it would be the appropriate solution for Python.)
On Thu, Sep 30, 2021 at 07:41:47AM -0300, Soni L. wrote:
You misnderstand exception hygiene. It isn't about "do the least stuff in try blocks", but about "don't shadow unrelated exceptions into your public API".
I disagree with that. I don't think that "exception hygiene" is a common term in software development (at least googling doesn't find many examples of the term) so it is not like there is a canonical definition beyond "clean use of exceptions", which is going to be at least partly subjective. I think that doing the least amount of work necessary inside a try block is a large part of exception hygiene. Likewise catching the smallest subset of exceptions that you know how to meaningfully handle. Large try blocks with a bunch of stuff inside them is an obvious Code Smell in the sense that Joel Spolsky talks about: https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ As soon as I see a big block of code inside a try block, it smells a bit fishy and I know that I need to think *really carefully* about the possibility that maybe it is catching too many exceptions. try: n = len(item) # followed by thirty more lines of code... except TypeError: print('item is not a sequence') fallback() I have to think really hard, am I *really sure* that none of those other thirty lines can raise TypeError, which will then be wrongly diagnosed and wrongly handled? That's a code smell. It's not a clean use of exceptions. It is dirty, not clean. It is *unhygienic*.
For example, generators don't allow you to manually raise StopIteration anymore: [...] This is a (limited) form of exception hygiene. Can we generalize it?
Sure. See the "exception guard" recipe I posted. If you go to the recipe, it links to some discussions.
This effectively means all generators *are* wrapped in a try/except, so your point about "too much stuff inside a try block" goes directly against accepted practice and even existing python features as they're implemented.
StopIteration is very special. It doesn't even inherit from Exception. It is used directly for flow control by the interpreter. (Which goes against accepted practice in other languages: https://softwareengineering.stackexchange.com/questions/189222/are-exception... One might ask, is Python doing it wrong, is everyone else doing it wrong, or are we asking the wrong questions? By I digress.) We might take the point of view that StopIteration is reserved for use by iterators, and that the interpreter reserves the use to treat *any* use of StopIteration in any other context as undefined behaviour, and that we ought to be grateful that the interpreter merely converts it to a RuntimeError instead of making demons fly out of your nose. http://catb.org/jargon/html/N/nasal-demons.html So I don't think we should be too gung-ho about generalising from StopIteration (a specific exception used by the interpreter for flow control) to arbitrary exceptions which represent error conditions. (More to follow.) -- Steve
On Fri, Oct 1, 2021 at 3:02 PM Steven D'Aprano <steve@pearwood.info> wrote:
We might take the point of view that StopIteration is reserved for use by iterators, and that the interpreter reserves the use to treat *any* use of StopIteration in any other context as undefined behaviour, and that we ought to be grateful that the interpreter merely converts it to a RuntimeError instead of making demons fly out of your nose.
http://catb.org/jargon/html/N/nasal-demons.html
So I don't think we should be too gung-ho about generalising from StopIteration (a specific exception used by the interpreter for flow control) to arbitrary exceptions which represent error conditions.
If you want to generalise StopIteration at all, the next closest would be AttributeError, which exists so that getattr and __getattr__ can return any object whatsoever, or can signal "there is no object to return". But there's no equivalent of generators for them, or if there is, it is generators themselves: def __getattr__(self, attr): for value in self.__getattr_gen__(attr): return attr raise AttributeError def __getattr_gen__(self, attr): if attr.startswith("spam"): yield "ham" + attr[4:] You could argue that, now, an AttrributeError inside the gen function should be turned into RuntimeError. But that just brings us right back to context managers or try/except: def __getattr_(self, attr): try: for value in self.__getattr_gen__(attr): return attr except AttributeError: raise RuntimeError("Exception leaked from implementation function") raise AttributeError("Attribute not found: " + str(attr)) So ultimately, the true specialness of StopIteration is actually generators, not flow control. (Also: If anyone implements "pip install nasal-demons", I would be very curious as to its hardware requirements.) ChrisA
Let's get to the fundamental problem with this. It is DWIM magic, and you haven't (as far as I have seen) yet specified how we are supposed to use it or predict how it is supposed to work. Here is your syntax again:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout # Line (A)
How does the compiler know that *only* ExceptionWeCareAbout originating in Line (A) should be re-raised, and any other location turned into RuntimeError?
Same way Rust decides whether to propagate or unwrap a Result: you *must* tell the compiler.
How do I tell the compiler? There is no sign in your syntax that Line (A) is special or different from the rest of the lines in the function.
What if I factor out those last two lines and make it:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() check_condition_or_raise(args.something, some_condition)
How does the compiler decide to re-raise exceptions originating in the last line but not the first two?
In this case, it explicitly doesn't.
So by refactoring a conditional raise (if condition: raise) into a function, I completely change the meaning of the code? That's just great. Not.
You explicitly told it the last line doesn't raise any exceptions that contribute to your API's exception surface.
I did? I wasn't aware that I told the interpeter *explicitly* anything about the last line. How does this work? That's what I'm trying to understand about your proposal: I write a function, and stick "with ExceptionName" into the function declaration line: def function(some, args) with ExceptionWeCareAbout: and then write a block of code under that declaration, and *somehow* in some completely unspecified way the compiler Does What I Mean by deciding that *some* of those lines which raise ExceptionWeCareAbout it should let the exception through while *other* lines that raise the same exception should be guarded against ExceptionWeCareAbout and have them raise RuntimeError instead. And I have no idea how it decides which lines are guarded and which are not. You tell me that I explicitly instructed the compiler which lines should be guarded, but I don't know how I did it.
You *must* use try: check_condition_or_raise(args.something, some_condition) except ExceptionWeCareAbout: raise
But that's not what the pre-refactoring code had. All I did was move: # The original version, before refactoring. if args.something and some_condition: raise ExceptionWeCareAbout into a named function and call that. In the original code, I didn't have to explicitly catch the exception and then immediately re-raise it: # This was never used. try: if args.something and some_condition: raise ExceptionWeCareAbout except ExceptionWeCareAbout: raise But now you are telling me that if I move the `if... raise` lines into a function I have to also wrap it in a try...except, catch the exception, and immediately and unconditionally re-raise it. This behaviour may be clear to *you* but I cannot read your mind. Unless you actually tell me how the compiler knows which part of the block need to be guarded and which parts don't, I have no clue how this is happening except "the compiler magically infers what I want it to do". -- Steve
On 2021-10-01 2:04 a.m., Steven D'Aprano wrote:
Let's get to the fundamental problem with this. It is DWIM magic, and you haven't (as far as I have seen) yet specified how we are supposed to use it or predict how it is supposed to work.
Here is your syntax again:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout # Line (A)
How does the compiler know that *only* ExceptionWeCareAbout originating in Line (A) should be re-raised, and any other location turned into RuntimeError?
Same way Rust decides whether to propagate or unwrap a Result: you *must* tell the compiler.
How do I tell the compiler? There is no sign in your syntax that Line (A) is special or different from the rest of the lines in the function.
What if I factor out those last two lines and make it:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: some.user_code() code_we_assume_is_safe() check_condition_or_raise(args.something, some_condition)
How does the compiler decide to re-raise exceptions originating in the last line but not the first two?
In this case, it explicitly doesn't.
So by refactoring a conditional raise (if condition: raise) into a function, I completely change the meaning of the code? That's just great. Not.
You explicitly told it the last line doesn't raise any exceptions that contribute to your API's exception surface.
I did? I wasn't aware that I told the interpeter *explicitly* anything about the last line. How does this work?
That's what I'm trying to understand about your proposal: I write a function, and stick "with ExceptionName" into the function declaration line:
def function(some, args) with ExceptionWeCareAbout:
and then write a block of code under that declaration, and *somehow* in some completely unspecified way the compiler Does What I Mean by deciding that *some* of those lines which raise ExceptionWeCareAbout it should let the exception through while *other* lines that raise the same exception should be guarded against ExceptionWeCareAbout and have them raise RuntimeError instead.
And I have no idea how it decides which lines are guarded and which are not. You tell me that I explicitly instructed the compiler which lines should be guarded, but I don't know how I did it.
You *must* use try: check_condition_or_raise(args.something, some_condition) except ExceptionWeCareAbout: raise
But that's not what the pre-refactoring code had. All I did was move:
# The original version, before refactoring. if args.something and some_condition: raise ExceptionWeCareAbout
into a named function and call that. In the original code, I didn't have to explicitly catch the exception and then immediately re-raise it:
# This was never used. try: if args.something and some_condition: raise ExceptionWeCareAbout except ExceptionWeCareAbout: raise
But now you are telling me that if I move the `if... raise` lines into a function I have to also wrap it in a try...except, catch the exception, and immediately and unconditionally re-raise it.
This behaviour may be clear to *you* but I cannot read your mind. Unless you actually tell me how the compiler knows which part of the block need to be guarded and which parts don't, I have no clue how this is happening except "the compiler magically infers what I want it to do".
Look for the clarification post we posted as a reply to the original post.
On further thought, maybe something like this is what you are looking for? https://code.activestate.com/recipes/580808-guard-against-an-exception-in-th... This allows you to replace one or more kinds of exceptions with another specified exception (defaults to RuntimeError) using a context manager. -- Steve
Alright, some ppl asked us to rephrase this, so: The plan is to take the function syntax: def name(args): and add an optional "with" to it: def name(args) with exceptions: these then get added to the function, similar to e.g. default args. when an exception is thrown*, the VM then checks these and converts relevant exceptions into RuntimeError, e.g.: def foo(): raise Bar def baz() with Bar: foo() baz() would make a RuntimeError, because foo raised a Bar and the VM sees that Bar is in baz's with. *except "raise" opcodes SKIP checking these (within the context of the function), so the following: def baz() with Bar: raise Bar baz() raises a Bar, not a RuntimeError from a Bar. You can then document your exceptions as you normally do: "This raises Bar when so and so are invalid", and then add "with Bar" to your function. the VM then makes sure that Bar is only raised when so and so are invalid, but under no other circumstances. e.g. someone monkeypatched your function to try and raise a Bar outside of the place it was meant to raise a Bar? it'll raise a RuntimeError instead. this gives you far more control over what your API exceptions, the exceptions you documented in your function's documentation, are, and especially under which conditions they'll actually be raised (and be catchable by the caller). this is what we mean by exception hygiene. On 2021-09-29 10:01 p.m., Soni L. wrote:
So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions:
def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout
It'd be nice if there was a way to... make this easier to deal with. Perhaps something where, something like this:
def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout
becomes:
def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() code_we_assume_is_safe() if args.something and some_condition: let_exception_through = True raise ExceptionWeCareAbout except ExceptionWeCareAbout as exc: if let_exception_through: raise else: raise RuntimeError from exc
(or equivalent)
and something like:
def foo() with Bar: try: baz() except Bar: raise
becomes:
def foo(): try: try: baz() except Bar: allow_exception = True raise except Bar as exc: if allow_exception: raise else: raise RuntimeError from exc
(thus allowing explicit exception propagation)
Additionally, note that it would only apply to the raise itself - something like `raise may_raise_the_same_error()` would desugar as if:
exception = may_raise_the_same_error() allow_exception = True raise exception
Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc. It does, however, encourage better exception hygiene. Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted. We know something like this would've saved us a lot of trouble, so we're sharing the idea in the hopes it can, in the future, help others.
On 2021-09-30 09:25, Soni L. wrote:
these then get added to the function, similar to e.g. default args. when an exception is thrown*, the VM then checks these and converts relevant exceptions into RuntimeError, e.g.:
def foo(): raise Bar def baz() with Bar: foo() baz()
would make a RuntimeError, because foo raised a Bar and the VM sees that Bar is in baz's with.
*except "raise" opcodes SKIP checking these (within the context of the function), so the following:
def baz() with Bar: raise Bar baz()
raises a Bar, not a RuntimeError from a Bar.
I think this illustrates what Chris said before about "you want exceptions to not be exceptions". The way exceptions work is that they propagate up the call stack. What you're describing is a mechanism whereby a Bar exception raised "directly" in the lexical function definition would behave differently from one raised deeper in the call stack. I think that breaks some pretty basic assumptions about how exceptions work, and I wouldn't support such a change. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On 30 Sep 2021, at 17:25, Soni L. <fakedme+py@gmail.com> wrote:
Alright, some ppl asked us to rephrase this, so:
The plan is to take the function syntax:
def name(args):
and add an optional "with" to it:
def name(args) with exceptions:
these then get added to the function, similar to e.g. default args. when an exception is thrown*, the VM then checks these and converts relevant exceptions into RuntimeError, e.g.:
def foo(): raise Bar def baz() with Bar: foo() baz()
would make a RuntimeError, because foo raised a Bar and the VM sees that Bar is in baz's with.
Does with Bar mean that Bar is expected? If so who cares if foo raises it? Are you really saying I cannot call functions to implement a complex algorithm that raises exceptions? The caller of baz is expecting Bar right?
*except "raise" opcodes SKIP checking these (within the context of the function), so the following:
def baz() with Bar: raise Bar baz()
raises a Bar, not a RuntimeError from a Bar.
You want to include the exceptions that a function can raise in its signature and have python enforce rules based on that information. C++ had/has this feature and it failed in practice so it has been deprecated. I'd be surprised that it will be useful in python given this experience in the C++ world. Barry
On 2021-09-30 5:34 p.m., Barry Scott wrote:
On 30 Sep 2021, at 17:25, Soni L. <fakedme+py@gmail.com <mailto:fakedme+py@gmail.com>> wrote:
Alright, some ppl asked us to rephrase this, so:
The plan is to take the function syntax:
def name(args):
and add an optional "with" to it:
def name(args) with exceptions:
these then get added to the function, similar to e.g. default args. when an exception is thrown*, the VM then checks these and converts relevant exceptions into RuntimeError, e.g.:
def foo(): raise Bar def baz() with Bar: foo() baz()
would make a RuntimeError, because foo raised a Bar and the VM sees that Bar is in baz's with.
Does with Bar mean that Bar is expected?
If so who cares if foo raises it? Are you really saying I cannot call functions to implement a complex algorithm that raises exceptions?
No, we're not saying that, altho it would be more verbose. Consider the example: def foo(): raise Bar def baz() with Bar: try: foo() except Bar: raise baz() This would successfully propagate foo's Bar to baz's caller. That's all - just making the exception propagation points explicit rather than implicit. (heh.)
The caller of baz is expecting Bar right?
*except "raise" opcodes SKIP checking these (within the context of the function), so the following:
def baz() with Bar: raise Bar baz()
raises a Bar, not a RuntimeError from a Bar.
You want to include the exceptions that a function can raise in its signature and have python enforce rules based on that information.
C++ had/has this feature and it failed in practice so it has been deprecated. I'd be surprised that it will be useful in python given this experience in the C++ world.
Unlike checked exceptions, this idea explicitly doesn't affect the caller. Take a look above - note how baz, being defined as "def baz() with Bar:", can still be called as "baz()" without any exception checking around it. We're trying to move away from checked exceptions here, it just happens that using similar syntax is surprisingly convenient. Sadly we do acknowledge the limitation that the best syntax for this is similar to what's used for checked exceptions in other languages. If you have any better syntax suggestions we're open to them tho. (And yeah, we do agree that *checked exceptions*, not to be confused with this idea, are a practical failure.)
Barry
On Thu, Sep 30, 2021 at 01:25:42PM -0300, Soni L. wrote:
def foo(): raise Bar def baz() with Bar: foo() baz()
would make a RuntimeError, because foo raised a Bar and the VM sees that Bar is in baz's with.
So the syntax "with SomeException" in a function declaration would mean that the function can not raise SomeException? Wouldn't "without SomeException" be a less misleading name then?
*except "raise" opcodes SKIP checking these (within the context of the function), so the following:
def baz() with Bar: raise Bar baz()
raises a Bar, not a RuntimeError from a Bar.
Every time I think I understand your proposal, I read some more about it, and get more confused. So the "with SomeException" syntax guarantees that the function *won't* raise SomeException (raising RuntimeError instead), unless it does raise SomeException, which under some unspecified conditions it will allow it through. And this syntax can only apply to an entire function at a time, you can't apply it to a block of code like a with-statement unless you move the entire block into its own function.
You can then document your exceptions as you normally do: "This raises Bar when so and so are invalid", and then add "with Bar" to your function. the VM then makes sure that Bar is only raised when so and so are invalid, but under no other circumstances.
Wait. This is even more magical. So if I document "this only raises Bar if P == NP" the compiler will solve the P versus NP problem? https://en.wikipedia.org/wiki/P_versus_NP_problem Okay, I jest a bit. A more realistic example: def myfunc(s:str) with ValueError: """This raises ValueError only if s is a palindrome.""" assert isinstance(s, str) do_stuff() # Guard: ValueError --> RuntimeError if s == s[::-1]: raise ValueError("spam") # No guard. n = math.sqrt(float(s)) # Guard: ValueError --> RuntimeError ... # more code here is still guarded Surely you aren't suggesting that the compiler infers meaning from the natural language docstring, but if not, how does it know which parts of the function to guard and which parts not to? How does the VM know that ValueError("spam") is okay but ValueError("math domain error") is not? What is the actual criteria used to allow some ValueErrors through and not others?
e.g. someone monkeypatched your function to try and raise a Bar outside of the place it was meant to raise a Bar? it'll raise a RuntimeError instead.
So if my function is this: def function(arg) with ValueError: """Raises ValueError if arg == 0.""" if arg = 0: raise ValueError return arg + 1 you are saying that if I monkeypatch it to this: def patched(arg): if arg == 0: return 1 raise ValueError function = patched function(99) that somehow it will raise RuntimeError? I don't want to believe that this is what you actually mean, but if it isn't this, then I don't know what you do mean. I wish your proposals and ideas would be more *precise* in their specifications. This is not the first time that I have found it very hard to work out precisely what you are suggesting and what you are not. -- Steve
On 2021-10-01 8:26 a.m., Steven D'Aprano wrote:
On Thu, Sep 30, 2021 at 01:25:42PM -0300, Soni L. wrote:
def foo(): raise Bar def baz() with Bar: foo() baz()
would make a RuntimeError, because foo raised a Bar and the VM sees that Bar is in baz's with.
So the syntax "with SomeException" in a function declaration would mean that the function can not raise SomeException?
Wouldn't "without SomeException" be a less misleading name then?
*except "raise" opcodes SKIP checking these (within the context of the function), so the following:
def baz() with Bar: raise Bar baz()
raises a Bar, not a RuntimeError from a Bar.
Every time I think I understand your proposal, I read some more about it, and get more confused.
So the "with SomeException" syntax guarantees that the function *won't* raise SomeException (raising RuntimeError instead), unless it does raise SomeException, which under some unspecified conditions it will allow it through.
And this syntax can only apply to an entire function at a time, you can't apply it to a block of code like a with-statement unless you move the entire block into its own function.
You can then document your exceptions as you normally do: "This raises Bar when so and so are invalid", and then add "with Bar" to your function. the VM then makes sure that Bar is only raised when so and so are invalid, but under no other circumstances.
Wait. This is even more magical. So if I document "this only raises Bar if P == NP" the compiler will solve the P versus NP problem?
https://en.wikipedia.org/wiki/P_versus_NP_problem
Okay, I jest a bit. A more realistic example:
def myfunc(s:str) with ValueError: """This raises ValueError only if s is a palindrome.""" assert isinstance(s, str) do_stuff() # Guard: ValueError --> RuntimeError if s == s[::-1]: raise ValueError("spam") # No guard. n = math.sqrt(float(s)) # Guard: ValueError --> RuntimeError ... # more code here is still guarded
Surely you aren't suggesting that the compiler infers meaning from the natural language docstring, but if not, how does it know which parts of the function to guard and which parts not to?
How does the VM know that ValueError("spam") is okay but ValueError("math domain error") is not? What is the actual criteria used to allow some ValueErrors through and not others?
It's not the ValueError(...). It's the location of the "raise".
e.g. someone monkeypatched your function to try and raise a Bar outside of the place it was meant to raise a Bar? it'll raise a RuntimeError instead.
So if my function is this:
def function(arg) with ValueError: """Raises ValueError if arg == 0.""" if arg = 0: raise ValueError return arg + 1
you are saying that if I monkeypatch it to this:
def patched(arg): if arg == 0: return 1 raise ValueError
function = patched function(99)
that somehow it will raise RuntimeError? I don't want to believe that this is what you actually mean, but if it isn't this, then I don't know what you do mean.
I wish your proposals and ideas would be more *precise* in their specifications. This is not the first time that I have found it very hard to work out precisely what you are suggesting and what you are not.
We did specify it: "raise" in body raises anything, regardless of the value matching something in the "with". deeper exceptions that match those in "with" get caught and wrapped in a RuntimeError. We're saying if your function is this: def function(arg) with ValueError: if arg == 0: raise ValueError return foo(arg) + 1 and someone monkeypatches it: def foo(arg): raise ValueError then your function will raise a RuntimeError. Could this be added to context managers instead? Could "raise" within "with" call the context manager with the exception? Probably! with foo: raise Bar # becomes semantically equivalent to with foo: exception = Bar # break out of the context without calling __exit__, and then... foo.__raise__(exception) (the default __raise__ would just call __exit__ with the exception and pretend like everything is normal.) This would be both more efficient and more accurate than doing runtime reflection to figure out where the exception came from. Specifically, this allows re-raising, whereas runtime reflection doesn't. But with generators having an implicit "with StopIteration", it seems to make more sense to generalize that to other exceptions than to tweak context managers to be able to optionally (not) intercept "raise" statements.
On 2021-10-01 8:26 a.m., Steven D'Aprano wrote:
I wish your proposals and ideas would be more *precise* in their specifications. This is not the first time that I have found it very hard to work out precisely what you are suggesting and what you are not.
Thanks for insisting on context managers. Does the attached context manager clarify what the proposal is about?
participants (5)
-
Barry Scott -
Brendan Barnwell -
Chris Angelico -
Soni L. -
Steven D'Aprano