Why not using "except: (...) raise" to cleanup on error?
Hi, I just read a recent bugfix in asyncio: https://github.com/python/cpython/commit/9602643120a509858d0bee4215d7f150e61... + try: + await waiter + except Exception: + transport.close() + raise Why only catching "except Exception:"? Why not also catching KeyboardInterrupt or MemoryError? Is it a special rule for asyncio, or a general policy in Python stdlib? For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example. Victor
On Tue, Jun 5, 2018 at 2:11 AM, Victor Stinner <vstinner@redhat.com> wrote:
Hi,
I just read a recent bugfix in asyncio:
https://github.com/python/cpython/commit/9602643120a509858d0bee4215d7f150e61...
+ try: + await waiter + except Exception: + transport.close() + raise
Why only catching "except Exception:"? Why not also catching KeyboardInterrupt or MemoryError? Is it a special rule for asyncio, or a general policy in Python stdlib?
For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example.
Pardon the dumb question, but why is try/finally unsuitable? ChrisA
On Mon, Jun 4, 2018 at 12:50 PM Chris Angelico <rosuav@gmail.com> wrote:
On Tue, Jun 5, 2018 at 2:11 AM, Victor Stinner <vstinner@redhat.com> wrote:
[..]
For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example.
Pardon the dumb question, but why is try/finally unsuitable?
Because try..finally isn't equivalent to try..except? Perhaps you should look at the actual code: https://github.com/python/cpython/blob/b609e687a076d77bdd687f5e4def85e29a044... Yury
On Tue, Jun 5, 2018 at 2:57 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Mon, Jun 4, 2018 at 12:50 PM Chris Angelico <rosuav@gmail.com> wrote:
On Tue, Jun 5, 2018 at 2:11 AM, Victor Stinner <vstinner@redhat.com> wrote:
[..]
For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example.
Pardon the dumb question, but why is try/finally unsuitable?
Because try..finally isn't equivalent to try..except? Perhaps you should look at the actual code: https://github.com/python/cpython/blob/b609e687a076d77bdd687f5e4def85e29a044...
Oh. Duh. Yep, it was a dumb question. Sorry! The transport should ONLY be closed on error. ChrisA
On 04.06.2018 20:11, Chris Angelico wrote:
On Tue, Jun 5, 2018 at 2:57 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Mon, Jun 4, 2018 at 12:50 PM Chris Angelico <rosuav@gmail.com> wrote:
On Tue, Jun 5, 2018 at 2:11 AM, Victor Stinner <vstinner@redhat.com> wrote: [..]
For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example. Pardon the dumb question, but why is try/finally unsuitable? Because try..finally isn't equivalent to try..except? Perhaps you should look at the actual code: https://github.com/python/cpython/blob/b609e687a076d77bdd687f5e4def85e29a044... Oh. Duh. Yep, it was a dumb question. Sorry! The transport should ONLY be closed on error.
I smell a big, big design violation here. The whole point of Exception vs BaseException is that anything not Exception is "not an error", has a completely different effect on the program than an error, and thus is to be dealt with completely differently. For example, warnings do not disrupt the control flow, and GeneratorExit is normally handled by the `for` loop machinery. That's the whole point why except: is strongly discouraged. Be _very_ careful because when a system has matured, the risk of making bad to disastrous design decisions skyrockets (because "the big picture" grows ever larger, and it's ever more difficult to account for all of it). The best solution I know of is an independent sanity-check against the project's core design principles: focus solely on them and say if the suggestion is in harmony with the existing big picture. This prevents the project from falling victim to https://en.wikipedia.org/wiki/Design_by_committee in the long run. This is easier to do for someone not intimately involved with the change and the affected area 'cuz they are less biased in favor of the change and less distracted by minute details. Someone may take up this role to "provide a unified vision" (to reduce the load on a single http://meatballwiki.org/wiki/BenevolentDictator , different projects have tried delegates (this can run afoul of https://en.wikipedia.org/wiki/Conway%27s_law though) and a round-robin approach (Apache)). The best way, however, would probably be for anyone dealing with a design change to remember to make this check. This is even easier in Python, 'cuz the core values are officially formulated as Python Zen, and any module has one or two governing principles at its core, tops, that can be extracted by skimming through its docs.
ChrisA _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
-- Regards, Ivan
On 04.06.2018 23:52, Ivan Pozdeev wrote:
On 04.06.2018 20:11, Chris Angelico wrote:
On Tue, Jun 5, 2018 at 2:57 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Mon, Jun 4, 2018 at 12:50 PM Chris Angelico <rosuav@gmail.com> wrote:
On Tue, Jun 5, 2018 at 2:11 AM, Victor Stinner <vstinner@redhat.com> wrote: [..]
For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example. Pardon the dumb question, but why is try/finally unsuitable? Because try..finally isn't equivalent to try..except? Perhaps you should look at the actual code: https://github.com/python/cpython/blob/b609e687a076d77bdd687f5e4def85e29a044...
In this particular code, it looks like just KeyboardInterrupt needs to be handled in addition to Exception -- and even that's not certain 'cuz KeyboardInterrupt is an abnormal termination and specifically designed to not be messed with by the code ("The exception inherits from |BaseException| <https://docs.python.org/3/library/exceptions.html?highlight=generatorexit#Ba...> so as to not be accidentally caught by code that catches |Exception| <https://docs.python.org/3/library/exceptions.html?highlight=generatorexit#Ex...> and thus prevent the interpreter from exiting."). It only makes sense to catch it in REPL interfaces where the user clearly wants to terminale the current command rather than the entire program. If e.g. a warning is upgraded to exception, this means that some code is broken from user's POV, but not from Python team's POV, so we can't really be sure if we can handle this situation gracefully: our cleanup code can fail just as well!
Oh. Duh. Yep, it was a dumb question. Sorry! The transport should ONLY be closed on error.
I smell a big, big design violation here. The whole point of Exception vs BaseException is that anything not Exception is "not an error", has a completely different effect on the program than an error, and thus is to be dealt with completely differently. For example, warnings do not disrupt the control flow, and GeneratorExit is normally handled by the `for` loop machinery. That's the whole point why except: is strongly discouraged.
Be _very_ careful because when a system has matured, the risk of making bad to disastrous design decisions skyrockets (because "the big picture" grows ever larger, and it's ever more difficult to account for all of it).
The best solution I know of is an independent sanity-check against the project's core design principles: focus solely on them and say if the suggestion is in harmony with the existing big picture. This prevents the project from falling victim to https://en.wikipedia.org/wiki/Design_by_committee in the long run. This is easier to do for someone not intimately involved with the change and the affected area 'cuz they are less biased in favor of the change and less distracted by minute details.
Someone may take up this role to "provide a unified vision" (to reduce the load on a single http://meatballwiki.org/wiki/BenevolentDictator , different projects have tried delegates (this can run afoul of https://en.wikipedia.org/wiki/Conway%27s_law though) and a round-robin approach (Apache)). The best way, however, would probably be for anyone dealing with a design change to remember to make this check.
This is even easier in Python, 'cuz the core values are officially formulated as Python Zen, and any module has one or two governing principles at its core, tops, that can be extracted by skimming through its docs.
ChrisA _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
-- Regards, Ivan
On 05.06.2018 0:54, Ivan Pozdeev wrote:
On 04.06.2018 23:52, Ivan Pozdeev wrote:
On 04.06.2018 20:11, Chris Angelico wrote:
On Tue, Jun 5, 2018 at 2:57 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Mon, Jun 4, 2018 at 12:50 PM Chris Angelico <rosuav@gmail.com> wrote:
On Tue, Jun 5, 2018 at 2:11 AM, Victor Stinner <vstinner@redhat.com> wrote: [..]
For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example. Pardon the dumb question, but why is try/finally unsuitable? Because try..finally isn't equivalent to try..except? Perhaps you should look at the actual code: https://github.com/python/cpython/blob/b609e687a076d77bdd687f5e4def85e29a044...
In this particular code, it looks like just KeyboardInterrupt needs to be handled in addition to Exception -- and even that's not certain 'cuz KeyboardInterrupt is an abnormal termination and specifically designed to not be messed with by the code ("The exception inherits from |BaseException| <https://docs.python.org/3/library/exceptions.html?highlight=generatorexit#Ba...> so as to not be accidentally caught by code that catches |Exception| <https://docs.python.org/3/library/exceptions.html?highlight=generatorexit#Ex...> and thus prevent the interpreter from exiting.").
It only makes sense to catch it in REPL interfaces where the user clearly wants to terminale the current command rather than the entire program.
Remembered `pip`, too -- there, it's justified by it working in transactions.
If e.g. a warning is upgraded to exception, this means that some code is broken from user's POV, but not from Python team's POV, so we can't really be sure if we can handle this situation gracefully: our cleanup code can fail just as well!
Oh. Duh. Yep, it was a dumb question. Sorry! The transport should ONLY be closed on error.
I smell a big, big design violation here. The whole point of Exception vs BaseException is that anything not Exception is "not an error", has a completely different effect on the program than an error, and thus is to be dealt with completely differently. For example, warnings do not disrupt the control flow, and GeneratorExit is normally handled by the `for` loop machinery. That's the whole point why except: is strongly discouraged.
Be _very_ careful because when a system has matured, the risk of making bad to disastrous design decisions skyrockets (because "the big picture" grows ever larger, and it's ever more difficult to account for all of it).
The best solution I know of is an independent sanity-check against the project's core design principles: focus solely on them and say if the suggestion is in harmony with the existing big picture. This prevents the project from falling victim to https://en.wikipedia.org/wiki/Design_by_committee in the long run. This is easier to do for someone not intimately involved with the change and the affected area 'cuz they are less biased in favor of the change and less distracted by minute details.
Someone may take up this role to "provide a unified vision" (to reduce the load on a single http://meatballwiki.org/wiki/BenevolentDictator , different projects have tried delegates (this can run afoul of https://en.wikipedia.org/wiki/Conway%27s_law though) and a round-robin approach (Apache)). The best way, however, would probably be for anyone dealing with a design change to remember to make this check.
This is even easier in Python, 'cuz the core values are officially formulated as Python Zen, and any module has one or two governing principles at its core, tops, that can be extracted by skimming through its docs.
ChrisA _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
-- Regards, Ivan
-- Regards, Ivan
It is currently a general convention in asyncio to only catch Exception, not BaseException. I consider this a flaw and we should fix it, but it's unfortunately not so easy -- the tests will fail if you replace all occurrences of Exception with BaseException, and it is not always clear what's the right thing to do. E.g. catching KeyboardInterrupt may actually make it harder to stop a runaway asyncio app. We should move this discussion to the issue tracker. On Mon, Jun 4, 2018 at 9:11 AM, Victor Stinner <vstinner@redhat.com> wrote:
Hi,
I just read a recent bugfix in asyncio:
https://github.com/python/cpython/commit/9602643120a509858d0bee4215d7f1 50e6125468
+ try: + await waiter + except Exception: + transport.close() + raise
Why only catching "except Exception:"? Why not also catching KeyboardInterrupt or MemoryError? Is it a special rule for asyncio, or a general policy in Python stdlib?
For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example.
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
It is currently a general convention in asyncio to only catch Exception, not BaseException. I consider this a flaw and we should fix it, but it's unfortunately not so easy -- the tests will fail if you replace all occurrences of Exception with BaseException, and it is not always clear what's the right thing to do. E.g. catching KeyboardInterrupt may actually make it harder to stop a runaway asyncio app.
Yes. Catching BaseExceptions or KeyboardInterrupts in start_tls() would be pointless. Currently asyncio's internal state isn't properly hardened to survive a BaseException in all other places it can occur. Fixing that is one of my goals for 3.8. Yury
2018-06-04 18:45 GMT+02:00 Guido van Rossum <guido@python.org>:
It is currently a general convention in asyncio to only catch Exception, not BaseException. I consider this a flaw and we should fix it, but it's unfortunately not so easy -- the tests will fail if you replace all occurrences of Exception with BaseException, and it is not always clear what's the right thing to do. E.g. catching KeyboardInterrupt may actually make it harder to stop a runaway asyncio app.
I recall vaguely something about loop.run_until_complete() which didn't behave "as expected" when interrupted by CTRL+c, like the following call to loop.run_until_complete() didn't work as expected. But this issue has been sorted out, no? I mean that maybe asyncio uses "except Exception:" for "historical reasons"? Victor
On Mon, Jun 4, 2018 at 3:38 PM Victor Stinner <vstinner@redhat.com> wrote:
2018-06-04 18:45 GMT+02:00 Guido van Rossum <guido@python.org>:
It is currently a general convention in asyncio to only catch Exception, not BaseException. I consider this a flaw and we should fix it, but it's unfortunately not so easy -- the tests will fail if you replace all occurrences of Exception with BaseException, and it is not always clear what's the right thing to do. E.g. catching KeyboardInterrupt may actually make it harder to stop a runaway asyncio app.
I recall vaguely something about loop.run_until_complete() which didn't behave "as expected" when interrupted by CTRL+c, like the following call to loop.run_until_complete() didn't work as expected. But this issue has been sorted out, no?
No, the issue is still there. And it's not an easy fix. Yury
participants (5)
-
Chris Angelico -
Guido van Rossum -
Ivan Pozdeev -
Victor Stinner -
Yury Selivanov