killing tasks that won't cancel
I have an asyncio question. In Python 3.7, is there a way to reliably end a task after having already tried calling cancel() on it and waiting for it to end? In Python 3.6, I did this with task.set_exception(), but in 3.7 that method was removed. --Chris
If the task's function swallows CancelledError exception -- it is a programming error. The same as generator object technically can swallow GeneratorExit (but such code is most likely buggy). On Tue, Feb 19, 2019 at 9:55 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have an asyncio question.
In Python 3.7, is there a way to reliably end a task after having already tried calling cancel() on it and waiting for it to end?
In Python 3.6, I did this with task.set_exception(), but in 3.7 that method was removed.
--Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
On Tue, Feb 19, 2019 at 12:10 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
If the task's function swallows CancelledError exception -- it is a programming error.
I was asking if there is a way to end such a task. Is there? The only approach I can think of without having something like set_exception() is to keep calling cancel() in a loop and waiting (but even that can fail under certain code), but I'm not sure off-hand if the API supports calling cancel() more than once. Also, I can see this happening even when there is no bug. Maybe the coroutine was properly written to cancel gracefully, but the caller doesn't want to continue waiting past a certain time. --Chris
The same as generator object technically can swallow GeneratorExit (but such code is most likely buggy).
On Tue, Feb 19, 2019 at 9:55 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have an asyncio question.
In Python 3.7, is there a way to reliably end a task after having already tried calling cancel() on it and waiting for it to end?
In Python 3.6, I did this with task.set_exception(), but in 3.7 that method was removed.
--Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
Let's continue discussion on the bug tracker: https://bugs.python.org/issue32363 On Tue, Feb 19, 2019 at 10:23 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Feb 19, 2019 at 12:10 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
If the task's function swallows CancelledError exception -- it is a programming error.
I was asking if there is a way to end such a task. Is there? The only approach I can think of without having something like set_exception() is to keep calling cancel() in a loop and waiting (but even that can fail under certain code), but I'm not sure off-hand if the API supports calling cancel() more than once.
Also, I can see this happening even when there is no bug. Maybe the coroutine was properly written to cancel gracefully, but the caller doesn't want to continue waiting past a certain time.
--Chris
The same as generator object technically can swallow GeneratorExit (but such code is most likely buggy).
On Tue, Feb 19, 2019 at 9:55 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have an asyncio question.
In Python 3.7, is there a way to reliably end a task after having already tried calling cancel() on it and waiting for it to end?
In Python 3.6, I did this with task.set_exception(), but in 3.7 that method was removed.
--Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
-- Thanks, Andrew Svetlov
On Feb 19, 2019, at 3:23 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Feb 19, 2019 at 12:10 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
If the task's function swallows CancelledError exception -- it is a programming error.
I was asking if there is a way to end such a task. Is there? The only approach I can think of without having something like set_exception() is to keep calling cancel() in a loop and waiting (but even that can fail under certain code), but I'm not sure off-hand if the API supports calling cancel() more than once.
Unfortunately asyncio isn't super flexible around "cancellation with a timeout" kind of scenarios. The current assumption is that once the cancellation is requested, the Task will start cancelling and will do so in a timely manner. Imposing a second layer of timeouts on the cancellation process itself isn't natively supported. But to properly address this we don't need a very broadly defined Task.set_exception(); we need to rethink the cancellation in asyncio (perhaps draw some inspiration from Trio and other frameworks). Yury
On Tue, Feb 19, 2019 at 12:33 PM Yury Selivanov <yselivanov@gmail.com> wrote:
Unfortunately asyncio isn't super flexible around "cancellation with a timeout" kind of scenarios. The current assumption is that once the cancellation is requested, the Task will start cancelling and will do so in a timely manner. Imposing a second layer of timeouts on the cancellation process itself isn't natively supported. But to properly address this we don't need a very broadly defined Task.set_exception();
Yes, I agree. I was just using Task.set_exception() because that is all that was available. (And I agree set_result() isn't needed.) --Chris
we need to rethink the cancellation in asyncio (perhaps draw some inspiration from Trio and other frameworks).
Yury
On Tue, Feb 19, 2019 at 12:33 PM Yury Selivanov <yselivanov@gmail.com> wrote:
Unfortunately asyncio isn't super flexible around "cancellation with a timeout" kind of scenarios. The current assumption is that once the cancellation is requested, the Task will start cancelling and will do so in a timely manner. Imposing a second layer of timeouts on the cancellation process itself isn't natively supported. But to properly address this we don't need a very broadly defined Task.set_exception(); we need to rethink the cancellation in asyncio (perhaps draw some inspiration from Trio and other frameworks).
What options have you already considered for asyncio's API? A couple naive things that occur to me are adding task.kill() with higher priority than task.cancel() (or equivalently task.graceful_cancel() with lower priority). Was task.cancel() meant more to have the meaning of "kill" or "graceful cancel"? In addition, the graceful version of the two (whichever that may be) could accept a timeout argument -- after which the exception of higher priority is raised. I realize this is a more simplistic model compared to the options trio is considering, but asyncio has already gone down the path of the simpler approach. --Chris
Yury
I'm not sure what a "higher priority" exception is...raising an exception is hard to miss. There are a few things Trio does differently here that might be relevant, but it depends on why Chris is having trouble cancelling tasks. 1. Trio's cancellation exception, trio.Cancelled, inherits from BaseException instead of Exception, like KeyboardInterrupt or StopIteration. So 'except Exception' doesn't catch it by accident. 2. Trio's cancellation is "stateful": if your code is in the cancelled state, then every time you try to do an async operation then it raises trio.Cancelled again. So you avoid the case where a tasks gets stuck, someone forces it to raise CancelledError, and then it has a 'finally' block that tries to do some cleanup... but the 'finally' block also gets stuck. In trio the 'finally' block can't accidentally get stuck. 3. Both of these features are somewhat dependent on trio using "delimited" cancellation. Before you can cancel something, you have to say how far you want to unwind. This means that there's never any reason for anyone to try to catch 'Cancelled' on purpose, because trio will catch it for you at the appropriate moment. And it's hard to do 'stateful' cancellation if you don't know how long the state is supposed to persist. And, you avoid cases where some code thinks it just threw in a CancelledError and is supposed to catch it, but actually it was thrown in from some other stack frame, and it ends up confusedly catching the wrong exception. I'm not sure how much of this could be adapted for asyncio. The obvious change would be to make asyncio.CancelledError a BaseException, though it seems borderline to me from a back-compat perspective. I think I remember Yury was thinking about changing it anyway, though? That would definitely help with the 'except Exception' kind of mistake. But the other issues are deeper. If you don't have a solid system for keeping track of what exactly is supposed to be cancelled, then it's easy to accidentally cancel too much, or cancel too little. Solving that requires a systematic approach. And unfortunately, asyncio already has 2 different sets of cancellation semantics (Future.cancel -> takes effect immediately, irrevocable & idempotent, doesn't necessarily cause the underlying machinery to stop processing, just stops it from reporting its result; Task.cancel -> doesn't take effect immediately or necessarily at all, can be called repeatedly and injects one CancelledError per call, tries to stop the underlying machinery, chains to other tasks/futures that the first task is await'ing). So if our goal is to make the system as a whole as reliable and predictable as possible within the constraints of back-compat... I don't know whether adding a third set of semantics would actually help, or make more code confused about what it was supposed to be catching. And I don't know if any of these actually address whatever problem you're having with uncancellable tasks. It's certainly possible to make an uncancellable task in Trio too. We just try to make it hard to do by accident. -n On Tue, Feb 19, 2019 at 3:53 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Feb 19, 2019 at 12:33 PM Yury Selivanov <yselivanov@gmail.com> wrote:
Unfortunately asyncio isn't super flexible around "cancellation with a timeout" kind of scenarios. The current assumption is that once the cancellation is requested, the Task will start cancelling and will do so in a timely manner. Imposing a second layer of timeouts on the cancellation process itself isn't natively supported. But to properly address this we don't need a very broadly defined Task.set_exception(); we need to rethink the cancellation in asyncio (perhaps draw some inspiration from Trio and other frameworks).
What options have you already considered for asyncio's API? A couple naive things that occur to me are adding task.kill() with higher priority than task.cancel() (or equivalently task.graceful_cancel() with lower priority). Was task.cancel() meant more to have the meaning of "kill" or "graceful cancel"? In addition, the graceful version of the two (whichever that may be) could accept a timeout argument -- after which the exception of higher priority is raised. I realize this is a more simplistic model compared to the options trio is considering, but asyncio has already gone down the path of the simpler approach.
--Chris
Yury
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Nathaniel J. Smith -- https://vorpus.org
On Tue, Feb 19, 2019 at 7:41 PM Nathaniel Smith <njs@pobox.com> wrote:
I'm not sure what a "higher priority" exception is...raising an exception is hard to miss.
Just quickly on this one point: that was just my colloquial way of saying superclass (or at least not a subclass), to emphasize that if you're e.g. catching CancelledError, this new exception would bubble up whereas CancelledError wouldn't. A similar example is KeyboardInterrupt not being caught by Exception. In the case of CancelledError, we probably would want the two exceptions to be comparable to one another rather than incomparable. I seem to recall for example there being an old discussion as to whether CancelledError should inherit from Exception or not. --Chris
There are a few things Trio does differently here that might be relevant, but it depends on why Chris is having trouble cancelling tasks.
1. Trio's cancellation exception, trio.Cancelled, inherits from BaseException instead of Exception, like KeyboardInterrupt or StopIteration. So 'except Exception' doesn't catch it by accident.
2. Trio's cancellation is "stateful": if your code is in the cancelled state, then every time you try to do an async operation then it raises trio.Cancelled again. So you avoid the case where a tasks gets stuck, someone forces it to raise CancelledError, and then it has a 'finally' block that tries to do some cleanup... but the 'finally' block also gets stuck. In trio the 'finally' block can't accidentally get stuck.
3. Both of these features are somewhat dependent on trio using "delimited" cancellation. Before you can cancel something, you have to say how far you want to unwind. This means that there's never any reason for anyone to try to catch 'Cancelled' on purpose, because trio will catch it for you at the appropriate moment. And it's hard to do 'stateful' cancellation if you don't know how long the state is supposed to persist. And, you avoid cases where some code thinks it just threw in a CancelledError and is supposed to catch it, but actually it was thrown in from some other stack frame, and it ends up confusedly catching the wrong exception.
I'm not sure how much of this could be adapted for asyncio. The obvious change would be to make asyncio.CancelledError a BaseException, though it seems borderline to me from a back-compat perspective. I think I remember Yury was thinking about changing it anyway, though? That would definitely help with the 'except Exception' kind of mistake.
But the other issues are deeper. If you don't have a solid system for keeping track of what exactly is supposed to be cancelled, then it's easy to accidentally cancel too much, or cancel too little. Solving that requires a systematic approach. And unfortunately, asyncio already has 2 different sets of cancellation semantics (Future.cancel -> takes effect immediately, irrevocable & idempotent, doesn't necessarily cause the underlying machinery to stop processing, just stops it from reporting its result; Task.cancel -> doesn't take effect immediately or necessarily at all, can be called repeatedly and injects one CancelledError per call, tries to stop the underlying machinery, chains to other tasks/futures that the first task is await'ing). So if our goal is to make the system as a whole as reliable and predictable as possible within the constraints of back-compat... I don't know whether adding a third set of semantics would actually help, or make more code confused about what it was supposed to be catching.
And I don't know if any of these actually address whatever problem you're having with uncancellable tasks. It's certainly possible to make an uncancellable task in Trio too. We just try to make it hard to do by accident.
-n
On Tue, Feb 19, 2019 at 3:53 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Feb 19, 2019 at 12:33 PM Yury Selivanov <yselivanov@gmail.com>
Unfortunately asyncio isn't super flexible around "cancellation with a
timeout" kind of scenarios. The current assumption is that once the cancellation is requested, the Task will start cancelling and will do so in a timely manner. Imposing a second layer of timeouts on the cancellation
wrote: process itself isn't natively supported. But to properly address this we don't need a very broadly defined Task.set_exception(); we need to rethink the cancellation in asyncio (perhaps draw some inspiration from Trio and other frameworks).
What options have you already considered for asyncio's API? A couple
naive things that occur to me are adding task.kill() with higher priority than task.cancel() (or equivalently task.graceful_cancel() with lower priority). Was task.cancel() meant more to have the meaning of "kill" or "graceful cancel"? In addition, the graceful version of the two (whichever that may be) could accept a timeout argument -- after which the exception of higher priority is raised. I realize this is a more simplistic model compared to the options trio is considering, but asyncio has already gone down the path of the simpler approach.
--Chris
Yury
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Nathaniel J. Smith -- https://vorpus.org
FYI Chris has started a parallel discussion on the same topic here: https://bugs.python.org/issue32363. Chris, let's keep this discussion in one place (now it's this list, I guess). It's hard to handle the same discussion in two different places. Please don't split discussions like this. I'll summarize what I said in the above referenced bpo here: 1. Task.set_result() and Task.set_exception() have never ever worked properly. They never actually communicated the set result/exception to the underlying coroutine. The fact that they were exposed at all was a simple oversight. I can guess how Task.set_exception() can be implemented in theory: the exception would be thrown into the wrapped coroutine. But I don't quite understand how Task.set_result() can be implemented at all. 2. Task and coroutine maintain a simple relationship: Task wraps its coroutine. The result of the coroutine is the result of the Task (not the other way around). The Task can request its coroutine to cancel. The coroutine may ignore that request by ignoring the asyncio.CancelledError exception. If the latter happens, the Task cannot terminate the coroutine, this is by design. Moreover, you can always write while True: try: await asyncio.sleep(1) except: pass and then nothing can terminate your coroutine. IOW, if your code chooses to ignore CancelledError the Task can do nothing about it. 3. For proper bi-directional communication between coroutines asyncio has queues. One can easily implement a message queue to implement injection of an exception or result into a coroutine. [Chris]
I was asking if there is a way to end such a task. Is there?
No, there's no way to end tasks like that. The key question here is: is this a theoretical problem you're concerned with? Or is this something that happens in real-world framework/library/code that you're dealing with? Yury
On Feb 19, 2019, at 2:53 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have an asyncio question.
In Python 3.7, is there a way to reliably end a task after having already tried calling cancel() on it and waiting for it to end?
In Python 3.6, I did this with task.set_exception(), but in 3.7 that method was removed.
--Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Tue, Feb 19, 2019 at 12:27 PM Yury Selivanov <yselivanov@gmail.com> wrote:
FYI Chris has started a parallel discussion on the same topic here: https://bugs.python.org/issue32363. Chris, let's keep this discussion in one place (now it's this list, I guess). It's hard to handle the same discussion in two different places. Please don't split discussions like this.
My apologies. My first comment was on the tracker, but then I realized this is a broader discussion, so I moved to this list. For example, it seems related to a discussion happening on the trio tracker re: graceful shutdown and "hard" and "soft" cancellation, etc: https://github.com/python-trio/trio/issues/147 I'm not sure how similar or different cancellation is across various async frameworks. --Chris
Thanks for referencing that issue, I'll check it out. I'm also quite curious what Nathaniel thinks about this problem and how he thinks he'll handle it in Trio. Yury
On Feb 19, 2019, at 3:36 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Feb 19, 2019 at 12:27 PM Yury Selivanov <yselivanov@gmail.com> wrote:
FYI Chris has started a parallel discussion on the same topic here: https://bugs.python.org/issue32363. Chris, let's keep this discussion in one place (now it's this list, I guess). It's hard to handle the same discussion in two different places. Please don't split discussions like this.
My apologies. My first comment was on the tracker, but then I realized this is a broader discussion, so I moved to this list. For example, it seems related to a discussion happening on the trio tracker re: graceful shutdown and "hard" and "soft" cancellation, etc: https://github.com/python-trio/trio/issues/147 I'm not sure how similar or different cancellation is across various async frameworks.
--Chris
participants (4)
-
Andrew Svetlov -
Chris Jerdonek -
Nathaniel Smith -
Yury Selivanov