Miguel Ángel Prosper wrote:
> Thank you so much for the work, I was very confused on how to even start
implementing it in the ProcessPoolExecutor, but you finished everything
super quick!
No problem! The ProcessPoolExecutor implementation wasn't immediately clear to me either, but after some experimentation (and some great suggestions from Brian Quinlain to improve it) I was able to get it functioning as intended. Thank you for the proposal, I think it will be a good addition for the Executor API.
Miguel Ángel Prosper wrote:
> I'm suppose that this might be better in another thread but... Could it
be possible to changing the context manager of the executor to cancel
futures on uncaught exceptions?
Hmm, it should be possible. Do you specifically mean cancelling the pending futures once a single one of the submitted functions raises an exception, or cancelling the pending futures when the Executor itself raises an exception (I.E. BrokenProcessPool)? I would assume the prior, since that seems more useful to me.
Miguel Ángel Prosper wrote:
> If not I could also be added as an optional parameter to the executor
constructor, "cancel_on_error" set to False to not changing anything.
Assuming we did implement something like this, I'd say it should be done as an optional parameter for the constructor (as stated above) rather than as the default option when an exception occurs, because it would significantly different from the current behavior. At the moment, the executor doesn't mind when exceptions are raised from submitted functions; it simply sets the exception to the future (via ``future.set_exception()``) and moves on to the next.
So, cancelling the pending futures after one raises an exception would be a significant behavioral change; one that could easily result in breaking existing code. As a result, I think it should be an optional parameter if anything.
As far as whether or not it should be implemented, I'm not 100% certain. While I'm not against the idea of it, I think it would be reasonable to wait until the new parameter is released in Python 3.9 and see what users think of it prior to adding a similar parameter to the executor constructor as well. The cost of adding a new feature demanded by users is relatively minimal, but once it's added we're stuck with maintaining it.
If the idea is approved, I'd be happy to help with implementing it. I just want to make sure that we actually want it in the first place.
What do you think, Guido? I've also CC'd Brian Quinlain and Antoine Pitrou, in case they have any feedback but didn't see the original thread.