Non-blocking concurrent.futures ThreadPoolExecutor
I ran into a problem which, according to a google search is not that uncommon: concurrent.futures ThreadPoolExecutor, installs an exit handler preventing program exit if any of the jobs it is running is blocked. (Which might be a surprising behaviour to someone using those). It is possible to workaround this problem by either unregistering the exit handler concurrent.futures.thread._python_exit or by subclassing the threadpoolExecutor and overriding the _adjust_thread_count method with one that does not register its threads queues. Both seems kinda ugly though. It would seems reasonable to have a separate _daemon_thread_queues dict (in contrast to _threads_queues) that would not get joined upon exit. A "daemon" (or any better name) parameter of ThreadPoolExecutor initialisation would allow to let it add its workers in this dict rather than the default one. Does this seems feasible ? Thanks !
It would seems reasonable to have a separate _daemon_thread_queues dict (in contrast to _threads_queues) that would not get joined upon exit. A "daemon" (or any better name) parameter of ThreadPoolExecutor initialisation would allow to let it add its workers in this dict rather than the default one.
We're currently looking into adjusting the current Executor implementation (for both TPE and PPE) to not use daemon threads at all for subinterpreter compatibility, see https://bugs.python.org/issue39812. Instead of using an atexit handler, there's some consideration for using a similar threading-specific "exit handler". Instead of being upon program exit, registered functions would be called just before all of the non-daemon threads are joined in `threading._shutdown()`. As a result, I suspect that we are unlikely to implement something like the above. But, as a convenient side effect, it should somewhat address the current potentially surprising behavior. (Note that this is currently in early draft stages, so it could end up being entirely different) On Mon, Mar 9, 2020 at 5:39 PM Remy NOEL <mocramis@gmail.com> wrote:
I ran into a problem which, according to a google search is not that uncommon: concurrent.futures ThreadPoolExecutor, installs an exit handler preventing program exit if any of the jobs it is running is blocked. (Which might be a surprising behaviour to someone using those).
It is possible to workaround this problem by either unregistering the exit handler concurrent.futures.thread._python_exit or by subclassing the threadpoolExecutor and overriding the _adjust_thread_count method with one that does not register its threads queues.
Both seems kinda ugly though. It would seems reasonable to have a separate _daemon_thread_queues dict (in contrast to _threads_queues) that would not get joined upon exit. A "daemon" (or any better name) parameter of ThreadPoolExecutor initialisation would allow to let it add its workers in this dict rather than the default one.
Does this seems feasible ?
Thanks !
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FH44JO... Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Mar 9, 2020 at 11:18 PM Kyle Stanley <aeros167@gmail.com> wrote:
We're currently looking into adjusting the current Executor implementation (for both TPE and PPE) to not use daemon threads at all for subinterpreter compatibility, see https://bugs.python.org/issue39812. Instead of using an atexit handler, there's some consideration for using a similar threading-specific "exit handler". Instead of being upon program exit, registered functions would be called just before all of the non-daemon threads are joined in `threading._shutdown()`.
As a result, I suspect that we are unlikely to implement something like the above. But, as a convenient side effect, it should somewhat address the current potentially surprising behavior.
There might still be some fun involved given that current implementation *might* lazy-spawn threads upon submit. This means that if a ThreadExecutor is called from both daemon and non-daemon threads (which is a very bad idea though), the kind of spawned threads might be unpredictable.
On Tue, 10 Mar 2020 15:16:06 +0100 Remy NOEL <mocramis@gmail.com> wrote:
On Mon, Mar 9, 2020 at 11:18 PM Kyle Stanley <aeros167@gmail.com> wrote:
We're currently looking into adjusting the current Executor implementation (for both TPE and PPE) to not use daemon threads at all for subinterpreter compatibility, see https://bugs.python.org/issue39812. Instead of using an atexit handler, there's some consideration for using a similar threading-specific "exit handler". Instead of being upon program exit, registered functions would be called just before all of the non-daemon threads are joined in `threading._shutdown()`.
As a result, I suspect that we are unlikely to implement something like the above. But, as a convenient side effect, it should somewhat address the current potentially surprising behavior.
There might still be some fun involved given that current implementation *might* lazy-spawn threads upon submit. This means that if a ThreadExecutor is called from both daemon and non-daemon threads (which is a very bad idea though), the kind of spawned threads might be unpredictable.
Daemon threads are already a bad idea, actually. It is really worth the effort to ensure your threads exit cleanly rather than expect the interpreter to kill them at shutdown. Regards Antoine.
This means that if a ThreadExecutor is called from both daemon and non-daemon threads (which is a very bad idea though), the kind of spawned
Remy NOEL wrote: threads might be unpredictable. IIUC, I don't think there's a way that we could reasonably support stability for provide any guarantees for spawning or interacting with the Executors from a daemon thread. It might even be worth considering to explicitly warn or prevent that in some capacity. Antoine Pitrou wrote:
Daemon threads are already a bad idea, actually. It is really worth the effort to ensure your threads exit cleanly rather than expect the interpreter to kill them at shutdown.
Agreed. Furthermore, if there's a thread in a program that can potentially hang indefinitely, converting them into daemon threads (allowing it to be killed on interpreter shutdown) just obscures the problem in most cases, IMO. There should be some consideration placed into preventing a thread from hanging indefinitely in the first place, such as with timeouts or some form of outside signaling (such as threading.Event). That's not to say that daemon threads aren't sometimes useful. Essentially, if you're okay with the thread being abruptly killed without allowing it's resources to be cleaned up gracefully, making it a daemon thread shouldn't be a significant issue. But, the core problem here is that ThreadPoolExecutor is an example of a resource that should be cleaned up gracefully, so spawning it in a daemon thread or trying to interact with it through one can interfere with that process. On Tue, Mar 10, 2020 at 3:40 PM Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 10 Mar 2020 15:16:06 +0100 Remy NOEL <mocramis@gmail.com> wrote:
On Mon, Mar 9, 2020 at 11:18 PM Kyle Stanley <aeros167@gmail.com> wrote:
We're currently looking into adjusting the current Executor implementation (for both TPE and PPE) to not use daemon threads at all for subinterpreter compatibility, see https://bugs.python.org/issue39812. Instead of using an atexit handler, there's some consideration for using a similar threading-specific "exit handler". Instead of being upon program exit, registered functions would be called just before all of the non-daemon threads are joined in `threading._shutdown()`.
As a result, I suspect that we are unlikely to implement something like the above. But, as a convenient side effect, it should somewhat address the current potentially surprising behavior.
There might still be some fun involved given that current implementation *might* lazy-spawn threads upon submit. This means that if a ThreadExecutor is called from both daemon and non-daemon threads (which is a very bad idea though), the kind of spawned threads might be unpredictable.
Daemon threads are already a bad idea, actually. It is really worth the effort to ensure your threads exit cleanly rather than expect the interpreter to kill them at shutdown.
Regards
Antoine.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CUY3K4... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Mar 11, 2020 at 3:20 AM Kyle Stanley <aeros167@gmail.com> wrote:
IIUC, I don't think there's a way that we could reasonably support stability for provide any guarantees for spawning or interacting with the Executors from a daemon thread. It might even be worth considering to explicitly warn or prevent that in some capacity.
We could however easily detect this case: we just need to check current_thread().daemon at submit against existing worker threads.
Antoine Pitrou wrote:
Daemon threads are already a bad idea, actually. It is really worth the effort to ensure your threads exit cleanly rather than expect the interpreter to kill them at shutdown.
Agreed. Furthermore, if there's a thread in a program that can potentially hang indefinitely, converting them into daemon threads (allowing it to be killed on interpreter shutdown) just obscures the problem in most cases, IMO. There should be some consideration placed into preventing a thread from hanging indefinitely in the first place, such as with timeouts or some form of outside signaling (such as threading.Event).
That's not to say that daemon threads aren't sometimes useful. Essentially, if you're okay with the thread being abruptly killed without allowing it's resources to be cleaned up gracefully, making it a daemon thread shouldn't be a significant issue. But, the core problem here is that ThreadPoolExecutor is an example of a resource that should be cleaned up gracefully, so spawning it in a daemon thread or trying to interact with it through one can interfere with that process.
I completely agree (even more so that i already ran into some undying daemon threads): This should be set up from the very start. Sadly, It is very hard to get rid of them when an entire existing codebase rely on those for exit :/ . Anyway, thanks for the information and advices !
On 09/03/2020 17:50, Remy NOEL wrote:
I ran into a problem which, according to a google search is not that uncommon: concurrent.futures ThreadPoolExecutor, installs an exit handler preventing program exit if any of the jobs it is running is blocked. (Which might be a surprising behaviour to someone using those).
It is possible to workaround this problem by either unregistering the exit handler concurrent.futures.thread._python_exit or by subclassing the threadpoolExecutor and overriding the _adjust_thread_count method with one that does not register its threads queues.
Both seems kinda ugly though.
Well, yes. The non-ugly thing would be to make sure your threads are well behaved ;-) -- Rhodri James *-* Kynesim Ltd
participants (4)
-
Antoine Pitrou -
Kyle Stanley -
Remy NOEL -
Rhodri James