threading and multiprocessing deadlock
Johannes Bauer
dfnsonfsduifb at gmx.de
Sun Dec 5 18:50:11 EST 2021
Hi there,
I'm a bit confused. In my scenario I a mixing threading with
multiprocessing. Threading by itself would be nice, but for GIL reasons
I need both, unfortunately. I've encountered a weird situation in which
multiprocessing Process()es which are started in a new thread don't
actually start and so they deadlock on join.
I've created a minimal example that demonstrates the issue. I'm running
on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
([GCC 10.3.0] on linux).
Here's the code:
import time
import multiprocessing
import threading
def myfnc():
print("myfnc")
def run(result_queue, callback):
result = callback()
result_queue.put(result)
def start(fnc):
def background_thread():
queue = multiprocessing.Queue()
proc = multiprocessing.Process(target = run, args = (queue, fnc))
proc.start()
print("join?")
proc.join()
print("joined.")
result = queue.get()
threading.Thread(target = background_thread).start()
start(myfnc)
start(myfnc)
start(myfnc)
start(myfnc)
while True:
time.sleep(1)
What you'll see is that "join?" and "joined." nondeterministically does
*not* appear in pairs. For example:
join?
join?
myfnc
myfnc
join?
join?
joined.
joined.
What's worse is that when this happens and I Ctrl-C out of Python, the
started Thread is still running in the background:
$ ps ax | grep minimal
370167 pts/0 S 0:00 python3 minimal.py
370175 pts/2 S+ 0:00 grep minimal
Can someone figure out what is going on there?
Best,
Johannes
More information about the Python-list
mailing list