threading and multiprocessing deadlock

[Barry Scott]

> On 5 Dec 2021, at 23:50, Johannes Bauer <dfnsonfsduifb at gmx.de> wrote:
> 
> Hi there,
> 
> I'm a bit confused. In my scenario I a mixing threading with
> multiprocessing. Threading by itself would be nice, but for GIL reasons
> I need both, unfortunately. I've encountered a weird situation in which
> multiprocessing Process()es which are started in a new thread don't
> actually start and so they deadlock on join.
> 
> I've created a minimal example that demonstrates the issue. I'm running
> on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
> ([GCC 10.3.0] on linux).
> 
> Here's the code:

I suggest that you include the threading.current_thread() in your messages.
Then you can see which thread is saying what.

Barry

> 
> 
> import time
> import multiprocessing
> import threading
> 
> def myfnc():
> 	print("myfnc")
> 
> def run(result_queue, callback):
> 	result = callback()
> 	result_queue.put(result)
> 
> def start(fnc):
> 	def background_thread():
> 		queue = multiprocessing.Queue()
> 		proc = multiprocessing.Process(target = run, args = (queue, fnc))
> 		proc.start()
> 		print("join?")
> 		proc.join()
> 		print("joined.")
> 		result = queue.get()
> 	threading.Thread(target = background_thread).start()
> 
> start(myfnc)
> start(myfnc)
> start(myfnc)
> start(myfnc)
> while True:
> 	time.sleep(1)
> 
> 
> What you'll see is that "join?" and "joined." nondeterministically does
> *not* appear in pairs. For example:
> 
> join?
> join?
> myfnc
> myfnc
> join?
> join?
> joined.
> joined.
> 
> What's worse is that when this happens and I Ctrl-C out of Python, the
> started Thread is still running in the background:
> 
> $ ps ax | grep minimal
> 370167 pts/0    S      0:00 python3 minimal.py
> 370175 pts/2    S+     0:00 grep minimal
> 
> Can someone figure out what is going on there?
> 
> Best,
> Johannes
> -- 
> https://mail.python.org/mailman/listinfo/python-list
>