threading and multiprocessing deadlock
Martin Di Paola
martinp.dipaola at gmail.com
Mon Dec 6 07:56:43 EST 2021
Hi!, in short your code should work.
I think that the join-joined problem is just an interpretation problem.
In pseudo code the background_thread function does:
def background_thread()
# bla
print("join?")
# bla
print("joined")
When running this function in parallel using threads, you will probably
get a few "join?" first before receiving any "joined?". That is because
the functions are running in parallel.
The order "join?" then "joined" is preserved within a thread but not
preserved globally.
Now, I see another issue in the output (and perhaps you was asking about
this one):
join?
join?
myfnc
myfnc
join?
join?
joined.
joined.
So you have 4 "join?" that correspond to the 4 background_thread
function calls in threads but only 2 "myfnc" and 2 "joined".
Could be possible that the output is truncated by accident?
I ran the same program and I got a reasonable output (4 "join?", "myfnc"
and "joined"):
join?
join?
myfnc
join?
myfnc
join?
joined.
myfnc
joined.
joined.
myfnc
joined.
Another issue that I see is that you are not joining the threads that
you spawned (background_thread functions).
I hope that this can guide you to fix or at least narrow the issue.
Thanks,
Martin.
On Mon, Dec 06, 2021 at 12:50:11AM +0100, Johannes Bauer wrote:
>Hi there,
>
>I'm a bit confused. In my scenario I a mixing threading with
>multiprocessing. Threading by itself would be nice, but for GIL reasons
>I need both, unfortunately. I've encountered a weird situation in which
>multiprocessing Process()es which are started in a new thread don't
>actually start and so they deadlock on join.
>
>I've created a minimal example that demonstrates the issue. I'm running
>on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
>([GCC 10.3.0] on linux).
>
>Here's the code:
>
>
>import time
>import multiprocessing
>import threading
>
>def myfnc():
> print("myfnc")
>
>def run(result_queue, callback):
> result = callback()
> result_queue.put(result)
>
>def start(fnc):
> def background_thread():
> queue = multiprocessing.Queue()
> proc = multiprocessing.Process(target = run, args = (queue, fnc))
> proc.start()
> print("join?")
> proc.join()
> print("joined.")
> result = queue.get()
> threading.Thread(target = background_thread).start()
>
>start(myfnc)
>start(myfnc)
>start(myfnc)
>start(myfnc)
>while True:
> time.sleep(1)
>
>
>What you'll see is that "join?" and "joined." nondeterministically does
>*not* appear in pairs. For example:
>
>join?
>join?
>myfnc
>myfnc
>join?
>join?
>joined.
>joined.
>
>What's worse is that when this happens and I Ctrl-C out of Python, the
>started Thread is still running in the background:
>
>$ ps ax | grep minimal
> 370167 pts/0 S 0:00 python3 minimal.py
> 370175 pts/2 S+ 0:00 grep minimal
>
>Can someone figure out what is going on there?
>
>Best,
>Johannes
>--
>https://mail.python.org/mailman/listinfo/python-list
More information about the Python-list
mailing list