[issue20319] concurrent.futures.wait() can block forever even if Futures have completed

[Sebastian Kreft]

Sebastian Kreft added the comment:

I'm using the Python 3.4.1 compiled from source and I'm may be hitting this issue.

My workload is launching two subprocess in parallel, and whenever one is ready, launches another one. In one of the runs, the whole process got stuck after launching about 3K subprocess, and the underlying processes had in fact finished.

To wait for the finished subprocesses, I'm using FIRST_COMPLETED. Below is the core of my workload:

for element in element_generator:
    while len(running) >= max_tasks:
        done, pending = concurrent.futures.wait(running, timeout=15.0, return_when=concurrent.futures.FIRST_COMPLETED)
        process_results(done)
        running = pending

    running.add(executor.submit(exe_subprocess, element)) 
 

I don't really know what's the best way to reproduce this, as I've run the same workload with different executables, more concurrency and faster response times, and I haven't seen the issue.

----------
nosy: +Sebastian.Kreft.Deezer

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20319>
_______________________________________