[pypy-issue] Issue #3111: Multiprocessing occasionally hangs (pypy/pypy)

[Aaron Wise]

New issue 3111: Multiprocessing occasionally hangs
https://bitbucket.org/pypy/pypy/issues/3111/multiprocessing-occasionally-hangs

Aaron Wise:

I’ve had an issue with some multiprocessed code occasionally failing, and have managed to work it to a \(fairly\) minimal example \(below\).  Running this code with pypy3.6-7.2 on linux \(ubuntu 18.04\) causes the loop to hang occasionally \(about 1 in 10 runs\).  When I ctrl\+c, I get the following error message \(interrupted by the KeyboardInterrupt\):

> Traceback \(most recent call last\):  
> File "/path/to/pypy3.6-7.2/lib-python/3/multiprocessing/pool.py", line 720, in next  
> item = self.\_items.popleft\(\)  
> IndexError: pop from an empty deque

As far as I can tell, if I do not define maxtasksperchild=1 \(and thus reuse processes\) the error does not occur.  \(But it may occur very rarely, I’ve only tested it about 1000 times.\)

‌

```python
import multiprocessing

def process_chunk(chunk_index):
    return ([],[],[])

def main():
    i = 0
    with multiprocessing.Pool(5, maxtasksperchild=1) as p:
        for sub_headers, sub_results, sub_scores in p.imap_unordered(process_chunk, range(5000)):
            print(i)
            i += 1

if __name__ == '__main__':
    main()
```

‌