[pypy-issue] Issue #3154: Multiprocessing leaking file descriptors (pypy/pypy)

[Andrew Robinson]

New issue 3154: Multiprocessing leaking file descriptors
https://bitbucket.org/pypy/pypy/issues/3154/multiprocessing-leaking-file-descriptors

Andrew Robinson:

Have migrated a multiprocessing project from PyPy 6.0 \(2.7\) to PyPy 7.3 \(3.6\) and have identified an issue with multiprocessing leaking descriptors. I believe this may also relate to the issue reported as #3060.

I have identified this issue as repeatable on both Ubuntu 18.04.2 and CentOS 7.0.

Simple code to reproduce issue \(Taken from #3111 which is raised a different issue, but current issue is reproduced with the same code sample\):

```
import multiprocessing

def process_chunk(chunk_index):
    return ([],[],[])

def main():
    i = 0
    with multiprocessing.Pool(5, maxtasksperchild=1) as p:
        for sub_headers, sub_results, sub_scores in p.imap_unordered(process_chunk, range(5000)):
            print(i)
            i += 1

if __name__ == '__main__':
    main()
```

If saving the above as “test.py” the descriptor leak can be verified by running “/path/to/pypy3 test.py” then taking the PID from ps and doing an “ls -l /proc/PID/fd” and comparing to the same when running native Python Ie. Running “/path/to/python3 test.py” taking PID and “ls -l /proc/PID/fd”.

Under native Python3 the descriptor list remains constant at 9 pipe entries.

Under PyPy the file descriptor list keeps growing until ulimit is reached and it will throw an exception as “OSError: \[Errno 24\] Too many open files“. In my case ulimit is set to 1024.

Traceback below:

‌

```
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/callation/pypy3/lib-python/3/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/callation/pypy3/lib-python/3/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 405, in _handle_workers
    pool._maintain_pool()
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 246, in _maintain_pool
    self._repopulate_pool()
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 239, in _repopulate_pool
    w.start()
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 65, in _launch
    parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files
```

‌