[New-bugs-announce] [issue42752] multiprocessing Queue leaks a file descriptor associated with the pipe writer (#33081 still a problem)

[Alex Orange]

New submission from Alex Orange <crazycasta at gmail.com>:

Didn't feel like necroing #33081, but this is basically that problem. The trouble is the cleanup that appeared to fix #33081 only kicks in once something has been put in the queue. So if for instance a Process function puts something in the queue and the parent gets it, then calls q.close() the writer on the parent side doesn't get culled until the object does. This is particularly a problem for PyPy and isn't exactly great for any weird corner cases if anyone holds onto Queue objects after they're closed for any reason (horders!).

Attached file test_queue.py is an example of how to trigger this problem. Run it without a command line argument "python test_queue.py" and it won't crash (though it will take a very long time to complete). Run with an argument "python test_queue.py fail" and it will fail once you run out of file descriptors (one leaked per queue).

My suggestion on how to handle this is to set self._close to something that will close self._writer. Then, when _start_thread is called, instead of directly passing the self._writer.close object, pass a small function that will switch out self._close to the Finalize method used later on and return self._writer. Finally, inside _feed, use this method to get the _writer object and wrap the outer while 1 with a contextlib.closer on this object.

This is a fair bit of stitching things together here and there so let me know if anyone has any suggestions on this before I get started.

----------
components: Library (Lib)
files: test_queue.py
messages: 383832
nosy: Alex.Orange
priority: normal
severity: normal
status: open
title: multiprocessing Queue leaks a file descriptor associated with the pipe writer (#33081 still a problem)
type: resource usage
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file49701/test_queue.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42752>
_______________________________________