[New-bugs-announce] [issue46391] Library multiprocess leaks named resources.

[XD Trol]

New submission from XD Trol <milestonejxd at gmail.com>:

Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python. 

Many Repo users have encountered resource leak warnings with Python 3.9. 
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5

I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.

To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )

```
import multiprocessing as mp

global_resource = mp.Semaphore()

def submain(): pass

if __name__ == '__main__':
    p = mp.Process(target=submain)
    p.start()
    p.join()
```

Tested on macOS with Python 3.9.7
> python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.

This bug will 100% reproduce when then main module uses named resources as global variables and uses `spawn` context, which is the case of Repo on macOS.

This is caused by multiprocess::BaseProcess::_bootstrap.

When a new process is started with multiprocessing.Process.start() in `spawn` context.
1. The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
2. Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.

When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.

And I uploaded a MR.

----------
components: Library (Lib)
messages: 410654
nosy: milestonejxd
priority: normal
severity: normal
status: open
title: Library multiprocess leaks named resources.
type: resource usage
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue46391>
_______________________________________