[New-bugs-announce] [issue6653] Potential memory leak in multiprocessing

[Jesse Noller]

New submission from Jesse Noller <jnoller at gmail.com>:

I have example code to show this. It creates a system-wide memory leak 
on Linux/Unix (present until the next reboot), unless the last statement 
in the target of mp.Process ensures a manual clean up of the globals.

The problem is line 353 in multiprocessing/forking.py. The function 
exit() is defined as os._exit on Linux and ExitProcess on Windows, none 
of which allows normal clean up.

>>> help(os._exit)
Help on built-in function _exit in module nt:

_exit(...)
   _exit(status)

   Exit to the system with specified status, without normal exit 
processing.


The problem is fixed if line 353 in forking.py is changed from

exit(exitcode)

to

sys.exit(exitcode)




Test run without bugfix:

G:\DEVELO~1\SHARED~2>python test.py
open handle to 569f439b24e24fc8a547b81932616066
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]
open handle to 0582d4b161c546f582c1c96e7bd0c39d
open handle to 569f439b24e24fc8a547b81932616066
modified array
closed handle to 569f439b24e24fc8a547b81932616066
[[ 1.  1.  1.  0.]
 [ 1.  1.  1.  0.]]
closed handle to 569f439b24e24fc8a547b81932616066


You can see here that opening and closing of handles are unmatched. This 
is on Windows, where the kernel ensures the clean-up, so it may not 
matter. But on Unix this would have created a permament (system wide) 
memory leak! What is happening here is globals not being cleaned up due 
to the use of os._exit instead of sys.exit.


Test run with bugfix:

G:\DEVELO~1\SHARED~2>python test.py
open handle to 930778d27b414253bc329f2b70adaa05
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]
open handle to 3f6cebf8c5de413685bb770d02ae9666
open handle to 930778d27b414253bc329f2b70adaa05
modified array
closed handle to 930778d27b414253bc329f2b70adaa05
closed handle to 3f6cebf8c5de413685bb770d02ae9666
[[ 1.  1.  1.  0.]
 [ 1.  1.  1.  0.]]
closed handle to 930778d27b414253bc329f2b70adaa05



Now all allocations and deallocations are matched.


Regards,
Sturla Molden

----------
files: test.zip
messages: 91332
nosy: jnoller
severity: normal
status: open
title: Potential memory leak in multiprocessing
Added file: http://bugs.python.org/file14660/test.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6653>
_______________________________________