[issue27535] Memory leaks when opening tons of files

[Александр Карпинский]

New submission from Александр Карпинский:

Actually, this issue is related to the warning module. The test script creates a lot of files with different names and deletes them. On the first pass the scripts calls the `f.close()` method for every file. On the second it doesn't.

As a result, on the first pass, the memory consumption is constant and is about 3.9 Mb for my environment. For the second pass, the memory consumption constantly grows up to 246 Mb for 1 million files. I.e. memory leak is about 254 bytes for every opened file.

This happens because of warning about not closed files. The memory is consumed for storing all possible warning messages generated for different file names to prevent further warnings with the same messages. Which is not the case if we are always opening new files.

While not calling f.close() **might** lead to memory issues, the warning which notifies about this **absolutely guaranteed** leads to memory issue. Although f.close() is highly recommended, this is still the issue for code which doesn't use it for some reasons and doesn't experience other problems with not closed files because files are actually always closed in IOBase.__del__.

This issue was discovered in this report: https://github.com/python-pillow/Pillow/issues/2019


The best way to fix this is excluding file name from warning message. As a result, the message for every file will be the same and warnings registry will not grow. For now, the message looks like this:

./memory.py:23: ResourceWarning: unclosed file <_io.FileIO name='__999.tmp' mode='wb' closefd=True>
  open_files(number, close)

It may looks like this:

./memory.py:23: ResourceWarning: unclosed file <FileIO mode='wb' closefd=True>
  open_files(number, close)

----------
components: IO
files: memory.py
messages: 270600
nosy: Александр Карпинский
priority: normal
severity: normal
status: open
title: Memory leaks when opening tons of files
type: resource usage
versions: Python 3.4, Python 3.5
Added file: http://bugs.python.org/file43762/memory.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue27535>
_______________________________________