[pypy-dev] Memory leaking with iterator.close() in PyPy3

[hubo]

This problem is caused by caching GeneratorExit exception:

https://bitbucket.org/pypy/pypy/src/8b43b50fbf61f46701d398bf514cf540201ffd03/pypy/interpreter/generator.py?at=py3.5&fileviewer=file-view-default#generator.py-454:457

In Python 3, an exception saves its traceback in __traceback__ attribute. When an exception object with __traceback__ is raised again, current frame is appended to the __traceback__. When the GeneratorExit is raised inside a iterator, the __traceback__ attribute of this object saves the frame, which prevents the iterator from been collected by GC.

Since __traceback__ attribute is mutable, it is generally a dangerous idea to reuse an exception object in Python 3, the problem includes: creating memory leaks with frames, wrong traceback, etc. I think remove the caching will solve the problem.

It's quite easy to be validated with the following script:

def test():
    try:
        yield 1
    except BaseException as e:
        global ex
        ex = e

for _ in range(10):
    t = test()
    next(t)
    t.close()

import traceback
traceback.print_tb(ex.__traceback__)

You will see many frames in the traceback.

Removing the __traceback__ attribute before each usage is not acceptable, because the exception may be used concurrently in different threads, or nested in a finalizer when GC is triggered inside the generator close.

2018-06-28 

hubo 



发件人："hubo" <hubo at jiedaibao.com>
发送时间：2018-06-28 13:06
主题：[pypy-dev] Memory leaking with iterator.close() in PyPy3
收件人："PyPy Developer Mailing List"<pypy-dev at python.org>
抄送：

In PyPy3, when an iterator is closed with "close()" method, the iterator leaks and cannot be collected.

Execute the following script in PyPy3, the memory usage is increasing very fast, and gc.collect() cannot collect the memory


def test():
    yield 1

while True:
    t = test()
    t.close()


The tested version:

Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04)
[PyPy 6.0.0 with GCC 6.2.0 20160901]

This is not reproduced in CPython 3.5 and PyPy2.

2018-06-28


hubo 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180628/d9695e0c/attachment.html>