[pypy-dev] Memory leaking with iterator.close() in PyPy3

[Carl Friedrich Bolz-Tereick]

Hi hubo,

Wow, that was a very good analysis, thanks a lot for finding that out! Armin is working on fixing the bug. 

Cheers, 
Carl Friedrich

On June 28, 2018 7:51:58 AM GMT+02:00, hubo <hubo at jiedaibao.com> wrote:
>This problem is caused by caching GeneratorExit exception:
>
>https://bitbucket.org/pypy/pypy/src/8b43b50fbf61f46701d398bf514cf540201ffd03/pypy/interpreter/generator.py?at=py3.5&fileviewer=file-view-default#generator.py-454:457
>
>In Python 3, an exception saves its traceback in __traceback__
>attribute. When an exception object with __traceback__ is raised again,
>current frame is appended to the __traceback__. When the GeneratorExit
>is raised inside a iterator, the __traceback__ attribute of this object
>saves the frame, which prevents the iterator from been collected by GC.
>
>Since __traceback__ attribute is mutable, it is generally a dangerous
>idea to reuse an exception object in Python 3, the problem includes:
>creating memory leaks with frames, wrong traceback, etc. I think remove
>the caching will solve the problem.
>
>It's quite easy to be validated with the following script:
>
>def test():
>    try:
>        yield 1
>    except BaseException as e:
>        global ex
>        ex = e
>
>for _ in range(10):
>    t = test()
>    next(t)
>    t.close()
>
>import traceback
>traceback.print_tb(ex.__traceback__)
>
>You will see many frames in the traceback.
>
>Removing the __traceback__ attribute before each usage is not
>acceptable, because the exception may be used concurrently in different
>threads, or nested in a finalizer when GC is triggered inside the
>generator close.
>
>2018-06-28 
>
>hubo 
>
>
>
>发件人："hubo" <hubo at jiedaibao.com>
>发送时间：2018-06-28 13:06
>主题：[pypy-dev] Memory leaking with iterator.close() in PyPy3
>收件人："PyPy Developer Mailing List"<pypy-dev at python.org>
>抄送：
>
>In PyPy3, when an iterator is closed with "close()" method, the
>iterator leaks and cannot be collected.
>
>Execute the following script in PyPy3, the memory usage is increasing
>very fast, and gc.collect() cannot collect the memory
>
>
>def test():
>    yield 1
>
>while True:
>    t = test()
>    t.close()
>
>
>The tested version:
>
>Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04)
>[PyPy 6.0.0 with GCC 6.2.0 20160901]
>
>This is not reproduced in CPython 3.5 and PyPy2.
>
>2018-06-28
>
>
>hubo 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180628/a0af4cc2/attachment-0001.html>