Memory leaking with iterator.close() in PyPy3
In PyPy3, when an iterator is closed with "close()" method, the iterator leaks and cannot be collected. Execute the following script in PyPy3, the memory usage is increasing very fast, and gc.collect() cannot collect the memory def test(): yield 1 while True: t = test() t.close() The tested version: Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04) [PyPy 6.0.0 with GCC 6.2.0 20160901] This is not reproduced in CPython 3.5 and PyPy2. 2018-06-28 hubo
This problem is caused by caching GeneratorExit exception: https://bitbucket.org/pypy/pypy/src/8b43b50fbf61f46701d398bf514cf540201ffd03/pypy/interpreter/generator.py?at=py3.5&fileviewer=file-view-default#generator.py-454:457 In Python 3, an exception saves its traceback in __traceback__ attribute. When an exception object with __traceback__ is raised again, current frame is appended to the __traceback__. When the GeneratorExit is raised inside a iterator, the __traceback__ attribute of this object saves the frame, which prevents the iterator from been collected by GC. Since __traceback__ attribute is mutable, it is generally a dangerous idea to reuse an exception object in Python 3, the problem includes: creating memory leaks with frames, wrong traceback, etc. I think remove the caching will solve the problem. It's quite easy to be validated with the following script: def test(): try: yield 1 except BaseException as e: global ex ex = e for _ in range(10): t = test() next(t) t.close() import traceback traceback.print_tb(ex.__traceback__) You will see many frames in the traceback. Removing the __traceback__ attribute before each usage is not acceptable, because the exception may be used concurrently in different threads, or nested in a finalizer when GC is triggered inside the generator close. 2018-06-28 hubo 发件人："hubo" <hubo@jiedaibao.com> 发送时间：2018-06-28 13:06 主题：[pypy-dev] Memory leaking with iterator.close() in PyPy3 收件人："PyPy Developer Mailing List"<pypy-dev@python.org> 抄送： In PyPy3, when an iterator is closed with "close()" method, the iterator leaks and cannot be collected. Execute the following script in PyPy3, the memory usage is increasing very fast, and gc.collect() cannot collect the memory def test(): yield 1 while True: t = test() t.close() The tested version: Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04) [PyPy 6.0.0 with GCC 6.2.0 20160901] This is not reproduced in CPython 3.5 and PyPy2. 2018-06-28 hubo
Hi hubo, Wow, that was a very good analysis, thanks a lot for finding that out! Armin is working on fixing the bug. Cheers, Carl Friedrich On June 28, 2018 7:51:58 AM GMT+02:00, hubo <hubo@jiedaibao.com> wrote:
This problem is caused by caching GeneratorExit exception:
In Python 3, an exception saves its traceback in __traceback__ attribute. When an exception object with __traceback__ is raised again, current frame is appended to the __traceback__. When the GeneratorExit is raised inside a iterator, the __traceback__ attribute of this object saves the frame, which prevents the iterator from been collected by GC.
Since __traceback__ attribute is mutable, it is generally a dangerous idea to reuse an exception object in Python 3, the problem includes: creating memory leaks with frames, wrong traceback, etc. I think remove the caching will solve the problem.
It's quite easy to be validated with the following script:
def test(): try: yield 1 except BaseException as e: global ex ex = e
for _ in range(10): t = test() next(t) t.close()
import traceback traceback.print_tb(ex.__traceback__)
You will see many frames in the traceback.
Removing the __traceback__ attribute before each usage is not acceptable, because the exception may be used concurrently in different threads, or nested in a finalizer when GC is triggered inside the generator close.
2018-06-28
hubo
发件人："hubo" <hubo@jiedaibao.com> 发送时间：2018-06-28 13:06 主题：[pypy-dev] Memory leaking with iterator.close() in PyPy3 收件人："PyPy Developer Mailing List"<pypy-dev@python.org> 抄送：
In PyPy3, when an iterator is closed with "close()" method, the iterator leaks and cannot be collected.
Execute the following script in PyPy3, the memory usage is increasing very fast, and gc.collect() cannot collect the memory
def test(): yield 1
while True: t = test() t.close()
The tested version:
Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04) [PyPy 6.0.0 with GCC 6.2.0 20160901]
This is not reproduced in CPython 3.5 and PyPy2.
2018-06-28
hubo
Hi Hubo, The bug is fixed now in 0017e9688738. It was apparently the only place in the source that made a prebuilt W_BaseException instance. Thanks a lot! Armin
Thanks a lot. How long will the next release be ready? In the mean while I can try the nightly build. 2018-06-28 hubo 发件人：Armin Rigo <armin.rigo@gmail.com> 发送时间：2018-06-28 16:55 主题：Re: [pypy-dev] Memory leaking with iterator.close() in PyPy3 收件人："Carl Friedrich Bolz-Tereick"<cfbolz@gmx.de> 抄送："hubo"<hubo@jiedaibao.com>,"PyPy Developer Mailing List"<pypy-dev@python.org> Hi Hubo, The bug is fixed now in 0017e9688738. It was apparently the only place in the source that made a prebuilt W_BaseException instance. Thanks a lot! Armin
participants (3)
-
Armin Rigo -
Carl Friedrich Bolz-Tereick -
hubo