This problem is caused by caching GeneratorExit exception:
In Python 3, an exception saves its traceback in __traceback__ attribute.
When an exception object with __traceback__ is raised again, current
frame is appended to the __traceback__. When the GeneratorExit is raised inside
a iterator, the __traceback__ attribute of this object saves the frame, which
prevents the iterator from been collected by GC.
Since __traceback__ attribute is mutable, it is generally a dangerous idea
to reuse an exception object in Python 3, the problem includes: creating memory
leaks with frames, wrong traceback, etc. I think remove the caching will solve
the problem.
It's quite easy to be validated with the following script:
def
test():
try:
yield 1
except BaseException as
e:
global
ex
ex = e
for _ in range(10):
t = test()
next(t)
t.close()
import traceback
traceback.print_tb(ex.__traceback__)
You will see many frames in the traceback.
Removing the __traceback__ attribute before each usage is not acceptable,
because the exception may be used concurrently in different threads,
or nested in a finalizer when GC is triggered inside the generator close.
2018-06-28
hubo
发件人："hubo" <hubo@jiedaibao.com>
发送时间：2018-06-28 13:06
主题：[pypy-dev] Memory leaking with iterator.close() in
PyPy3
收件人："PyPy Developer Mailing
List"<pypy-dev@python.org>
抄送：
In PyPy3, when an iterator is closed with "close()" method, the
iterator leaks and cannot be collected.
Execute the following script in PyPy3, the memory usage is increasing
very fast, and gc.collect() cannot collect the memory
def test():
yield 1
while True:
t = test()
t.close()
The tested version:
Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04)
[PyPy 6.0.0 with
GCC 6.2.0 20160901]
This is not reproduced in CPython 3.5 and PyPy2.