[New-bugs-announce] [issue45830] Custom pickler memory leak

[Douglas Raillard]

New submission from Douglas Raillard <douglas.raillard at arm.com>:

The following script exhibits the memory leak. It only happens if "dispatch_table" is set _before_ calling super().__init__, which is pretty unexpected.


    import pickle
    import io
    import gc
    import tracemalloc

    tracemalloc.start(10)
    snap = tracemalloc.take_snapshot()

    class MyPickler(pickle.Pickler):
        def __init__(self, *args, **kwargs):
            # Swapping the next 2 lines "solves" the memory leak for some reason
            self.dispatch_table = dict()
            super().__init__(*args, **kwargs)

    l=[]
    for i in range(10000):
        if i % 1000 == 0:
            print('='*80)
            snap2 = tracemalloc.take_snapshot()
            stats=snap2.compare_to(snap, 'lineno')
            for s in stats[:10]:
                print(s)
            snap = snap2

        f = io.BytesIO()
        MyPickler(f)
        gc.collect()


The output of the last iteration is as follow. The leak of 562 kiB is apparent:

	testmem.py:12: size=562 KiB (+62.5 KiB), count=9000 (+1000), average=64 B
	/usr/lib/python3.10/tracemalloc.py:125: size=2376 B (-72 B), count=33 (-1), average=72 B
	/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1 (+1), average=72 B
	/usr/lib/python3.10/tracemalloc.py:502: size=252 B (+28 B), count=9 (+1), average=28 B
	/usr/lib/python3.10/tracemalloc.py:498: size=2104 B (+0 B), count=36 (+0), average=58 B
	/home/dourai01/Work/lisa/lisa/testmem.py:10: size=1844 B (+0 B), count=9 (+0), average=205 B
	/usr/lib/python3.10/tracemalloc.py:193: size=1680 B (+0 B), count=35 (+0), average=48 B
	/usr/lib/python3.10/tracemalloc.py:547: size=1256 B (+0 B), count=3 (+0), average=419 B
	/usr/lib/python3.10/tracemalloc.py:226: size=832 B (+0 B), count=2 (+0), average=416 B
	/usr/lib/python3.10/tracemalloc.py:173: size=800 B (+0 B), count=2 (+0), average=400 B

If "dispatch_table" is set after calling super().__init__, there is no leak anymore:

	/usr/lib/python3.10/tracemalloc.py:135: size=740 B (+740 B), count=7 (+7), average=106 B
	/usr/lib/python3.10/tracemalloc.py:125: size=2088 B (-656 B), count=29 (-4), average=72 B
	/usr/lib/python3.10/tracemalloc.py:136: size=320 B (+320 B), count=1 (+1), average=320 B
	/usr/lib/python3.10/tracemalloc.py:132: size=0 B (-256 B), count=0 (-1)
	/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1 (+1), average=72 B
	/usr/lib/python3.10/tracemalloc.py:498: size=2008 B (+48 B), count=34 (+1), average=59 B
	/usr/lib/python3.10/tracemalloc.py:193: size=1584 B (+48 B), count=33 (+1), average=48 B
	/usr/lib/python3.10/tracemalloc.py:502: size=196 B (-28 B), count=7 (-1), average=28 B
	/usr/lib/python3.10/tracemalloc.py:126: size=84 B (+28 B), count=3 (+1), average=28 B

----------
components: Library (Lib)
messages: 406478
nosy: douglas-raillard-arm
priority: normal
severity: normal
status: open
title: Custom pickler memory leak
type: resource usage
versions: Python 3.10

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45830>
_______________________________________