Program uses twice as much memory in Python 3.6 than in Python 3.5
jan at hyper-world.de
Mon Mar 27 17:57:18 EDT 2017
I have a program which uses twice as much memory when I run it in Python
3.6 than when I run it in Python 3.5 (about 60GB instead of 30GB). I
tried to find the reason for that, but the cumulated size (measured with
sys.getsizeof) of all objects returned by gc.get_objects accumulates
only to about 17GB in both cases. The program also uses NumPy and I
tried tracking allocations with the NumPy allocation tracker in the
relevant part of the program, but again the number of allocations are
almost identical and the reported maximum memory usage perfectly agrees
(it is about 18GB).
Any ideas where this significant increase in memory consumption could
come from? Or any ideas how to further debug this?
Looking at the changelog it seems that the only change in Python 3.6
affecting memory usage is the new dict implementation which is supposed
to be more memory efficient. In fact, this is what I find if I run the
program with a smaller test cases. There the memory consumption is less
with Python 3.6.
More information about the Python-list