[Python-Dev] Reduce memory footprint of Python

[Victor Stinner]

2013/10/6  <martin at v.loewis.de>:
> Quoting Antoine Pitrou <solipsis at pitrou.net>:
>
>> The linecache module is used implicitly by the traceback and warnings
>> module, so perhaps quite a bit more than one would imagine :-)
>>
>> I think limiting the linecache cache size would be good enough.
>
> So what specific limit would you suggest?

Before using tracemalloc, I didn't even know that lineache has a
cache. And Antoine wrote, linecache is used indirectly by many
modules. Even if you get a single line, the module does all the whole
file content. I was surprised to see that the cache size can be up to
2 MB.

(To answer to your question, yes, our application on embedded device
does format exceptions and tracebacks, so use linecache.)

The re cache is limited. I remember that I saw it once bigger than 700
KB, it was to build a huge range of Unicode characters... but I'm now
unable to reproduce the case.

I'm quite sure that the cache has an impact on performances. For
example, if you format a traceback where most frames come from the
same file, you don't want to decode the file from Unicode for each
line. I don't want to remove the cache, and I'm unable to configure
its size.

I suggested a function to clear "all" caches, because *if* you care of
memory, you would like to flush all caches (even if you don't know
that linecache or re have caches), not anytime, but when your
application is idle. I propose a registry because IMO the module
should register itself to the registry, the registry should not have
to know all modules. (It should not the module to clear its cache ;-))

I worked on a project spending weeks to try to reduce the memory
footprint by 1 MB. And it was hard :-) Biggest objects were in the
application but it would help to be able to free Python internal
caches.

Victor