[Python-Dev] Reduce memory footprint of Python

[Nick Coghlan]

On 7 Oct 2013 07:28, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> 2013/10/6  <martin at v.loewis.de>:
> > Quoting Antoine Pitrou <solipsis at pitrou.net>:
> >
> >> The linecache module is used implicitly by the traceback and warnings
> >> module, so perhaps quite a bit more than one would imagine :-)
> >>
> >> I think limiting the linecache cache size would be good enough.
> >
> > So what specific limit would you suggest?
>
> Before using tracemalloc, I didn't even know that lineache has a
> cache. And Antoine wrote, linecache is used indirectly by many
> modules. Even if you get a single line, the module does all the whole
> file content. I was surprised to see that the cache size can be up to
> 2 MB.
>
> (To answer to your question, yes, our application on embedded device
> does format exceptions and tracebacks, so use linecache.)
>
> The re cache is limited. I remember that I saw it once bigger than 700
> KB, it was to build a huge range of Unicode characters... but I'm now
> unable to reproduce the case.
>
> I'm quite sure that the cache has an impact on performances. For
> example, if you format a traceback where most frames come from the
> same file, you don't want to decode the file from Unicode for each
> line. I don't want to remove the cache, and I'm unable to configure
> its size.
>
> I suggested a function to clear "all" caches, because *if* you care of
> memory, you would like to flush all caches (even if you don't know
> that linecache or re have caches), not anytime, but when your
> application is idle. I propose a registry because IMO the module
> should register itself to the registry, the registry should not have
> to know all modules. (It should not the module to clear its cache ;-))

We kind of have such a manual registry already: the reference leak hunting
in regrtest needs to know about a lot of these caches so it can clear them.

Something like a "gc.purge_caches" operation with a "gc.atpurge"
registration function sounds like a reasonable operation to me (especially
if it offered a way to get a list of which modules had registered cache
purging functions).

Like any import time side effects, we'd need to look carefully at the
impact it has on the lifecycle of the module globals. We'd also need to be
sure that any caches we registered really were pure caches.

Cheers,
Nick.

>
> I worked on a project spending weeks to try to reduce the memory
> footprint by 1 MB. And it was hard :-) Biggest objects were in the
> application but it would help to be able to free Python internal
> caches.
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20131007/7142dab8/attachment.html>