[Python-Dev] Reduce memory footprint of Python

[Antoine Pitrou]

Hi,

On Sun, 6 Oct 2013 17:32:37 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> Slowly, I'm trying to see if it would be possible to reduce the memory
> footprint of Python using the tracemalloc module.
> 
> First, I noticed that linecache can allocate more than 2 MB. What do
> you think of adding a registry of "clear cache" functions? For
> exemple, re.purge() and linecache.clearcache(). gc.collect() clears
> free lists. I don't know if gc.collect() should be related to this new
> registy (clear all caches) or not.

Rather than "clear" the cache, perhaps limit its size?
AFAICT, linecache is used in the warnings and traceback modules.

> The dictionary of interned Unicode strings can be large: up to 1.5 MB
> (with +30,000 strings). Just the dictionary, excluding size of
> strings. Is the size normal or not? Using tracemalloc, this dictionary
> is usually to largest memory block.

You don't tell us what the total memory occupation is. A 1.5MB overhead
is not the same on a 10MB total or on a 50MB total.
(also, how many modules were loaded? large third-party libraries?)

The interned strings dictionary could perhaps be replaced with a set?

> unittest doesn't look to release memory (the TestCase class) after the
> execution of a test.
> 
> test_import.test_module_with_large_stack() creates a large Python
> module and import it, but it does not unload it.

That's not really relevant to memory use on production systems, though.

Regards

Antoine.