[Python-Dev] tracemalloc: add an optional memory limit

[Victor Stinner]

Hi,

2013/12/9 Serhiy Storchaka <storchaka at gmail.com>:
> But tracemalloc doesn't count memory allocated besides Python allocators
> (e.g. memory for executable, static variables and stack, memory allocated by
> extensions and C lib, unallocated but not returned to OS dynamical memory).
> When you want investigate how your program works on low memory, e.g. 500 MB,
> ulimit -v 500000 is much closer to real life than tracemalloc.

Well, both limits are useful, but also (completly?) different.

"memory for executable, static variables and stack," : these values
are fixed, so they don't matter at all. Substract them from the RSS
when you choose the memory limit for tracemalloc.

"memory allocated by extensions" : I started to modify some Python
extensions (ex: zlib, bzip2, lzma) to reuse Python memory allocation,
so the memory is also traced by tracemalloc. Memory allocated by other
extensions is not traced.

"and C lib," : correct, this memory is not traced by tracemalloc and
so not counted in the memory limit.

"unallocated but not returned to OS dynamical memory" : it's really
hard to estimate and understand the fragmentation of the heap memory
:-( But this may only concern short peak, and the free memory can be
reused later. If the peak is still lower than the limit, the
fragmentation does not matter

> Limiting memory in tracemalloc can be used only for testing obscure corner
> cases in Python interpreter itself.

In my previous job in embedded device, we hard an *hard* limit for
memory. Python had its own dedicated memory: an hardcoded 30 MB chunk
of memory, reserved for Python. Only memory directly allocated by
Python was stored in this 30 MB chunk. Other allocations ("memory
allocated by extensions, C lib" in your list) was allocated in the
system memory. And I was only concerned of memory directly allocated
by Python.

So being able to run the application on a PC and set a "Python" memory
limit of 30 MB makes sense in such use case.

> It will be very rarely used since all
> bugs will be fixed first time (and thank you for your current work). So
> there is more sense implemented it as auxiliary utility than as feature of
> the stdlib module.

The memory limit is used to test if the application doesn't leak
memory nor allocate more than the limit. Said differently, it's a
convinient tool to simulate an embeded device :-)

Victor