[Python-ideas] Preventing out of memory conditions

[Max Moroz]

Sometimes, I have the flexibility to reduce the memory used by my
program (e.g., by destroying large cached objects, etc.). It would be
great if I could ask Python interpreter to notify me when memory is
running out, so I can take such actions.

Of course, it's nearly impossible for Python to know in advance if the
OS would run out of memory with the next malloc call. Furthermore,
Python shouldn't guess which memory (physical, virtual, etc.) is
relevant in the particular situation (for instance, in my case, I only
care about physical memory, since swapping to disk makes my
application as good as frozen). So the problem as stated above is
unsolvable.

But let's say I am willing to do some work to estimate the maximum
amount of memory my application can be allowed to use. If I provide
that number to Python interpreter, it may be possible for it to notify
me when the next memory allocation would exceed this limit by calling
a function I provide it (hopefully passing as arguments the amount of
memory being requested, as well as the amount currently in use). My
callback function could then destroy some objects, and return True to
indicate that some objects were destroyed. At that point, the
intepreter could run its standard garbage collection routines to
release the memory that corresponded to those objects - before
proceeding with whatever it was trying to do originally. (If I
returned False, or if I didn't provide a callback function at all, the
interpreter would simply behave as it does today.) Any memory
allocations that happen while the callback function itself is
executing, would not trigger further calls to it. The whole mechanism
would be disabled for the rest of the session if the memory freed by
the callback function was insufficient to prevent going over the
memory limit.

Would this be worth considering for a future language extension? How
hard would it be to implement?

Max