Preventing out of memory conditions
Sometimes, I have the flexibility to reduce the memory used by my program (e.g., by destroying large cached objects, etc.). It would be great if I could ask Python interpreter to notify me when memory is running out, so I can take such actions. Of course, it's nearly impossible for Python to know in advance if the OS would run out of memory with the next malloc call. Furthermore, Python shouldn't guess which memory (physical, virtual, etc.) is relevant in the particular situation (for instance, in my case, I only care about physical memory, since swapping to disk makes my application as good as frozen). So the problem as stated above is unsolvable. But let's say I am willing to do some work to estimate the maximum amount of memory my application can be allowed to use. If I provide that number to Python interpreter, it may be possible for it to notify me when the next memory allocation would exceed this limit by calling a function I provide it (hopefully passing as arguments the amount of memory being requested, as well as the amount currently in use). My callback function could then destroy some objects, and return True to indicate that some objects were destroyed. At that point, the intepreter could run its standard garbage collection routines to release the memory that corresponded to those objects - before proceeding with whatever it was trying to do originally. (If I returned False, or if I didn't provide a callback function at all, the interpreter would simply behave as it does today.) Any memory allocations that happen while the callback function itself is executing, would not trigger further calls to it. The whole mechanism would be disabled for the rest of the session if the memory freed by the callback function was insufficient to prevent going over the memory limit. Would this be worth considering for a future language extension? How hard would it be to implement? Max
participants (7)
-
Ben Hoyt
-
Gregory P. Smith
-
Jim Jewett
-
Max Moroz
-
Random832
-
Serhiy Storchaka
-
Stephen J. Turnbull