More details in MemoryError
I propose to add new optional attributes to MemoryError, which show how many memory was required in failed allocation and how many memory was used at this moment.
On Mon, Jan 21, 2013 at 09:20:08PM +0200, Serhiy Storchaka <storchaka@gmail.com> wrote:
I'd very much like to see a situation when a program can survive MemoryError. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
2013/1/21 Oleg Broytman <phd@phdru.name>:
I'd very much like to see a situation when a program can survive MemoryError.
Let's say your using an image processing program. You have several images open on which you've been working for a couple minutes/hours. You open a new one, and it's so large that it results in MemoryError : instead of just losing all your current work (yeah, the program should support auto-save anyway, but let's pretend it doesn't), the program catches MemoryError, and displays a popup saying "No enough memory to process this image". Now, sure, there are cases where an OOM condition will result in thrashing to death, or simply because of overcommit malloc() will never return NULL and you'll get nuked by the OOM killer, but depending on your operating system and allocation pattern, there are times when you can reasonably recover from a MemoryError. Also, a memory allocation failure doesn't necessarily mean you're OOM, it could be that youve exhausted your address space (on 32-bit), or hit RLIMIT_VM/RLIMIT_DATA. 2013/1/21 Benjamin Peterson <benjamin@python.org>:
What is this useful for?
Even if the exception isn't caught, if the extra information gets dumped in the traceback, it can be used for post-mortem debugging (to help distinguish between OOM, address space exhaustion, heap fragmentation, overflow in computation of malloc() argument, etc). So I think it could probably be useful, but I see two problems: - right now, the amount of memory isn't tracked. IIRC, Antoine added recently a counter for allocated blocks, not bytes - the exception is raised at the calling site where the allocation routine failed (this comes from Modules/_pickle.c): """ PyMemoTable *memo = PyMem_MALLOC(sizeof(PyMemoTable)); if (memo == NULL) { PyErr_NoMemory(); return NULL; } """ So we can't easily capture the current allocated memory and the requested memory (the former could probably be retrieved in PyErr_NoMemory(), but the later would require modifying every call site and repeating it).
There's a bigger reason memory error must be stateless: we preallocate and reuse it. -- Sent from my phone, thus the relative brevity :)
Le Tue, 22 Jan 2013 18:53:49 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
There's a bigger reason memory error must be stateless: we preallocate and reuse it.
Not anymore, it's a freelist now: http://hg.python.org/cpython/file/e8f40d4f497c/Objects/exceptions.c#l2123 The "stateless" part was bogus in Python 3, because of the embedded traceback and context. Regards Antoine.
On Tue, Jan 22, 2013 at 7:12 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Oh cool, I forgot about that change. In that case, +0 for at least reporting how much memory was being requested for the call that failed, even if that only turns out to be useful in our own test suite. -0 for the "currently allocated" suggestion though, as I don't see how we can provide a meaningful value for that (too much memory usage can be outside of the controller of the Python memory allocator, and we don't even track our own usage all that closely in non-debug builds). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le Tue, 22 Jan 2013 20:42:38 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
Windows makes it easy to retrive the current process' memory statistics: http://hg.python.org/benchmarks/file/43f8a0f5edd3/perf.py#l240 As usual, though, POSIX platforms are stupidly painful to work with: http://hg.python.org/benchmarks/file/43f8a0f5edd3/perf.py#l202 Regards Antoine.
On 22/01/13 09:12, Benjamin Peterson wrote:
After locking up a production machine with a foolishly large list multiplication (I left it thrashing overnight, and 16+ hours later gave up and power-cycled the machine), I have come to appreciate ulimit on Linux systems. That means I often see MemoryErrors while testing. [steve@ando ~]$ ulimit -v 20000 [steve@ando ~]$ python3.3 Python 3.3.0rc3 (default, Sep 27 2012, 18:44:58) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux Type "help", "copyright", "credits" or "license" for more information. === startup script executed === py> x = [0]*1000000 py> x = [0]*123456789012 # oops what was I thinking? Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError For interactive use, it would be really useful in such a situation to see how much memory was requested and how much was available. That would allow me to roughly estimate (say) how big a list I could make in the available memory, instead of tediously trying larger and smaller lists. Something like this could be used to decide whether or not to flush unimportant in-memory caches, compact data structures, etc., or just give up and exit. -- Steven
On Tue, Jan 22, 2013 at 11:42 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That's a nice idea, but unless the requested allocation was fairly large, there's a good chance you don't have room to allocate anything more. That may make it a bit tricky to do a compaction operation. But if there's some sort of "automatically freeable memory" (simple example: exception-triggered stack unwinding results in a whole bunch of locals disappearing), and you can stay within that, then you might be able to recover. Would require some tightrope-walking in the exception handler, but ought to be possible. ChrisA
On Wed, Jan 23, 2013 at 5:27 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Depends on the workflow. Something that allocates an immediate block of memory, yes, but if you're progressively building a complex structure, individual allocations mightn't themselves be significant. ChrisA
On 2013-01-22 13:04, Chris Angelico wrote:
FYI, allocating memory specially for such cases is sometimes called a "memory parachute". I wonder whether you could have a subclass of MemoryError called LowMemoryError. If allocation fails and there's a parachute, it would free the parachute and raise LowMemoryError. That would gave you a chance to tidy up before quitting or even, perhaps, free enough stuff to make a new parachute and continue working. If allocation fails and there's no parachute, it would raise MemoryError as at present. With LowMemoryError as a subclass of MemoryError, existing code would still work the same.
On Mon, Jan 21, 2013 at 09:20:08PM +0200, Serhiy Storchaka <storchaka@gmail.com> wrote:
I'd very much like to see a situation when a program can survive MemoryError. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
2013/1/21 Oleg Broytman <phd@phdru.name>:
I'd very much like to see a situation when a program can survive MemoryError.
Let's say your using an image processing program. You have several images open on which you've been working for a couple minutes/hours. You open a new one, and it's so large that it results in MemoryError : instead of just losing all your current work (yeah, the program should support auto-save anyway, but let's pretend it doesn't), the program catches MemoryError, and displays a popup saying "No enough memory to process this image". Now, sure, there are cases where an OOM condition will result in thrashing to death, or simply because of overcommit malloc() will never return NULL and you'll get nuked by the OOM killer, but depending on your operating system and allocation pattern, there are times when you can reasonably recover from a MemoryError. Also, a memory allocation failure doesn't necessarily mean you're OOM, it could be that youve exhausted your address space (on 32-bit), or hit RLIMIT_VM/RLIMIT_DATA. 2013/1/21 Benjamin Peterson <benjamin@python.org>:
What is this useful for?
Even if the exception isn't caught, if the extra information gets dumped in the traceback, it can be used for post-mortem debugging (to help distinguish between OOM, address space exhaustion, heap fragmentation, overflow in computation of malloc() argument, etc). So I think it could probably be useful, but I see two problems: - right now, the amount of memory isn't tracked. IIRC, Antoine added recently a counter for allocated blocks, not bytes - the exception is raised at the calling site where the allocation routine failed (this comes from Modules/_pickle.c): """ PyMemoTable *memo = PyMem_MALLOC(sizeof(PyMemoTable)); if (memo == NULL) { PyErr_NoMemory(); return NULL; } """ So we can't easily capture the current allocated memory and the requested memory (the former could probably be retrieved in PyErr_NoMemory(), but the later would require modifying every call site and repeating it).
There's a bigger reason memory error must be stateless: we preallocate and reuse it. -- Sent from my phone, thus the relative brevity :)
Le Tue, 22 Jan 2013 18:53:49 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
There's a bigger reason memory error must be stateless: we preallocate and reuse it.
Not anymore, it's a freelist now: http://hg.python.org/cpython/file/e8f40d4f497c/Objects/exceptions.c#l2123 The "stateless" part was bogus in Python 3, because of the embedded traceback and context. Regards Antoine.
On Tue, Jan 22, 2013 at 7:12 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Oh cool, I forgot about that change. In that case, +0 for at least reporting how much memory was being requested for the call that failed, even if that only turns out to be useful in our own test suite. -0 for the "currently allocated" suggestion though, as I don't see how we can provide a meaningful value for that (too much memory usage can be outside of the controller of the Python memory allocator, and we don't even track our own usage all that closely in non-debug builds). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le Tue, 22 Jan 2013 20:42:38 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
Windows makes it easy to retrive the current process' memory statistics: http://hg.python.org/benchmarks/file/43f8a0f5edd3/perf.py#l240 As usual, though, POSIX platforms are stupidly painful to work with: http://hg.python.org/benchmarks/file/43f8a0f5edd3/perf.py#l202 Regards Antoine.
On 22/01/13 09:12, Benjamin Peterson wrote:
After locking up a production machine with a foolishly large list multiplication (I left it thrashing overnight, and 16+ hours later gave up and power-cycled the machine), I have come to appreciate ulimit on Linux systems. That means I often see MemoryErrors while testing. [steve@ando ~]$ ulimit -v 20000 [steve@ando ~]$ python3.3 Python 3.3.0rc3 (default, Sep 27 2012, 18:44:58) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux Type "help", "copyright", "credits" or "license" for more information. === startup script executed === py> x = [0]*1000000 py> x = [0]*123456789012 # oops what was I thinking? Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError For interactive use, it would be really useful in such a situation to see how much memory was requested and how much was available. That would allow me to roughly estimate (say) how big a list I could make in the available memory, instead of tediously trying larger and smaller lists. Something like this could be used to decide whether or not to flush unimportant in-memory caches, compact data structures, etc., or just give up and exit. -- Steven
On Tue, Jan 22, 2013 at 11:42 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That's a nice idea, but unless the requested allocation was fairly large, there's a good chance you don't have room to allocate anything more. That may make it a bit tricky to do a compaction operation. But if there's some sort of "automatically freeable memory" (simple example: exception-triggered stack unwinding results in a whole bunch of locals disappearing), and you can stay within that, then you might be able to recover. Would require some tightrope-walking in the exception handler, but ought to be possible. ChrisA
On Wed, Jan 23, 2013 at 5:27 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Depends on the workflow. Something that allocates an immediate block of memory, yes, but if you're progressively building a complex structure, individual allocations mightn't themselves be significant. ChrisA
On 2013-01-22 13:04, Chris Angelico wrote:
FYI, allocating memory specially for such cases is sometimes called a "memory parachute". I wonder whether you could have a subclass of MemoryError called LowMemoryError. If allocation fails and there's a parachute, it would free the parachute and raise LowMemoryError. That would gave you a chance to tidy up before quitting or even, perhaps, free enough stuff to make a new parachute and continue working. If allocation fails and there's no parachute, it would raise MemoryError as at present. With LowMemoryError as a subclass of MemoryError, existing code would still work the same.
participants (9)
-
Antoine Pitrou -
Benjamin Peterson -
Charles-François Natali -
Chris Angelico -
MRAB -
Nick Coghlan -
Oleg Broytman -
Serhiy Storchaka -
Steven D'Aprano