2013/1/21 Oleg Broytman :

I'd very much like to see a situation when a program can survive MemoryError.

Let's say your using an image processing program. You have several images open on which you've been working for a couple minutes/hours. You open a new one, and it's so large that it results in MemoryError : instead of just losing all your current work (yeah, the program should support auto-save anyway, but let's pretend it doesn't), the program catches MemoryError, and displays a popup saying "No enough memory to process this image". Now, sure, there are cases where an OOM condition will result in thrashing to death, or simply because of overcommit malloc() will never return NULL and you'll get nuked by the OOM killer, but depending on your operating system and allocation pattern, there are times when you can reasonably recover from a MemoryError. Also, a memory allocation failure doesn't necessarily mean you're OOM, it could be that youve exhausted your address space (on 32-bit), or hit RLIMIT_VM/RLIMIT_DATA. 2013/1/21 Benjamin Peterson :

What is this useful for?

Even if the exception isn't caught, if the extra information gets dumped in the traceback, it can be used for post-mortem debugging (to help distinguish between OOM, address space exhaustion, heap fragmentation, overflow in computation of malloc() argument, etc). So I think it could probably be useful, but I see two problems: - right now, the amount of memory isn't tracked. IIRC, Antoine added recently a counter for allocated blocks, not bytes - the exception is raised at the calling site where the allocation routine failed (this comes from Modules/_pickle.c): """ PyMemoTable *memo = PyMem_MALLOC(sizeof(PyMemoTable)); if (memo == NULL) { PyErr_NoMemory(); return NULL; } """ So we can't easily capture the current allocated memory and the requested memory (the former could probably be retrieved in PyErr_NoMemory(), but the later would require modifying every call site and repeating it).

Le Tue, 22 Jan 2013 18:53:49 +1000, Nick Coghlan a écrit :

There's a bigger reason memory error must be stateless: we preallocate and reuse it.

Not anymore, it's a freelist now: http://hg.python.org/cpython/file/e8f40d4f497c/Objects/exceptions.c#l2123 The "stateless" part was bogus in Python 3, because of the embedded traceback and context. Regards Antoine.

Le Tue, 22 Jan 2013 20:42:38 +1000, Nick Coghlan a écrit :

In that case, +0 for at least reporting how much memory was being requested for the call that failed, even if that only turns out to be useful in our own test suite. -0 for the "currently allocated" suggestion though, as I don't see how we can provide a meaningful value for that (too much memory usage can be outside of the controller of the Python memory allocator, and we don't even track our own usage all that closely in non-debug builds).

Windows makes it easy to retrive the current process' memory statistics: http://hg.python.org/benchmarks/file/43f8a0f5edd3/perf.py#l240 As usual, though, POSIX platforms are stupidly painful to work with: http://hg.python.org/benchmarks/file/43f8a0f5edd3/perf.py#l202 Regards Antoine.

On 22.01.13 00:12, Benjamin Peterson wrote:

Serhiy Storchaka writes:

...

I propose to add new optional attributes to MemoryError, which show how many memory was required in failed allocation and how many memory was used at this moment.

What is this useful for?

Bigmem testing.

On Tue, Jan 22, 2013 at 11:42 PM, Steven D'Aprano wrote:

Something like this could be used to decide whether or not to flush unimportant in-memory caches, compact data structures, etc., or just give up and exit.

That's a nice idea, but unless the requested allocation was fairly large, there's a good chance you don't have room to allocate anything more. That may make it a bit tricky to do a compaction operation. But if there's some sort of "automatically freeable memory" (simple example: exception-triggered stack unwinding results in a whole bunch of locals disappearing), and you can stay within that, then you might be able to recover. Would require some tightrope-walking in the exception handler, but ought to be possible. ChrisA

On Wed, Jan 23, 2013 at 5:27 AM, Antoine Pitrou wrote:

On Wed, 23 Jan 2013 00:04:15 +1100 Chris Angelico wrote:

...

On Tue, Jan 22, 2013 at 11:42 PM, Steven D'Aprano wrote:

...

Something like this could be used to decide whether or not to flush unimportant in-memory caches, compact data structures, etc., or just give up and exit.

That's a nice idea, but unless the requested allocation was fairly large, there's a good chance you don't have room to allocate anything more.

I wouldn't be surprised if most cases of MemoryErrors were on fairly large allocation requests ;-)

Depends on the workflow. Something that allocates an immediate block of memory, yes, but if you're progressively building a complex structure, individual allocations mightn't themselves be significant. ChrisA