I'm still getting MemoryError exceptions in RPython, as I mentioned a few months back. I only knew of one way to reproduce it back then, but now I'm seeing different ways. So, inspired, I wrote a short program to quickly exhaust memory in the hopes of reproducing the problem. It seems to work! Fast! If you run the code pasted at the bottom with CPython, it should end printing 'test passed'. If you run the code with pypy 1.9 (I tried Windows), you should get: RPython traceback: File "translator_goal_targetpypystandalone.c", line 1048, in entry_point File "interpreter_function.c", line 929, in funccall__star_1 File "interpreter_function.c", line 958, in funccall__star_1 File "rpython_memory_gc_minimark.c", line 2518, in MiniMarkGC_collect_and_rese rve File "rpython_memory_gc_minimark.c", line 2220, in MiniMarkGC_minor_collection File "rpython_memory_gc_minimark.c", line 4258, in MiniMarkGC_collect_oldrefs_ to_nursery File "rpython_memory_gc_base.c", line 1799, in trace___trace_drag_out File "rpython_memory_gc_minimark.c", line 6667, in MiniMarkGC__malloc_out_of_n ursery_nonsmall Fatal RPython error: MemoryError Can someone else at least confirm this? Thanks, -Roger Flores exhaust-mem.py: """ Code to exhaust memory and generate a MemoryError exception """ size = 1 # find the largest memory allocation that succeeds try: while True: new_size = size * 2 d = [0] * size size = new_size except MemoryError as msg: pass l = [] # exhaust memory while True: try: # add the allocations to a list so references to them remain. # otherwise the garbage collector will clean them out! l.append([0] * size) print '{:,}'.format(size) except MemoryError as msg: if size > 1: # can't allocate this much memory anymore. Try a smaller size size = size / 2 else: print 'test passed' break
Hi Roger, On Tue, Aug 7, 2012 at 7:41 AM, Roger Flores <aidembb@yahoo.com> wrote:
Can someone else at least confirm this?
Yes, I can reproduce it easily running in a 32-bit chroot on a machine with more than 4GB of RAM. The problem is the same, and is still not solved: we run out of memory when doing a minor cycle of garbage collection because we don't have enough RAM to even copy the surviving objects outside the nursery. We can try to fix this particular problem, which requires looking in details through the GC for what occurs if any malloc fails during a collection. A bientôt, Armin.
Re-hi, I tried to debug it more precisely, and it seems that the problem is even more basic than I thought. It is very hard to solve in general. The issue is that when we are really out of memory, then *every* single allocation is going to fail. The difference with CPython is that in the same situation, the latter can still (randomly) satisfy a small number of allocations of different sizes. This is an internal detail of how the memory allocators work. What occurs then in PyPy is that when we are out of memory, we can really not allocate any single object at more. So we cannot even execute anything from the "except MemoryError" block, because when catching the exception, we try to allocate a small internal object --- which re-raises MemoryError. I tried to look if we would go anywhere by making sure that no small internal object is allocated, but I doubt it, because then it would crash in the first line (e.g. "size = size / 2", which of course allocates a new "long" object). In the end you get this "fatal RPython error: MemoryError" because even printing the traceback requires some memory (and thus re-raises MemoryError instead). Generally, we have no hope to pass cleanly the test you gave. Even in CPython it works by chance; I can tweak it in "reasonable" ways and have it fail too (e.g. if you need to do a bit more than "size = size / 2" here, then you're likely to get a MemoryError in the except handler too). So, I don't think this is ever going to be fixed (or fixable)... A bientôt, Armin.
What occurs then in PyPy is that when we are out of memory, we can really not allocate any single object at more.
outside of programming there is the concept of having a secret backup ("nest egg"). Would'nt it be a solution to preallocate some bytes on startup for really, really bad times? As "really, really bad times" this kind of memory error would qualify. So there should be made a special case "if you are out of memory, to handle this out of memory error, use the preallocated bytes from the secret backup you made on startup". Harald -- GHUM GmbH Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971
Hi Harald, On Tue, Aug 7, 2012 at 10:23 AM, Massa, Harald Armin <chef@ghum.de> wrote:
outside of programming there is the concept of having a secret backup ("nest egg"). Would'nt it be a solution to preallocate some bytes on startup for really, really bad times?
It might help in general, but not in this case: this example causes and catches multiple MemoryErrors. If you unveil your secret backup at the first MemoryError, you gained nothing, because the crash occurs only at the ~25th MemoryError. A bientôt, Armin.
Hi Armin, Harald, On 08/07/2012 11:28 AM, Armin Rigo wrote:
Hi Harald,
On Tue, Aug 7, 2012 at 10:23 AM, Massa, Harald Armin<chef@ghum.de> wrote:
outside of programming there is the concept of having a secret backup ("nest egg"). Would'nt it be a solution to preallocate some bytes on startup for really, really bad times?
It might help in general, but not in this case: this example causes and catches multiple MemoryErrors. If you unveil your secret backup at the first MemoryError, you gained nothing, because the crash occurs only at the ~25th MemoryError.
I'm under the Impression that that Kind of Error could be evaded by reserving a special part of the memory pool for strings and internal objects *after* the general Pool is used up. My assumption is that most of the time the Memory errors are caused semi-large allocations/reallocation (like allocate or re-size a list/array/matrix) in which case there will still be some memory left to at least print a stack-trace at interpreter level and quit. However in some cases (like the example) the memory error is caused by accumulating more and more smaller objects, in which case the memory one would use for a Stack-trace is used up and the extra pool would be used. In that case i presume any allocation not needed for a trace would cause just another memory error in the exception handler. That should be able to handle most of the normal cases of people trying to handle something and print out some extra information before failing Its of course not safe from malicious code or people just deliberately refusing to fail if there is a memory-error. but really "evil" code is fine to receive a rpython traceback. -- Ronny
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Hi Ronny, On Tue, Aug 7, 2012 at 1:56 PM, Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de> wrote:
However in some cases (like the example) the memory error is caused by accumulating more and more smaller objects, in which case the memory one would use for a Stack-trace is used up and the extra pool would be used.
I'm quite unsure how you propose to do that. We could have a reserve pool that we activate when trying to print the fatal MemoryError, but that doesn't really help: the normal stack trace cannot be even *built* in memory, as it requires allocating PyTraceback objects. That's the reason why the RPython traceback reported at the start of this thread is so short: because all other levels in the stack caught the RPython MemoryError, but failed very quickly when trying to handle it, throwing *another* RPython MemoryError to the caller. If you can come up with a more precise scheme, you're welcome. The issue is to know when it's ok to reserve from that pool and when we should raise an RPython MemoryError instead. A possible answer would look like "when allocating internal vs user objects", for some unspecified-yet distinction of internal vs user objects. A bientôt, Armin.
On Tue, Aug 7, 2012 at 6:33 AM, Armin Rigo <arigo@tunes.org> wrote:
If you can come up with a more precise scheme, you're welcome. The issue is to know when it's ok to reserve from that pool and when we should raise an RPython MemoryError instead. A possible answer would look like "when allocating internal vs user objects", for some unspecified-yet distinction of internal vs user objects.
Would having all allocations inside an except MemoryError: block be eligible to use the reserved pool be workable? Or perhaps "inside an except: block, while a MemoryError is the current exception"? It might even be ok to just have it be "while inside an except: block". To my naive eyes, it seems like something simple like one of the above would handle the normal use cases, and allow for some real-world use to provide information about if the criteria need tightening, etc. Cheers, Eli
On Tue, Aug 7, 2012 at 6:33 AM, Armin Rigo <arigo@tunes.org> wrote:
If you can come up with a more precise scheme, you're welcome. The issue is to know when it's ok to reserve from that pool and when we should raise an RPython MemoryError instead. A possible answer would
I'm not convinced this MemoryError is unsolvable yet. Perhaps it can be broken into a couple of steps. When the app requests memory that would leave too little remaining, the memory request needs to be refused and a MemoryError sent instead. This informs the app that it can no longer make memory requests like that and expect them to succeed. The app then has a few options. Leave the exception unhandled and get terminated. Reduce it's memory usage by pruning it's structures. Stop an operation and report the problem, i.e. "Document too large too load". The key point is that the RPython memory system signals the app's memory system so it can respond as needed. If the app tries to handle the MemoryError it will need more memory requests to succeed, for temporaries like Armin points out, to get it's job done. Either the free memory available will be restored by the app to an amount sufficient to avoid further MemoryErrors, or the app basically had it's chance and now RPython needs the remaining memory to successfully describe the apps early demise (print MemoryError and a stack trace showing what the app was doing). So we don't need special pools of memory, new APIs or anything special. We do need to have two out of memory thresholds. The first is enough for RPython to terminate the app and describe the problem. The second, higher threshold, is used to notify the app that it can no longer request memory like that unless it reduces it's usage. Only one MemoryError is sent to the app until either the app frees enough memory or gets terminated. Imaginary code could look something like this: def alloc(size): if free_mem - size < app_reserve + rpython_reserve: if not raised_MemoryError: raised_MemoryError = True raise MemoryError() if free_mem - size < rpython_reserve: print 'MemoryError' print stack_trace() # of app else: raised_MemoryError = False app_reserve and rpython_reserve might be 1K, 10K, 100K, idk. Apps might want a variable they can tweak to further pad the reserve because they know they need extra room to prune their memory systems. My main question is can the RPython memory system do a collection while using only rpython_reserve of memory? Note the MemoryError when RPython terminates the app isn't really a MemoryError. It's not an exception. It can't be caught (it's too late for that). A different name really should be used to avoid confusion. Basically, this design sends one MemoryError to my test code to give it a chance to do something about it. And it will let it keep going for a while. Soon though, it will give up on the test code, terminate it, and report why. No RPython traceback is needed. To me, the key issue is that apps get at least one MemoryError so they can do something about it. The current state, that they may or may not get to do *anything* is kind of scary. -Roger
Hi Eli, hi Roger, Both of your solutions might work. Roger, you overlook a problem in this pseudo-code:
print 'MemoryError' print stack_trace() # of app
We can't suddenly print the stack trace of the application from inside the GC. In order to have a regular stack trace printed, an exception must be raised, which is caught and re-raised by the various levels, collecting stack information in (new) PyTraceback objects; then, in PyPy, we need to import (at app-level) the "traceback" module and call a function from there. This takes a lot of extra memory. Well, overall we need first someone to care about sanitizing the GC first. When the GC raises an RPython MemoryError in the middle of a minor collection, it probably leaves things in a broken state. It didn't show up so far because you can't do anything anyway but get the crash. A bientôt, Armin.
Well, overall we need first someone to care about sanitizing the GC first. When the GC raises an RPython MemoryError in the middle of a minor collection, it probably leaves things in a broken state. It
Again, perhaps this could be avoided for now. Could the rpython_reserve be set high enough so that minor collections always succeed? 10KB? 100KB? Basically get it working as desired first, then go back and optimize memory sizes and collection cycles when someone can. Fun stat: pypy for my PPM app can store twice as much data as python in the 2GB of RAM! So using a silly huge number for the reserve memories still puts pypy well ahead in the memory issues. That is, if python doesn't need to be more careful as well.
We can't suddenly print the stack trace of the application from inside ...
a function from there. This takes a lot of extra memory.
OK. It sounds like it already works, just needs a large rpython_reserve. Again, pick a large number, show it works, optimize later? -Roger ________________________________ From: Armin Rigo <arigo@tunes.org> To: Roger Flores <aidembb@yahoo.com> Cc: "pypy-dev@python.org" <pypy-dev@python.org> Sent: Wednesday, August 8, 2012 1:09 AM Subject: Re: [pypy-dev] Fatal RPython error: MemoryError Hi Eli, hi Roger, Both of your solutions might work. Roger, you overlook a problem in this pseudo-code:
print 'MemoryError' print stack_trace() # of app
We can't suddenly print the stack trace of the application from inside the GC. In order to have a regular stack trace printed, an exception must be raised, which is caught and re-raised by the various levels, collecting stack information in (new) PyTraceback objects; then, in PyPy, we need to import (at app-level) the "traceback" module and call a function from there. This takes a lot of extra memory. Well, overall we need first someone to care about sanitizing the GC first. When the GC raises an RPython MemoryError in the middle of a minor collection, it probably leaves things in a broken state. It didn't show up so far because you can't do anything anyway but get the crash. A bientôt, Armin.
Tuesday 07 August 2012 you wrote:
Re-hi,
I tried to debug it more precisely, and it seems that the problem is even more basic than I thought. It is very hard to solve in general. The issue is that when we are really out of memory, then *every* single allocation is going to fail. The difference with CPython is that in the same situation, the latter can still (randomly) satisfy a small number of allocations of different sizes. This is an internal detail of how the memory allocators work.
What occurs then in PyPy is that when we are out of memory, we can really not allocate any single object at more. So we cannot even execute anything from the "except MemoryError" block, because when catching the exception, we try to allocate a small internal object --- which re-raises MemoryError. I tried to look if we would go anywhere by making sure that no small internal object is allocated, but I doubt it, because then it would crash in the first line (e.g. "size = size / 2", which of course allocates a new "long" object). In the end you get this "fatal RPython error: MemoryError" because even printing the traceback requires some memory (and thus re-raises MemoryError instead).
Generally, we have no hope to pass cleanly the test you gave. Even in CPython it works by chance; I can tweak it in "reasonable" ways and have it fail too (e.g. if you need to do a bit more than "size = size / 2" here, then you're likely to get a MemoryError in the except handler too).
So, I don't think this is ever going to be fixed (or fixable)...
Can't you have a chunk of reserve memory allocated from the start of the GC, which you free if a memory allocation fails? Jacob
participants (6)
-
Armin Rigo -
Eli Stevens (Gmail) -
Jacob Hallén -
Massa, Harald Armin -
Roger Flores -
Ronny Pfannschmidt