Mark Hammond wrote:
From the newsgroup. Any thoughts?
Mark.
"Phil Mayes" <nospam@bitbucket.com> wrote in message news:<olD35.82$_%.4481@newsfeed.avtel.net>...
The following program progressively allocates all memory: size = 1 << 20 list = [None,] while size: try: mem = [None] * size mem[0] = list list = mem except: size = size / 2
It fails with an access violation.
The only immediate thought I have is that Python does not have builtin emergency procedures for such extreme situations. Although there are some safety checks, it is fairly easy to crash the interpreter in a number of ways. For example, with the introduction of the mmapmodule, there are even more "risks", like managing to mmap the python executable and screw it up completely (not speaking of the possibility to introduce Trojan horses and similar if this is done on purpose). There would probably be more checks, but complete safety is hard to achieve in the current state. The proposed solution is not a solution. The example program dumps core in different locations on my machine. We just do not have enough control from Python for managing such extreme cases. Sad, but true... -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
Mark Hammond wrote:
From the newsgroup. Any thoughts?
Mark.
"Phil Mayes" <nospam@bitbucket.com> wrote in message news:<olD35.82$_%.4481@newsfeed.avtel.net>...
The following program progressively allocates all memory: size = 1 << 20 list = [None,] while size: try: mem = [None] * size mem[0] = list list = mem except: size = size / 2
It fails with an access violation. (Turn off virtual memory to provoke it.)
I had a closer look at this report. It revealed a buglet in Instance_New and raises a question on how to handle the possible NULL values pushed on the stack in ceval.c, at the exception handling code block:
3. the NULL pointer at tb is pushed here: ceval.c line 1816: PUSH(tb); ceval.c line 1817: PUSH(val); ceval.c line 1818: PUSH(exc);
1. In PyInstance_New, when the inst->in_dict allocation fails, there's a DECREF(inst) which triggers the instance_dealloc function. The latter goes on checking whether __del__ is defined in the instance's dict (by calling getattr2) whithout checking first whether inst->in_dict is not NULL. I.e. the dealloc_function assumes the instance has been initialized successfully. The net effect is a segfault in PyDict_GetItem which is called with a NULL dict. There are two possible fixes, and I'm not sure which one is better: a) Fix the Instance_New code, by freeing the instance on inst->in_dict allocation failure (without decref'ing the instance). b) Fix the dealloc code, by checking whether in_dict is not NULL before proceeding with the __del__ finalization logic. Opinions? 2. In ceval.c, there are indeed cases where 'tb' and 'exc' are NULL on low-memory conditions. They are pushed on the stack, then popped with a segfault. The only reasonable behavior I get, after some testing, is by setting the tb and exc to Py_None if any of them is NULL before pushing them onto the stack. The program succeeds or the process is killed by the OS (tested on Linux). However, I'm not sure that setting tb and/or exc to Py_None, if NULL, makes any sense and won't cause side effects. Any other suggestions? -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
Vladimir Marangozov writes:
There are two possible fixes, and I'm not sure which one is better:
a) Fix the Instance_New code, by freeing the instance on inst->in_dict allocation failure (without decref'ing the instance).
I think this is the right one; this *shold* be all that's ever needed, and isolates the check to the cod ethat can experiance failure.
b) Fix the dealloc code, by checking whether in_dict is not NULL before proceeding with the __del__ finalization logic.
This protects against bad C code elsewhere that makes a mess of existing objects by stepping around the API (such as by setting inst->in_dict to NULL). I'm not sure we want to protect against that since it could mask bugs (I doubt anyone would do that deliberatly, so I feel safe calling it a bug!).
Opinions?
And for free! So infer what you will about their worth. :)
2. In ceval.c, there are indeed cases where 'tb' and 'exc' are NULL on low-memory conditions. They are pushed on the stack, then popped with a segfault. The only reasonable behavior I get, after some testing, is by setting the tb and exc to Py_None if any of them is NULL before pushing them onto the stack. The program succeeds or the process is killed by the OS (tested on Linux).
However, I'm not sure that setting tb and/or exc to Py_None, if NULL, makes any sense and won't cause side effects.
This doesn't seem like a situation where you'd be masking some other kind of bug by using Py_None, and it avoids the core dump. Since you're not masking anything, I'd go ahead and use Py_None here. -Fred -- Fred L. Drake, Jr. <fdrake at beopen.com> BeOpen PythonLabs Team Member
participants (3)
-
Fred L. Drake, Jr. -
Mark Hammond -
Vladimir.Marangozov＠inrialpes.fr