FW: [PythonLabs] Re: [Python-Dev] RE: Painful death in debug build
FYI, for anyone else looking at this. Guido tricked me into sending this to an internal PythonLabs address. -----Original Message----- From: Zope.Com Python Labs Sent: Saturday, October 06, 2001 12:47 AM To: Guido van Rossum Cc: Zope.Com Python Labs Subject: [PythonLabs] Re: [Python-Dev] RE: Painful death in debug build [Guido]
I tried efence on Linux (which traps bad stores using VM tricks) and couldn't get it to fail either.
Thanks for trying! I'll pursue here then. VM tricks may not be enough, unless they're very fine-grained. The MS debug malloc overallocates by 8 bytes, 4 on each side of the blocks it returns. These get filled with 0xFD bytes. When the debug free() gets a block back, it first checks these bytes. The four before the block are OK (still 0xFD). The first two bytes after the block have been zeroed, but the two bytes after those are still 0xFD. So who's storing 0 into a short via a computed address suffering an off-by-one error <0.9 wink>? (*) When I crank up enough energy, it should be more-or-less straightforward to set a data breakpoint on one of the byte addresses getting corrupted, and so catch the culprit in the act (the potential problem is that memory gets recycled, and on occasion I've had to give up on this approach after wading thru 100s of false alarms). (*) Hmm! Cogitating: the traceback pointed at a subclass of long, and the internal representation of longs is indeed one of the few places Python uses an array of shorts (Unicode is another -- others?). A dollar bets this will prove to be relevant.
This is enough to reproduce the problem under a debug Windows build: --------------------------------------------------------------- class X(long): pass x = X(0xffffL) print "before del" del x print "after del" --------------------------------------------------------------- It does not fail if the instance is created via x = X(0x7fffL) instead. It does fail on x = X(0x8000L) The relevant difference is that Python uses 15-bit "digits" internally for longs, so 0x8000 may be the smallest value that will cause it to allocate more memory than already comes for free with the _longobject header: struct _longobject { PyObject_HEAD int ob_size; digit ob_digit[1]; }; but-too-tired-to-do-more-now-ly y'rs - tim
[Tim]
This is enough to reproduce the problem under a debug Windows build:
--------------------------------------------------------------- class X(long): pass
x = X(0xffffL) print "before del" del x print "after del" ---------------------------------------------------------------
It does not fail if the instance is created via
x = X(0x7fffL)
instead. It does fail on
x = X(0x8000L)
The relevant difference is that Python uses 15-bit "digits" internally for longs, so 0x8000 may be the smallest value that will cause it to allocate more memory than already comes for free with the _longobject header
Now I'm in: PyObject * PyType_GenericAlloc(PyTypeObject *type, int nitems) { #define PTRSIZE (sizeof(PyObject *)) int size; PyObject *obj; /* Inline PyObject_New() so we can zero the memory */ That comment doesn't seem to make much sense; the code following does a whole bunch that PyObject_New() doesn't do. size = _PyObject_VAR_SIZE(type, nitems); Then size = basicsize + 2 * itemsize = 26 + 2*2 = 30. /* Round up size, if necessary, so we fully zero out __dict__ */ if (type->tp_itemsize % PTRSIZE != 0) { size += PTRSIZE - 1; size /= PTRSIZE; size *= PTRSIZE; } I don't understand what that comment is trying to say; the code following it bumps size up to 32, and that turns out to be "the problem": if (PyType_IS_GC(type)) { obj = _PyObject_GC_Malloc(type, nitems); } else { obj = PyObject_MALLOC(size); } The subtype is a GC type, so _PyObject_GC_Malloc allocates sizeof(PyGC_Head) + (size_t)_PyObject_VAR_SIZE(tp, size) = 12 + 30 = 42 bytes. Note that the second term is the 30 originally computed for size, not the 32 that size was later bumped up to. if (obj == NULL) return PyErr_NoMemory(); Nope. memset(obj, '\0', size); Bingo! obj is at offset 12 from the allocated block, and size is 32, so we zero out the last 32 of the trailing 30 (=42-12) bytes allocated; i.e., we zero out two bytes beyond the end of the allocated block. Everything else follows from that. Seems pretty clear that this doesn't have anything specific to do with longs. Yup:
class X(str): ... pass ... [6960 refs] x = X("ab") [6969 refs] x = 1
That also blows up. Since I'm not sure what the roundup of size is trying to accomplish, I'm not sure what to do about it. If I ignore the comment, I *imagine* we want subtype pointer fields (like tp_dict) following the base-type prefix to be naturally aligned for the architecture, and then padding makes sense to me.
So the problem is that one size calculation rounds up to multiples of a pointer size and the other doesn't; unfortunately the one that doesn't gets passed to malloc. The reason for the rounding up --a recent addition-- is that when you derive a subclass from tuple, string or long, the __dict__ of the subclass instance (and its weakref list head) are allocated after the variable-length portion of the instance. I propose to fix the other size calculation to also round up. I'll leave it to you to fix this, since you've almost nailed this already (and since I'm not feeling so great right now). --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (2)
-
Guido van Rossum -
Tim Peters