: [Python-Dev] RE: Painful death in debug build

[Tim Peters]

[Tim]
> This is enough to reproduce the problem under a debug Windows build:
>
> ---------------------------------------------------------------
> class X(long):
>     pass
>
> x = X(0xffffL)
> print "before del"
> del x
> print "after del"
> ---------------------------------------------------------------
>
> It does not fail if the instance is created via
>
>     x = X(0x7fffL)
>
> instead.  It does fail on
>
>     x = X(0x8000L)
>
> The relevant difference is that Python uses 15-bit "digits" internally
> for longs, so 0x8000 may be the smallest value that will cause it to
> allocate more memory than already comes for free with the _longobject
> header

Now I'm in:

PyObject *
PyType_GenericAlloc(PyTypeObject *type, int nitems)
{
#define PTRSIZE (sizeof(PyObject *))

	int size;
	PyObject *obj;

	/* Inline PyObject_New() so we can zero the memory */

That comment doesn't seem to make much sense; the code following does a
whole bunch that PyObject_New() doesn't do.

	size = _PyObject_VAR_SIZE(type, nitems);

Then size = basicsize + 2 * itemsize = 26 + 2*2 = 30.

	/* Round up size, if necessary, so we fully zero out __dict__ */
	if (type->tp_itemsize % PTRSIZE != 0) {
		size += PTRSIZE - 1;
		size /= PTRSIZE;
		size *= PTRSIZE;
	}

I don't understand what that comment is trying to say; the code following it
bumps size up to 32, and that turns out to be "the problem":

	if (PyType_IS_GC(type)) {
		obj = _PyObject_GC_Malloc(type, nitems);
	}
	else {
		obj = PyObject_MALLOC(size);
	}

The subtype is a GC type, so _PyObject_GC_Malloc allocates

   sizeof(PyGC_Head) + (size_t)_PyObject_VAR_SIZE(tp, size) =
   12                + 30 =

42 bytes.  Note that the second term is the 30 originally computed for size,
not the 32 that size was later bumped up to.

	if (obj == NULL)
		return PyErr_NoMemory();

Nope.

	memset(obj, '\0', size);

Bingo!  obj is at offset 12 from the allocated block, and size is 32, so we
zero out the last 32 of the trailing 30 (=42-12) bytes allocated; i.e., we
zero out two bytes beyond the end of the allocated block.  Everything else
follows from that.  Seems pretty clear that this doesn't have anything
specific to do with longs.  Yup:

>>> class X(str):
...     pass
...
[6960 refs]
>>> x = X("ab")
[6969 refs]
>>> x = 1

That also blows up.

Since I'm not sure what the roundup of size is trying to accomplish, I'm not
sure what to do about it.  If I ignore the comment, I *imagine* we want
subtype pointer fields (like tp_dict) following the base-type prefix to be
naturally aligned for the architecture, and then padding makes sense to me.