[Python-Dev] PyMem_Realloc corner case

[Tim Peters]

[Jason Orendorff]
> I just got bit by the following:
>
> On Windows,

Which flavor of Windows?  Which compiler?  Which version of Python?

> PyMem_Realloc() doesn't seem to work as advertised in
>   http://www.python.org/doc/current/api/memoryInterface.html
>
> In particular, "if n is equal to zero, the memory block is
> resized but is not freed, and the returned pointer is non-NULL".

I expect Fred made that up <wink>.  Whatever, it isn't true on all
platforms.

> In Objects/object.c, lines 1878-1896, PyMem_Malloc() and
> PyMem_Realloc() seem to try to guarantee this behavior.
> Somehow it isn't working for me.

What does "isn't working" mean?  What were your inputs and what was your
output?  What did you expect instead?

> Does this mean MALLOC_ZERO_RETURNS_NULL should be defined
> in PC/pyconfig.h?  Or do I have an off version of the CRT
> that causes problems nobody else has <sigh>?

Probably no and no:  realloc'ing to 0 bytes is stupid, so I'd rather you
stopped hitting yourself over the head and that Python stopped encouraging
you to do so <wink>.  You have control over one of those, anyway.

MS malloc(0) does not return NULL, which is why MALLOC_ZERO_RETURNS_NULL
should not be #define'd.  However, MS realloc(p, 0) returns NULL if and only
if p is not NULL (well, MS realloc(NULL, 0) can return NULL if there's not
enough heap space remaining to allocate a dummy object), and if p is not
NULL does free the block.  All of this is fine by the C standard.

If people think this needs to "be fixed", fine, but I woould strongly oppose
#define'ing MALLOC_ZERO_RETURNS_NULL on Windows:  adding the silly 1 to all
malloc calls then can cause overallocation by 8 bytes, that's a horrible
waste, and whoever assumed malloc(0) behavior was correlated with
realloc(not-NULL, 0) behavior to begin with was wrong.  Better to get rid of
MALLOC_ZERO_RETURNS_NULL, and get rid of _PyMem_EXTRA, instead
special-casing the snot out of a 0 argumentl e.g. instead of

void *
PyMem_Realloc(void *p, size_t nbytes)
{
#if _PyMem_EXTRA > 0
	if (nbytes == 0)
		nbytes = _PyMem_EXTRA;
#endif
	return PyMem_REALLOC(p, nbytes);
}

do

void *
PyMem_Realloc(void *p, size_t nbytes)
{
	return PyMem_REALLOC(p, nbytes ? nbytes : 1);
}

instead.

Note:  The pymalloc realloc works just like the MS realloc here, so it's not
"just Windows"; from obmalloc.c's _THIS_REALLOC:

			/* Don't bother if a smaller size was requested
			   except for realloc(p, 0) == free(p), ret NULL */
			if (nbytes == 0) {
				_THIS_FREE(p);
				bp = NULL;

In fact, I believe it's *common* for realloc(not-NULL, 0) to act like
free(not-NULL), and that's allowed but not required by C.  The overly gross
MALLOC_ZERO_RETURNS_NULL is thus likely not defined on lots of platforms
with this realloc behavior.

Can we call it a doc error?