[Python-Dev] Use of PyObject_NEW

[Tim Peters]

[Martin v. Loewis]
> I tried to understand the various memory allocation function and
> macros in Python, and found that there is probably a misunderstanding
> in what PyObject_NEW does.
>
> For example, PyRange_New says
>
> 	rangeobject *obj = PyObject_NEW(rangeobject, &PyRange_Type);
>
> 	if (obj == NULL)
> 		return NULL;
>
> The assumption apparently is that somebody will raise a MemoryError
> and return NULL when allocation fails. However, this code expands to
>
>      rangeobject *obj = ((rangeobject*)PyObject_Init(
>                (PyObject *) malloc(((&PyRange_Type)->tp_basicsize)),
>                (&PyRange_Type)));
>
> 	if (obj == ((void *)0) )
> 		return ((void *)0) ;
>
> malloc will just return NULL in case of failure, and PyObject_Init
> starts with
>
> 	if (op == NULL) {
> 		PyErr_SetString(PyExc_SystemError,
> 				"NULL object passed to PyObject_Init");
> 		return op;
>   	}
>
> So instead of a MemoryError, you will get a SystemError if the system
> runs out of memory. Is that intentional?

Not credible -- it's a bug.  PyObject_NEW and PyObject_New should behave
alike, and PyObject_New does set MemoryError in this case (via
_PyObject_New().

> The documentation says
>
>   Macro version of \cfunction{PyObject_New()}, to gain performance at
>   the expense of safety.  This does not check \var{type} for a \NULL{}
>   value.
>
> This is incorrect: It does check for NULL.

No, it's correct:  as stated, it does not check \var{type} for a NULL.  Note
that the docs name the *second* argument "type", while the macro names the
first argument "type".  Neither PyObject_New nor PyObject_NEW check their
second argument for NULL (so it's a minor mystery too why the docs bother to
point this out only for PyObject_NEW).

> It also does not help to gain performance - PyObject_New has three
> calls (_PyObject_New, malloc, _Py_NewReference), and so does
> PyObject_NEW (malloc, PyObject_Init, _Py_NewReference).
>
> I recommend to deprecate PyObject_NEW (and correspondingly
> PyObject_NEW_VAR, PyObject_DEL).

I think that's pointless -- lots of code uses these things, and we have no
effective way to deprecate pieces of the C API anyway.  I'd be in favor of
redefining

#define PyObject_NEW PyObject_New

to get rid of the MemoryError bug, though.