[Python-Dev] Use of PyObject_NEW

[Guido van Rossum]

> I tried to understand the various memory allocation function and
> macros in Python, and found that there is probably a misunderstanding
> in what PyObject_NEW does.
> 
> For example, PyRange_New says
> 
> 	rangeobject *obj = PyObject_NEW(rangeobject, &PyRange_Type);
> 
> 	if (obj == NULL)
> 		return NULL;
> 
> The assumption apparently is that somebody will raise a MemoryError
> and return NULL when allocation fails. However, this code expands to
> 
>      rangeobject *obj = ((rangeobject*)PyObject_Init(
>                (PyObject *) malloc(((&PyRange_Type)->tp_basicsize)),
>                (&PyRange_Type)));
> 
> 	if (obj == ((void *)0) )
> 		return ((void *)0) ;
> 
> malloc will just return NULL in case of failure, and PyObject_Init
> starts with
> 
> 	if (op == NULL) {
> 		PyErr_SetString(PyExc_SystemError,
> 				"NULL object passed to PyObject_Init");
> 		return op;
>   	}
> 
> So instead of a MemoryError, you will get a SystemError if the system
> runs out of memory. Is that intentional?

Oh, good catch.  PyObject_Init() should only raise its own exception
when PyErr_Occurred() returns NULL.

> The documentation says
> 
>   Macro version of \cfunction{PyObject_New()}, to gain performance at
>   the expense of safety.  This does not check \var{type} for a \NULL{}
>   value.
> 
> This is incorrect: It does check for NULL.

It checks the allocated storage for NULL, but it doesn't check the
'type' argument for NULL.  Of course it doesn't!  The function doesn't
either.  The type is always the address of a statically allocated
variable so there's no worry about NULL here.  So there's something
else wrong with the docs -- why does it say this at all?

> It also does not help to
> gain performance - PyObject_New has three calls (_PyObject_New, malloc,
> _Py_NewReference), and so does PyObject_NEW (malloc, PyObject_Init,
> _Py_NewReference).

Curious.  You seem to be right.  I'm pretty sure it *used* to be
faster to use the macros, but I believe correctness requirements have
made the point moot.

> I recommend to deprecate PyObject_NEW (and correspondingly
> PyObject_NEW_VAR, PyObject_DEL).

And in the mean time, define them as aliases for the functions -- that
should save some code size.

--Guido van Rossum (home page: http://www.python.org/~guido/)