[New-bugs-announce] [issue37200] PyType_GenericAlloc might over-allocate memory

[Neil Schemenauer]

New submission from Neil Schemenauer <nas-python at arctrix.com>:

In the process of working on some garbage collector/obmalloc experiments, I noticed what seems to be a quirk about PyType_GenericAlloc().  It calls:

size = _PyObject_VAR_SIZE(type, nitems+1);

Note the "+1" which is documented as "for the sentinel".  That code dates back to change "e5c691abe3946ddbaa00730b92f3b96f96903f7d" when Guido added support for heap types.  This extra item is not added by _PyObject_GC_NewVar().  Also, the documentation for tp_alloc says that the size of the allocated block should be:

  tp_basicsize + nitems*tp_itemsize, rounded up to a multiple of sizeof(void*);

The "+1" for the sentinel is definitely needed in certain cases.  I think it might only be needed if 'type' is a subtype of 'type'.  I.e. if Py_TPFLAGS_TYPE_SUBCLASS is set on 'type'.

I haven't done enough analysis to fully understand this quirk yet but I think we are allocating extra memory quite regularly.  Quite a lot of types use tp_alloc = PyType_GenericAlloc.  E.g. the 'list' type or a subclass of the tuple type.

It seems with the attached patch, unit tests still pass.  Perhaps the +1 could be removed on the non-GC branch of the code as well.

----------
components: Interpreter Core
files: generic_alloc_sentinel.txt
messages: 345002
nosy: nascheme
priority: normal
severity: normal
status: open
title: PyType_GenericAlloc might over-allocate memory
type: performance
Added file: https://bugs.python.org/file48403/generic_alloc_sentinel.txt

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37200>
_______________________________________