Hi Julien, Isn't this a similar problem that you have with regular malloc? When you call malloc() with some size, malloc actually will reserve more than that for alignment purposes and for bookkeeping and apart from some platform-specific APIs like malloc_usable_size() you cannot query that value. Unless I am missing something here, there is also some problem of what you propose: the Python allocators deal with memory and they do not know what that memory is going to be used so when you say:
This is a bummer as there then no safe way that I can think of to know ifan allocated memory space is gc-tracked or gc-untracked.
My answer would be that that's because memory itself cannot be gc tracked,
only objects can and those belonging to different
categories. For example, notice that the tracemalloc module does not report
objects, it only reports memory blocks and you
cannot ask tracemalloc to give you a list of all the objects that were
allocated because it does not have the notion of what
an object is.
Adding the functionality that you suggest would couple the allocator with
other parts of the VM and it also won't give you the
the answer that you need to because what's underneath (pymalloc or libc
malloc or whatever else) may actually allocate even more
than what you asked.
Regards from cloudy London,
Pablo Galindo Salgado
On Thu, 14 Jan 2021 at 16:10, Julien Danjou
Hi there,
I've spent quite some time on memory profiling for Python now. I'm struggling to get more information from the allocated memory right now for what looks like a sad reason. :(
Supposedly PyObject_Malloc() returns some memory space to store a PyObject. If that was true all the time, that would allow anyone to introspect the allocated memory and understand why it's being used.
Unfortunately, this is not the case. Objects whose types are tracked by the GC go through _PyObject_GC_Alloc() which changes the underlying memory structure to be (PyGC_HEAD + PyObject).
This is a bummer as there then no safe way that I can think of to know if an allocated memory space is gc-tracked or gc-untracked. It makes it therefore impossible to introspect the memory allocated by PyObject_Malloc().
There are multiple ways to solve this, but I don't see any way right now of doing this without slightly changing CPython.
Has anyone any idea on how to workaround this or the kind of change that could be acceptable to mitigate the issue?
Thanks!
Cheers, -- Julien Danjou # Free Software hacker # https://julien.danjou.info _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QRX6U5XB... Code of Conduct: http://python.org/psf/codeofconduct/