C extensions "should" use PyMem_Malloc/PyMem_Free to benefit of PEP 445 hooks like debug hooks and tracemalloc. * Debug hooks (enabled by -X dev) helps to debug buffer overflow: https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks * tracemalloc: https://docs.python.org/dev/library/tracemalloc.html * PEP 445, API for memory allocators: https://www.python.org/dev/peps/pep-0445/ tracemalloc is used more and more often in CPython internals to provide more information when an issue is detected. Example with ResourceWarning: https://docs.python.org/dev/library/devmode.html#resourcewarning-example Buffer overflow: "On error, the debug hooks use the tracemalloc module to get the traceback where a memory block was allocated. The traceback is only displayed if tracemalloc is tracing Python memory allocations and the memory block was traced." Also, PyMem_Malloc/PyMem_Free now use pymalloc memory allocator which is more efficient than libc malloc/free for memory blocks smaller than or equal to 512 bytes. When I modified them to use pymalloc, I saw a global speedup around 10% faster. But these functions require the caller to hold the GIL. Use PyMem_RawMalloc/PyMem_RawFree if you don't hold the GIL. Note: The Python ssl module doesn't use PyMem_Malloc/PyMem_Free for OpenSSL. Last time I tried, the OpenSSL function to set a custom memory allocator didn't work at all. Maybe it has been fixed in the meanwhile. https://bugs.python.org/issue18227#msg191610 See https://bugs.python.org/issue18227#msg191250 for the list of C extensions which may be modified to use Python memory allocators. Victor Le mer. 13 mai 2020 à 14:23, Pablo Galindo Salgado <pablogsal@gmail.com> a écrit :
But again this is for PyObjects only.
Not really, we check also memory blocks:
https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.p...
as long as you don't directly call malloc and use one of the Python specific APIs like PyMem_Malloc then the reflect code should catch that.
On Wed, 13 May 2020 at 12:29, Giampaolo Rodola' <g.rodola@gmail.com> wrote:
On Wed, May 13, 2020 at 9:17 AM Ammar Askar <ammar@ammaraskar.com> wrote:
Py_DECREF calls in the C code
I think this part specifically is already covered through refleak checks: https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.p...
Since it can involve the repetition of tests many times, these aren't run on the CI though, they do get run on the refleak buildbots so issues get caught eventually: https://buildbot.python.org/all/#/builders?tags=%2Brefleak
But again this is for PyObjects only. Were you able to find any memory leaks with your proof-of-concept? I don't think there's a lot of chances of someone missing a PyMem_Free call and there's not a lot of other manual memory management but I could be wrong. Anything found there could help motivate adding this a bit more.
Yeah, I agree it depends on how many PyMem_* occurrences are there, and it probably makes more sense to cover those ones only. Under Modules/* I found:
- 24 occurrences for PyMem_RawMalloc - 2 for PyMem_RawCalloc - 106 for PyMem_Malloc - 12 for PyMem_Calloc - 39 for PyMem_New - 5 for " = malloc("
I spent an hour covering around 20 of those and didn't find any leak. It's a boring work. I will try to work on it over the next few weeks.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/C7ZZRDPG... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YNBGAWNC... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.