[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
>> Perhaps if you add some guards somewhere :-)
>
> We have runtime checks but only implemented in debug mode for efficiency.
>
> By the way, I proposed once to add an environment variable to allow to
> enable these checks without having to recompile Python.  Since the PEP
> 445, it became easy to implement this. What do you think?
> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable

Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable:

   http://bugs.python.org/issue26516

PYTHONMALLOC=debug installs debug hooks to:

* detect API violations, ex: PyObject_Free() called on a buffer
allocated by PyMem_Malloc()
* detect write before the start of the buffer (buffer underflow)
* detect write after the end of the buffer (buffer overflow)

https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks

The main advantage of this variable is that you don't have to
recompile Python in debug mode to benefit of these checks.

Recompiling Python in debug mode requires to recompile *all*
extensions modules since the debug ABI is incompatible. When I played
with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/
), I had such issues, it was very annoying with non-trivial extension
modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to
recompile extension modules anymore!


With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool
suite to "debug memory"!

My motivation for PYTHONMALLOC=debug is to detect API violations to
prepare my change on PyMem_Malloc() allocator (
http://bugs.python.org/issue26249 ), but also to help users to detect
bugs.

It's common that users report a bug: "Python crashed", but have no
idea of the responsible of the crash. I hope that detection of buffer
underflow & overflow will help them to detect bugs in their own
extension modules.


Moreover, I added PYTHONMALLOC=malloc to ease the use of external
memory debugger on Python. By default, Python uses pymalloc allocator
for PyObject_Malloc() which raises a lot of false positive in
Valgrind. We even have a configuration (--with-valgrind) and a
Valgrind suppressino file to be able to skip these false alarms in
Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind
(or other tools).

Victor