[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance
brett at python.org
Wed Mar 9 12:54:18 EST 2016
On Wed, 9 Mar 2016 at 06:57 Victor Stinner <victor.stinner at gmail.com> wrote:
> 2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> >> Perhaps if you add some guards somewhere :-)
> > We have runtime checks but only implemented in debug mode for efficiency.
> > By the way, I proposed once to add an environment variable to allow to
> > enable these checks without having to recompile Python. Since the PEP
> > 445, it became easy to implement this. What do you think?
> Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable:
> PYTHONMALLOC=debug installs debug hooks to:
> * detect API violations, ex: PyObject_Free() called on a buffer
> allocated by PyMem_Malloc()
> * detect write before the start of the buffer (buffer underflow)
> * detect write after the end of the buffer (buffer overflow)
> The main advantage of this variable is that you don't have to
> recompile Python in debug mode to benefit of these checks.
I just wanted to say this all sounds awesome! Thanks for all the hard work
on making our memory management story easier to work with, Victor.
> Recompiling Python in debug mode requires to recompile *all*
> extensions modules since the debug ABI is incompatible. When I played
> with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/
> ), I had such issues, it was very annoying with non-trivial extension
> modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to
> recompile extension modules anymore!
> With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool
> suite to "debug memory"!
> My motivation for PYTHONMALLOC=debug is to detect API violations to
> prepare my change on PyMem_Malloc() allocator (
> http://bugs.python.org/issue26249 ), but also to help users to detect
> It's common that users report a bug: "Python crashed", but have no
> idea of the responsible of the crash. I hope that detection of buffer
> underflow & overflow will help them to detect bugs in their own
> extension modules.
> Moreover, I added PYTHONMALLOC=malloc to ease the use of external
> memory debugger on Python. By default, Python uses pymalloc allocator
> for PyObject_Malloc() which raises a lot of false positive in
> Valgrind. We even have a configuration (--with-valgrind) and a
> Valgrind suppressino file to be able to skip these false alarms in
> Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind
> (or other tools).
> Python-Dev mailing list
> Python-Dev at python.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev