[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Brett Cannon brett at python.org
Wed Mar 9 12:54:18 EST 2016

On Wed, 9 Mar 2016 at 06:57 Victor Stinner <victor.stinner at gmail.com> wrote:

> 2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> >> Perhaps if you add some guards somewhere :-)
> >
> > We have runtime checks but only implemented in debug mode for efficiency.
> >
> > By the way, I proposed once to add an environment variable to allow to
> > enable these checks without having to recompile Python.  Since the PEP
> > 445, it became easy to implement this. What do you think?
> >
> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable
> Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable:
>    http://bugs.python.org/issue26516
> PYTHONMALLOC=debug installs debug hooks to:
> * detect API violations, ex: PyObject_Free() called on a buffer
> allocated by PyMem_Malloc()
> * detect write before the start of the buffer (buffer underflow)
> * detect write after the end of the buffer (buffer overflow)
> https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks
> The main advantage of this variable is that you don't have to
> recompile Python in debug mode to benefit of these checks.

I just wanted to say this all sounds awesome! Thanks for all the hard work
on making our memory management story easier to work with, Victor.


> Recompiling Python in debug mode requires to recompile *all*
> extensions modules since the debug ABI is incompatible. When I played
> with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/
> ), I had such issues, it was very annoying with non-trivial extension
> modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to
> recompile extension modules anymore!
> With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool
> suite to "debug memory"!
> My motivation for PYTHONMALLOC=debug is to detect API violations to
> prepare my change on PyMem_Malloc() allocator (
> http://bugs.python.org/issue26249 ), but also to help users to detect
> bugs.
> It's common that users report a bug: "Python crashed", but have no
> idea of the responsible of the crash. I hope that detection of buffer
> underflow & overflow will help them to detect bugs in their own
> extension modules.
> Moreover, I added PYTHONMALLOC=malloc to ease the use of external
> memory debugger on Python. By default, Python uses pymalloc allocator
> for PyObject_Malloc() which raises a lot of false positive in
> Valgrind. We even have a configuration (--with-valgrind) and a
> Valgrind suppressino file to be able to skip these false alarms in
> Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind
> (or other tools).
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160309/61eb5cdf/attachment.html>

More information about the Python-Dev mailing list