[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Michael Sullivan sully at msully.net
Tue Apr 16 22:05:50 EDT 2019


On Tue, Apr 16, 2019 at 2:11 AM Victor Stinner <vstinner at redhat.com> wrote:

> Hi Michael,
>
> Do you know the tracemalloc module? Did you try it? It works on a
> regular Python (compiled in debug mode).
>
> I would be curious to know if tracemalloc also allows you to track the
> memory leak.
>
> Playing around with it a little it does not seem super helpful here
(unless I am missing something): it tracks the allocations based on the
python call stack, which doesn't help here, in a C extension module
generated from python code.

Though, in the the mypyc case, we could implement a debug option for
creating dummy frames so that we always have a useful call stack. That
seems like less of an option for actual hand-written extension modules,
though. (Though on the flip side, the python call stacks might be more
useful there.)

sys.getobjects() is just a list of objects. Do you have a tool written
> on top of it to track memory leaks? If yes, how?
>
> Not really.

We have a very simple memory profiler built on top of gc.get_objects() that
just reports how many of different types of objects there are and how much
memory they are using:
https://github.com/python/mypy/blob/master/mypy/memprofile.py.
I swapped out gc.get_objects() for sys.getobjects(), observed that we were
leaking int objects, and inspected the live int objects, which gave a
pretty good clue where the leak was.


> Victor
>
> Le mar. 16 avr. 2019 à 00:28, Michael Sullivan <sully at msully.net> a écrit
> :
> >
> > > The main question is if anyone ever used Py_TRACE_REFS? Does someone
> > > use sys.getobjects() or PYTHONDUMPREFS environment variable?
> >
> > I used sys.getobjects() today to track down a memory leak in the
> mypyc-compiled version of mypy.
> >
> > We were leaking memory badly but no sign of the leak was showing up in
> mypy's gc.get_objects() based profiler. Using a debug build and switching
> to sys.getobjects() showed that we were badly leaking int objects. A quick
> inspection of the values in question (large and random looking) suggested
> we were leaking hash values, and that quickly pointed me to
> https://github.com/mypyc/mypyc/pull/562.
> >
> > I don't have any strong feelings about whether to keep it in the
> "default" debug build, though. I was using a debug build that I built
> myself with every debug feature that seemed potentially useful.
> >
> > -sully
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190416/498140d5/attachment.html>


More information about the Python-Dev mailing list