[Numpy-discussion] tracing numpy data allocation with python callbacks

[Thouis (Ray) Jones]

On Thu, May 17, 2012 at 9:52 PM, Nathaniel Smith <njs at pobox.com> wrote:
> I'd be tempted to just see if I could get by with massif or another
> "real" heap profiler -- unfortunately the ones I know are C oriented,
> but might still be useful...

I got some very useful information from Fabien's technique, which led
me to wanting to know more, particularly about spikes in ephemeral
allocations, such as for the subexpressions of "a = b * c + d * e".

The main benefit to what I posted over heap profilers in general was
that it gave a way for tracing at the python level, so I can easily do
things like record the allocations (long-lived and ephemeral) by what
line in the python code call stack is responsible.

Perhaps I'll rewrite it to be cleaner, something like:
numpy.trace_data_allocations(malloc_callback=None, free_callback=None,
realloc_callback=None)

Where "None" turns off tracing of that particular call.  It should be
efficient when not activated (check a single flag).

Ray Jones