[Numpy-discussion] Profiling (was GSoC : Performance parity between numpy arrays and Python scalars)

Nathaniel Smith njs at pobox.com
Mon Jun 17 12:34:47 EDT 2013

On Mon, Jun 17, 2013 at 4:29 PM, Arink Verma <arinkverma at gmail.com> wrote:
> Hi Nathaniel
>> It's a probabilistic sampling profiler, so if it doesn't have enough
>> samples then it can miss things. 227 samples is way way too low. You need to
>> run the profiled code for longer (a few seconds at least), and if that's not
>> enough then maybe increase the sampling rate too (though be careful because
>> setting this too high can also add noise).
> I ran code '1000000000' times, which record 229115 samples. Callgraph[1]
> generated is converging to PyArray_DESCR, and rest are unconnected.
> Does it mean anything?
> [1]https://docs.google.com/file/d/0B3Pqyp8kuQw0MzRoTTNVbmlaNFE/edit?usp=sharing

I think it means that pprof is failing to walk to the stack to compute
callers. That's consistent with PyArray_DESCR being the only "call"
that it can find, because PyArray_DESCR isn't a real function, it
always gets inlined, so detecting the caller doesn't require walking
the stack.

Is your numpy compiled with -fomit-frame-pointer or something like
that? Any other weird build options used while building it? Is your
binary stripped? If you're using a distro version, do you have the
debug symbols installed? Did you compile this numpy yourself? (If not,
the simplest thing to do would be to just build it yourself using the
default settings and see if that helps.) What OS are you on?

When I run your profiling code (but being lazier and only running
12000 samples), then do 'google-pprof --pdf /path/to/python
/path/to/my/profile.prof', then I get the graph attached below.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: prof.pdf
Type: application/pdf
Size: 20466 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/02e8051b/attachment.pdf>

More information about the NumPy-Discussion mailing list