[Numpy-discussion] profiling line by line

[Ondrej Certik]

On Thu, Sep 18, 2008 at 4:12 PM, Ryan May <rmay31 at gmail.com> wrote:
> Ondrej Certik wrote:
>> On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman <cimrman3 at ntc.zcu.cz> wrote:
>>>>
>>>> It requires Cython and a C compiler to build. I'm still debating
>>>> myself about the desired workflow for using it, but for now, it only
>>>> profiles functions which you have registered with it. I have made the
>>>> profiler work as a decorator to make this easy. E.g.,
>>> many thanks for this! I have wanted to try out the profiler but failed
>>> to build it (changeset 6 0de294aa75bf):
>>>
>>> $ python setup.py install --root=/home/share/software/
>>> running install
>>> running build
>>> running build_py
>>> creating build
>>> creating build/lib.linux-i686-2.4
>>> copying line_profiler.py -> build/lib.linux-i686-2.4
>>> running build_ext
>>> cythoning _line_profiler.pyx to _line_profiler.c
>>> building '_line_profiler' extension
>>> creating build/temp.linux-i686-2.4
>>> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC
>>> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c
>>> -o build/temp.linux-i686-2.4/_line_profiler.o
>>> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a
>>> function)
>>> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1
>>>
>>> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine.
>>
>> I am telling you all the time Robert to use Debian that it just works
>> and you say, no no, gentoo is the best. :)
>
> And what's wrong with that? :)  Once you get over the learning curve,
> Gentoo works just fine.  Must be Robert K.'s fault. :)

Well, I think if Robert C. hasn't yet get over the learning curve
after so many years of hard work, maybe the learning curve is too
steep. :)

Anyway, back to work: Robert K., I noticed that if I profile some
function, I get results like this for example:

----
    40      3072        46952      6.6      [x,w] = p_roots(n)
    41      3072        24192      3.4      x = real(x)
    42      3072        34470      4.8      ainf, binf = map(isinf,(a,b))
    43      3072         6565      0.9      if ainf or binf:
    44                                          raise ValueError,
"Gaussian quadrature is only available for " \
    45                                                "finite limits."
    46      3072         5093      0.7      if not reference:
    47                                          x = (b-a)*(x+1)/2.0 + a
    48      3072       594190     83.5      return
(b-a)/2.0*sum(w*func(x,*args),0)
---

Then if I turn profiling of the func() method, I get this:

--------------------
    40      3072        46999      4.6      [x,w] = p_roots(n)
    41      3072        24313      2.4      x = real(x)
    42      3072        34327      3.4      ainf, binf = map(isinf,(a,b))
    43      3072         6190      0.6      if ainf or binf:
    44                                          raise ValueError,
"Gaussian quadrature is only available for " \
    45                                                "finite limits."
    46      3072         4918      0.5      if not reference:
    47                                          x = (b-a)*(x+1)/2.0 + a
    48      3072       906876     88.6      return
(b-a)/2.0*sum(w*func(x,*args),0)
-----------

So the timing raises a lot. For obvious reasons, that's the overhead
of the profiler. But the problem is that then the timings just don't
fit, e.g. if I sum the total time spent in subfunctions, it doesn't
account for all the time printed on the respective line in the parent
function.

I don't know if there is any way to fix it, or even worth fixing. So I
guess one should just use the profiling info to determine which line
to fix.

Do you think it's worthy to implement line profiling for all lines and
then make sure that all timings match? What is your experience.

Ondrej