[Python-ideas] Line-by-line profiling

[Nick Coghlan]

On 20 December 2014 at 18:28, Todd <toddrjen at gmail.com> wrote:

> Currently, the python profiler works at the function or method level.
> This is good for identifying slow functions, but not so good for
> identifying things like slow for loops inside larger functions.  You can
> use timers, but putting a lot of times in its cumbersome and makes direct
> comparison between potential hotspots difficult.  And although we should,
> in practice it isn't airways feasible to refactor or code so there is only
> one major hotspot per function or method.
>
> A solution to this is to have a profiler that shows the execution time for
> each line (cumulative and per-call).  Currently there is the third-party
> line-profiler, but at least to me this seems like the sort of common
> functionality that belongs in the standard library.  line-profiler also
> requires the use of a decorator since it has a large performance penalty,
> and perhaps a more deeply integrated line-by-line profiler could avoid this.
>
It's worth noting that since Python 3.4 and 2.7.9, the PyPI line_profiler
module is only a "pip install line_profiler" away for most users.

While bringing line_profiler into the standard library would technically be
feasible, it creates some additional challenges:

* line_profiler uses Cython for its build process, so including it would
mean being able to bootstrap Cython in a CPython source checkout in order
to completely build the standard library

* a line_profiler module would still be needed on PyPI for older Python
versions, so this would involve either forking a separately maintained copy
for the standard library, or else persuading Robert Kern to accept the
additional maintenance burden associated with inclusion in the standard
library as well as being published on PyPI

A potentially lower cost alternative may be to add a profiling HOWTO guide,
or even just some "See Also" links in the documentation for the profile
module, and point out the available of the line_profiler and kernprof
utilities. (The SnakeViz visualiser I was recently introduced to at the
Brisbane Python User Group is another useful utility that could be worth
making more readily discoverable: https://jiffyclub.github.io/snakeviz/)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141220/8a127a5c/attachment.html>