On 20 December 2014 at 18:28, Todd <toddrjen@gmail.com> wrote:

Currently, the python profiler works at the function or method level. This is good for identifying slow functions, but not so good for identifying things like slow for loops inside larger functions. You can use timers, but putting a lot of times in its cumbersome and makes direct comparison between potential hotspots difficult. And although we should, in practice it isn't airways feasible to refactor or code so there is only one major hotspot per function or method.

A solution to this is to have a profiler that shows the execution time for each line (cumulative and per-call). Currently there is the third-party line-profiler, but at least to me this seems like the sort of common functionality that belongs in the standard library. line-profiler also requires the use of a decorator since it has a large performance penalty, and perhaps a more deeply integrated line-by-line profiler could avoid this.

It's worth noting that since Python 3.4 and 2.7.9, the PyPI line_profiler module is only a "pip install line_profiler" away for most users. While bringing line_profiler into the standard library would technically be feasible, it creates some additional challenges: * line_profiler uses Cython for its build process, so including it would mean being able to bootstrap Cython in a CPython source checkout in order to completely build the standard library * a line_profiler module would still be needed on PyPI for older Python versions, so this would involve either forking a separately maintained copy for the standard library, or else persuading Robert Kern to accept the additional maintenance burden associated with inclusion in the standard library as well as being published on PyPI A potentially lower cost alternative may be to add a profiling HOWTO guide, or even just some "See Also" links in the documentation for the profile module, and point out the available of the line_profiler and kernprof utilities. (The SnakeViz visualiser I was recently introduced to at the Brisbane Python User Group is another useful utility that could be worth making more readily discoverable: https://jiffyclub.github.io/snakeviz/) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 20 December 2014 at 18:28, Todd <toddrjen@gmail.com> wrote:

Currently, the python profiler works at the function or method level. This is good for identifying slow functions, but not so good for identifying things like slow for loops inside larger functions. You can use timers, but putting a lot of times in its cumbersome and makes direct comparison between potential hotspots difficult. And although we should, in practice it isn't airways feasible to refactor or code so there is only one major hotspot per function or method.

A solution to this is to have a profiler that shows the execution time for each line (cumulative and per-call). Currently there is the third-party line-profiler, but at least to me this seems like the sort of common functionality that belongs in the standard library. line-profiler also requires the use of a decorator since it has a large performance penalty, and perhaps a more deeply integrated line-by-line profiler could avoid this.

It's worth noting that since Python 3.4 and 2.7.9, the PyPI line_profiler module is only a "pip install line_profiler" away for most users. While bringing line_profiler into the standard library would technically be feasible, it creates some additional challenges: * line_profiler uses Cython for its build process, so including it would mean being able to bootstrap Cython in a CPython source checkout in order to completely build the standard library * a line_profiler module would still be needed on PyPI for older Python versions, so this would involve either forking a separately maintained copy for the standard library, or else persuading Robert Kern to accept the additional maintenance burden associated with inclusion in the standard library as well as being published on PyPI A potentially lower cost alternative may be to add a profiling HOWTO guide, or even just some "See Also" links in the documentation for the profile module, and point out the available of the line_profiler and kernprof utilities. (The SnakeViz visualiser I was recently introduced to at the Brisbane Python User Group is another useful utility that could be worth making more readily discoverable: https://jiffyclub.github.io/snakeviz/) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia