[Tutor] performance loss -- profiling

[Kent Johnson]

On Mon, May 18, 2009 at 2:49 PM, spir <denis.spir at free.fr> wrote:
> Hello,
>
> I have an big performance problem with an app I'm currently working on.
> It "suddenly" runs at least 5 times slower that it used to. The issue is, beeing in a finalization phase, I'm all the time touching thingies here and there. But performance change is visible only when running on big test data, which I haven't done for a few days.
>
> As a consequence I'm unable to find what bug I've introduced that causes such a change. If you of any rational method to diagnose such problems, thank you very much.
>
> Also, it's the first time I really have to cope with machine-time; so I'm totally new to technics like using a profiler. Any hints on the topics heartfully welcome :-)

Are you using a source code control system? If so, you can back out
changes and see where the performance changes. Some SCCS will even
help you with this, for example the Mercurial bisect command automates
a binary search through the repository changesets.

The Python profiler is not hard to run. Interpreting the results is
more difficult :-) See the docs to get started:
http://docs.python.org/library/profile.html

Because profiling can slow your code down considerably, if you have a
general idea where the slowdown is, you can just profile that.

The output of the profiler is a file which you can load in an
interactive session using pstats.Stats. You will want to sort the
stats by cumulative time and print. Look for something that is taking
an unexpectedly large amount of the time, or that is being called more
than expected.

RunSnakeRun looks like a useful program for visualizing the profile results:
http://www.vrplumber.com/programming/runsnakerun/

PProfUI is a simpler profile viewer that I have found useful:
http://webpages.charter.net/erburley/pprofui.html

Once you find a hot spot, you have to figure out why it is hot. Is it
being called to much? Is there a better algorithm or data structure?
The profiler won't help with this part. line_profiler might be useful,
I haven't used it:
http://packages.python.org/line_profiler/

Kent