Is there a standard or best methodology for tracking python code performance? I see in the intertubes that one can use the time module, or timeit or other things. How about memory usage? Or is this typically handled by debugging tools? What I really want to do is isolate logjams in python code by function call etc. in a fairly non-intrusive way. Thanks for any advice. Eric -- Eric Hallman Tech-X Corporation hallman@txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 254-5833 Boulder, CO 80303 Fax: (303) 448-7756 --
Hi Eric, On Thu, May 3, 2012 at 1:41 PM, Eric Hallman <hallman@txcorp.com> wrote:
Is there a standard or best methodology for tracking python code performance? I see in the intertubes that one can use the time module, or timeit or other things. How about memory usage? Or is this typically handled by debugging tools?
What I really want to do is isolate logjams in python code by function call etc. in a fairly non-intrusive way.
Thanks for any advice.
If you're using yt, there are three methods you can call upon. If not, I'd recommend using the cProfile module as a start, moving on to what was kernprof and is now lineprof, and ending at Bryan O'Sullivan's recent statprof. For visualizing, "pip install jinja2 pyprof2html" will get you pyprof2html, which turns a cProfile file to HTML which is nicely viewable. In yt, performance timers (written by Stephen) can help, but for very simple stuff: with parallel_profile("some_prefix"): some_expensive_operation() will output .cprof files with processor IDs in them, which can be run through pyprof2html. For memory, get_memory_usage() is a yt function that gives you the current resident size of your PID. There are a few more memory debuggers, but thye get a bit tricky and I can't comment on their success-rates. -Matt
Eric -- Eric Hallman Tech-X Corporation hallman@txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 254-5833 Boulder, CO 80303 Fax: (303) 448-7756 --
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Matt, OK. Thanks a lot for your advice on this. This looks like a perfect place to start! Eric On May 3, 2012, at 1:44 PM, Matthew Turk wrote:
Hi Eric,
On Thu, May 3, 2012 at 1:41 PM, Eric Hallman <hallman@txcorp.com> wrote:
Is there a standard or best methodology for tracking python code performance? I see in the intertubes that one can use the time module, or timeit or other things. How about memory usage? Or is this typically handled by debugging tools?
What I really want to do is isolate logjams in python code by function call etc. in a fairly non-intrusive way.
Thanks for any advice.
If you're using yt, there are three methods you can call upon. If not, I'd recommend using the cProfile module as a start, moving on to what was kernprof and is now lineprof, and ending at Bryan O'Sullivan's recent statprof. For visualizing, "pip install jinja2 pyprof2html" will get you pyprof2html, which turns a cProfile file to HTML which is nicely viewable.
In yt, performance timers (written by Stephen) can help, but for very simple stuff:
with parallel_profile("some_prefix"): some_expensive_operation()
will output .cprof files with processor IDs in them, which can be run through pyprof2html. For memory, get_memory_usage() is a yt function that gives you the current resident size of your PID.
There are a few more memory debuggers, but thye get a bit tricky and I can't comment on their success-rates.
-Matt
Eric -- Eric Hallman Tech-X Corporation hallman@txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 254-5833 Boulder, CO 80303 Fax: (303) 448-7756 --
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Eric Hallman Tech-X Corporation hallman@txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 254-5833 Boulder, CO 80303 Fax: (303) 448-7756 --
Have you heard of RunSnakeRun? It's sort of a profiler that gives you a nice graphical interface to see time spend in modules. It doesn't do memory profiling, though… Cheers, Marcel -- +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Marcel R. Haas Postdoc, Astrophysics Space Telescope Science Institute, Baltimore, MD http://www.marcelhaas.com<http://www.marcelhaas.com/> mhaas@stsci.edu<mailto:mhaas@stsci.edu> Address: STScI 3700 San Martin Drive Baltimore, MD, 21218 Office P611, tel +1 (410) 338 5012 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ From: Eric Hallman <hallman@txcorp.com<mailto:hallman@txcorp.com>> Reply-To: Discussion of the yt analysis package <yt-users@lists.spacepope.org<mailto:yt-users@lists.spacepope.org>> Date: Thu, 3 May 2012 13:41:21 -0400 To: Discussion of the yt analysis package <yt-users@lists.spacepope.org<mailto:yt-users@lists.spacepope.org>> Subject: [yt-users] python timing Is there a standard or best methodology for tracking python code performance? I see in the intertubes that one can use the time module, or timeit or other things. How about memory usage? Or is this typically handled by debugging tools? What I really want to do is isolate logjams in python code by function call etc. in a fairly non-intrusive way. Thanks for any advice. Eric -- Eric Hallman Tech-X Corporation hallman@txcorp.com<mailto:kruger@txcorp.com> 5621 Arapahoe Ave, Suite A Phone: (720) 254-5833 Boulder, CO 80303 Fax: (303) 448-7756 -- _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org<mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (3)
-
Eric Hallman
-
Marcel Haas
-
Matthew Turk