unit-profiling, similar to unit-testing

[Roy Smith]

In article <mailman.2810.1321562763.27778.python-list at python.org>,
 Tycho Andersen <tycho at tycho.ws> wrote:

> While I agree there's a lot of things you can't control for, you can
> get a more accurate picture by using CPU time instead of wall time
> (e.g. the clock() system call). If what you care about is mostly CPU
> time [...]

That's a big if.  In some cases, CPU time is important, but more often, 
wall-clock time is more critical.  Let's say I've got two versions of a 
program.  Here's some results for my test run:

Version     CPU Time     Wall-Clock Time
   1         2 hours       2.5 hours
   2         1.5 hours     5.0 hours

Between versions, I reduced the CPU time to complete the given task, but 
increased the wall clock time.  Perhaps I doubled the size of some hash 
table.  Now I get a lot fewer hash collisions (so I spend less CPU time 
re-hashing), but my memory usage went up so I'm paging a lot and my 
locality of reference went down so my main memory cache hit rate is 
worse.

Which is better?  I think most people would say version 1 is better.

CPU time is only important in a situation where the system is CPU bound.  
In many real-life cases, that's not at all true.  Things can be memory 
bound.  Or I/O bound (which, when you consider paging, is often the same 
thing as memory bound).  Or lock-contention bound.

Before you starting measuring things, it's usually a good idea to know 
what you want to measure, and why :-)