[Numpy-discussion] Performance testing in unit tests

[Robert Kern]

On Fri, Aug 28, 2009 at 00:44, Gael
Varoquaux<gael.varoquaux at normalesup.org> wrote:
> On Thu, Aug 27, 2009 at 03:33:30PM -0700, Robert Kern wrote:
>> From my experience, doing performance tests inside of your normal test
>> suite is entirely unreliable. Performance testing requires rigorous
>> control over external factors that you cannot do inside of your test
>> suite. Your tests will fail when run in the entire test suite and pass
>> when run by themselves, or vice versa.
>
>> It can also be hard to run two similar tests serially thanks to any
>> number of caches that might be in effect, but this is often
>> manageable.
>
> That is why it can be useful to repeat the measure several times, isn't
> it?

If warm-cache performance is what you want to measure, yes.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco