[Python-Dev] Re: [Python-checkins] python/dist/src/Lib timeit.py, 1.15, 1.16

[Guido van Rossum]

> > > Make timings more consistent by temporarily disabling GC.
> > 
> > Shouldn't this be an option at best? 
> 
> I think not.  The default behavior should be the one that
> gives the most consistent timings between runs.

If the timed code's malloc behavior is predictable (and I say it
usually is!) then the GC behavior should (in theory) also be
predictable, at least if you do a full GC before starting.

But I haven't experimented with this, and from your reversal I take it
that you have.

> > What if I want to time some code
> > that expects GC to be on?
> 
> >>> Timer(stmt, 'import gc; gc.enable()'.timeit()
> 
> If desired, I can add a note to that effect in the docs.

Please do.

> However, GC is such a random event (in terms of when it occurs,
> what it affects, how much time it takes, and the state of the 
> system) that you would always want it off unless you're specifically
> measuring GC performance across two sets of code that handle GC
> differently.  

If I have a piece of code that allocates so much as to trigger GC
regularly, I'd want the GC behavior included in the timings.

> GC performance does make a difference when measuring the performance
> of a whole, memory intensive application; however, that tends to
> fall outside the scope and purpose of the module.

Usually.  That's why I'm asking for a way to change the default
behavior.

> If you really want an option, I can put it in, but why clutter
> the API with something that few people care about or understand.

Your setup suggestion is fine.

> An alternative is to stick with my original idea of running
> a gc.collect() before each timing pass.  I would have done that
> but Tim's comment left me with doubts about whether it was the 
> right thing to do.  On the surface it makes sense and empirical
> tests show that the timings do become more stable.  However, 
> trashing the cache before each run may create timings that don't
> reflect a normal operating environment (though the timeit environment
> is already anything but normal).

Here I agree with Tim.

--Guido van Rossum (home page: http://www.python.org/~guido/)