[Python-Dev] Benchmarking Python 3.3 against Python 2.7 (wide build)

[Maciej Fijalkowski]

On Mon, Oct 1, 2012 at 7:07 AM, Robert Collins
<robertc at robertcollins.net> wrote:
> On Mon, Oct 1, 2012 at 2:35 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>> On Sun, Sep 30, 2012 at 07:12:47PM -0400, Brett Cannon wrote:
>>
>>> > python3 perf.py -T --basedir ../benchmarks -f -b py3k
>>> ../cpython/builds/2.7-wide/bin/python ../cpython/builds/3.3/bin/python3.3
>>
>>> ### call_method ###
>>> Min: 0.491433 -> 0.414841: 1.18x faster
>>> Avg: 0.493640 -> 0.416564: 1.19x faster
>>> Significant (t=127.21)
>>> Stddev: 0.00170 -> 0.00162: 1.0513x smaller
>>
>> I'm not sure if this is the right place to discuss this, but what is the
>> justification for recording the average and std deviation of the
>> benchmarks?
>>
>> If the benchmarks are based on timeit, the timeit docs warn against
>> taking any statistic other than the minimum.
>
> Also because timeit is wrong to give that recommendation.
>
> There are factors - such as garbage collection - that affect
> operations on average, even though they may not kick in in every run.
> If you want to know how something will perform as part of a larger
> system, taking the best possible and extrapolating from it is a
> mistake. As a concrete example, consider an algorithm that generates
> cycles with several hundred MB of memory in them. Best case the RAM is
> available, nothing swaps, and gc doesn't kick in during the
> algorithm's execution. However, the larger program has to deal with
> those several hundred MB of memory sitting around until gc *does* kick
> in, has to pay the price of a gc run over a large heap, and deal with
> the impact on disk read cache. When you do enough runs to see those
> effects *that will affect the whole program* kick in, then you can
> extrapolate from that basis. e.g. the question timeit optimises itself
> to answer isn't the question most folk need most of the time.
>
> -Rob
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

Timeit disables the GC for good measure (which is very bad IMO, but it
was already discussed)