[Python-Dev] performance testing recommendations in devguide

Maciej Fijalkowski fijall at gmail.com
Wed May 29 22:04:34 CEST 2013

On Wed, May 29, 2013 at 9:19 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Hi,
> On Wed, 29 May 2013 21:59:21 +0300
> Carlos Nepomuceno <carlosnepomuceno at outlook.com> wrote:
>> [1] "pybench - run the standard Python PyBench benchmark suite. This is considered
>> an unreliable, unrepresentative benchmark; do not base decisions
>> off it. It is included only for completeness."
> "unrepresentative" is the main criticism against pybench. PyBench is a
> suite of micro-benchmarks (almost nano-benchmarks, actually :-)) that
> don't try to simulate any real-world situation.
> PyBench may also be unreliable, because its tests are so static that
> they could be optimized away by a clever enough (JIT) compiler.
> Regards
> Antoine.

For what is worth PyBench is bad because it's micro-only. A lot of
stuff only shows up in larger examples, especially on an optimizing
compiler. The proposed list contains also only micro-benchmarks, which
will have the exact same problem as pybench.

More information about the Python-Dev mailing list