[Python-Dev] performance testing recommendations in devguide

[Carlos Nepomuceno]

----------------------------------------
> Date: Wed, 29 May 2013 12:00:44 -0600
> From: ericsnowcurrently at gmail.com
> To: python-dev at python.org
> Subject: [Python-Dev] performance testing recommendations in devguide
>
> The devguide doesn't have anything on performance testing that I could
> find. We do have a number of relatively useful resources in this
> space though, like pybench and (eventually) speed.python.org. I'd
> like to add a page to the devguide on performance testing, including
> an explanation of our performance goals, how to test for them, and
> what tools are available.

Thanks Eric! I was looking for that kind of place! ;)

> Tools I'm aware of:
> * pybench (relatively limited in real-world usefulness)
> * timeit module (for quick comparisions)
> * benchmarks repo (real-world performance test suite)
> * speed.python.org (would omit for now)

Why PyBench isn't considered reliable[1]?

What do you mean by "benchmarks repo"? http://hg.python.org/benchmarks ?

> Things to test:
> * speed
> * memory (tools? tests?)
>
> Critically sensitive performance subjects
> * interpreter start-up time
> * module import overhead
> * attribute lookup overhead (including MRO traversal)
> * function call overhead
> * instance creation overhead
> * dict performance (the underlying namespace type)
> * tuple performance (packing/unpacking, integral container type)
> * string performance
>
> What would be important to say in the devguide regarding Python
> performance and testing it?

I've just discovered insertion at the end is faster than at the start of a list.
I'd like to see things like that not only in the devguide but also in the docs (http://docs.python.org/).
I found it on Dan's presentation[2] but I'm not sure it isn't in the docs somewhere.

> What would you add/subtract from the
> above?

Threading performance!

> How important is testing memory performance? How do we avoid
> performance regressions? Thanks!

Testing and making it faster! ;)

Offcourse we need a baseline (benchmarks database) to compare and check improvements.

> -eric


[1] "pybench - run the standard Python PyBench benchmark suite. This is considered
an unreliable, unrepresentative benchmark; do not base decisions
off it. It is included only for completeness."
Source: http://hg.python.org/benchmarks/file/dccd52b95a71/README.txt

[2] http://stromberg.dnsalias.org/~dstromberg/Intro-to-Python/Intro%20to%20Python%202010.pdf