On Sun, Sep 30, 2012 at 8:07 PM, Alexandre Vassalotti

wrote:

On Sun, Sep 30, 2012 at 4:50 PM, Brett Cannon wrote:

...

I accidentally left out the telco benchmark, which is bad since cdecimal makes it just scream on Python 3.3 (and I verified with Python 3.2 that this is an actual speedup and not some silly screw-up like I initially had with spectral_norm):

You could also make the pickle benchmark use the C accelerator module by passing the --use_cpickle flag. The Python 3 version should be a lot faster.

perf.py already uses --use_cpickle: Running fastpickle... INFO:root:Running ../cpython/builds/3.3/bin/python3.3 performance/bm_pickle.py -n 5 --use_cpickle pickle INFO:root:Running ../cpython/builds/2.7-wide/bin/python performance/bm_pickle.py -n 5 --use_cpickle pickle One thing that might make a change is using -1 for the protocol instead of 2, but that means losing the perk of perf.py doing all of the calculations, etc.

On Sun, Sep 30, 2012 at 8:14 PM, Gregory P. Smith wrote:

Interesting results!

Another data point for the benchmarks that would be interesting is memory consumption of the python process during the runs.

In 3.3 a reasonable place to gather this would be to add a callback to the new gc.callbacks and save a snapshot of the process's memory usage before every collection to gather peak, average and median usage over the life of the process. 2.7 doesn't have this feature but there is a backport of this to 2.7 in the bugtracker.

I guess I should join speed@ :)

There is already support in perf.py to track memory: -m, --track_memory Track memory usage. This only works on Linux.

On Sun, Sep 30, 2012 at 8:28 PM, Antoine Pitrou wrote:

On Sun, 30 Sep 2012 19:12:47 -0400 Brett Cannon wrote:

...

### mako_v2 ### Min: 0.137584 -> 0.287701: 2.09x slower Avg: 0.140620 -> 0.293204: 2.09x slower Significant (t=-296.14) Stddev: 0.00243 -> 0.00272: 1.1195x larger

Note that Mako can use the Markupsafe library for faster operation. This will skew the result if one of your Pythons has Markupsafe installed and the other does not.

Should probably have the benchmark print out a warning when markupsafe is used. Turns out I have it installed in my user directory for Python 2.7 so that probably came into play.

Perhaps the benchmark runner should launch its subtests in a controlled environment to avoid such issues?

If we had venv in Python 2.7 that might be easy to do, but otherwise is there an easy way without having to try to pull in virtualenv or something crazy like a chroot or something? -Brett (shipping a copy of Markupsafe would not be very practical, because it

has a C extension)

Regards

Antoine.

-- Software development and contracting: http://pro.pitrou.net

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

On Mon, Oct 1, 2012 at 7:54 AM, Antoine Pitrou wrote:

On Sun, 30 Sep 2012 21:49:20 -0400 Brett Cannon wrote:

...

...

Note that Mako can use the Markupsafe library for faster operation. This will skew the result if one of your Pythons has Markupsafe installed and the other does not.

Should probably have the benchmark print out a warning when markupsafe is used. Turns out I have it installed in my user directory for Python 2.7 so that probably came into play.

...

Perhaps the benchmark runner should launch its subtests in a controlled environment to avoid such issues?

If we had venv in Python 2.7 that might be easy to do, but otherwise is there an easy way without having to try to pull in virtualenv or something crazy like a chroot or something?

The mako benchmark could manually exclude markupsafe from sys.modules. That only addresses that specific benchmark, though)

Good point. Might be a good short term fix but it would be nice to have a solution to prevent similar issues in the future.

On Sun, Sep 30, 2012 at 9:35 PM, Steven D'Aprano wrote:

On Sun, Sep 30, 2012 at 07:12:47PM -0400, Brett Cannon wrote:

...

...

python3 perf.py -T --basedir ../benchmarks -f -b py3k ../cpython/builds/2.7-wide/bin/python ../cpython/builds/3.3/bin/python3.3

...

### call_method ### Min: 0.491433 -> 0.414841: 1.18x faster Avg: 0.493640 -> 0.416564: 1.19x faster Significant (t=127.21) Stddev: 0.00170 -> 0.00162: 1.0513x smaller

I'm not sure if this is the right place to discuss this,

The speed mailing list would be best.

but what is the justification for recording the average and std deviation of the benchmarks?

Because the tests, when run in a more rigorous fashion, run many more iterations so the average is used to even out bumps thanks to executing, e.g. 50 times. And the stddev is there to know how variable the results were in the end.

If the benchmarks are based on timeit, the timeit docs warn against taking any statistic other than the minimum.

They don't use timeit.

On Mon, Oct 1, 2012 at 7:07 AM, Robert Collins wrote:

On Mon, Oct 1, 2012 at 2:35 PM, Steven D'Aprano wrote:

...

On Sun, Sep 30, 2012 at 07:12:47PM -0400, Brett Cannon wrote:

...

...

python3 perf.py -T --basedir ../benchmarks -f -b py3k ../cpython/builds/2.7-wide/bin/python ../cpython/builds/3.3/bin/python3.3

...

### call_method ### Min: 0.491433 -> 0.414841: 1.18x faster Avg: 0.493640 -> 0.416564: 1.19x faster Significant (t=127.21) Stddev: 0.00170 -> 0.00162: 1.0513x smaller

I'm not sure if this is the right place to discuss this, but what is the justification for recording the average and std deviation of the benchmarks?

If the benchmarks are based on timeit, the timeit docs warn against taking any statistic other than the minimum.

Also because timeit is wrong to give that recommendation.

There are factors - such as garbage collection - that affect operations on average, even though they may not kick in in every run. If you want to know how something will perform as part of a larger system, taking the best possible and extrapolating from it is a mistake. As a concrete example, consider an algorithm that generates cycles with several hundred MB of memory in them. Best case the RAM is available, nothing swaps, and gc doesn't kick in during the algorithm's execution. However, the larger program has to deal with those several hundred MB of memory sitting around until gc *does* kick in, has to pay the price of a gc run over a large heap, and deal with the impact on disk read cache. When you do enough runs to see those effects *that will affect the whole program* kick in, then you can extrapolate from that basis. e.g. the question timeit optimises itself to answer isn't the question most folk need most of the time.

-Rob _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

Timeit disables the GC for good measure (which is very bad IMO, but it was already discussed)

On Mon, Oct 1, 2012 at 3:44 AM, Maciej Fijalkowski wrote:

...

I am presenting the talk "Python 3.3: Trust Me, It's Better Than 2.7" as PyCon Argentina and Brasil (and US if they accept the talk). As part of

On Mon, Oct 1, 2012 at 1:12 AM, Brett Cannon wrote: that

...

talk I need to be able to benchmark Python 3.3 against 2.7 (both from tip) using the unladen benchmarks (which now include benchmarks from PyPy that can be relatively easily ported to Python 3).

Hi Brett.

*If* you're talking about benchmarks, would be cool if you mention that pypy is actually much faster on most of them.

I will definitely mention that PyPy is actively working on Python 3 support and people should help out where they can (whether it be technical or financial) since PyPy will be faster than CPython in this regard and if you needed a good chance to switch interpreters this would be it. BTW, now that 3.3 is out is Antonio going to aim for 3.3 compatibility for the initial release or stay back on 3.2?

Also a very sad fact is that a lot of actually interesting benchmarks don't work on py3k (although a growing number). Twisted and sympy are very informative for example

As soon as those projects are ported we can obviously add those benchmarks.