Hi Victor,
On 2 November 2016 at 16:53, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> 2016-11-02 15:20 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>:
>> Is that really the kind of examples you want to put forward?
>
> I am not a big fan of timeit, but we must use it sometimes to
> micro-optimizations in CPython to check if an optimize really makes
> CPython faster or not. I am only trying to enhance timeit.
> Understanding results require to understand how the statements are
> executed.
Don't get me wrong, I understand the point of the following usage of timeit:
python2 -m perf timeit '[1,2]*1000' --duplicate=1000
What I'm criticizing here is this instead:
python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy
because you're very unlikely to get any relevant information from such
a comparison. I stand by my original remark: I would say it should be
an error or at least a big fat warning to use --duplicate and PyPy in
the same invocation. This is as opposed to silently ignoring
--duplicate for PyPy, which is just adding more confusion imho.
A bientôt,
Armin.
2016-11-02 15:20 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>:
> Is that really the kind of examples you want to put forward?
I am not a big fan of timeit, but we must use it sometimes to
micro-optimizations in CPython to check if an optimize really makes
CPython faster or not. I am only trying to enhance timeit.
Understanding results require to understand how the statements are
executed.
> This example means "compare CPython where the data cache gets extra pressure from reading a strangely large code object,
I wrote --duplicate option to benchmark "x+y" with "x=1; y=2". I know,
it's an extreme and stupid benchmark, but many people spend a lot of
time on trying to optimize this in Python/ceval.c:
https://bugs.python.org/issue21955
I tried multiple values of --duplicate when benchmarking x+y, and x+y
seems "faster" when using a larger --duplicate value. I understand
that the cost of the outer loop is higher than the cost of "reading a
strangely large code object".
I provide a tool and I try to document how to use it. But it's hard to
prevent users to use it for stupid things.
For example, recently I spent time trying to optimize bytes%args in
Python 3 after reading an article, but then I realized that the Python
2 benchmark was meaningless:
https://atleastfornow.net/blog/not-all-bytes/
def bytes_plus():
b"hi" + b" " + b"there"
... benchmark(bytes_plus) ...
bytes_plus() is optimized by the _compiler_, so the benchmark measure
the cost of LOAD_CONST :-)
The issue was not the tool but the usage of the tool :-D
Victor
Hi Victor,
On 2 November 2016 at 13:00, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> On CPython, --duplicate makes sense (no?). So for example, the
> following command should use duplicate on CPython but not on PyPy:
>
> python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy
This example means "compare CPython where the data cache gets extra
pressure from reading a strangely large code object, and PyPy where
the multiplication might be entirely removed for all I know".
Is that really the kind of examples you want to put forward?
A bientôt,
Armin.
Hum, so for an usability point of view, I think that the best to do is
to ignore the option if Python has a JIT.
On CPython, --duplicate makes sense (no?). So for example, the
following command should use duplicate on CPython but not on PyPy:
python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy
Victor
Hi Victor,
On 2 November 2016 at 11:50, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> 2016-11-02 11:04 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>:
>> On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote:
>>> 3) new --duplication option to perf timeit
>>
>> This is never a good idea on top of PyPy, so I wouldn't mind if using
>> this option on top of PyPy threw an error.
>
> Can you please elaborate?
Yes, exactly :-) Consider a benchmark written like that:
for i in range(lots):
z = a + b
z = a + b
z = a + b
z = a + b
z = a + b
What you are really measuring by running PyPy on this is completely
different from what you *think* you are measuring---in this case,
mostly everything is optimized away. If you try to make it actually
do something so that it's not optimized away, then the problem of
duplicating lines becomes of making the tracing JIT compiler not happy
at all. If you duplicate the lines too many times, the loop body
becomes too long for the JIT compiler to swallow---never duplicates
1000 times, that's always too much! But even if you duplicate only 10
times, then the more subtle problem is: assume that each line can
follow *two* control flow paths (even internally, e.g. because of some
condition done in RPython). (It is likely the case, if you try to do
something non-trivially-optimizable-away.) Then if you duplicate the
line 10 times, there are suddenly 2**10 control flow paths. That
means the JIT will never be able to warm up completely. Suddenly you
are measuring the JIT compiler's performance and not at all your
code's.
The --duplication option on PyPy is thus either useless or limited to
use cases where you definitely know there is only one code path ever
followed, and don't duplicate too much, and know for sure that
multiple repetitions of the same line won't cause cross-line
optimizations. That's not possible to explain without going very
technical.
A bientôt,
Armin.
2016-11-02 11:04 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>:
> On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote:
>> 3) new --duplication option to perf timeit
>
> This is never a good idea on top of PyPy, so I wouldn't mind if using
> this option on top of PyPy threw an error.
Can you please elaborate?
Victor
Hi Victor,
On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> 3) new --duplication option to perf timeit
This is never a good idea on top of PyPy, so I wouldn't mind if using
this option on top of PyPy threw an error.
A bientôt,
Armin.
Hi,
Thanks first for that, it's very interesting.
About to enrich benchmark suite, I might have a suggestion: We might add
REST/JSON scenarios, because a lot of people use Python for that.
It isn't certainly not the best REST/JSON scenarios, because they have a
small payload, but better than nothing:
https://www.techempower.com/benchmarks/#section=code&hw=peak&test=fortune
Moreover, we already have several implementations for the most populars Web
frameworks:
https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks...
The drawback is that a lot of tests need a database.
I can help if you're interested in.
Have a nice week.
