Re: [pypy-dev] PyPy 1.1 beta release
It is also hard for people to process fractional numbers when they are thinking about speed. '2 times the speed' feels a lot easier to understand than '2.1' times the speed. And once you get to numbers less than 1, things break down altogether. If you want to tell me that something is slower, I don't expect to hear it as 'some number less than 1' times the speed. I want a very hard break at the point 0, and for you then to go about telling me how many times slower than something that something else is. For most measurements, I would be happy if nobody mentioned the words 'speed', 'faster' and 'slower' at all. What I am _really_ interested, is a measurement of time. And I have a much easier time understanding time quantities, which I am used to dealing with, than speed quantites which rarely show up in life. So while I am always a bit hazy on what 'x times the speed' really means, when you change this to 'this program runs in half the time, one quarter of the time, twice the time, or even .8 of the time' I have a much easier time of it. I'm used to measuring time, and I expect it to be linear. I'm not used to measuring speed, and I keep worrying 'is this linear'? 'is this logarithmic?' 'is this exponential?'. It is only when I get to measure the actual times taken to do some sort of task, say a benchmark, that I get any real sense of whether a change seems to be a trivial small improvement, or a colossal major one. I wonder if others feel the same way. Laura
I think that's also the reason that speed (performance) is usually measured compared to something else, in our case CPython, which would get index of 100. If this is the index of average script execution time, PyPy index is then 80 to 200, and lower is better. Regards, Gasper Zejn On Tuesday 21 April 2009 17:34:17 Laura Creighton wrote:
On 04/21/2009 08:34 AM Laura Creighton wrote:
IMHO 'speed' is distance/time physically, and time still belongs in the denominator for measures of computing performance reasonably called 'speed', e.g. mips == millions (of) instructions (executed) /second, or gigaflops, which is giga (billions) of floating point operations /second. When software gets involved on top of the hardware, we have measures like pystones/second: -- Python 2.5.1 (r251:54863, May 4 2007, 16:52:23) [GCC 4.1.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
When it comes to benchmark test performance, maybe one could have bogoteps -- bogus test executions / second -- maybe with milli or kilo etc prefixed and specific test identifiers postfixed? ;-) So cpython 2.5.1 on my laptop does 55555.6 bogoteps-pystn? Relative comparisons can then be percentages, as in cpython pystones/sec being 1500% (??) of pypy pystones/sec, or analogously for whatever specific test. Using the speed measures will depend, as in physical speed, on what question you want to answer, e.g., how long will it take to get to the beach if we average 50 mph, vs how fast do we have to average driving to get to the beach in two hours. Either way, you can't drive faster than your car will go, and that's distance/time ;-) How 'fast' is your car? How 'fast' is your computing platform? Put time in denominator ;-) Regards, Bengt Richter
On Wed, Apr 22, 2009 at 04:17, Steven H. Rogers <steve@shrogers.com> wrote:
No. 100% faster takes half the time. First, PyPy takes (up to) twice the time of CPython.
Second, If A is 100% _slower_ than B, I take twice as much as your time; but in that case, you can't infer that B takes 100% less time - percentage sentences are not reversible like that; it takes 50% less time than B, so I'd say it's 50% faster. Analogously (or for the same reason), if from an amount of 100 €, you subtract 50% and you add the 50% of the result, you get 50 + 25 = 75 €. At least, when you write "20 percent slower", you were referring to the 0.8x factor (but I still think you should have written "PyPy can be from 20% faster to 100% slower than CPython", to be absolutely clear. By linear scaling, I guess, 20% faster would mean "new runtime = 0.9 old runtime", but that doesn't make sense.
-- Paolo Giarrusso
I would have thought a natural reading of "A is 100% faster than B" is that A's speed is B's speed + 100% (of B's speed), i.e. 2x B's speed. Meaning it makes twice as much progress in the same amount of time, or equivalently takes half the time to make the same amount of progress. Slower/faster are comparisons of speed. 100% faster = +100% speed = 2x speed. 100% slower = -50% speed = 0.5x speed? That seems illogical. I guess you could say X% faster/slower means -/+X% to the time. That leads to concluding that 90% faster than 100m/s is not 190m/s, but 100m/0.1s = 1000m/s, which is very counter-intuitive to me.
I would probably interpret "PyPy is 100% slower than CPython" to mean that PyPy's runtime is CPython's runtime +100% of CPython's runtime, i.e. half the speed, but only because the alternative interpretation is that it's CPython's speed -100%, i.e. speed of 0, which doesn't make sense. But that leaves "PyPy is 90% slower than CPython" meaning either it needs 1.9 times the runtime, or 10 times the runtime, which is a huge difference to leave to the assumption that the reader interprets "slower" the same way as the writer. I think the original formulation in terms of speed is clear (but said the wrong thing). Saying something like "PyPy takes 0.8-2x the time CPython takes to run the same code" is even clearer. Using "slower" and "faster" will get misinterpreted unless you actually define how you're using them. -- Ben Mellor
Firstly I want to congratulate everyone on the PyPy project, you made real progress from pypy 1.0 to 1.1. I specially liked the cuttings on features that where not being used. The next sped would be to move some other stuff out of the pypy repository, like all of lang subtree. Great and now lets focus on the bugfixes for the best pypy release ever! (I got this meme from the cherokee releases, they are always the best cherokee release ever). About the performance measurements: On Apr 22, 2009, at 12:24 AM, Ben Mellor wrote:
+1 This is very clear and is still concise. A very long discussion for a very simple topic... -- Leonardo Santagada santagada at gmail.com
I think that's also the reason that speed (performance) is usually measured compared to something else, in our case CPython, which would get index of 100. If this is the index of average script execution time, PyPy index is then 80 to 200, and lower is better. Regards, Gasper Zejn On Tuesday 21 April 2009 17:34:17 Laura Creighton wrote:
On 04/21/2009 08:34 AM Laura Creighton wrote:
IMHO 'speed' is distance/time physically, and time still belongs in the denominator for measures of computing performance reasonably called 'speed', e.g. mips == millions (of) instructions (executed) /second, or gigaflops, which is giga (billions) of floating point operations /second. When software gets involved on top of the hardware, we have measures like pystones/second: -- Python 2.5.1 (r251:54863, May 4 2007, 16:52:23) [GCC 4.1.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
When it comes to benchmark test performance, maybe one could have bogoteps -- bogus test executions / second -- maybe with milli or kilo etc prefixed and specific test identifiers postfixed? ;-) So cpython 2.5.1 on my laptop does 55555.6 bogoteps-pystn? Relative comparisons can then be percentages, as in cpython pystones/sec being 1500% (??) of pypy pystones/sec, or analogously for whatever specific test. Using the speed measures will depend, as in physical speed, on what question you want to answer, e.g., how long will it take to get to the beach if we average 50 mph, vs how fast do we have to average driving to get to the beach in two hours. Either way, you can't drive faster than your car will go, and that's distance/time ;-) How 'fast' is your car? How 'fast' is your computing platform? Put time in denominator ;-) Regards, Bengt Richter
On Wed, Apr 22, 2009 at 04:17, Steven H. Rogers <steve@shrogers.com> wrote:
No. 100% faster takes half the time. First, PyPy takes (up to) twice the time of CPython.
Second, If A is 100% _slower_ than B, I take twice as much as your time; but in that case, you can't infer that B takes 100% less time - percentage sentences are not reversible like that; it takes 50% less time than B, so I'd say it's 50% faster. Analogously (or for the same reason), if from an amount of 100 €, you subtract 50% and you add the 50% of the result, you get 50 + 25 = 75 €. At least, when you write "20 percent slower", you were referring to the 0.8x factor (but I still think you should have written "PyPy can be from 20% faster to 100% slower than CPython", to be absolutely clear. By linear scaling, I guess, 20% faster would mean "new runtime = 0.9 old runtime", but that doesn't make sense.
-- Paolo Giarrusso
I would have thought a natural reading of "A is 100% faster than B" is that A's speed is B's speed + 100% (of B's speed), i.e. 2x B's speed. Meaning it makes twice as much progress in the same amount of time, or equivalently takes half the time to make the same amount of progress. Slower/faster are comparisons of speed. 100% faster = +100% speed = 2x speed. 100% slower = -50% speed = 0.5x speed? That seems illogical. I guess you could say X% faster/slower means -/+X% to the time. That leads to concluding that 90% faster than 100m/s is not 190m/s, but 100m/0.1s = 1000m/s, which is very counter-intuitive to me.
I would probably interpret "PyPy is 100% slower than CPython" to mean that PyPy's runtime is CPython's runtime +100% of CPython's runtime, i.e. half the speed, but only because the alternative interpretation is that it's CPython's speed -100%, i.e. speed of 0, which doesn't make sense. But that leaves "PyPy is 90% slower than CPython" meaning either it needs 1.9 times the runtime, or 10 times the runtime, which is a huge difference to leave to the assumption that the reader interprets "slower" the same way as the writer. I think the original formulation in terms of speed is clear (but said the wrong thing). Saying something like "PyPy takes 0.8-2x the time CPython takes to run the same code" is even clearer. Using "slower" and "faster" will get misinterpreted unless you actually define how you're using them. -- Ben Mellor
Firstly I want to congratulate everyone on the PyPy project, you made real progress from pypy 1.0 to 1.1. I specially liked the cuttings on features that where not being used. The next sped would be to move some other stuff out of the pypy repository, like all of lang subtree. Great and now lets focus on the bugfixes for the best pypy release ever! (I got this meme from the cherokee releases, they are always the best cherokee release ever). About the performance measurements: On Apr 22, 2009, at 12:24 AM, Ben Mellor wrote:
+1 This is very clear and is still concise. A very long discussion for a very simple topic... -- Leonardo Santagada santagada at gmail.com
participants (7)
-
Ben Mellor -
Bengt Richter -
Gasper Zejn -
Laura Creighton -
Leonardo Santagada -
Paolo Giarrusso -
Steven H. Rogers