[IronPython] pybench results for CPython and IronPython
Michael Foord
fuzzyman at voidspace.org.uk
Sun Apr 22 16:19:19 CEST 2007
Michael Foord wrote:
> Jim Hugunin wrote:
>
>> I'd like to point out one fact about these benchmarks that seems to have been missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation. The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.
>>
>>
> And on close examination the results are impressive. Nice one.
>
> The anomaly with raising exceptions is very weird though (IronPython is
> around 30 times slower!) :
>
> TryExcept: 7ms 100ms -93.4% 11ms 102ms -89.4%
> TryRaiseExcept: 3413ms 114ms +2896.2% 3453ms 115ms +2910.6%
>
Sorry for all the emails - I see that you addressed this in your
comments Jim.
Michael
> All the best,
>
> Michael Foord
>
>
>
>> I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now. I compared with CPython-2.5.1 (the latest version).
>>
>> Running on .NET, I find that IronPython is faster on some tests and CPython is faster on others. We know that we still have performance work to do. After all, IronPython just reached its 1.1 release and almost all of our recent work has been about compatibility and completeness. However, the story is already quite interesting. Out of the 51 tests in pybench, CPython is more than 2x faster on 10 of the tests and IronPython is more than 2x faster on 9 of the tests. Depending on what your code does, either implementation could run faster.
>>
>> The most interesting cases to me are the 5 tests where CPython is more than 3x faster than IronPython and the other 5 tests where IronPython is more than 3x faster than CPython. CPython's strongest performance is in dictionaries with integer and string keys, list slicing, small tuples and code that actually throws and catches exceptions. IronPython's strongest performance is in calling builtin functions, if/then/else blocks, calling python functions, deep recursion, and try/except blocks that don't actually catch an exception.
>>
>> The places where IronPython is performing strongest are where it can use the .NET code generation and JIT optimization most effectively. The places where it is slowest are mainly in runtime library implementation of core datatypes. CPython's list and dictionary datatypes are written in C and have been hand-tuned for Python-style workloads over the past 10 years. IronPython's datatypes are much newer and clearly need more tuning as the implementation matures.
>>
>> The only two tests that really stand out as showing a deep performance issue are the two exception handling ones. These are the only tests where either implementation is more than 4x faster than the other. IronPython is 10x faster on the try/catch without an exception and CPython is 30x faster when an exception is actually raised. This is a deliberate design decision within .NET to make code that doesn't throw exceptions run faster - even if that means slowing down code that does throw exceptions. I'm fairly confident this was the right decision - and even remember the day long ago when Guido was discussing Python's exception system and explained that he'd accept almost any slow-down to the exceptional case in return for removing a single instruction from the non-exceptional path.
>>
>> Thanks - Jim
>>
>> My results
>> -------------------------------------------------------------------------------
>> PYBENCH 2.0
>> -------------------------------------------------------------------------------
>> * using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int
>> el)]
>> * disabled garbage collection
>> * system check interval set to maximum: 2147483647
>> * using timer: time.clock
>>
>> -------------------------------------------------------------------------------
>> Benchmark: ipy11.pybench
>> -------------------------------------------------------------------------------
>>
>> Rounds: 10
>> Warp: 10
>> Timer: time.time
>>
>> Machine Details:
>> Platform ID: cli-32bit
>> Processor:
>>
>> Python:
>> Implementation: Python
>> Executable: c:\ironpython-1.1\ipy.exe
>> Version: 1.1.0
>> Compiler: X
>> Bits: 32bit
>> Build: 0 0 (#0)
>> Unicode: UCS2
>>
>>
>> -------------------------------------------------------------------------------
>> Comparing with: py25.pybench
>> -------------------------------------------------------------------------------
>>
>> Rounds: 10
>> Warp: 10
>> Timer: time.clock
>>
>> Machine Details:
>> Platform ID: Microsoft-Windows-32bit-WindowsPE
>> Processor:
>>
>> Python:
>> Implementation: Python
>> Executable: c:\python25\python.exe
>> Version: 2.5.1
>> Compiler: MSC v.1310 32 bit (Intel)
>> Bits: 32bit
>> Build: Apr 18 2007 08:51:08 (#r251:54863)
>> Unicode: UCS2
>>
>>
>> Test minimum run-time average run-time
>> this other diff this other diff
>> -------------------------------------------------------------------------------
>> BuiltinFunctionCalls: 44ms 181ms -75.7% 48ms 182ms -73.7%
>> BuiltinMethodLookup: 335ms 160ms +109.1% 344ms 162ms +112.0%
>> CompareFloats: 99ms 111ms -11.3% 105ms 112ms -6.0%
>> CompareFloatsIntegers: 63ms 125ms -49.7% 65ms 126ms -48.7%
>> CompareIntegers: 102ms 114ms -10.4% 105ms 115ms -8.9%
>> CompareInternedStrings: 175ms 126ms +39.0% 180ms 127ms +41.6%
>> CompareLongs: 111ms 105ms +5.3% 114ms 106ms +7.2%
>> CompareStrings: 167ms 126ms +32.7% 172ms 127ms +35.1%
>> CompareUnicode: 121ms 125ms -3.4% 123ms 126ms -2.0%
>> ConcatStrings: 482ms 272ms +77.2% 533ms 275ms +93.4%
>> ConcatUnicode: 305ms 216ms +41.0% 350ms 217ms +61.5%
>> CreateInstances: 102ms 148ms -31.1% 106ms 149ms -28.9%
>> CreateNewInstances: 326ms 131ms +148.0% 335ms 133ms +151.9%
>> CreateStringsWithConcat: 228ms 147ms +55.9% 238ms 148ms +61.0%
>> CreateUnicodeWithConcat: 91ms 154ms -41.2% 94ms 157ms -40.4%
>> DictCreation: 137ms 105ms +30.0% 143ms 106ms +33.9%
>> DictWithFloatKeys: 301ms 230ms +30.8% 306ms 232ms +32.0%
>> DictWithIntegerKeys: 343ms 106ms +223.0% 351ms 108ms +224.7%
>> DictWithStringKeys: 388ms 102ms +282.1% 395ms 102ms +285.1%
>> ForLoops: 39ms 95ms -59.0% 40ms 97ms -58.5%
>> IfThenElse: 35ms 111ms -68.9% 37ms 112ms -66.8%
>> ListSlicing: 468ms 155ms +201.3% 477ms 157ms +204.4%
>> NestedForLoops: 55ms 120ms -54.1% 57ms 122ms -53.0%
>> NormalClassAttribute: 276ms 127ms +117.7% 282ms 129ms +119.7%
>> NormalInstanceAttribute: 106ms 119ms -11.0% 109ms 120ms -9.7%
>> PythonFunctionCalls: 37ms 130ms -71.7% 40ms 131ms -69.5%
>> PythonMethodCalls: 127ms 158ms -19.6% 132ms 159ms -16.8%
>> Recursion: 49ms 177ms -72.5% 50ms 179ms -71.8%
>> SecondImport: 189ms 126ms +50.4% 195ms 127ms +53.6%
>> SecondPackageImport: 196ms 134ms +45.8% 202ms 136ms +48.5%
>> SecondSubmoduleImport: 266ms 177ms +50.6% 272ms 180ms +51.3%
>> SimpleComplexArithmetic: 97ms 148ms -34.4% 101ms 149ms -32.0%
>> SimpleDictManipulation: 311ms 118ms +164.4% 318ms 118ms +168.0%
>> SimpleFloatArithmetic: 64ms 119ms -46.6% 69ms 121ms -43.2%
>> SimpleIntFloatArithmetic: 41ms 100ms -59.4% 43ms 102ms -58.2%
>> SimpleIntegerArithmetic: 43ms 100ms -57.2% 44ms 101ms -56.4%
>> SimpleListManipulation: 121ms 104ms +16.6% 124ms 105ms +18.3%
>> SimpleLongArithmetic: 126ms 113ms +11.2% 131ms 115ms +14.1%
>> SmallLists: 223ms 154ms +45.0% 229ms 156ms +47.0%
>> SmallTuples: 508ms 144ms +252.2% 523ms 145ms +259.7%
>> SpecialClassAttribute: 275ms 124ms +121.2% 283ms 126ms +124.5%
>> SpecialInstanceAttribute: 105ms 210ms -50.1% 109ms 212ms -48.5%
>> StringMappings: 342ms 633ms -46.0% 351ms 637ms -44.9%
>> StringPredicates: 222ms 217ms +2.3% 228ms 219ms +4.4%
>> StringSlicing: 245ms 162ms +51.4% 260ms 163ms +59.4%
>> TryExcept: 7ms 100ms -93.4% 11ms 102ms -89.4%
>> TryRaiseExcept: 3413ms 114ms +2896.2% 3453ms 115ms +2910.6%
>> TupleSlicing: 227ms 150ms +51.6% 235ms 151ms +54.9%
>> UnicodeMappings: 227ms 126ms +79.6% 233ms 129ms +81.4%
>> UnicodePredicates: 218ms 129ms +68.8% 224ms 132ms +70.4%
>> UnicodeSlicing: 211ms 182ms +15.8% 223ms 184ms +21.3%
>> -------------------------------------------------------------------------------
>> Totals: 12781ms 7659ms +66.9% 13191ms 7741ms +70.4%
>>
>> (this=ipy11.pybench, other=py25.pybench)
>> _______________________________________________
>> users mailing list
>> users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
>>
>>
>
> _______________________________________________
> users mailing list
> users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
More information about the Ironpython-users
mailing list