[IronPython] Microsoft .NET vs. Mono on running IronPython

[Sanghyeon Seo]

As Jim published pybench results running on Microsoft .NET (Thanks,
Jim!), we can now compare Microsoft .NET with Mono on performance of
running IronPython. We always assumed .NET is faster than Mono, but
how do they compare in details?

As Jim's results were against CPython 2.5, I will use the result
against CPython 2.5 too. Let's assume that CPython 2.5's performance
characteristics are same on both Windows ans Linux, that is, Windows
CPython 2.5 doesn't perform (say) integer arithmetics faster than
Linux CPython 2.5. This could be wrong, for example, Visual C++ maybe
optimize better than GCC, or vice versa.

Indeed, if I compare Windows CPython 2.5 numbers and Linux CPython 2.5
numbers, Windows CPython 2.5 is ~1.2 times faster, most likely because
Jim's laptop is faster. With a single exception. StringMappings test,
calling .upper() and .lower() methods on string, is 4 times faster on
Linux. This could be glibc doing smart optimization. I decided to
exclude this test in comparisons below.

Numbers below are for minimum runtime.

Summary:

>From these tests, Mono seems to perform comparisons and exception
handlings better than .NET for running IronPython.

Mono is 6 times slower on string and unicode concatenation and 4 times
slower on slicing. It's also 3 to 4 times slower on IronPython
instance creation.

# Transcript

mono = [[168, 214], [538, 207], [81, 153], [61, 159], [105, 139],
[134, 168], [145, 123], [208, 170], [157, 155], [3626, 314], [2166,
244], [273, 194], [1260, 171], [1071, 180], [402, 192], [678, 148],
[362, 245], [467, 138], [521, 148], [69, 110], [38, 126], [1369, 141],
[108, 153], [413, 172], [143, 149], [114, 184], [190, 221], [57, 264],
[506, 163], [507, 168], [721, 210], [254, 153], [458, 148], [193,
170], [57, 125], [57, 132], [176, 139], [372, 158], [583, 162], [1452,
152], [413, 169], [143, 271], [377, 198], [1097, 179], [6, 103],
[2680, 136], [619, 161], [302, 147], [324, 176], [977, 197]]

net = [[44, 181], [335, 160], [99, 111], [63, 125], [102, 114], [175,
126], [111, 105], [167, 126], [121, 125], [482, 272], [305, 216],
[102, 148], [326, 131], [228, 147], [91, 154], [137, 105], [301, 230],
[343, 106], [388, 102], [39, 95], [35, 111], [468, 155], [55, 120],
[276, 127], [106, 119], [37, 130], [127, 158], [49, 177], [189, 126],
[196, 134], [266, 177], [97, 148], [311, 118], [64, 119], [41, 100],
[43, 100], [121, 104], [126, 113], [223, 154], [508, 144], [275, 124],
[105, 210], [222, 217], [245, 162], [7, 100], [3413, 114], [227, 150],
[227, 126], [218, 129], [211, 182]]

# Linux IronPython against Windows IronPython, adjusted by multiplying
Windows CPython against Linux CPython.
k = [float(li)/wi*wc/lc for ((li, lc), (wi, wc)) in zip(mono, net)]

# How many tests Mono run fast?
>>> len([1 for x in k if x < 1])
10

These tests are: CompareFloats, CompareFloatsIntegers,
CompareIntegers, CompareInternedStrings, CompareStrings,
DictWithStringKeys, IfThenElse, Recursion, TryExcept, TryRaiseExcept.

# And .NET?
>>> len([1 for x in k if x > 1])
40

Well, sure.

# How many tests .NET run more than two times faster?
>>> len([1 for x in k if x > 2])
21

# Three times?
>>> len([1 for x in k if x > 2])
9

These tests, embarassing for Mono are: BuiltinFunctionCalls,
ConcatStrings, ConcatUnicode, CreateInstances, CreateNewInstances,
CreateStringsWithConcat, ListSlicing, StringSlicing, UnicodeSlicing.

-- 
Seo Sanghyeon