[IronPython] Performance of IronPython 2 Beta 4 and IronPython 1

[Michael Foord]

Just for fun I also compared with CPython. The results are interesting, 
I'll turn it into a blog post of course...

Results in milliseconds with a granularity of about 15ms and so an 
accuracy of +/- ~60ms.
All testing with 10 000 000 operations unless otherwise stated.

The version of Python I compared against was Python 2.4.

Empty loop (overhead):
    IP1: 422
    IP2: 438
    Py: 3578
   
Create instance newstyle:
    IP1: 20360
    IP2:  1109
    Py:   4063
   
Create instance oldstyle:
    IP1: 3766
    IP2: 3359
    Py:  4797
   
Function call:
    IP1: 937
    IP2: 906
    Py: 3313
   
Create function: 25% slower
    IP1: 2828
    IP2: 3640
    Py:  2766
   
Define newstyle (1 000 000):
    IP1: 42047
    IP2: 20484
    Py:  23921
   
Define oldstyle (1 000 000): 33% slower
    IP1: 1781
    IP2: 2671
    Py:  2108

Comparing (== and !=):
    IP1: 278597
    IP2: 117662
    Py:   62423
   
Sets:
    IP1: 37095
    IP2: 30860
    Py:   8047

Lists (10 000): 50% slower
    IP1: 10422
    IP2: 16109
    Py:   6094

Recursion (10 000):
    IP1: 1125
    IP2: 1000
    Py:  3609
  
Sets2 (100 000): 500% slower
    IP1:  4984
    IP2: 30547
    Py:   1203

func_with_args:
    IP1: 6312
    IP2: 5906
    Py: 11250

method_with_args:
    IP1: 20594
    IP2: 11813
    Py:  14875
   
method_with_kwargs:
    IP1: 27953
    IP2: 11187
    Py:  20032
   
import: 15% slower
    IP1: 28469
    IP2: 32000
    Py:  25782

global: 20% slower
    IP1: 1047
    IP2: 1203
    Py:  4141

Exceptions (100000): 40% slower
    IP1: 4703
    IP2: 6125
    Py:   266
   
Engine execution: 8000% slower!!
    IP1: 1600
    IP2: 115002
 

Michael Foord wrote:
> Hello all,
>
> I've ported Resolver One to run on IronPython 2 Beta 4 to check for 
> any potential problems (we will only do a *proper* port once IP 2 is 
> out of beta).
>
> The basic porting was straightforward and several bugs have been fixed 
> since IP 2 B3 - many thanks to the IronPython team.
>
> The good news is that Resolver One is only 30-50% slower than Resolver 
> One on IronPython 1! (It was 300 - 400% slower on top of IP 2 B3.) 
> Resolver One is fairly heavily optimised around the performance 
> hotspots of IronPython 1, so we expect to have to do a fair bit of 
> profiling and refactoring to readjust to the performance profile of IP 2.
>
> Having said that, there are a few oddities (and the areas that slow 
> down vary tremendously depending on which spreadsheet we use to 
> benchmark it - making it fairly difficult to track down the hotspots).
>
> We have one particular phase of spreadsheet calculation that takes 
> 0.4seconds on IP1 and around 6 seconds on IP2, so I have been doing 
> some micro-benchmarking to try and identify the hotspot. I've 
> certainly found part of the problem.
>
> For those that are interested I've attached the very basic 
> microbenchmarks I've been using. The nice thing is that in *general* 
> IP2 does outperform IP1.
>
> The results that stand out in the other direction are:
>
> Using sets with custom classes (that define '__eq__', '__ne__' and 
> '__hash__') seems to be 6 times slower in IronPython 2.
>
> Adding lists together is about 50% slower.
>
> Defining functions seems to be 25% slower and defining old style 
> classes about 33% slower. (Creating instances of new style classes is 
> massively faster though - thanks!)
>
> The code I used to test sets (sets2.py) is as follows:
>
> from System import DateTime
>
> class Thing(object):
>    def __init__(self, val):
>        self.val = val
>      def __eq__(self, other):
>        return self.val == other.val
>
>    def __neq__(self):
>        return not self.__eq__(other)
>          def __hash__(self):
>        return hash(self.val)
>             def test(s):
>    a = set()
>    for i in xrange(100000):
>        a.add(Thing(i))
>        a.add(Thing(i+1))
>        Thing(i) in a
>        Thing(i+2) in a
>    return (DateTime.Now -s).TotalMilliseconds
>   s = DateTime.Now
> print test(s)
>
>
> Interestingly the time taken is exactly the same if I remove the 
> definition of '__hash__'.
>
> The full set of results below:
>
> Results in milliseconds with a granularity of about 15ms and so an 
> accuracy of +/- ~60ms.
> All testing with 10 000 000 operations unless otherwise stated.
>
> Empty loop (overhead):
>    IP1: 421.9
>    IP2: 438
>   Create instance newstyle:
>    IP1: 20360
>    IP2: 1109
>   Create instance oldstyle:
>    IP1: 3766
>    IP2: 3359
>   Function call:
>    IP1: 937
>    IP2: 906
>   Create function: 25% slower
>    IP1: 2828
>    IP2: 3640
>   Define newstyle (1 000 000):
>    IP1: 42047
>    IP2: 20484
>   Define oldstyle (1 000 000): 33% slower
>    IP1: 1781
>    IP2: 2671
>
> Comparing (== and !=):
>    IP1: 278597
>    IP2: 117662
>   Sets (with numbers):
>    IP1: 37095
>    IP2: 30860
>
> Lists (10 000): 50% slower
>    IP1: 10422
>    IP2: 16109
>
> Recursion (10 000):
>    IP1: 1125
>    IP2: 1000
>  
> Sets2 (100 000): 600% slower
>    IP1: 4984
>    IP2: 30547
>
>
> I'll be doing more as the 600% slow down for sets and the 50% slow 
> down for lists accounts for some of the dependency analysis problem 
> but not all of it.
>
> Many Thanks
>
> Michael Foord
> -- 
> http://www.resolversystems.com
> http://www.ironpythoninaction.com
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.resolverhacks.net/
http://www.theotherdelia.co.uk/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ip2tests.zip
Type: application/octet-stream
Size: 7272 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/5b871dca/attachment.obj>