[IronPython] Slow Performance of CPython libs?

[Jon Cooper]

I like the ANTS Profiler, and there's also one in VS2005 Team that is supposed to be okay.

From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Birsch
Sent: Thursday, February 21, 2008 7:36 AM
To: Discussion of IronPython
Subject: Re: [IronPython] Slow Performance of CPython libs?

Thanks Michael and Dino.

I'll prof and send update. Got a good profiler recommendation for .Net?
Meanwhile I noticed the sample site below causes BeautifulSoup to generate quite a few [python] exceptions during __init__. Does IronPython handle exceptions significantly slower than CPtyhon?

Repro code is simple (just build a BeautifulSoup obj with mininova's home page).
Here are the .py and .cs I used to time the diffs:

bstest.py:
#Bypass CPython default socket implementation with IPCE/FePy
import imp, os, sys
sys.modules['socket'] = module = imp.new_module('socket')
execfile('socket.py', module.__dict__)

from BeautifulSoup import BeautifulSoup
from urllib import urlopen
import datetime

def getContent(url):
    #Download html data
    startTime = datetime.datetime.now()
    print "Getting url", url
    html = urlopen(url).read()
    print "Time taken:", datetime.datetime.now() - startTime

    #Make soup
    startTime = datetime.datetime.now()
    print "Making soup..."
    soup = BeautifulSoup(markup=html)
    print "Time taken:", datetime.datetime.now() - startTime

if __name__ == "__main__":
    print getContent("www.mininova.org<http://www.mininova.org>")


C#:
using System;
using System.Collections.Generic;
using System.Text;
using IronPython.Hosting;

namespace IronPythonBeautifulSoupTest
{
    public class Program
    {
        public static void Main(string[] args)
        {
            //Init
            System.Console.WriteLine("Starting...");
            DateTime start = DateTime.Now;
            PythonEngine engine = new PythonEngine();

            //Add paths:
            //BeautifulSoup.py, socket.py, bstest.py located on exe dir
            engine.AddToPath(@".");
            //CPython Lib (replace with your own)
            engine.AddToPath(@"D:\Dev\Python\Lib");

            //Import and load
            TimeSpan span = DateTime.Now - start;
            System.Console.WriteLine("[1] Import: " + span.TotalSeconds);
            DateTime d = DateTime.Now;
            engine.ExecuteFile(@"bstest.py");
            span = DateTime.Now - d;
            System.Console.WriteLine("[2] Load: " + span.TotalSeconds);

            //Execute
            d = DateTime.Now;
            engine.Execute("getContent(\"http://www.mininova.org\<http://www.mininova.org/>")");
            span = DateTime.Now - d;
            System.Console.WriteLine("[3] Execute: " + span.TotalSeconds);
            span = DateTime.Now - start;
            System.Console.WriteLine("Total: " + span.TotalSeconds);
        }
    }
}


On Wed, Feb 20, 2008 at 6:57 PM, Dino Viehland <dinov at exchange.microsoft.com<mailto:dinov at exchange.microsoft.com>> wrote:
We've actually had this issue reported once before a long time ago - it's a very low CodePlex ID - http://www.codeplex.com/IronPython/WorkItem/View.aspx?WorkItemId=651

We haven't had a chance to investigate the end-to-end scenario.  If someone could come up with a smaller simpler repro that'd be great.  Otherwise we haven't forgotten about it we've just had more immediately pressing issues to work on :(.

-----Original Message-----
From: users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com> [mailto:users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com>] On Behalf Of Michael Foord
Sent: Wednesday, February 20, 2008 5:20 AM
To: Discussion of IronPython
Subject: Re: [IronPython] Slow Performance of CPython libs?

Birsch wrote:
> Hi - We've been using IronPython successfully to allow extensibility
> of our application.
>
> Overall we are happy with the performance, with the exception of
> BeautifulSoup which seems to run very slowly: x5 or more time to
> execute compared to CPython.
>
> Most of the time seems to be spent during __init__() of BS, where the
> markup is parsed.
>
> We suspect this has to do with the fact that our CPython env is
> executing .pyc files and can precompile its libs, while the IronPython
> environment compiles each iteration. We couldn't find a way to
> pre-compile the libs and then introduce them into the code, but in any
> case this will result in a large management overhead since the amount
> of CPython libs we expose to our users contains 100's of modules.
>
> Any ideas on how to optimize?

I think it is worth doing real profiling to find out where the time is
being spent during parsing.

If it is spending most of the time in '__init__' then the time is
probably not spent in importing - so compilation isn't relevant and it
is a runtime performance issue. (Importing is much slower with
IronPython and at Resolver Systems we do use precompiled binaries - but
strangely enough it doesn't provide much of a performance gain.)

Michael
http://www.manning.com/foord

>
> Thanks,
> -Birsch
>
> Note: we're using FePy/IPCE libs with regular IP v1.1.1 runtime DLLs
> (this was done to overcome library incompatibilities and network
> errors). However, the relevant slow .py code (mainly SGMLParser and
> BeautifulSoup) is the same.
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com<mailto:Users at lists.ironpython.com>
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>

_______________________________________________
Users mailing list
Users at lists.ironpython.com<mailto:Users at lists.ironpython.com>
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
_______________________________________________
Users mailing list
Users at lists.ironpython.com<mailto:Users at lists.ironpython.com>
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com


________________________________
Arrowgrass Capital Partners (US) LP is a limited partnership registered in the State of Delaware.

This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any transmission errors. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080221/83593b50/attachment.html>