[IronPython] parallel importing

[Dino Viehland]

Speeding up code gen is certainly an issue that has been brought up internally - but I don't think there's anything concrete there yet.  Anyways that'll require a .NET framework update so it's a good long term solution but not so good for the short term.  The CLR's solution to this in the past has been ngen hence the reason we're leveraging that for the short term.

From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Vernon Cole
Sent: Monday, October 13, 2008 9:58 AM
To: Discussion of IronPython
Subject: Re: [IronPython] parallel importing

Help me out here.  I have seen many references to schemes to optimize import speeds on Iron Python. Isn't this an attempt to make up for the snail-like startup speeds suffered by all .net applications? It seems to me that when Iron Python finally starts running, it imports and performs very well. The slowdown is in getting the .net engine started. Try comparing startup times for "Hello World" in C Python and Iron.
   It is my experience that I can identify .net programs in general by the fact that I have double-clicked its desktop icon three times (thinking I had missed it somehow) before the first instance displays its splash screen, and then the next two display an "already running" dialog.
Any chance of getting the .net people to help out with this?
--
Vernon Cole
On Mon, Oct 13, 2008 at 10:00 AM, Dino Viehland <dinov at microsoft.com<mailto:dinov at microsoft.com>> wrote:
This is all still on 1.x, right?  It looks like #1 is fixed in 2.0 (we are locking but on the wrong object in 1.x).

#2 is still broken in 2.x though as well.

-----Original Message-----
From: users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com> [mailto:users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com>] On Behalf Of Kamil Dworakowski
Sent: Monday, October 13, 2008 4:45 AM
To: users at lists.ironpython.com<mailto:users at lists.ironpython.com>
Subject: [IronPython] parallel importing

Importing time is such a pain that I wanted to do it in parallel on
many threads. I figured that if the modules where imported in such a
way that:

no two threads import the same module at the same time

everything would be fine. To ensure that condition, it is enough to
build a dependency graph and import based on that.

I did it: Resolver One start up time improved by 20% on a two-core
machine. But it crashes, surprisingly on single core machines it is
more often (6 crashes on 200 starts).

So far I have identified two causes for crashes:

1. One thread imports a module with class B inside while another is
importing a module with class C inside. If B and C are subclasses of
A, it can result in IndexOutOfRangeException being raised, when, under
the hood, IronPython.Runtime.Types.DynamicType.AddSubclass is being
executed.

2. Attributes on .NET modules are loaded lazily, so importing
namespaces only is not enough. Attribute getting from reflected
packages is not thread safe. Looks like I would have to import every
class explicitly (would that be enough?).

Second cause would be pretty easy to address, but I'm not so sure
about the first one. Are there any more potential points of problems?
I am beginning to think I was to optimistic about all of this
importing on multiple cores, but if these are the only ones it could
probably be still fixed.

If anyone is interested the code for it is on github:
http://github.com/luntain/ipy-parallel-import.

--
Kamil Dworakowski
Resolver Systems Ltd.
_______________________________________________
Users mailing list
Users at lists.ironpython.com<mailto:Users at lists.ironpython.com>
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
_______________________________________________
Users mailing list
Users at lists.ironpython.com<mailto:Users at lists.ironpython.com>
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20081013/d477940f/attachment.html>