[IronPython] parallel importing

[Kamil Dworakowski]

Would that be easy to backport fix for #2 to 1.x branch?

On Mon, Oct 13, 2008 at 5:00 PM, Dino Viehland <dinov at microsoft.com> wrote:
> This is all still on 1.x, right?  It looks like #1 is fixed in 2.0 (we are locking but on the wrong object in 1.x).
>
> #2 is still broken in 2.x though as well.
>
> -----Original Message-----
> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kamil Dworakowski
> Sent: Monday, October 13, 2008 4:45 AM
> To: users at lists.ironpython.com
> Subject: [IronPython] parallel importing
>
> Importing time is such a pain that I wanted to do it in parallel on
> many threads. I figured that if the modules where imported in such a
> way that:
>
> no two threads import the same module at the same time
>
> everything would be fine. To ensure that condition, it is enough to
> build a dependency graph and import based on that.
>
> I did it: Resolver One start up time improved by 20% on a two-core
> machine. But it crashes, surprisingly on single core machines it is
> more often (6 crashes on 200 starts).
>
> So far I have identified two causes for crashes:
>
> 1. One thread imports a module with class B inside while another is
> importing a module with class C inside. If B and C are subclasses of
> A, it can result in IndexOutOfRangeException being raised, when, under
> the hood, IronPython.Runtime.Types.DynamicType.AddSubclass is being
> executed.
>
> 2. Attributes on .NET modules are loaded lazily, so importing
> namespaces only is not enough. Attribute getting from reflected
> packages is not thread safe. Looks like I would have to import every
> class explicitly (would that be enough?).
>
> Second cause would be pretty easy to address, but I'm not so sure
> about the first one. Are there any more potential points of problems?
> I am beginning to think I was to optimistic about all of this
> importing on multiple cores, but if these are the only ones it could
> probably be still fixed.
>
> If anyone is interested the code for it is on github:
> http://github.com/luntain/ipy-parallel-import.
>
> --
> Kamil Dworakowski
> Resolver Systems Ltd.
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>