[IronPython] parallel importing

[Dino Viehland]

Attached is a unified diff of the changes (supposedly equivalent to diff -u).  Let me know if some other diff format would be better.

-----Original Message-----
From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kamil Dworakowski
Sent: Monday, October 13, 2008 9:42 AM
To: Discussion of IronPython
Subject: Re: [IronPython] parallel importing

Could you provide a patch?

On Mon, Oct 13, 2008 at 5:31 PM, Dino Viehland <dinov at microsoft.com> wrote:
> Making the code changes is easy - the hard part is really doing a new 1.x release and all of the sign off work that entails.  We haven't ruled it out but we sure would like to avoid it if possible.
>
> -----Original Message-----
> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kamil Dworakowski
> Sent: Monday, October 13, 2008 9:07 AM
> To: Discussion of IronPython
> Subject: Re: [IronPython] parallel importing
>
> Would that be easy to backport fix for #2 to 1.x branch?
>
> On Mon, Oct 13, 2008 at 5:00 PM, Dino Viehland <dinov at microsoft.com> wrote:
>> This is all still on 1.x, right?  It looks like #1 is fixed in 2.0 (we are locking but on the wrong object in 1.x).
>>
>> #2 is still broken in 2.x though as well.
>>
>> -----Original Message-----
>> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kamil Dworakowski
>> Sent: Monday, October 13, 2008 4:45 AM
>> To: users at lists.ironpython.com
>> Subject: [IronPython] parallel importing
>>
>> Importing time is such a pain that I wanted to do it in parallel on
>> many threads. I figured that if the modules where imported in such a
>> way that:
>>
>> no two threads import the same module at the same time
>>
>> everything would be fine. To ensure that condition, it is enough to
>> build a dependency graph and import based on that.
>>
>> I did it: Resolver One start up time improved by 20% on a two-core
>> machine. But it crashes, surprisingly on single core machines it is
>> more often (6 crashes on 200 starts).
>>
>> So far I have identified two causes for crashes:
>>
>> 1. One thread imports a module with class B inside while another is
>> importing a module with class C inside. If B and C are subclasses of
>> A, it can result in IndexOutOfRangeException being raised, when, under
>> the hood, IronPython.Runtime.Types.DynamicType.AddSubclass is being
>> executed.
>>
>> 2. Attributes on .NET modules are loaded lazily, so importing
>> namespaces only is not enough. Attribute getting from reflected
>> packages is not thread safe. Looks like I would have to import every
>> class explicitly (would that be enough?).
>>
>> Second cause would be pretty easy to address, but I'm not so sure
>> about the first one. Are there any more potential points of problems?
>> I am beginning to think I was to optimistic about all of this
>> importing on multiple cores, but if these are the only ones it could
>> probably be still fixed.
>>
>> If anyone is interested the code for it is on github:
>> http://github.com/luntain/ipy-parallel-import.
>>
>> --
>> Kamil Dworakowski
>> Resolver Systems Ltd.
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
_______________________________________________
Users mailing list
Users at lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: diff.txt
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20081013/fb56ff7d/attachment.txt>