[IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)

Fuzzyman fuzzyman at voidspace.org.uk
Tue Jun 5 10:30:00 CEST 2007 

Shri Borde wrote:

>Michael, how does multiple engines help you when several pieces of user-code are running concurrently? It seems that if you are willing to live with some restrictions (like every thread having a single output stream), you should be able to live without multiple engines. We could make this easier by formalizing this design pattern, but you may be able to achieve your required semantics of only redirecting stdout without that.
>  
>

The users may have several user interface windows open at one time, each 
relating to a different data-set.

Although only one window is 'active',  several can be processing data at 
the same time - and executing user code as a result.

Each window has an IronPython engine associated with it, in which the 
data-sets are processed. This way output from each data-set goes to its 
window.

User code *can* spin off new threads if they want, but because they are 
executing 'inside' their engines - output goes to the right place.

If you can suggest a way we can achieve this without multiple engines, 
then we are interested.

All the best,


Michael Foord
http://www.voidspace.org.uk/ironpython/index.shtml

>We have not finalized on the single engine story yet, but we are leaning towards it at the moment.
>
>Shri Borde said:
>  
>
>>Have you looked at moving to 2.0? Dino had mentioned a workaround (of maintaining a thread-local static) which would enable you to selectively redirect console output. Given that you are running into a number of issues and that you might have a stream of such issues going forward, it will be hard for us to continually port all those fixes to 1.1.
>>    
>>
>
>Michael Foord said:
>  
>
>>That is the problem: if we have several pieces of user-code running concurrently then we wouldn't know which output stream it belonged to.
>>    
>>
>(If we only had one stream at a time we could override sys.stdout from the Python side.)
>
>-----Original Message-----
>From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland
>Sent: Wednesday, May 02, 2007 1:48 PM
>To: Discussion of IronPython
>Subject: Re: [IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)
>
>Both you and Tony bring up a very good point.  I believe that we can solve the thread problem by using .NET's ExecutionContext.  I've never actually tried that, I just know it exists, so I'll need to look into that. That will cause the context to be flown to thread pool threads, newly created threads, and other various async points within the CLR.  One reason for its existence today is for security so the CLR is pretty good at making sure this always gets flowed (there are various ways to do operations which don't flow it, but they all have "Unsafe" in their name).
>
>Based upon Tony's description of I'm starting to believe the swappable SystemStates rather than multiple PythonEngines are the right solution - especially given that CPython is using a similar technique and even more so if ExecutionContext solves the async/threading problem.
>
>Thanks for all your feedback and let me know if you have any more.  I'll take it all to our discussion about the hosting APIs next week.
>
>-----Original Message-----
>From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Foord
>Sent: Wednesday, May 02, 2007 12:19 PM
>To: Discussion of IronPython
>Subject: Re: [IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)
>
>Dino Viehland wrote:
>  
>
>>I didn't even realize posting at the top was considered a no-no on mailing list etiquette...  But indeed, I am using Outlook.  I wonder if there's an option to change that somewhere...
>>
>>    
>>
>
>I was a bit tongue in cheek - but top-posting suffers from the same
>readability issues on mailing lists as it does in usenet posts.
>
>  
>
>>Thanks for your feedback Michael.  This thread has diverged a little bit so let me know if you have any thoughts on the rest of it.
>>
>>    
>>
>It occurs to me that your suggestion won't work for us if the user
>spawns any threads of their own (which is highly likely) - unless we can
>tell what thread it was spawned from...
>
>Meanwhile I'll try and digest the rest of *this* thread...
>
>Michael
>
>
>  
>
>>-----Original Message-----
>>From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Foord
>>Sent: Wednesday, May 02, 2007 9:06 AM
>>To: Discussion of IronPython
>>Subject: Re: [IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)
>>
>>My guess is that you have to use outlook. It does seem to encourage
>>top-posting. ;-)
>>
>>Dino Viehland wrote:
>>
>>    
>>
>>>The scripts are running on multiple threads?
>>>
>>>      
>>>
>>Usually - although sometimes code is executed on the GUI thread, but in
>>this case we always know where to send the output.
>>
>>
>>    
>>
>>>The easy way to do this in v2.0 is to set console output (we no longer maintain our own output streams) to be a stream which looks at a thread static variable which is the real stream to output to.  Would that solve the entire isolation problem for you?
>>>
>>>      
>>>
>>Our current code is nice and elegant, whilst yours sounds hacky. :-p
>>
>>Actually its just a solution we didn't think of, although it isn't quite
>>as nice as running them in separate engines which we do now, and does
>>give us *some* measure of isolation. (It is slightly less likely that a
>>badly written user script will now kill the whole application, although
>>obviously still very easy.)
>>
>>
>>
>>    
>>
>>>The only problem w/ this is if user code sets sys.stdout they'll hijack all the other scripts.
>>>
>>>
>>>      
>>>
>>Which is a problem - but we can always say "don't do that then"...
>>
>>I think we still like our current use of multiple engines, and would
>>prefer to see that supported in IronPython 2.0, even with a shared
>>SystemState.
>>
>>At least you have given us an alternative without *having* to use
>>AppDomains if this isn't possible.
>>
>>Thanks
>>
>>Michael Foord
>>http://www.voidspace.org.uk/ironpython/index.shtml
>>
>>
>>    
>>
>>>-----Original Message-----
>>>From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Foord
>>>Sent: Wednesday, May 02, 2007 5:58 AM
>>>To: Discussion of IronPython
>>>Subject: Re: [IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)
>>>
>>>Michael Foord wrote:
>>>
>>>
>>>      
>>>
>>>>Dino Viehland wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>I'm not actually the one working on the engine APIs so that's the reason I've tended to be vague.  I'll talk to the people doing it and let you know what I hear.
>>>>>
>>>>>But the more info you can give us the better decision we'll be able to make.  For example what do you actually need to be isolated?  Do you need multiple system states so they get their own modules, console, etc...  do you need everything in sys isolated?  Do you need to guarantee the isolation even if .NET code is called (e.g. they could smuggled data via a static field).  If they do need some rather high level of isolation are app domains good enough?  Do you need to marshal a lot of data in/out?  Or is the effort to spin up and use app domains correctly?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>At Resolver we are currently using multiple IronPython engines. Moving
>>>>to AppDomains is a long term goal for us, but is actually quite a lot of
>>>>work (we would have *lots* of cross-domain calls and so to avoid that we
>>>>have to find an efficient way of pumping lots of data in and then out of
>>>>the app domain).
>>>>
>>>>Switching to app domains is not a high priority task for us, and in the
>>>>meantime we *can't* upgrade to IronPython 2 if it doesn't support
>>>>multiple engines.
>>>>
>>>>Isolation of engines is only a minor benefit (it is a positive side
>>>>effect - but not the reason we started using them) for us at the moment,
>>>>and an isolated system state (although nice) is not vital.
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>In fact, the reason we started using multiple engines was to divert the
>>>standard output of simultaneously running user scripts to different
>>>output windows...
>>>
>>>Michael
>>>
>>>
>>>
>>>
>>>      
>>>
>>>>All the best,
>>>>
>>>>Michael Foord
>>>>http://www.voidspace.org.uk/ironpython/index.shtml
>>>>
>>>>_______________________________________________
>>>>users mailing list
>>>>users at lists.ironpython.com
>>>>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>_______________________________________________
>>>users mailing list
>>>users at lists.ironpython.com
>>>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>_______________________________________________
>>>users mailing list
>>>users at lists.ironpython.com
>>>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>
>>>
>>>
>>>      
>>>
>>_______________________________________________
>>users mailing list
>>users at lists.ironpython.com
>>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>_______________________________________________
>>users mailing list
>>users at lists.ironpython.com
>>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
>>
>>    
>>
>
>_______________________________________________
>users mailing list
>users at lists.ironpython.com
>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>_______________________________________________
>users mailing list
>users at lists.ironpython.com
>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>_______________________________________________
>users mailing list
>users at lists.ironpython.com
>http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>  
>

More information about the Ironpython-users mailing list