[IronPython] Implementing Greenlets API in IronPython

Mon Jun 28 20:04:43 CEST 2010

I think that's a fair summary and ties into what I was saying about cross function switching.  So basically this means every method needs to be re-written to support this including dispatching calls via returning back to an outer loop and then dispatching to the called object with the new arguments (including the perf hit that would come with that).  I think that can certainly be done with a DLR tree AST re-write and replacing normal invocation with this trampoline mechanism.  Obviously it'll come with a significant perf hit as we need to allocate memory for local variables, calls with be slower as they need to return and call, etc...

But if you can get the tree re-write going to make the code entirely stackless from there you could opt into to the tree re-write dynamically.  That could be done by doing a simpler tree re-write to track where you are in the function and then throwing a .NET exception when the 1st switch occurs.  There would need to be try/catch/rethrow blocks to save all of the relevant state (which would need to be in locals) and then all of the functions on the stack would need to be re-written with the full stackless re-write.

I do agree that it sounds more difficult then what is going on with the CPython implementation.

From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Tristan Zajonc
Sent: Friday, June 25, 2010 4:45 AM
To: Discussion of IronPython
Subject: Re: [IronPython] Implementing Greenlets API in IronPython

I have been trying to hire somebody to implement the greenlet extension module, which would enable modules like eventlet and gevent to work on IronPython.  The person I currently have looking into it, however, isn't hopeful. A condensed version of his perspective is below.  Perhaps the IronPython list (Dino?) has an idea for a solution?

== Response to Greenlets on IronPython ==

I looked at various solutions but none of them seem to arrive at the correct solution. I did try generating the DLR using AST again to make the function stackless but the limitation is with python and .NET.

Let me explain the issue.
1) .NET lanugages (C#. VB.NET<http://VB.NET> etc.) are strongly typed. In other words, if I get a variable I know what its type is at compile time. IronPython actually compiles the functions into CLR and runs them on the fly. A function is compiled once and can be executed again and again. One must know that the function needs to be stackless at compile time and a function cannot be made stackless at run time.
2) Python is a typeless language. A variable can assume any type at any point in execution at run-time. Now consider the following code in Python using the greenlet API:

a = greenlet(foo)

def abc():
    print "X"
    a.switch()
    print "Y"

Although variable a is defined, it does not have a strong type and during execution, a can be assigned a different object which can also implement the switch function and the code will execute fine.

As mentioned above, IronPython needs to know at compile time that the function has to be made stackless. In python since a is not typed, a can be greenlet in one execution of abc and any other object in the second call. Also, there is no way to determine at compile that the function abc needs to be made stackless since it needs to switch since the type of a is unknown till it is actually called at runtime.

Hence is it next to impossible to implement the switch because of the above implementation.

CPython dos not compile the functions and does not use the yield keyword (which is not avilable in C/C++ anyways) to do it. It does this by using context switching which is built into python and so the API is fairly straightforward in CPython. CPython basically has the stack of all previous functions calls and saves the stack so that it can resume later from that point. It needs to store the context and uses the inherent context switching built into Python to do it.

== End Response ==

Ideas?

Tristan

On Mon, May 31, 2010 at 1:28 PM, Dino Viehland <dinov at microsoft.com<mailto:dinov at microsoft.com>> wrote:
You'll want to first look at GeneratorRewriter.cs - it's responsible for taking a DLR AST and re-writing it so that it can re-start at an arbitrary location and so that it's stackless.  There's one of these for debugging and one that's used for IronPython's generators, you can probably adapt either one.  You can get the AST to re-write from the function code object but that's not public right now (we'll make it public if anyone has a compelling scenario to do so - this could be it but you'll probably need more public surface area as well).

>From there I think you just need to worry about having to save multiple frames (which I get the impression is supported from the docs - eg f() calls g() and then g switches).  For that you'll probably want to re-write all of the dynamic expressions in the generator (these show up in several forms though based upon how we're generating code-  there's a ReduciableDynamicExpression and some PythonDynamicExpression's floating around now).  If greenlets only works across calls then you just need to intercept the ones which are using invoke binders.  If you can have a descriptor who's __get__ or __set__ switches then you'd also need to handle gets/sets/etc...  If you can do something like import a module which hasn't been loaded yet and then have the top-level module code switch then you'll need to start getting really creative :)  You'll then presumably need to change these into using some trampoline to do the actual invocation.

Hopefully that will get you started - I'm sure there'll be lots of little issues so if you have questions feel free to ask.  If you get it all working we can figure out what exactly we need to make public and how we should expose that so you can do this as an extension - but I'm sure you'll initially be using a bunch of internal APIs.

From: users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com> [mailto:users-bounces at lists.ironpython.com<mailto:users-bounces at lists.ironpython.com>] On Behalf Of Tristan Zajonc
Sent: Sunday, May 30, 2010 6:59 PM
To: Discussion of IronPython
Subject: [IronPython] Implementing Greenlets API in IronPython

Hi -

The greenlets C-extension module (http://packages.python.org/greenlet/)  provides a very basic api that is used by eventlet and gevent to provide asynchronous programming constructs.  It would be nice to have a reasonably performant version of the greenlet API in IronPython.

Before I start playing around with possibilities, is there an obvious approach to implement the greenlets API in IronPython?

Thanks
Tristan

_______________________________________________
Users mailing list
Users at lists.ironpython.com<mailto:Users at lists.ironpython.com>
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20100628/fa67475f/attachment.html>