[IronPython] Implementing Greenlets API in IronPython

Mon Jun 28 21:26:03 CEST 2010

Thanks for the comments, I'll pass this along.

On Mon, Jun 28, 2010 at 2:04 PM, Dino Viehland <dinov at microsoft.com> wrote:

>  I think that’s a fair summary and ties into what I was saying about cross
> function switching.  So basically this means every method needs to be
> re-written to support this including dispatching calls via returning back to
> an outer loop and then dispatching to the called object with the new
> arguments (including the perf hit that would come with that).  I think that
> can certainly be done with a DLR tree AST re-write and replacing normal
> invocation with this trampoline mechanism.  Obviously it’ll come with a
> significant perf hit as we need to allocate memory for local variables,
> calls with be slower as they need to return and call, etc…
>
>
>
> But if you can get the tree re-write going to make the code entirely
> stackless from there you could opt into to the tree re-write dynamically.
> That could be done by doing a simpler tree re-write to track where you are
> in the function and then throwing a .NET exception when the 1st switch
> occurs.  There would need to be try/catch/rethrow blocks to save all of the
> relevant state (which would need to be in locals) and then all of the
> functions on the stack would need to be re-written with the full stackless
> re-write.
>
>
>
> I do agree that it sounds more difficult then what is going on with the
> CPython implementation.
>
>
>
> *From:* users-bounces at lists.ironpython.com [mailto:
> users-bounces at lists.ironpython.com] *On Behalf Of *Tristan Zajonc
> *Sent:* Friday, June 25, 2010 4:45 AM
> *To:* Discussion of IronPython
> *Subject:* Re: [IronPython] Implementing Greenlets API in IronPython
>
>
>
> I have been trying to hire somebody to implement the greenlet extension
> module, which would enable modules like eventlet and gevent to work on
> IronPython.  The person I currently have looking into it, however, isn't
> hopeful. A condensed version of his perspective is below.  Perhaps the
> IronPython list (Dino?) has an idea for a solution?
>
>
>
> == Response to Greenlets on IronPython ==
>
>
>
> I looked at various solutions but none of them seem to arrive at the
> correct solution. I did try generating the DLR using AST again to make the
> function stackless but the limitation is with python and .NET.
>
>
>
> Let me explain the issue.
>
> 1) .NET lanugages (C#. VB.NET etc.) are strongly typed. In other words, if
> I get a variable I know what its type is at compile time. IronPython
> actually compiles the functions into CLR and runs them on the fly. A
> function is compiled once and can be executed again and again. One must know
> that the function needs to be stackless at compile time and a function
> cannot be made stackless at run time.
>
> 2) Python is a typeless language. A variable can assume any type at any
> point in execution at run-time. Now consider the following code in Python
> using the greenlet API:
>
> a = greenlet(foo)
>
> def abc():
>     print "X"
>     a.switch()
>     print "Y"
>
> Although variable a is defined, it does not have a strong type and during
> execution, a can be assigned a different object which can also implement the
> switch function and the code will execute fine.
>
> As mentioned above, IronPython needs to know at compile time that the
> function has to be made stackless. In python since a is not typed, a can be
> greenlet in one execution of abc and any other object in the second call.
> Also, there is no way to determine at compile that the function abc needs to
> be made stackless since it needs to switch since the type of a is unknown
> till it is actually called at runtime.
>
> Hence is it next to impossible to implement the switch because of the above
> implementation.
>
>
> CPython dos not compile the functions and does not use the yield keyword
> (which is not avilable in C/C++ anyways) to do it. It does this by
> using context switching which is built into python and so the API is
> fairly straightforward in CPython. CPython basically has the stack of all
> previous functions calls and saves the stack so that it can resume later
> from that point. It needs to store the context and uses the inherent context
> switching built into Python to do it.
>
>
>
> == End Response ==
>
>
>
> Ideas?
>
>
>
> Tristan
>
>
>
> On Mon, May 31, 2010 at 1:28 PM, Dino Viehland <dinov at microsoft.com>
> wrote:
>
> You’ll want to first look at GeneratorRewriter.cs - it’s responsible for
> taking a DLR AST and re-writing it so that it can re-start at an arbitrary
> location and so that it’s stackless.  There’s one of these for debugging and
> one that’s used for IronPython’s generators, you can probably adapt either
> one.  You can get the AST to re-write from the function code object but
> that’s not public right now (we’ll make it public if anyone has a compelling
> scenario to do so – this could be it but you’ll probably need more public
> surface area as well).
>
>
>
> From there I think you just need to worry about having to save multiple
> frames (which I get the impression is supported from the docs – eg f() calls
> g() and then g switches).  For that you’ll probably want to re-write all of
> the dynamic expressions in the generator (these show up in several forms
> though based upon how we’re generating code-  there’s a
> ReduciableDynamicExpression and some PythonDynamicExpression’s floating
> around now).  If greenlets only works across calls then you just need to
> intercept the ones which are using invoke binders.  If you can have a
> descriptor who’s __get__ or __set__ switches then you’d also need to handle
> gets/sets/etc…  If you can do something like import a module which hasn’t
> been loaded yet and then have the top-level module code switch then you’ll
> need to start getting really creative J  You’ll then presumably need to
> change these into using some trampoline to do the actual invocation.
>
>
>
> Hopefully that will get you started – I’m sure there’ll be lots of little
> issues so if you have questions feel free to ask.  If you get it all working
> we can figure out what exactly we need to make public and how we should
> expose that so you can do this as an extension – but I’m sure you’ll
> initially be using a bunch of internal APIs.
>
>
>
>
>
> *From:* users-bounces at lists.ironpython.com [mailto:
> users-bounces at lists.ironpython.com] *On Behalf Of *Tristan Zajonc
> *Sent:* Sunday, May 30, 2010 6:59 PM
> *To:* Discussion of IronPython
> *Subject:* [IronPython] Implementing Greenlets API in IronPython
>
>
>
> Hi -
>
>
>
> The greenlets C-extension module (http://packages.python.org/greenlet/)
>  provides a very basic api that is used by eventlet and gevent to
> provide asynchronous programming constructs.  It would be nice to have a
> reasonably performant version of the greenlet API in IronPython.
>
>
>
> Before I start playing around with possibilities, is there an obvious
> approach to implement the greenlets API in IronPython?
>
>
>
> Thanks
>
> Tristan
>
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20100628/e6ac7535/attachment.html>