unladen/swallow, pyjs+pyv8, pyjs+python-spidermonkey: accessing c python modules from optimised python compilers
http://groups.google.com/group/unladen-swallow/browse_thread/thread/6f6bfac7... i thought you might like to know that discussions which i believe may be of benefit to pypy are underway. you will have heard of unladen/swallow, an effort to replace the stack machine of http://python.org with a LLVM JIT compiler. you may also have heard of the experiments to combine http://pyjs.org with http://code.google.com/p/pyv8 and also of the experiment to combine http://pyjs.org with python-spidermonkey. the "fly in the ointment" is that for "full" optimisation to occur, it is necessary to "break out" from the prison that intobject.c, longobject.c etc. make. once this prison is opened, by turning the hard-coded python types into a more flexible and dynamically-overridable architecture, you (the pypy developers) will be free to write modules that will allow interoperability between [admittedly recompiled] c-based python modules, with no effort [other than recompilation] required on the part of the users. if you believe that this is something that would be of benefit to pypy, you should consider participating in the discussion and design of the dynamic architecture which will allow _all_ the "python optimisers" to be free of the restrictions imposed by the "original" c-based python implementation. the google engineers of unladen/swallow have absolutely no qualms about making, and maintaining, a fork of the "original" http://python.org and so it will be a simple matter to tell pypy users "yes, go get the version of python unladen/swallow, recompile your c-based module and then pypy will be able to talk to your c-based module, unmodified." l.
Hi Luke, On Sun, Apr 05, 2009 at 06:10:28PM +0000, Luke Kenneth Casson Leighton wrote:
the "fly in the ointment" is that for "full" optimisation to occur, it is necessary to "break out" from the prison that intobject.c, longobject.c etc. make.
once this prison is opened, by turning the hard-coded python types into a more flexible and dynamically-overridable architecture, you (the pypy developers) will be free to write modules that will allow interoperability between [admittedly recompiled] c-based python modules, with no effort [other than recompilation] required on the part of the users.
I am not sure what the point you are trying to make is, just by reading a bit of the URL you pointed to. Maybe I should read more... :-/ I should just point out that supporting C extension modules in PyPy has been discussed, but the obvious conclusion was that you can't just recompile the ones from CPython and hope that they work (unless you do completely evil tricks, e.g. with the garbage collector). A bientot, Armin.
I am not sure what the point you are trying to make is, just by reading a bit of the URL you pointed to. Maybe I should read more... :-/
neeh - i seem to have managed a sensible explanation just earlier today on pypy-dev, and i think also that people here have been exploring modifications to CPython already.
I should just point out that supporting C extension modules in PyPy has been discussed, but the obvious conclusion was that you can't just recompile the ones from CPython and hope that they work (unless you do completely evil tricks, e.g. with the garbage collector).
*sigh* yes i had forgotten about GC - thank you for reminding me. here's the thing: the unladen/swallow team, for phase 2, are _going_ to "unbox" the CPython base types (int, long, string, dict, list etc.). that means that they are _going_ to face the evil tricks, and more. the question i invite the pypy-dev team to consider is this: given that unladen/swallow team is going to tackle these issues, does the pypy-team want to leverage that opportunity, given that it faces exactly the same issue (as does the pyjs+pyv8 experiment) ? l.
Hi, On Thu, Apr 23, 2009 at 05:34:57PM +0000, Luke Kenneth Casson Leighton wrote:
the question i invite the pypy-dev team to consider is this: given that unladen/swallow team is going to tackle these issues, does the pypy-team want to leverage that opportunity, given that it faces exactly the same issue (as does the pyjs+pyv8 experiment) ?
No. Please read a bit more about PyPy before making random comments. For example, this blog post: http://morepypy.blogspot.com/2009/03/good-news-everyone.html would tell you that we are achieving the same results as unboxing at a completely different level. We have been trying (since the start of PyPy) a different path that unladen/swallow and won't change it completely just because you mention it. Please stop discussing unladen/swallow issues on this list if they have no relationship whatsoever with PyPy issues :-) A bientot, Armin.
different level. We have been trying (since the start of PyPy) a different path that unladen/swallow and won't change it completely just because you mention it. Please stop discussing unladen/swallow issues on this list if they have no relationship whatsoever with PyPy issues :-)
okie :)
A bientot,
Armin.
2009/4/7 Armin Rigo <arigo@tunes.org>
Hi Luke,
On Sun, Apr 05, 2009 at 06:10:28PM +0000, Luke Kenneth Casson Leighton wrote:
the "fly in the ointment" is that for "full" optimisation to occur, it is necessary to "break out" from the prison that intobject.c, longobject.c etc. make.
once this prison is opened, by turning the hard-coded python types into a more flexible and dynamically-overridable architecture, you (the pypy developers) will be free to write modules that will allow interoperability between [admittedly recompiled] c-based python modules, with no effort [other than recompilation] required on the part of the users.
I am not sure what the point you are trying to make is, just by reading a bit of the URL you pointed to. Maybe I should read more... :-/
I should just point out that supporting C extension modules in PyPy has been discussed, but the obvious conclusion was that you can't just recompile the ones from CPython and hope that they work (unless you do completely evil tricks, e.g. with the garbage collector).
Well, we've achieved binary compatibility with a large proportion of the Python C-API for IronPython (including GC and GIL issues) with Ironclad. http://resolversystems.com/documentation/index.php/Ironclad It certainly *can* be done. It's a lot of work to reimplement the Python C-API though. :-) Michael
A bientot,
Armin. _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
On Thu, Apr 23, 2009 at 5:51 PM, Michael Foord <fuzzyman@gmail.com> wrote:
2009/4/7 Armin Rigo <arigo@tunes.org>
Hi Luke,
On Sun, Apr 05, 2009 at 06:10:28PM +0000, Luke Kenneth Casson Leighton wrote:
the "fly in the ointment" is that for "full" optimisation to occur, it is necessary to "break out" from the prison that intobject.c, longobject.c etc. make.
once this prison is opened, by turning the hard-coded python types into a more flexible and dynamically-overridable architecture, you (the pypy developers) will be free to write modules that will allow interoperability between [admittedly recompiled] c-based python modules, with no effort [other than recompilation] required on the part of the users.
I am not sure what the point you are trying to make is, just by reading a bit of the URL you pointed to. Maybe I should read more... :-/
I should just point out that supporting C extension modules in PyPy has been discussed, but the obvious conclusion was that you can't just recompile the ones from CPython and hope that they work (unless you do completely evil tricks, e.g. with the garbage collector).
Well, we've achieved binary compatibility with a large proportion of the Python C-API for IronPython (including GC and GIL issues) with Ironclad.
_cool_. excellent.
It certainly *can* be done. It's a lot of work to reimplement the Python C-API though. :-)
nffh. hmm - do the unladen/swallow team know what you've managed to do?
2009/4/23 Luke Kenneth Casson Leighton <lkcl@lkcl.net>
On Thu, Apr 23, 2009 at 5:51 PM, Michael Foord <fuzzyman@gmail.com> wrote:
2009/4/7 Armin Rigo <arigo@tunes.org>
Hi Luke,
On Sun, Apr 05, 2009 at 06:10:28PM +0000, Luke Kenneth Casson Leighton wrote:
the "fly in the ointment" is that for "full" optimisation to occur, it is necessary to "break out" from the prison that intobject.c, longobject.c etc. make.
once this prison is opened, by turning the hard-coded python types into a more flexible and dynamically-overridable architecture, you (the pypy developers) will be free to write modules that will allow interoperability between [admittedly recompiled] c-based python modules, with no effort [other than recompilation] required on the part of the users.
I am not sure what the point you are trying to make is, just by reading a bit of the URL you pointed to. Maybe I should read more... :-/
I should just point out that supporting C extension modules in PyPy has been discussed, but the obvious conclusion was that you can't just recompile the ones from CPython and hope that they work (unless you do completely evil tricks, e.g. with the garbage collector).
Well, we've achieved binary compatibility with a large proportion of the Python C-API for IronPython (including GC and GIL issues) with Ironclad.
_cool_. excellent.
It certainly *can* be done. It's a lot of work to reimplement the Python C-API though. :-)
nffh.
hmm - do the unladen/swallow team know what you've managed to do?
Absolutely no idea. :-) Michael -- http://www.ironpythoninaction.com/
Well, we've achieved binary compatibility with a large proportion of the Python C-API for IronPython (including GC and GIL issues) with Ironclad.
*stunned*. holy crap. correct me if i'm wrong, but it looks like ... you ... you... reimplemented the entire CPython library API, um... um... i think the word i'm looking for is... "RESPECT". you _crazy_ fools :)
2009/4/23 Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Well, we've achieved binary compatibility with a large proportion of the Python C-API for IronPython (including GC and GIL issues) with Ironclad.
*stunned*. holy crap. correct me if i'm wrong, but it looks like ... you ... you... reimplemented the entire CPython library API, um... um... i think the word i'm looking for is... "RESPECT".
you _crazy_ fools :)
Heh heh, that's an appropriate reaction - almost all the work was done by a developer called William Reade. Huge chunks of SciPy and Numpy can be used with IronPython through Ironclad. It's not (yet...) the entire API, but it is a big chunk of the Python C API. Michael Foord -- http://www.ironpythoninaction.com/
participants (3)
-
Armin Rigo -
Luke Kenneth Casson Leighton -
Michael Foord