Re: [pypy-dev] External RPython mailing list
Porting ShedSkin to use the PyPy translation toolchain on the surface sounds like a good idea, but its not if we look at the details. The first issue is legal, PyPy is MIT licensed, which works very well when integrated by commerical software. But Shedskin uses the GNU GPL3, so importing any of its code (or code that imports GPL code, etc) into the user's compiled program also binds it to the GPL - which is no good for commerical software. Shedskin within the PyPy toolchain may taint the users program with GPL code because some RPython programs will import from rlib (which may in someway depend on Mark's GPL code). The second issue is technical, not much is gained for the likely great amount of effort it would take to merge ShedSkin. Lets look at what is gained: 1. muteable globals 2. None and (int,float) are intermixable as attributes on an instance because ShedSkin has some limited support for dynamic-sub-types. (PyPy can not mix None with int and float) 3. operator overloading (except for __iter__ and __call__), PyPy only allows overloading __init__ and __del__ #1, would be nice to have but its an easy workaround to use singleton instances. #2, no big advantage. #3, this is a big advantage, i wish i could at least overload __getattr__ in PyPy Rpy. ShedSkin is behind PyPy Rpy in the following areas: 1. no getattr, hasattr etc. 2. *args (Mark says he can bring it back but only for homogenous types (PyPy supports *args with non-homogenous types)) 3. passing method references 4. no interface to C 5. mixed-type tuples are limited to length two (PyPy allows for any length) 6. multiple inheritance The ShedSkin readme itself states that " the type inference techniques employed by Shed Skin currently do not scale very well beyond several hundred lines of code" and recommends ShedSkin only for small programs. Not having the 6 items above are additional reasons why ShedSkin is not ideal for writting large programs, these are serious limitations for a large program, especially #4 - not having a easy way to interface with C is a huge show-stopper; PyPy Rpy has an amazingly simple way to interface with C (rffi). The other dissadvantages of ShedSkin are: slow object allocation (Phil Hassey did a test showing ShedSkin 30% slower than RPython), and it only translates to C++ while PyPy can translate to C, Java, and #C. There are simple ways to improve PyPy Rpy that will benifit both the PyPy project and those who strictly want to use the translation toolchain. I've been following the progress of this years Google Summer of Code projects, and i see a big stumbling block for everybody was RPython. PyPy today would have a better 64bit JIT, faster ctypes, and better numpy support if RPython was itself better. RPython Wishlist: . documentation . iteration over tuples of any length (with any mixed types) . overloading __getattr__, __setattr__ . pickle support, if its limited thats ok. . rstruct is incomplete . llvm backend, what happened to llvm support? . not having to define dummy functions on the base class to prevent 'demotion' . not having to use the hack `assert isinstance(a,MySubClass)` to call methods with incompatible signatures. . we already have the decorator: @specialize.argtype(1), why can't we have @specialize.argtype(*) so that all arguments can have flexible types? . methods stored in a list for easy dispatch can not have mismatched signatures. I aggree with Fijal, CPython module extension should be a low priority. There is a big speed overhead when passing data back and forth from CPython, and speed is the whole point of going through the trouble of writting in RPython. Those who are less concerned with speed and want CPython module extensions can use Cython, which is well tested. Those who are interested in RPython and want a simple way to get started can use ShedSkin to make an extension module, and then migrate their module to a standalone app with PyPy if they choose. -hart
On 16 September 2010 10:44, Hart's Antler <bhartsho@yahoo.com> wrote:
[the GPL] is no good for commerical software.
Please don't go there.
... what is gained: 2. None and (int,float) are intermixable as attributes on an instance because ShedSkin has some limited support for dynamic-sub-types. (PyPy can not mix None with int and float)
And it's not obvious we would want this anyway, as such type information has to live somewhere, and having to carry that around makes using ints and floats more expensive. In pypy-c, small applevel ints are already tagged in the obvious way (the lowest bit of a pointer is always unset, so setting it tags an app-level integer), but rpython ints shouldn't have to live with the performance hit and additional accuracy loss where tagged ints are cast to native ints.
3. operator overloading (except for __iter__ and __call__), PyPy only allows overloading __init__ and __del__
Makes rtyping more work. It takes long enough as it is.
ShedSkin is behind PyPy Rpy in the following areas: 1. no getattr, hasattr etc.
Does rpython really have this? As it appears in, say, the main python eval loop, it's constant folded. This is a good example of python usage in metaprogramming rpython.
6. multiple inheritance
That would limit rpython's ability to target the CLI and JVM if implemented naeively. It's doable, but means we need to use interfaces on ootype targets and need to do more work to look up methods in the C backend - possibly implementing hotspot-style itables or virtual inheritance or passing a dictionary around. It could get ugly quickly, not only for the backends, but also for the JIT generator, which already has too much to think about in terms of the model it is implementing.
. not having to define dummy functions on the base class to prevent 'demotion'
Some concept of an interface would be handy. -- William Leslie
If anyone wants to pay developers to work on rpython it should probably follow this wishlist and not focus on trying to merge shedskin into pypy (for the all the reasons given above). Here is my version of the list On Wed, Sep 15, 2010 at 9:44 PM, Hart's Antler <bhartsho@yahoo.com> wrote:
RPython Wishlist: . documentation Yes, this should be the highest priority. I could add documentation, doctests explaining stuff, templates for new modules and other things, better error messages.
. iteration over tuples of any length (with any mixed types) . overloading __getattr__, __setattr__ . pickle support, if its limited thats ok. did anyone ever really needed this? Would be cool I think, but unnecessary.
. rstruct is incomplete . llvm backend, what happened to llvm support? also I don't know about the utility of this for rpython itself, would be interesting if there was a jit backend that used llvm to better jit everything.
. not having to define dummy functions on the base class to prevent 'demotion' . not having to use the hack `assert isinstance(a,MySubClass)` to call methods with incompatible signatures. both are nice to have in a language with type inferences. Without this explicit in the code maybe the error messages would be even more complex to deal with.
. we already have the decorator: @specialize.argtype(1), why can't we have @specialize.argtype(*) so that all arguments can have flexible types? could be @specilize.argtype(42)
. methods stored in a list for easy dispatch can not have mismatched signatures.
I would add: . tool to automatically generate stubs for c libraries . c++ support (to be able to use c++ libs directly on rpython) . better java support (because the jvm matters). . better tooling (profiling, debugging etc) . separate compilation and/or some cache to speed up compilation. -- Leonardo Santagada
----- Original Message ----
From: Hart's Antler <bhartsho@yahoo.com> To: pypy-dev@codespeak.net Cc: Saravanan Shanmugham <sarvi@yahoo.com> Sent: Wed, September 15, 2010 5:44:35 PM Subject: Re: External RPython mailing list
Porting ShedSkin to use the PyPy translation toolchain on the surface sounds like a good idea, but its not if we look at the details. The first issue is legal, PyPy is MIT licensed, which works very well when integrated by commerical software. But Shedskin uses the GNU GPL3, so importing any of its code (or code that imports GPL code, etc) into the user's compiled program also binds it to the GPL - which is no good for commerical software. Shedskin within the PyPy toolchain may taint the users program with GPL code because some RPython programs will import from rlib (which may in someway depend on Mark's GPL code).
Sarvi: Good point. I hadn't noticed that the generated C++ code was GPL. I think Mark might be open to MIT licesing the generated C++ code. Coz, without it Shedskin as a tool chain would make no sense. But then thats a different story. Either way, it looks like there i not much enthusiasm for porting Shedskin on PyPy and have pypy generate a compiler instead of an interpreter.
From various threads on python.org as well pypy itself, I see a lot of interest in a compiler for a staticaly typed subset of python. I also feel that a statically typed subset of python can be faster than the dynamic superset.
Which is why I have exploring options to spur some interest to drive some momentum in this area. Hasn't been easy. I can see why some might feel that RPython is not for general use and only for language development. But what totally surprises me though is that as a language developer, I would want RPython to be as flexible as possibile within feasibility of course. Anyway, I am going to keep it simple and just start exploring just expanding on PyPy's RPython compiler to make it more general purpose. yes, on a separate branch :-)) of ofcourse. :-) I'll start with some of the items below. Lets see where that can go. Sarvi
The second issue is technical, not much is gained for the likely great amount of effort it would take to merge ShedSkin. Lets look at what is gained: 1. muteable globals 2. None and (int,float) are intermixable as attributes on an instance because ShedSkin has some limited support for dynamic-sub-types. (PyPy can not mix None with int and float) 3. operator overloading (except for __iter__ and __call__), PyPy only allows overloading __init__ and __del__
#1, would be nice to have but its an easy workaround to use singleton instances. #2, no big advantage. #3, this is a big advantage, i wish i could at least overload __getattr__ in PyPy Rpy.
ShedSkin is behind PyPy Rpy in the following areas: 1. no getattr, hasattr etc. 2. *args (Mark says he can bring it back but only for homogenous types (PyPy supports *args with non-homogenous types)) 3. passing method references 4. no interface to C 5. mixed-type tuples are limited to length two (PyPy allows for any length) 6. multiple inheritance
The ShedSkin readme itself states that " the type inference techniques employed by Shed Skin currently do not scale very well beyond several hundred lines of code" and recommends ShedSkin only for small programs. Not having the 6 items above are additional reasons why ShedSkin is not ideal for writting large programs, these are serious limitations for a large program, especially #4 - not having a easy way to interface with C is a huge show-stopper; PyPy Rpy has an amazingly simple way to interface with C (rffi). The other dissadvantages of ShedSkin are: slow object allocation (Phil Hassey did a test showing ShedSkin 30% slower than RPython), and it only translates to C++ while PyPy can translate to C, Java, and #C.
There are simple ways to improve PyPy Rpy that will benifit both the PyPy project and those who strictly want to use the translation toolchain. I've been following the progress of this years Google Summer of Code projects, and i see a big stumbling block for everybody was RPython. PyPy today would have a better 64bit JIT, faster ctypes, and better numpy support if RPython was itself better.
RPython Wishlist: . documentation . iteration over tuples of any length (with any mixed types) . overloading __getattr__, __setattr__ . pickle support, if its limited thats ok. . rstruct is incomplete . llvm backend, what happened to llvm support? . not having to define dummy functions on the base class to prevent 'demotion' . not having to use the hack `assert isinstance(a,MySubClass)` to call methods with incompatible signatures. . we already have the decorator: @specialize.argtype(1), why can't we have @specialize.argtype(*) so that all arguments can have flexible types? . methods stored in a list for easy dispatch can not have mismatched signatures.
I aggree with Fijal, CPython module extension should be a low priority. There is a big speed overhead when passing data back and forth from CPython, and speed is the whole point of going through the trouble of writting in RPython. Those who are less concerned with speed and want CPython module extensions can use Cython, which is well tested. Those who are interested in RPython and want a simple way to get started can use ShedSkin to make an extension module, and then migrate their module to a standalone app with PyPy if they choose.
-hart
On 16 September 2010 13:03, Saravanan Shanmugham <sarvi@yahoo.com> wrote:
Either way, it looks like there i not much enthusiasm for porting Shedskin on PyPy and have pypy generate a compiler instead of an interpreter.
In a sense, it already does :). And of course translation is compilation, too.
From various threads on python.org as well pypy itself, I see a lot of interest in a compiler for a staticaly typed subset of python. I also feel that a statically typed subset of python can be faster than the dynamic superset.
It can be, there's nothing stopping you from dynamically compiling a static language, and feeding back profiling information is easy too. It's just probably not going to be *that* much faster to be of value at the end of the day. Then again, the end of the day could be a long way away.
I can see why some might feel that RPython is not for general use and only for language development.
But what totally surprises me though is that as a language developer, I would want RPython to be as flexible as possibile within feasibility of course.
You also have to look at it from the other perspective - that of someone implementing a backend or translation aspect, such as a garbage collector or a JIT compiler generator. This is my perspective coming to pypy - I am experimenting with a range of optimisations based on extensive region and effect analysis, and I fear rpython already makes this difficult. For example, the use of abstract interpretation to generate the flowgraph IR means that you now have no information about which loop is the 'outer' one, and that information can be useful in generating heuristics. Similar things could be said about the JIT and generators, which is not something I have looked at extensively, but dealing with the generator case would have been implicit from the start if the IR used a CPS transform to represent all instruction flow. In short: rpython is complicated enough already. It happens to do the job it was created for, but not a whole lot more than that. It happens to be well suited to my experiments for two unrelated reasons*. But I can't imagine choosing it to write extension modules or inner loops - there are plenty of languages that do it better, like cython, pyrex, D, cyclone, SML, etc. * it's (memory) safe and the rffi is sane, particularly about letting native code deal with rpython objects. And thanks to the pypy python interpreter, there's a large body of code to test it on. -- William Leslie
participants (4)
-
Hart's Antler -
Leonardo Santagada -
Saravanan Shanmugham -
William Leslie