RPython: write generic native library / write CPython C module
Hi, I got the idea to use RPython to use my Python code to write a generic native library. I have looked roughly at the RPython translator code and it seems like there are ways to export custom symbols. Has that already been done? Or is it possible to do that? --- Then, I got the further idea to write a CPython C extension module with RPython. (Somehow the opposite of cpyext.) I thought I could just export the PyMODINIT_FUNC init*() symbol and do the common logic there. I.e., export the functions and other objects/classes. The next step would be to define wrapper functions which get native CPython PyObject structures and transform them into the RPython equivalents. Maybe cpyext can be useful for this task. Maybe someone could point me to the direction whether this is feasible and where to start? Thanks and regards, Albert
Hi Albert, You may or may not get help with your questions: see http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-r... Armin
Hi Armin, Well, not quite. I think I know about the limitations and restrictions of RPython. But that is why I think it is especially useful in some cases where I have written some library in Python with some strict subset of Python (or already mostly RPython). I need a native version of the same library and don't really see the point in translating it 1:1 to C by hand. Also, this is maybe more for playing around and just testing it. Maybe I stumble upon other problems which hinder the whole idea. -Albert On Sun, Jun 16, 2013 at 8:45 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Albert,
You may or may not get help with your questions: see http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-r...
Armin
On Sun, Jun 16, 2013 at 7:37 PM, Albert Zeyer <albzey@googlemail.com> wrote:
Hi Armin,
Well, not quite. I think I know about the limitations and restrictions of RPython. But that is why I think it is especially useful in some cases where I have written some library in Python with some strict subset of Python (or already mostly RPython). I need a native version of the same library and don't really see the point in translating it 1:1 to C by hand.
Also, this is maybe more for playing around and just testing it. Maybe I stumble upon other problems which hinder the whole idea.
-Albert
so let's reverse the question. Why do you need such a library?
Hi Albert, On Sun, Jun 16, 2013 at 1:37 PM, Albert Zeyer <albzey@googlemail.com> wrote:
You may or may not get help with your questions: see http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-r...
Well, not quite.
No, sorry for the misunderstanding :-) What I meant is that your questions may be possibly valid, but I was pointing you to this FAQ entry that explains why we (= the general PyPy developers) are not really interested in this direction, and which lists some of our reasons for that. There is notably the fact that RPython is not meant as a general language used to produce stand-alone C libraries. It's certainly possible to make one in theory, but cumbersome. That's what I meant with "you may not get help with your questions". A bientôt, Armin.
Hi Armin, Thanks for the clarification! I was wondering: Why is it that RPython is not a good general purpose language? In the original paper (http://rpython.googlecode.com/svn/trunk/doc/rpython-DLS08.pdf), it is said:
The result is a language that is more expressive than C# and Java, but which does not compromise runtime efficiency. RPython was initially designed for the specific purpose of implementing PyPy [25] (a Python interpreter written in Python), but it has grown into a full-fledged language in its own right.
Currently, RPython can be used in many contexts: to develop stand-alone programs, such as the Standard Interpreter itself; to write highly efficient extension modules for CPython, which could only be written in C in the past; to develop dynamic web applications without the need to write JavaScript code; to produce efficient libraries of classes and functions to be used by other .NET and Java programs. In particular, RPython can be the ideal companion for all those CPython, IronPython and Jython developers that so far have been forced to write the parts of their programs that need high performance in C, C# or Java.
I think, in general, RPython looks quite nice. I have static types via automatic type inference. I could use C or C++ for the same tasks but I like the Python syntax. I could also use Cython but I don't really see much the difference between Cython and RPython, except that RPython does automatic type inference and is a strict subset of Python, while Cython is explicit manifest typing. (Note that I also asked the same question here: http://stackoverflow.com/questions/17134479/why-do-people-say-that-rpython-i... ) Regards, Albert On Mon, Jun 17, 2013 at 9:23 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Albert,
On Sun, Jun 16, 2013 at 1:37 PM, Albert Zeyer <albzey@googlemail.com> wrote:
You may or may not get help with your questions: see http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-r...
Well, not quite.
No, sorry for the misunderstanding :-) What I meant is that your questions may be possibly valid, but I was pointing you to this FAQ entry that explains why we (= the general PyPy developers) are not really interested in this direction, and which lists some of our reasons for that. There is notably the fact that RPython is not meant as a general language used to produce stand-alone C libraries. It's certainly possible to make one in theory, but cumbersome. That's what I meant with "you may not get help with your questions".
A bientôt,
Armin.
Hi Albert, On Mon, Jun 17, 2013 at 11:20 AM, Albert Zeyer <albzey@googlemail.com> wrote:
I was wondering: Why is it that RPython is not a good general purpose language? In the original paper (http://rpython.googlecode.com/svn/trunk/doc/rpython-DLS08.pdf), it is said:
RPython dates further back. It is perhaps first documented in the 2006 paper "PyPy's Approach to Virtual Machine Construction". In there you'll already find paragraphs like this: """ We should mention that restricting oneself to write RPython code instead of Python is still a burden, and that we are not proposing in any way that the Python language itself should evolve in this direction, nor even that RPython usage should become widespread. It is a tool designed with a specific goal in mind, which is the ability to produce reasonably efficient, stand-alone code in a large variety of environment. """ The paper you cite was written by an academic group that was focusing on OO languages (Java and CLI), building on the "variety of environment" concept. However the two paragraphs you cite feel a bit like advertisement today, given that interest for Java and CLI dried out in PyPy... Anyway, let's try to answer your questions in more details. RPython is great at one thing: being a language in which you express just the right amount of details in your interpreter for the purpose of turning this interpreter into a full VM automatically. For any other "more traditional" usage --- whatever the programming level --- there is another language out there that already exists and is much more well-supported than RPython in terms of features, tools, debuggers, group of people, etc. When people look at RPython, an obvious feature is that it is syntactically identical to Python. "RPython must be an easy language, given that it has got the syntax of Python, which is easy". This is a common misconception. In fact, pleasing the automatic type inference process can be difficult. It requires the programmer keeping in his head the global types of his whole program, and carefully writing code according to these implicit types. The process is much harder for newcomers, which don't have any written-down example to learn how to manipulate the types --- precisely because they are implicit. So this is the reason we are recommending against RPython now (and for many years now). Anybody who thinks RPython is as easy as Python is someone who will happily try out RPython and be burned alive by it. So "by default" we answer negatively to people who might be at the level of newcomers, to keep them away from this in-house, not-easy-to-program, not-easy-to-debug subset of Python which we call RPython. Obviously, we don't think that RPython is completely bogus. It is a language with use cases, and probably the one language with the most advanced meta-JIT framework :-) But, at least, I don't want to spend three years of my time trying to turn it into an good language for everybody (and, likely, getting out only an ok-ish language). Personally, I find it far more interesting to spend three years of my time (or 10!) on other things, like giving a good alternative implementation of the full unmodified Python language :-) A bientôt, Armin. PS: if you go to the STUPS Group in Softwaretechnik und Programmiersprachen, you might find David Schneider still in Düsseldorf :-)
On 06/16/2013 01:37 PM, Albert Zeyer wrote:
Hi Armin,
Well, not quite. I think I know about the limitations and restrictions of RPython. But that is why I think it is especially useful in some cases where I have written some library in Python with some strict subset of Python (or already mostly RPython). I need a native version of the same library and don't really see the point in translating it 1:1 to C by hand.
One of the reasons why the RPython project does not care much about this use case anymore is that we *had* a way to make CPython extensions. It was called the "extension compiler". It turned out to be a) extremely annoying to implement b) not very useful in practice. Thus a totally frustrating experience overall. The reasons for slowness was mainly: when compiling RPython code to be a C extension module, reference counting is used to reclaim garbage. That is extremely slow. Therefore the modules never got the performance anybody expected. Cheers, Carl Friedrich
Oh, that is interesting. I searched a bit around but did not find too much. Is that what I can see in test_wrapping.py for example [here](https://bitbucket.org/william_ml_leslie/pypy-effect-analysis/src/4ba135c9c31... I also read about the [RPythonic project](https://code.google.com/p/rpythonic/) which seems to do something similar. In my case, I think it would already be fast if for every call with attributes of immutable objects (strings, numbers, tuples), it would just copy them and otherwise, it would use some wrapper objects. That way, RPython would not have to rely on reference counting (except that the wrapper objects would hold one reference to a native PyObject). For some other objects, e.g. dicts, it maybe could also copy them if they are not going to be modified. I'm interested in trying it out. Or do you think it would be a waste of time? (Btw., I saw that you have worked at Heinrich Heine Düsseldorf. I'm just sitting there right now in the library. :)) On Mon, Jun 17, 2013 at 11:38 AM, Carl Friedrich Bolz <cfbolz@gmx.de> wrote:
On 06/16/2013 01:37 PM, Albert Zeyer wrote:
Hi Armin,
Well, not quite. I think I know about the limitations and restrictions of RPython. But that is why I think it is especially useful in some cases where I have written some library in Python with some strict subset of Python (or already mostly RPython). I need a native version of the same library and don't really see the point in translating it 1:1 to C by hand.
One of the reasons why the RPython project does not care much about this use case anymore is that we *had* a way to make CPython extensions. It was called the "extension compiler". It turned out to be a) extremely annoying to implement b) not very useful in practice. Thus a totally frustrating experience overall.
The reasons for slowness was mainly: when compiling RPython code to be a C extension module, reference counting is used to reclaim garbage. That is extremely slow. Therefore the modules never got the performance anybody expected.
Cheers,
Carl Friedrich _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
participants (4)
-
Albert Zeyer -
Armin Rigo -
Carl Friedrich Bolz -
Maciej Fijalkowski