First question - why did you choose to implement this as a compiler to python bytecode? It does sound like an interpreter written in rpython would have both a much better performance and a much easier implementation (compiler vs interpreter).

A few reasons for this. Mostly I didn't want to have to build up an entire standard lib. Clojure is a bit unique in that it doesn't define a standard library beyond the ~200 functions found in core.clj. This means that Clojure leaves IO, GUI, etc, completely up to the VM. So for stock Clojure this means you drop to Java interop whenever you want to do IO. If, however, I develop the entire thing off of CPython/pypy, I can use all the libraries for these platforms that are already quite documented and stable. "Don't re-invent the wheel" is more or less the mantra of Clojure implementation developers. Long term though, I plan on implementing part (if not all) of Clojure-py in RPython. This may be as simple as doing a pull request to pypy asking to have my immutable structures adopted into the stock VM, or I may attempt to build a from-scratch interpreter, we'll see. So I guess it's like this: I could go with a custom VM, but when I'm done, there's really not a whole lot my VM could do besides run benchmarks. Even as clojure-py stands now, you could probably sit down in one night and write a full blown Qt app via PySide with it, the interop with Python is that good. So as it stands, we can write apps with Django, PySide, numpy, etc. in Clojure and only after about 3 months worth of work! The other thing that has been bugging me more and more lately, is what benefit a RPython interpreter would get me. Besides Python's lack of overloaded functions, there's really no features in Python I can't find a use for in the Clojure compiler, and there's really nothing I'm lacking in the pypy VM. Timothy

Hi Timothy, I'm new in the list and I was very recently planning to do a pet project in implementing a clojure interpreter in Rpython. Of course, I share the same concerns as you about the standard library issue (although your body of work on the issue gives you a much better understanding of all the concerns, which at the moment I can only guess), so I was planning as starting with a very simple dummy interpreter and then build on top of that more features as time and skill allow. I will try to check your clojure implementation this weekend. Hopefully I will manage to do something useful :) A more generally targeted question, besides the sample rpython tutorial for brainfuck, what are the recommended readings (parts of the pypy code, papers, etc), tools and/or magic for working at rpython on an interpreter? I saw that there will be a sprint soon in Berlin (and it not being that far from Prague, if the dates allow, it might be possible for me to take some vacation days and assist). In regards to that, having only experience with python and almost none in interpreter writing, will my presence there be of any value? I can imagine that you must want to do a lot of work in such an event and that a newbie like me might not be the thing to have around. Thus, I would like to ask if with the readings that I ask in the previous paragraph there would be some easy task that I could work on to get acquainted with the software and assist another future sprint. Best regards, Antoni Segura Puimedon On Fri, Feb 17, 2012 at 3:18 PM, Timothy Baldridge <tbaldridge@gmail.com>wrote:

Oh yeah, I forgot. The other nice thing about doing clojure-py in Python is that I should be able to write RPython code in Clojure. Lisp macros FTW!

Timothy

-- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev

On Fri, Feb 17, 2012 at 5:11 PM, Timothy Baldridge <tbaldridge@gmail.com> wrote:

...

A more generally targeted question, besides the sample rpython tutorial for brainfuck, what are the recommended readings (parts of the pypy code, papers, etc), tools and/or magic for working at rpython on an interpreter?

Now, most of this code is more or less crap, but I do suggest taking a look at my first stab at a RPython interpreter for Clojure

https://github.com/halgari/clj-pypy

IIRC, the target complied okay, and you could run the code in scratchspace.clj.

Basically I got to the point where I realized that if I had lisp macros, I could write RPython code way faster. Half of the structures in Clojure follow this pattern:

class Foo(object):    def __init__(self, foo bar, baz):         self.foo = foo         self.bar = bar         self.baz = baz    def addBarBaz(self):         return bar + baz

I could write that for that, or, in Clojure I could write myself a macro and just do:

(deftype Foo [foo bar baz]      (addBarBaz[self] (+ bar baz)))

Just a sidenote - RPython allows you to do metaprogramming. It's not as easy and pleasant, but you can generate code. In fact we do it all over the place in pypy either using string and exec() or closures that return slightly different classes for each set of given parameters.

On Fri, Feb 17, 2012 at 4:11 PM, Timothy Baldridge <tbaldridge@gmail.com>wrote:

...

A more generally targeted question, besides the sample rpython tutorial for brainfuck, what are the recommended readings (parts of the pypy code, papers, etc), tools and/or magic for working at rpython on an interpreter?

Now, most of this code is more or less crap, but I do suggest taking a look at my first stab at a RPython interpreter for Clojure

https://github.com/halgari/clj-pypy

IIRC, the target complied okay, and you could run the code in scratchspace.clj.

Basically I got to the point where I realized that if I had lisp macros, I could write RPython code way faster. Half of the structures in Clojure follow this pattern:

class Foo(object): def __init__(self, foo bar, baz): self.foo = foo self.bar = bar self.baz = baz def addBarBaz(self): return bar + baz

I could write that for that, or, in Clojure I could write myself a macro and just do:

(deftype Foo [foo bar baz] (addBarBaz[self] (+ bar baz)))

So basically I've found that I can write the same code in the following languages with this line ratio:

Clojure: 1 Python: 3 Java: 6

This is the reason I shelved the RPython idea. If I want to dramatically re-define how types are handled in Clojure-py all I have to do is re-write a single macro.

<over generalization>

"Why write a jit when you can have RPython do it for you?"

"Why write a typesystem, when you can have Clojure macros do it for you?"

This would be the clojurescript approach, right?

</over generalization>

Timothy

-- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth)

Hi Antoni, On Fri, Feb 17, 2012 at 15:51, Antoni Segura Puimedon <celebdor@gmail.com> wrote:

I saw that there will be a sprint soon in Berlin (and it not being that far from Prague, if the dates allow, it might be possible for me to take some vacation days and assist).

You are most welcome to our sprints. No previous experience is needed, although as you say, it's useful if you have previously read about or done something about some part of PyPy; or at least have good Python knowledge. We are planning so far to have a sprint not in Berlin (unless I missed this one), but in Leipzig, Germany, near the end of June. A bientôt, Armin.

It's factorial(), so it's handling large "longs".  So it's about 2x as slow as CPython.  End of the story.

Funny enough, had just thought of that and given it a try. I switched it to calculating (fact 20) and (times 200000) and PyPy is now 3x faster than CPython. Thanks for the help! Timothy

Hi Armin, On Sat, Feb 18, 2012 at 12:54 PM, Armin Rigo <arigo@tunes.org> wrote:

Hi Antoni,

On Fri, Feb 17, 2012 at 18:49, Antoni Segura Puimedon <celebdor@gmail.com> wrote:

...

Any suggestion on where to start reading pypy code/material?

In your case, I'd suggest that you should first write a Clojure interpreter in regular Python, and then worry about making it RPython. This is done mostly by following the guidelines in http://doc.pypy.org/en/latest/coding-guide.html .

Thanks a lot for the advice. I will do that then, start from the basics.

Note also that, from what I've understood, it might be useful to integrate the Clojure interpreter with PyPy to allow calls to the rest of the Python standard library and built-in modules. This is relatively easy to do, and doesn't prevent the JIT from JITting the clojure interpreter. (You can have multiple JITs in the same process.) I imagine it would be done as a PyPy built-in module, which can be called from Python code in order to compile and execute Clojure code.

To do this you would need to follow the general structure shown for example in pypy/module/_demo/. This is explained in http://doc.pypy.org/en/latest/coding-guide.html#mixed-modules .

I will read it, but I expect to have a lot of questions about the integration :P

A bientôt,

Armin.

Best regards, Antoni Segura Puimedon