[pypy-dev] PyPy and production use
faassen at startifact.com
Fri Jun 15 12:41:06 CEST 2007
I'm changing the title of this thread as this is another topic. It's one
I care about as an observer of PyPy.
Antonio Cuni wrote:
> Finally, a note which is not related with the work plan but to a
> question that appeared in the logs; someone asked whether we want to
> "sell" RPython as a stand-alone product.
I think you should. I'll explain why below.
> Last sunday I gave a PyPy talk at Pycon italy and, surprisingly enough,
> RPython was perceived as the only (or the most relevant) usable product
> of the pypy project. I tried hardly to explain that rpython is basically
> only an accident and not very usable, but people are still impressed by
> the 300000% speedup.
> It would be interesting to know if this perception is world-wide or only
> limited to the attendants of my talk (maybe because I didn't stress
> enough the pros of the other pypy goals or the cons of rpython).
I got this impression back at the presentation last year at EuroPython.
I therefore think this impression is more widespread than just your
audience in Italy. This is not because you've communicated the state or
future potential of PyPy wrongly, but because the audience likely has a
different perspective than you do.
I realize that PyPy is much more than RPython and that RPython is
somewhat incidental to the project. For people working on production
projects right now, these other features are less relevant at this point
in time, however. Let's take PyPy's generated interpreter. If I'm
currently using the CPython platform (as most Python programmers are),
choosing the PyPy-C interpreter would mean that I have far less
extension module support, and a slightly slower interpreter. I'd need
some pretty unusual use cases currently for the benefits of PyPy-C to
outweigh its drawbacks.
The PyPy developers themselves of course *do* have unusual use cases.
They want to experiment with interpreter generation, different language
features, different platforms, and so on.
I expect this to change over time. Once the PyPy (generated) JIT
matures, PyPy will become a lot more attractive. If you could then
somehow figure out a way to use CPython modules with PyPy's interpreter
you'd get even more people. Right now, unless I really really need some
unusual language feature in my production project, I don't have a
motivation to use PyPy's interpreter.
Maturing the PyPy interpreter will take quite a lot of time. You've come
a long way and it's quite impressive, but at the same time I don't
expect to be using a PyPy-generated interpreter to develop web
applications any time soon.
Now RPython. RPython generated modules are usable from CPython. This
means I can write relatively plain Python code, generate a module, and
start using this module in my web application today. The benefit is
vastly increased speed for my module.
Using RPython with CPython is attractive for the same reasons Pyrex is
attractive, with the benefits that it's more well-engineered than Pyrex.
Pyrex is currently more mature to use. Using RPython in this way is a
bit of a struggle still in comparison. This is however tool support that
I feel can improve relatively quickly.
An open source projects needs mindshare. It needs users. PyPy is an open
source project, and I'd recommend getting any users you can get.
Granted, you'll get opportunists like me who are only there for
production benefits. These people are less interested in language
experimentation and new features - they're there for the speed
primarily: the main thing that PyPy, through RPython, can offer to a
Python programmer today. But if your aim is to provide a platform that
is suitable for production use, you *need* opportunists like that. They
can help mature the platform. Open source projects thrive by attracting
developers that are in it for their own (enlightened) self-interest.
Some of these opportunists might actually help out beyond just RPython.
They might in fact be interested in such language experimentation. It
might however not be a priority for them. For instance, I'm interested,
but I need an excuse to spend more time on PyPy. Production benefits are
a great excuse. Using RPython extensions with CPython is the shortest
route to production benefits.
Besides gaining new users, supporting RPython as a product will help
PyPy in other ways as well. It will get you more module support. New
users will come in and help you write more extension modules in RPython.
They'll be doing this initially so they can use them in CPython, but
it's a nice side effect.
Currently I'm working with Godefroid Chapelle (well, he's been doing the
most work so far) to try to make an RPython version of a web templating
engine (Zope's page template engine). We hope we will be able to use
this technology to speed up the generation of web templates. We'll be
using this within a CPython server.
We see the potential of PyPy. Godefroid made the observation that a web
templating engine implements an interpreter. PyPy has technology to
generate a JIT for arbitrary interpreters in RPython. This means that in
the future we might be able to have a JIT-ed templating language,
hopefully boosting performance even more. (besides the huge coolness factor)
Just the potential for future JITting isn't motivating enough to
experiment with PyPy right now though. The ability to use RPython with
CPython is the critical bit. This is what gives us the hope we'll get a
payoff in our production work. The JIT is a nice bonus that might become
So if people conclude that RPython is currently the most relevant
product of the PyPy project for them, it doesn't mean that they haven't
understood the potential of PyPy in the future. It's because they're
thinking about how PyPy is relevant to their own work today.
PyPy has always been a unique project: it's a research project and it's
an open source project aiming for real-world use. It takes ideas from
ivory towers and applies them to a pragmatic, real-world language like
Python. I think this balance between pragmatic and research goals is one
of this project's greatest strengths. I encourage PyPy developers to
continue to embrace both goals.
More information about the Pypy-dev