Running PyPy on top of CPython
Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html) that PyPy is runnable on top of CPython. How do I start PyPy so it runs in CPython?
IINM, Running Pypy overtop of CPython is mostly useful for bootstrapping Pypy - but it's generally easier to download a Pypy binary from the official website. If you just want CPython+speed, maybe try https://stromberg.dnsalias.org/~strombrg/speeding-python/ ? On Sun, Jan 9, 2022 at 11:51 AM M A <teammember0x01@gmail.com> wrote:
On 10/1/22 4:31 am, M A wrote:
Well I am developing for PyPy and was hoping to try my code out without having to build PyPy first.
You asked this question on the issue tracker as well. Running PyPy on top of CPython is not a replacement for getting stress tests in the issue comment [0] to pass. [0] https://foss.heptapod.net/pypy/pypy/-/issues/3566#note_167193
Hi, On Mon, 10 Jan 2022, 3:31 AM M A <teammember0x01@gmail.com> wrote:
Well I am developing for PyPy and was hoping to try my code out without having to build PyPy first.
I tried twice to tell you how we develop PyPy in the issue: by running tests. You seem to be ignoring that. Sorry but I will ignore further requests from you until you acknowledge you have at least tried what I recommend first. Armin
I think you are confused. I just read in PyPy's documentation that PyPy could be run from CPython. This sounds like something that could help save me time by seeing if my changes work. I'm am not sure why you think I am ignoring the tests. Yes I have tried them. I am seeing if there is a more efficient way of trying out my code without having to wait a long time of PyPy to recompile.
Hi, On Mon, 10 Jan 2022 at 15:56, M A <teammember0x01@gmail.com> wrote:
I think you are confused. I just read in PyPy's documentation that PyPy could be run from CPython. This sounds like something that could help save me time by seeing if my changes work. I'm am not sure why you think I am ignoring the tests. Yes I have tried them. I am seeing if there is a more efficient way of trying out my code without having to wait a long time of PyPy to recompile.
Sorry for not explaining myself clearly. I've been trying all along to tell you that you don't need to recompile PyPy, ever. As long as not all the tests I listed are passing, then there is basically no point. (except that it feels good to see PyPy half-working instead of not working at all, of course, but don't try to debug that) The documentation says "PyPy can be run on top of CPython": that's almost what all the tests are doing. They run not the whole PyPy, but instead some small example RPython code (sometimes written as RPython in the test, sometimes at a lower level). The point is that they test the JIT backend by running it as normal Python code, on top of CPython. When working on the JIT backend, you don't want to run the whole PyPy on top of CPython running with the JIT; while possible in theory, it is far too slow to be practical. Instead, run the tests, which test the same thing but with a small bit of RPython code instead of the many-thousand-of-lines-of-code of PyPy. Let me try to explain myself in a different way. What we're trying to do here is fix a JIT compiler emitting machine code, at the level of RPython instead of Python. That means that any error is likely to cause not just a problem for some specific Python function which you can find by running Python code; instead, bugs cause random hard-to-debug crashes that show up at half-random points, with no obvious one-to-one correspondence to what Python code is running. This could be caused for example by the wrong registers being used or accidentally overwritten in some cases, leading to a nonsense value propagating further, until the point of crash. Most tests listed above check all basic cases and problems we encountered so far. The last test is producing and running random exemples, in an attempt to find the random rare-case issues. It has been the case that this last test found a remaining bug after running for hours, but I think that after half a day we can conclude that there is no bug left to be found. In the past, there were very rare bugs that went uncaught by random testing too; these were a real pain to debug---I once spent more than two weeks straight on a pair of them, running "gdb" inside a translated PyPy, hacking at the generated C code to dump more information to try to figure it out. But these bugs were more than just in the JIT backend (e.g. some interaction between the JIT and the GC), and they are fixed now. I'm telling this story to explain why I do not recommend going down that path for the simpler bugs that are already caught by the tests! So, once more, I recommend working on this by running the tests, and fixing them, until all tests pass. Once all tests pass (and not before) you can try to translate the whole PyPy, and at this point it will likely work on the first try. Sorry for not managing to get this message across to you in the past. I hope I have done so now. A bientôt, Armin.
Hi Omer. I think much of what Armin wrote is already in the documentation like https://doc.pypy.org/en/latest/contributing.html#architecture and https://doc.pypy.org/en/latest/contributing.html#test-driven-development. Could you see where things can be improved? Matti On 11/1/22 4:17 pm, Omer Katz wrote:
On 9/1/22 9:50 pm, M A wrote:
Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html) that PyPy is runnable on top of CPython. How do I start PyPy so it runs in CPython?
python2 pypy/bin/pyinteractive.py You can get help with python2 pypy/bin/pyinteractive.py --help It takes ~20 secs to startup. To drop into rpython, hit ctrl-c, to return to interpreted PyPy on top of CPython hit ctrl-d In rpython you have access to "space". Matti
IINM, Running Pypy overtop of CPython is mostly useful for bootstrapping Pypy - but it's generally easier to download a Pypy binary from the official website. If you just want CPython+speed, maybe try https://stromberg.dnsalias.org/~strombrg/speeding-python/ ? On Sun, Jan 9, 2022 at 11:51 AM M A <teammember0x01@gmail.com> wrote:
On 10/1/22 4:31 am, M A wrote:
Well I am developing for PyPy and was hoping to try my code out without having to build PyPy first.
You asked this question on the issue tracker as well. Running PyPy on top of CPython is not a replacement for getting stress tests in the issue comment [0] to pass. [0] https://foss.heptapod.net/pypy/pypy/-/issues/3566#note_167193
Hi, On Mon, 10 Jan 2022, 3:31 AM M A <teammember0x01@gmail.com> wrote:
Well I am developing for PyPy and was hoping to try my code out without having to build PyPy first.
I tried twice to tell you how we develop PyPy in the issue: by running tests. You seem to be ignoring that. Sorry but I will ignore further requests from you until you acknowledge you have at least tried what I recommend first. Armin
I think you are confused. I just read in PyPy's documentation that PyPy could be run from CPython. This sounds like something that could help save me time by seeing if my changes work. I'm am not sure why you think I am ignoring the tests. Yes I have tried them. I am seeing if there is a more efficient way of trying out my code without having to wait a long time of PyPy to recompile.
Hi, On Mon, 10 Jan 2022 at 15:56, M A <teammember0x01@gmail.com> wrote:
I think you are confused. I just read in PyPy's documentation that PyPy could be run from CPython. This sounds like something that could help save me time by seeing if my changes work. I'm am not sure why you think I am ignoring the tests. Yes I have tried them. I am seeing if there is a more efficient way of trying out my code without having to wait a long time of PyPy to recompile.
Sorry for not explaining myself clearly. I've been trying all along to tell you that you don't need to recompile PyPy, ever. As long as not all the tests I listed are passing, then there is basically no point. (except that it feels good to see PyPy half-working instead of not working at all, of course, but don't try to debug that) The documentation says "PyPy can be run on top of CPython": that's almost what all the tests are doing. They run not the whole PyPy, but instead some small example RPython code (sometimes written as RPython in the test, sometimes at a lower level). The point is that they test the JIT backend by running it as normal Python code, on top of CPython. When working on the JIT backend, you don't want to run the whole PyPy on top of CPython running with the JIT; while possible in theory, it is far too slow to be practical. Instead, run the tests, which test the same thing but with a small bit of RPython code instead of the many-thousand-of-lines-of-code of PyPy. Let me try to explain myself in a different way. What we're trying to do here is fix a JIT compiler emitting machine code, at the level of RPython instead of Python. That means that any error is likely to cause not just a problem for some specific Python function which you can find by running Python code; instead, bugs cause random hard-to-debug crashes that show up at half-random points, with no obvious one-to-one correspondence to what Python code is running. This could be caused for example by the wrong registers being used or accidentally overwritten in some cases, leading to a nonsense value propagating further, until the point of crash. Most tests listed above check all basic cases and problems we encountered so far. The last test is producing and running random exemples, in an attempt to find the random rare-case issues. It has been the case that this last test found a remaining bug after running for hours, but I think that after half a day we can conclude that there is no bug left to be found. In the past, there were very rare bugs that went uncaught by random testing too; these were a real pain to debug---I once spent more than two weeks straight on a pair of them, running "gdb" inside a translated PyPy, hacking at the generated C code to dump more information to try to figure it out. But these bugs were more than just in the JIT backend (e.g. some interaction between the JIT and the GC), and they are fixed now. I'm telling this story to explain why I do not recommend going down that path for the simpler bugs that are already caught by the tests! So, once more, I recommend working on this by running the tests, and fixing them, until all tests pass. Once all tests pass (and not before) you can try to translate the whole PyPy, and at this point it will likely work on the first try. Sorry for not managing to get this message across to you in the past. I hope I have done so now. A bientôt, Armin.
Hi Omer. I think much of what Armin wrote is already in the documentation like https://doc.pypy.org/en/latest/contributing.html#architecture and https://doc.pypy.org/en/latest/contributing.html#test-driven-development. Could you see where things can be improved? Matti On 11/1/22 4:17 pm, Omer Katz wrote:
On 9/1/22 9:50 pm, M A wrote:
Hello, I read here (https://rpython.readthedocs.io/en/latest/rpython.html) that PyPy is runnable on top of CPython. How do I start PyPy so it runs in CPython?
python2 pypy/bin/pyinteractive.py You can get help with python2 pypy/bin/pyinteractive.py --help It takes ~20 secs to startup. To drop into rpython, hit ctrl-c, to return to interpreted PyPy on top of CPython hit ctrl-d In rpython you have access to "space". Matti
participants (5)
-
Armin Rigo -
Dan Stromberg -
M A -
Matti Picus -
Omer Katz