How hard would it be to generate PyPy for the Dalvik Virtual Machine?
This showed up on another list I read. ------- Forwarded Message From: Roland Orre <roland.orre@gmail.com> Date: Wed, 22 Apr 2009 10:05:15 +0200 To: cafe@ffii.org Subject: Re: [Cafe] android g1 X-BeenThere: cafe@ffii.org I also have a Google G1 now with a developers account. It is cool to have a telephone which I can ssh from and where I can set up an ftp server. I intended to start porting some applications like some HP calculator (I have HP42S and HP48GX), and probably openssl, openssh and a vpn client. One tricky aspect with the Android system is that it has it's own noncompatible VM which is the Dalvik virtual machine, which shold be more memory efficient than the standard JVM. This JVM is written by Dan Bornstein and released with Apache license v2. The tricky thing is that this gets around Sun's grip on the JVM but also implies that platform independent code is no longer platform independent... JVM bytecode can be converted to DalvikVM code though, but there is no just in time compiler for JVM to DalvikVM yet though, so e.g. Jython is not really nice implemented on it yet, although there is jythonroid http://code.google.com/p/jythonroid/ The current ARM chip is claimed to have hardware support for JVM though, even though Android should not be platform dependent of course. Any opinion about the JVM - DalvikVM issue? /Roland On Wed, Apr 22, 2009 at 08:39, Ivan F. Villanueva B. <ivan@ffii.org> wrote:
_______________________________________________ Cafe mailing list Cafe@ffii.org https://lists.ffii.org/mailman/listinfo/cafe ------- End of Forwarded Message
I'll address Jython: we plan support for Android in our 2.5.1 release. Supporting Android is similar to supporting unsigned applets: we either compile Python code in advance to Java bytecode, or compile it on the fly to Python bytecode (PBC) and execute with our PBC VM (just a straight port of ceval.c). Any class proxies also need to be compiled to Java bytecode in advance too, although we could support Java dynamic proxies for interface inheritance only (a significant but hopefully workable restriction, given that we prefer interfaces in Java, but of course only impacts Java integration). Packaging for Dalvik or as applet then becomes similar to say packaging a WAR file for Django or Pylons. The only missing pieces are support for the deployment piece, dynamic proxies, and a PBC compiler, all of which we have started work on, but put on hold for getting 2.5.0 out. - Jim On Wed, Apr 22, 2009 at 2:24 AM, Laura Creighton <lac@openend.se> wrote:
-- Jim Baker jbaker@zyasoft.com
Hi Jim, Jim Baker wrote:
this is very interesting. Is the PBC VM already working? If so, how do its performances compare against the normally compiled code? I and Niko did some benchmarking on pypy-jvm and we discovered that we spent (unsurprisingly) the largest amount of time inside the main eval loop (up to 68% of the total on some benchmark, IIRC). On average, the dispatch overhead is something like 50%. Is it the same for jython? Btw, did you consider the possibility of reusing the pypy eval loop instead of rewriting yet another one from scratch? ciao, Anto
The PBC-VM is already working (or was, see below). On pystone, it runs about 50% of the speed of normally compiled code, and this applies to when it's both cold (50,000 passes) and hotspotted (500,000). I have not looked at anything else, for our use case, this is certainly fast enough. The implementation is in org.python.core.PyBytecode, plus support modules _marshal.java and pycimport.py. We did not consider PyPy, it never occurred to me this was something you had done. In normal Java bytecode, we don't see these sort of dispatch issues. Instead it's attribute lookup, call path, and parts related to frame maintenance that seem to be the biggest issues. Again, not something that has been studied deeply. But we expect to do just that post 2.5.0. Unfortunately, the PBC-VM is not working now. I was passing the entire regr test with it, as of the merge from the pbcvm branch, excluding some aspects around code introspection and differences in float reprs for CPython and Jython, but now there's something trivial breaking it. Looks I need to add a regression test that uses a well-known chunk of pyc to just verify that the calling conventions are fine. Naturally, the PBC-VM will be much easier to test once we implement a PBC compiler. For the moment, we just use the pycimport module, which provides for an alternate load path. - Jim On Wed, Apr 22, 2009 at 1:17 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
-- Jim Baker jbaker@zyasoft.com
Hi Jim, On Wed, Apr 22, 2009 at 01:53:08PM -0600, Jim Baker wrote:
Then I can (again) suggest that you look at PyPy instead of rewriting your own .pyc compiler from scratch -- but that might not be quite what you are looking for, given that (as I understand it) writing PBC would be mostly an extension of your own Python-to-Java compiler. Nevertheless, I'm sure that if you want to try that path, you can try to extract code from PyPy, and plug it in Jython either in the form of pure Python code or compiled Java code. Armin
Armin, I'll take a look at PyPy. However, it's likely we will use our existing naive compiler as the basis for the PBC compiler, since it works within the context of our Antlr grammar, including supporting infrastructure like incremental parses, as well as some other scaffolding like scopes compilation. Thanks! - Jim On Thu, Apr 23, 2009 at 1:42 AM, Armin Rigo <arigo@tunes.org> wrote:
-- Jim Baker jbaker@zyasoft.com
Laura Creighton wrote:
This showed up on another list I read. [cut]
if the JVM-to-davlik converter is not buggy and if the target device is powerful enough, I think that pypy-jvm should work out of the box, as so far it does not do any jit-compilation at all. On the other hand, I have no clue about performances and memory consumption. ciao, Anto
I'll address Jython: we plan support for Android in our 2.5.1 release. Supporting Android is similar to supporting unsigned applets: we either compile Python code in advance to Java bytecode, or compile it on the fly to Python bytecode (PBC) and execute with our PBC VM (just a straight port of ceval.c). Any class proxies also need to be compiled to Java bytecode in advance too, although we could support Java dynamic proxies for interface inheritance only (a significant but hopefully workable restriction, given that we prefer interfaces in Java, but of course only impacts Java integration). Packaging for Dalvik or as applet then becomes similar to say packaging a WAR file for Django or Pylons. The only missing pieces are support for the deployment piece, dynamic proxies, and a PBC compiler, all of which we have started work on, but put on hold for getting 2.5.0 out. - Jim On Wed, Apr 22, 2009 at 2:24 AM, Laura Creighton <lac@openend.se> wrote:
-- Jim Baker jbaker@zyasoft.com
Hi Jim, Jim Baker wrote:
this is very interesting. Is the PBC VM already working? If so, how do its performances compare against the normally compiled code? I and Niko did some benchmarking on pypy-jvm and we discovered that we spent (unsurprisingly) the largest amount of time inside the main eval loop (up to 68% of the total on some benchmark, IIRC). On average, the dispatch overhead is something like 50%. Is it the same for jython? Btw, did you consider the possibility of reusing the pypy eval loop instead of rewriting yet another one from scratch? ciao, Anto
The PBC-VM is already working (or was, see below). On pystone, it runs about 50% of the speed of normally compiled code, and this applies to when it's both cold (50,000 passes) and hotspotted (500,000). I have not looked at anything else, for our use case, this is certainly fast enough. The implementation is in org.python.core.PyBytecode, plus support modules _marshal.java and pycimport.py. We did not consider PyPy, it never occurred to me this was something you had done. In normal Java bytecode, we don't see these sort of dispatch issues. Instead it's attribute lookup, call path, and parts related to frame maintenance that seem to be the biggest issues. Again, not something that has been studied deeply. But we expect to do just that post 2.5.0. Unfortunately, the PBC-VM is not working now. I was passing the entire regr test with it, as of the merge from the pbcvm branch, excluding some aspects around code introspection and differences in float reprs for CPython and Jython, but now there's something trivial breaking it. Looks I need to add a regression test that uses a well-known chunk of pyc to just verify that the calling conventions are fine. Naturally, the PBC-VM will be much easier to test once we implement a PBC compiler. For the moment, we just use the pycimport module, which provides for an alternate load path. - Jim On Wed, Apr 22, 2009 at 1:17 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
-- Jim Baker jbaker@zyasoft.com
Hi Jim, On Wed, Apr 22, 2009 at 01:53:08PM -0600, Jim Baker wrote:
Then I can (again) suggest that you look at PyPy instead of rewriting your own .pyc compiler from scratch -- but that might not be quite what you are looking for, given that (as I understand it) writing PBC would be mostly an extension of your own Python-to-Java compiler. Nevertheless, I'm sure that if you want to try that path, you can try to extract code from PyPy, and plug it in Jython either in the form of pure Python code or compiled Java code. Armin
Armin, I'll take a look at PyPy. However, it's likely we will use our existing naive compiler as the basis for the PBC compiler, since it works within the context of our Antlr grammar, including supporting infrastructure like incremental parses, as well as some other scaffolding like scopes compilation. Thanks! - Jim On Thu, Apr 23, 2009 at 1:42 AM, Armin Rigo <arigo@tunes.org> wrote:
-- Jim Baker jbaker@zyasoft.com
Laura Creighton wrote:
This showed up on another list I read. [cut]
if the JVM-to-davlik converter is not buggy and if the target device is powerful enough, I think that pypy-jvm should work out of the box, as so far it does not do any jit-compilation at all. On the other hand, I have no clue about performances and memory consumption. ciao, Anto
participants (5)
-
Antonio Cuni -
Armin Rigo -
Jim Baker -
Laura Creighton -
Maciej Fijalkowski