[pypy-dev] Work on the JVM backend

[Michal Bendowski]

2012/1/15 Maciej Fijalkowski <fijall at gmail.com>:
> On Sun, Jan 15, 2012 at 11:28 PM, Michał Bendowski <michal at bendowski.pl> wrote:
>>
>>
>> On Sunday, 15 January 2012 at 0:41 , Michał Bendowski wrote:
>>
>>> On Saturday, 14 January 2012 at 22:28 , Maciej Fijalkowski wrote:
>>>
>>> > On Sat, Jan 14, 2012 at 11:18 PM, Michał Bendowski <michal at bendowski.pl (mailto:michal at bendowski.pl)> wrote:
>>> > >
>>> > >
>>> > > On Friday, 13 January 2012 at 16:02 , Antonio Cuni wrote:
>>> > >
>>> > > > Hello Michał,
>>> > > >
>>> > > > On 01/12/2012 09:24 PM, Michał Bendowski wrote:
>>> > > > > Hello everyone,
>>> > > > >
>>> > > > > Back in the summer I asked on this mailing list if there's interest in moving the JVM backend forward. Back then there was some enthusiasm, so I got back to it when I had the chance, which unfortunately was a few months later. The suggestion back then was to look into using JPype to integrate more closely with Java-side code, and that's what I would like to do.
>>> > > > >
>>> > > > > But before that, I noticed that the JVM backend fails to translate the standard interpreter and spent some time lately getting to know the code and trying to get it to work. What I have right now is a version that outputs valid Jasmin files, which unfortunately still contain some invalid bytecodes (longs vs ints from what I've seen, I'll look into it next).
>>> > > >
>>> > > >
>>> > > > the long vs int problems are likely due to the fact that you are translating
>>> > > > on a 64 bit machine. The translator toolchain assumes that the "native" long
>>> > > > type of the target platform is the same as the source one, but this is not the
>>> > > > case if you are targeting the JVM (where long is 32 bit) on a 64 bit linux
>>> > > > (where long is 64 bit).
>>> > > >
>>> > > > This problem is not easily solvable, so my suggestion is just to translate
>>> > > > pypy-jvm inside a 32bit chroot for now.
>>> > > >
>>> > > > > It would be awesome if someone could take a look at my changes. What's the best way to submit them? Bitbucket pull requests? They will need to go through some review - do you have a workflow for that?
>>> > > >
>>> > > >
>>> > > > we don't have any precise workflow, although a bitbucket pull request might be
>>> > > > the easiest thing to do. I'll be glad to review it.
>>> > > >
>>> > > > > Here's a short list of stuff I found and fixed (hopefully):
>>> > > > > - support the ll_getlength method of StringBuilders in ootype,
>>> > > > > - make compute_unique_id work on built-ins (StringBuilders again).
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > not sure what you mean here. What is the relation between compute_unique_id
>>> > > > and StringBuilder?
>>> > > >
>>> > > > > - provide oo implementations (or stubs) for pypy__rotateLeft, pypy__longlong2float etc.
>>> > > > > - handle rffi.SHORT and rffi.INT showing up in graphs. For now I try to emit something that makes sense (seemed easier), but the right solution is probably to see if the code in question (rbigint, rsha) can be implemented on the java level.
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > yes, this is another issue that has been around for a long time. In theory, we
>>> > > > would like to be able to write per-backend specific code which overrides the
>>> > > > default implementation. This would be useful for rbigint and rsha, but also
>>> > > > e.g. for rlib.streamio. However, we never wrote the infrastructure to do that.
>>> > > >
>>> > > > > - handle the jit_is_virtual opcode - I had no idea how to "safely ignore" it for now, is False the safe answer?
>>> > > >
>>> > > >
>>> > > > yes. Look at translator/c/funcgen.py:848: this is how jit_is_virtual is
>>> > > > implemented by the C backend, you can see that it always returns 0/
>>> > > >
>>> > > > > I hope someone can help me to submit the changes and maybe guide with further work.
>>> > > >
>>> > > >
>>> > > > Please put your work on bitbucket, I'll review it. I'd greatly appreciate if
>>> > > > you committed small checkins (one for each fix/feature you are doing) instead
>>> > > > of one giant commit with all the changes :-)
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > OK, I got myself a 32bit environment and created the pull request (. I'll be grateful for any feedback. One thing I didn't do was to create regression tests against the problems I found - I didn't know where to put the tests and what (and how) exactly to test. If you can shed some light on it, that would be awesome.
>>> >
>>> >
>>> > Lack of tests is a no-no in PyPy world :) Look how current tests are
>>> > implemented in pypy/translator/jvm/test/ and either extend those or
>>> > the base classes. You run them using py.test (which comes included
>>> > with pypy), refer to py.test documentation for details
>>>
>>>
>>>
>>>
>>> I'll look into it, looks like a whole new codebase to grep through (and I already found a bug in my code). I'll create a new pull request when I'm ready with the tests :)
>> OK - I have create another pull requests here: https://bitbucket.org/pypy/pypy/pull-request/20/improvements-to-the-jvm-backend-this-time
>>
>> The previous one should be rejected/deleted, it seems impossible from my side. I will be grateful for comments about the changes.
>>
>> Michał
>>
>>
>
> That sounds like a good step forward, however, why the tests are
> skipped? They should be passing now.

What do you mean? I didn't add any skipping code (except for
append_charpsize). What I did find out was that on a 64 bit system all
JVM tests get skipped (because of pypy/translator/jvm/conftest.py) -
is that what you mean?

> Also primitives (like float2longlong) miss tests I think.

They also miss implementations. Because JVM lacks the unsigned types,
the whole problem of translating the RFFI code for rbigint etc. seems
complex. For now I wanted to move the translation process forward, and
worry about the numeric calculations when we have something running at
all. Should I write tests that skip with "not implemented yet"
message?

Michał