Mailman 3 November 2007 - pypy-dev

Removal of RCTypes, the extension compiler, support for PyObjects
by Carl Friedrich Bolz 21 Nov '07

21 Nov '07

Hi all, Richard and I are about to finally remove support for the extension compiler, RCTypes and the ability to support CPython Objects in the translation. We will do this work on the new-genc-tests-wrapper branch (which now has a slightly strange name). If anybody has good arguments for keeping these things we would like to hear about it very soon. The RCTypes functionality is implemented by rffi these days. The extension compiler can be reimplemented with rffi if someone is interested in doing that. The current incarnation is not giving fast modules anyway (and is based on RCTypes and various strange hacks). Cheers, Carl Friedrich & Richard

8 19

Re: [pypy-dev] Removal of RCTypes, the extension compiler, support for PyObjects
by Martijn Faassen 21 Nov '07

21 Nov '07

Hey, On Nov 20, 2007 5:49 PM, Richard Emslie <rxe(a)ukshells.co.uk> wrote: [snip] > To be more accurate - what was found was that the overhead of calling in > and out of the of the generated extension module outweighed the gain in > speed generated by the rpython code. Also the generated code was not 100% > stable and even though I spent a few weeks hacking the generated code, it > was still segfaulting (whereas the non extension module was fine.) All in > all I agree that removing the current implementation is the best course of > action for moving forward. Great! I also understood from the previous discussions the old way was broken. But you talk about moving forward - is the plan to get back to establish the original feature again with new technology? That's not the impression I got from the email, but it'd be great if my impression was wrong. Regards, Martijn

1 0

Re: [pypy-dev] Removal of RCTypes, the extension compiler, support for PyObjects
by Carl Friedrich Bolz 21 Nov '07

21 Nov '07

this message of richard didn't make it to the list, I am forwarding it. ---------- Forwarded message ---------- From: Richard Emslie <rxe(a)ukshells.co.uk> Date: 20.11.2007 17:49 Subject: Re: [pypy-dev] Removal of RCTypes, the extension compiler, support for PyObjects To: Carl Friedrich Bolz <cfbolz(a)gmx.de> Cc: Martijn Faassen <faassen(a)startifact.com>, pypy-dev(a)codespeak.net On Tue, 20 Nov 2007, Carl Friedrich Bolz wrote: > Hi Martijn, > > Martijn Faassen wrote: >> I'll just note what this might mean to you in terms of project >> perception and perception of commitment: >> >> You're replacing one of the few areas of PyPy that at least *seemed* to >> be useful in the near term for production use (even though it evidently >> wasn't really) with "the extension compiler can be reimplemented with >> rffi if someone is interested in doing that". >> >> I.e. you're removing your implicit commitment (in the form of code that >> more or less worked) and replacing it with an explicit lack of >> commitment or interest in making this work. > > I really thought we were through with that. So, again: > > First let me note that I think the extcompiler is a promising idea. I > agree that the idea could lead to something generally useful. However: > The current implementation is not useful at all. It wasn't used > seriously by anybody. Richard tried to use it and he found that it > wasn't creating anything useful, since the resulting CPython > extensions weren't really any faster than just writing the stuff in > pure Python to begin with. > To be more accurate - what was found was that the overhead of calling in and out of the of the generated extension module outweighed the gain in speed generated by the rpython code. Also the generated code was not 100% stable and even though I spent a few weeks hacking the generated code, it was still segfaulting (whereas the non extension module was fine.) All in all I agree that removing the current implementation is the best course of action for moving forward. Cheers - Richard

1 0

pycon 2008
by Simon Burton 20 Nov '07

20 Nov '07

Is anyone planning on going ? doing a talk ? Simon.

3 2

making PyPy production ready: followup
by Martijn Faassen 20 Nov '07

20 Nov '07

Hi there, I'll continue in my role as the unofficial nag of this project. So we've had a useful discussion about making PyPy production ready. Excellent! Now we need to follow up on this. We need to distill a list of concrete tasks from it. We need to figure out who does what. Who in the PyPy project is going to make this list? Who will make sure that it gets done (or decides bits *won't* get done right now)? This brings me to commitment again, and leadership. I know I've asked about commitment before, in connection to the RPython for CPython modules facility. I got the answer "we'll need to discuss this at EuroPython", and there was some discussion there. I got the general impression that finishing the interpreter was considered to be of more importance. The current state of this support is rather in limbo: it's still there, but should be reworked to use new stuff that's there now, and it's unclear who will do it or when. Anyway, that's fine, as the main goal, I take it, is the interpreter. Good. And then I heard nothing much more until this current discussion... Perhaps I missed something? So what about the interpreter? What is the plan to bring it into production? Will the first goal be the PyPy-JVM idea that was thrown around at EuroPython? The C-backend? Something else? Someone needs to show leadership and make a plan. There needs to be a certain level of commitment to this plan from the project and project members. Even explicitly saying you are not going to do X and Y (or not right now) is valuable, as it helps people to know what to expect and where to help. Does the PyPy project commit to working on a Python interpreter in any shape and form that people can use in production at all? One that is released? If so, what's the timeline? When is the alpha going to be, when the beta? Do you only commit to this if you get volunteer contributors? Can you consider lowering ambition and do it without further volunteers? If you need further volunteers, will you give them the responsibility and authority to actually make hard decisions to make a release possible? Note that you'll get more volunteers that contribute if they have something to expect in the foreseeable future. I already mentioned release management. If you don't have some form of leadership on this, you risk scenarios like: * "we can't release right now as we're still working on X" * X is done. in the mean time, someone started Y " "We can't release right now as we're still working on Y" * Y is done. in the mean time, someone started Z * "We can't release right now as we're still working on Z" * and so on. You need to have someone or some group of people who are aware of these issues, who have the aim to make a release, and who can clearly say: yeah, I *know* we really need Y, but we are in stabilization mode right now, so please wait or do it on a branch. Making a release is *hard*. It's lots of frustrating work. You know, as you made the releases for the research part of the project. After this, the releases stopped. If you want a successful open source project, you need to make regular releases that people can use. You need to make new release announcements, as people read them. You also need to show some commitment towards production use to the public, and releases in this direction is the best way to do it. I believe doing this will be of tremendous open source marketing value to the project. You've gone all the way up to 1.0, but you've stated explicitly it's not intended for production use. Now is the time to let the world know you're progressing beyond this. Let the internet know that the PyPy project is a serious contender in the Python interpreter world, instead of an experiment. So, what's the plan? Regards, Martijn P.S. I'm leaving the funding discussion out of this explicitly. I assume you want to make PyPy work as an open source project. I'm also assuming that there is a way to get a production-ready interpreter by volunteer effort without external funding. If the project decides that there is *no* such way, then please let everybody know that too.

2 1

missing things for making PyPy "production" ready (for some value of production)
by Carl Friedrich Bolz 19 Nov '07

19 Nov '07

Hi all, I thought I would start a new thread for discussing what PyPy needs to become "production ready" (whatever that is) and succeed as a Python implementation. This is something were outside opinions are very welcome (of course there is no guarantee we can implement all of your wishes :-) ). Obvious problems are: - some missing extension modules of the stdlib. no clue which of them are really important, depends on the application you want to run, I guess. This can be worked on by outsiders. - lack of 3rd-party extension-module support. This is hard to change. Once we implement ctypes the problem will be mitigated, but not solved: it's no fun at all to wrap a large C++ library with ctypes, and there are lot's of existing non-ctypes based modules out there. - Speed. The JIT is still not in a state where it really speeds up arbitrary Python code. I expect this to change sooner or later. However, it's not an area were a lot of people can help. - for PyPy-JVM: bindings to allow the interaction with arbitrary Java libraries, threading support ... Comments? Cheers, Carl Friedrich

13 27

Re: [pypy-dev] pycon 2008
by Laura Creighton 17 Nov '07

17 Nov '07

In a message of Sat, 17 Nov 2007 02:46:19 +0100, Christian Tismer writes: >Simon Burton wrote: >> Is anyone planning on going ? doing a talk ? > Some people from Open End will be coming. How many is not set yet. Laura

1 0

Modern GC
by Florian Schulze 17 Nov '07

17 Nov '07

Interesting stuff about a modern garbage collector: http://www.cs.technion.ac.il/~erez/projects.html Regards, Florian Schulze

1 0

more problems building PyPy-JVM
by Martijn Faassen 16 Nov '07

16 Nov '07

Hi there, To start off, please note I haven't worked with Java in the past. I'm the proper kind of newbie to try out PyPy-JVM. :) I now get build process process to complete, but then I run into a new problem. Apparently it needs something called 'jasmin'. It was in my Ubuntu Feisty, so I installed it. Now when I run it again, I get: translation:info] Wrote JVM code [translation:info] Compiling JVM source... Invoking jasmin on ['/tmp/usession-1/pypy/pypy/InterlinkImplementation_6.j', '/tmp/usession-1/pypy/pypy/throwUnicodeDecodeError_1.j', '/tmp/usession-1/pypy/pypy/exceptions/ValueError_10.j', '/tmp/usession-1/pypy/pypy/Object_14.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeDecodeError_meta_18.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeError_meta_19.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeDecodeError_8.j', '/tmp/usession-1/pypy/pypy/exceptions/Exception_12.j', '/tmp/usession-1/pypy/pypy/Object_meta_16.j', '/tmp/usession-1/pypy/pypy/ll_raise_OSError__Signed_5.j', '/tmp/usession-1/pypy/pypy/exceptions/EnvironmentError_meta_30.j', '/tmp/usession-1/pypy/pypy/throwZeroDivisionError_4.j', '/tmp/usession-1/pypy/pypy/exceptions/EnvironmentError_27.j', '/tmp/usession-1/pypy/pypy/throwValueError_0.j', '/tmp/usession-1/pypy/pypy/exceptions/ValueError_meta_20.j', '/tmp/usession-1/pypy/pypy/Object_15.j', '/tmp/usession-1/pypy/pypy/throwOverflowError_3.j', '/tmp/usession-1/pypy/pypy/throwIndexError_2.j', '/tmp/usession-1/pypy/pypy/exceptions/OSError_26.j', '/tmp/usession-1/pypy/pypy/Object_13.j', '/tmp/usession-1/pypy/pypy/exceptions/OSError_meta_29.j', '/tmp/usession-1/pypy/pypy/exceptions/OverflowError_meta_45.j', '/tmp/usession-1/pypy/pypy/exceptions/Exception_meta_22.j', '/tmp/usession-1/pypy/pypy/exceptions/StandardError_meta_21.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeError_9.j', '/tmp/usession-1/pypy/pypy/Main.j', '/tmp/usession-1/pypy/pypy/exceptions/StandardError_11.j', '/tmp/usession-1/pypy/pypy/exceptions/LookupError_meta_52.j'] So far so good, I think... Then however: [Timer] Timings: [Timer] annotate - 329.6 s [Timer] rtype_ootype - 104.8 s [Timer] backendopt_ootype - 77.2 s [Timer] source_jvm - 96.8 s [Timer] compile_jvm - 2.0 s [Timer] ======================================== [Timer] Total: - 610.4 s [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 273, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/faassen/install/pypy-dist/pypy/translator/driver.py", line 810, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/faassen/install/pypy-dist/pypy/translator/tool/taskengine.py", line 112, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/faassen/install/pypy-dist/pypy/translator/driver.py", line 281, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/faassen/install/pypy-dist/pypy/translator/driver.py", line 746, in task_compile_jvm [translation:ERROR] self.jvmsource.compile() [translation:ERROR] File "/home/faassen/install/pypy-dist/pypy/translator/jvm/genjvm.py", line 165, in compile [translation:ERROR] self._invoke(jascmd + files, False) [translation:ERROR] File "/home/faassen/install/pypy-dist/pypy/translator/jvm/genjvm.py", line 107, in _invoke [translation:ERROR] raise JvmSubprogramError(res, args, stdout, stderr) [translation:ERROR] JvmSubprogramError: Error code 0 running ['jasmin', '-d', '/tmp/usession-1/pypy', '/tmp/usession-1/pypy/pypy/InterlinkImplementation_6.j', '/tmp/usession-1/pypy/pypy/throwUnicodeDecodeError_1.j', '/tmp/usession-1/pypy/pypy/exceptions/ValueError_10.j', '/tmp/usession-1/pypy/pypy/Object_14.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeDecodeError_meta_18.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeError_meta_19.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeDecodeError_8.j', '/tmp/usession-1/pypy/pypy/exceptions/Exception_12.j', '/tmp/usession-1/pypy/pypy/Object_meta_16.j', '/tmp/usession-1/pypy/pypy/ll_raise_OSError__Signed_5.j', '/tmp/usession-1/pypy/pypy/exceptions/EnvironmentError_meta_30.j', '/tmp/usession-1/pypy/pypy/throwZeroDivisionError_4.j', '/tmp/usession-1/pypy/pypy/exceptions/EnvironmentError_27.j', '/tmp/usession-1/pypy/pypy/throwValueError_0.j', '/tmp/usession-1/pypy/pypy/exceptions/ValueError_meta_20.j', '/tmp/usession-1/pypy/pypy/Object_15.j', '/tmp/usession-1/pypy/pypy/throwOverflowError_3.j', '/tmp/usession-1/pypy/pypy/throwIndexError_2.j', '/tmp/usession-1/pypy/pypy/exceptions/OSError_26.j', '/tmp/usession-1/pypy/pypy/Object_13.j', '/tmp/usession-1/pypy/pypy/exceptions/OSError_meta_29.j', '/tmp/usession-1/pypy/pypy/exceptions/OverflowError_meta_45.j', '/tmp/usession-1/pypy/pypy/exceptions/Exception_meta_22.j', '/tmp/usession-1/pypy/pypy/exceptions/StandardError_meta_21.j', '/tmp/usession-1/pypy/pypy/exceptions/UnicodeError_9.j', '/tmp/usession-1/pypy/pypy/Main.j', '/tmp/usession-1/pypy/pypy/exceptions/StandardError_11.j', '/tmp/usession-1/pypy/pypy/exceptions/LookupError_meta_52.j'] Do I need a different version of jasmin? 'jasmin -version' reports: Jasmin version: v sable-1.2 Also for my next step, assuming I get it to work after this, what'd I need do to start it up to run a script or see a Python prompt? The pypy docs on the website still have the JVM project being in an earlier stage than it really is, so it doesn't really say yet. Regards, Martijn

2 1

another pypy blog entry from sun
by Stephan Diehl 16 Nov '07

16 Nov '07

just in case you don't read programming.reddit.com: there is another blog enty from somebody at Sun (John Rose) http://blogs.sun.com/jrose/entry/a_day_with_pypy Have fun Stephan

3 4