Mailman 3 October 2009 - pypy-dev

Re: [pypy-dev] [pypy-svn] r68612 - pypy/trunk/pypy/rpython/lltypesystem
by Maciej Fijalkowski 17 Oct '09

17 Oct '09

> On Linux, to clear large regions of memory, the best way seems > to be madvise(MADV_DONTNEED). Indeed, the previous trick of > reading /dev/zero does not seem to have the correct effect of > lazily-allocating pages on all Linux systems (e.g. bigdog-vm1). > Note that this usage of madvise() is purely Linux-specific. > Note that bigdogvm1 is a vm, it might behave differently...

1 0

Mac osx patches
by Leonardo Santagada 15 Oct '09

15 Oct '09

To make compilation and asmgcroot happy on osx I made some changes, I would like to know if someone has anything against applying this stuff to trunk? Index: pypy/translator/c/gcc/trackgcroot.py =================================================================== --- pypy/translator/c/gcc/trackgcroot.py (revision 68036) +++ pypy/translator/c/gcc/trackgcroot.py (working copy) @@ -22,7 +22,11 @@ r_sectionstart = re.compile(r"\t\.("+'|'.join(OTHERSECTIONS) +").*$") r_functionstart_darwin = re.compile(r"_(\w+):\s*$") -OFFSET_LABELS = 2**30 +# darwin's ld complain about this hack +if sys.platform == 'darwin': + OFFSET_LABELS = 0 +else: + OFFSET_LABELS = 2**30 # inside functions LABEL = r'([.]?[\w$@]+)' Index: pypy/translator/platform/test/test_darwin.py =================================================================== --- pypy/translator/platform/test/test_darwin.py (revision 68036) +++ pypy/translator/platform/test/test_darwin.py (working copy) @@ -2,8 +2,8 @@ """ File containing darwin platform tests """ -import py, os -if os.name != 'darwin': +import py, sys +if sys.platform != 'darwin': py.test.skip("Darwin only") from pypy.tool.udir import udir Index: pypy/translator/platform/darwin.py =================================================================== --- pypy/translator/platform/darwin.py (revision 68036) +++ pypy/translator/platform/darwin.py (working copy) @@ -18,7 +18,7 @@ self.cc = cc def _args_for_shared(self, args): - return (self.shared_only + ['-bundle', '-undefined', 'dynamic_lookup'] + return (self.shared_only + ['-dynamiclib', '-undefined', 'dynamic_lookup'] + args) def include_dirs_for_libffi(self): -- Leonardo Santagada santagada at gmail.com

4 7

Fuzzing with Fusil and JIT crash
by Victor Stinner 10 Oct '09

10 Oct '09

Hi, Because of the last posts on your blog (a JIT better than just a PoC), I choosed to give a new try to PyPy. I'm trying PyPy with Fusil the fuzzer because it's my project, so I can easily fix bugs in Fusil, and also because it's a great tool to find bugs in PyPy :-) I already found some bugs/crashs (see PyPy's bugtracker), but it's not bad (there are few bugs if you compare it to other projects). Since PyPy is open source, I can write patches to fix the issues ;-) Slowly, I know PyPy enough to fix RPython bugs (I mean to fix modules written in RPython). But now my problem are the JIT crashes. I'm unable to locate the crash. There is not enough informations: BROKEN PROFILER DATA! ~~~ Crash in JIT! ~~~ <AssertionError object at 0x8449878> BROKEN PROFILER DATA! Fatal RPython error: CrashInJIT Ok, nice, a crash. But where? What is the assertion? Sometimes, it fails with a TypeError (same output except the exception type). gdb doesn't help: (gdb) where #0 0xb7fe1424 in __kernel_vsyscall () #1 0xb7c1d3d0 in raise () from /lib/i686/cmov/libc.so.6 #2 0xb7c20a85 in abort () from /lib/i686/cmov/libc.so.6 #3 0x0804ce1e in main () "main()", that's all. No file name or line number :-/ On IRC, we asked me to recompiled PyPy using the hidden "make debug_exc" command. But it doesn't change anything. I'm not sure that the make did really enable the hidden debug features. How can I check it? Does it output something special? Always or only on a crash. On a crash, the output is the same using pypy-c-jit ("make") or pypy-c-jit-debug ("make debug_exc"). I will retry to recompile PyPy from scratch using directly the right commmand ("make debug_exc"). Note: Don't try to compile PyPy C backend using -O0 or -O1. Only "-O3" and "- O2 -fomit-frame-pointer" are supported. Note2: "make clean" doesn't remove testing_1.s. Someone told me that it should be fixed now. Oh, another problem: on make failure, all .s files are removed. It's not possible to debug the error. I added ".PRECIOUS: %.s" in the Makefile. Can it be done by default? --- If you would like to try Fusil, it's available in Debian, Ubuntu, Mandriva, OpenEmbedded, Arch Linux, MacPort, Gentoo, ... For better performances and fewer false positives, clone the Mercurial repository. Fusil website: http://bitbucket.org/haypo/fusil/wiki/Home To reproduce the JIT crash, it takes 20 to 60 seconds using the following command: export PYTHONPATH=~/prog/HG/fusil/:~/prog/HG/python-ptrace/ ./pypy-c-jit-debug --jit threshold=10 \ ~/prog/HG/fusil/fuzzers/fusil-python \ --success=100 --blacklist=logging,compileall,this,socket (socket is blacklisted because of the issue #465, a crash in _ssl) You don't need to install Fusil if you change the PYTHONPATH. -- Victor Stinner http://www.haypocalc.com/

1 0

Help with "Exception: unexpected prebuilt constant"
by Philip Guo 09 Oct '09

09 Oct '09

Sorry for another newbish question, but I'm having trouble getting PyPy translated into C code. I added some calls to open() throughout the interpreter codebase (to log some info to files), but it failed translation with the following error: [translation:ERROR] Exception: unexpected prebuilt constant: <built-in function open> I then tried doing 'import os' and then using os.fdopen, but with the same error: [translation:ERROR] Exception: unexpected prebuilt constant: <built-in function fdopen> In general, what Python standard library calls can I make from my modified PyPy code and still have it translate properly? Specifically, how do I read/write files from within PyPy? Thanks! Philip

2 3

Re: [pypy-dev] Nightly Benchmarks
by Anders Hammarquist 08 Oct '09

08 Oct '09

In a message of Mon, 05 Oct 2009 23:04:16 +0200, Miquel Torres writes: >There are a couple of things I want to discuss: > >- An extra table (class) may be needed to store the hardware the benchmark >was run on. pypy will probably need to be tested on other architectures. Keeping track of the architecture makes sense. Possibly it could go as a field on the interpreter instead. >- The interpreter table needs extra fields: > - revision: absolutely necessary My original thoughts for the revision would be to put it on the Run, but if you are going to add several different languages it probably makes more sense to put it on the interpreter. This would be the version number for released (non-PyPy) interpreters. > - tag: this is for 1.0, 1.1, beta, trunk, snapshot, or what ever. This >is needed to know what pypy revisions to list in the web page that allows >comparison to other python implementations (most importantly cpython). > - Furthermore: You state that compile options are not needed, that they >would be different interpreters. >I agree that we can have pypy-c, pypy-crl and pypy-c-jit (for example) as >different interpreters. But things can get messy if we add garbage >collectors and other options: >pypy-c-68150-gc=hybrid--opt=3--no-allworkingmodules I think we misunderstood eachother on this one. I would certainly add a field for it on the interpreter object, keeping it in the name could quickly get messy. I thought you were talking about adding compile options to the Run. /Anders

1 0

Does PyPy supports Java ME?
by Jan Wedel 08 Oct '09

08 Oct '09

Hi, I was currently working on a port of PyMite (8-Bit MCU port of CPython) to Java ME. I already got the interpreter working so that basic Python scripts could run as code images on a mobile java IMP device. Actually we need Python interpreters on different embedded platforms supporting either C or Java ME (IMP). I discovered a lot of problems including the missing class implementation of PyMite. Then, by recommendation, I found PyPy. It looks as if it is exactly what I was looking for and I read that it is specifically designed to support embedded platforms. But I want to clarify a few things: 1.) Does PyPy generate both C and Java Bytecode for the Interpreter out of the box or with low effort? 2.) Is the java code compatible with Java ME or is it possible to modify the PyPy-code/Interpreter-code so that Java ME is supported? 3.) What standard libraries are supported? I coulnd't find a list. Especially the communication (such as socket) functionality is important. 4.) Do you think PyPy is stable enough for commercial use? The MIT license allows commercial use as far as I read. Thank you very much! Regards, Jan

3 4

Re: [pypy-dev] Does PyPy supports Java ME?
by Antonio Cuni 08 Oct '09

08 Oct '09

Jan Wedel wrote: > Hi Anto! > > Thanks for your answer. Hi Jan, (I'm cc-ing pypy-dev again) > Although it's now made by Sun, it shows the key points: > - No class loading > - No JNI > - No reflection > > Which means you cannot load dynamic code during run-time. On top, there > are a lot of libraries missing (XML parsing, Regex, logging...) so far pypy-jvm doesn't use any of those, so it should not be a problem. However, in the future reflection will be surely needed to access java libraries (but you don't seem to be interested in it) and class loading for the jit. > THAT might be a problem since we are working on memory constrained > systems with a maximum of a few MB of flash and approx. 200kB RAM. Are > there any estimations about the RAM usage of PyPy using classes and > other memory-consuming features? I think that 200kB are definitely not enough for pypy-jvm. I don't know for sure how to measure memory usage of java programs, but if I just launch pypy-jvm and measure the amount of memory used on linux, I get something like ~400 MB: this also includes the overhead of the jvm itself, but it's still a lot. > And is this 8-10MB including only the core interpreter? Or are there any > ways to strip it down by excluding unnecessary libs? it includes the core interpreter and the builtin modules; some of them can be stripped down, but I don't know how much you can save. > I didn't make myself clear. I meant python libs/modules. E.g. sys, > socket, sre etc... Because if these modules are supported I won't need a > lot of Java Libraries. As I explained, I already ported PyMite to Java > and I discovered that numerous python modules have a C counterpart such > as _socket, _sre etc. So If you want to run these libs, you need to > translate these C files. ah, now I understand. This is the complete list of builtin modules that are enabled for pypy-jvm: [translation] [usemodules] [translation] __pypy__ = True [translation] _ast = True [translation] _codecs = True [translation] _pickle_support = True [translation] _random = True [translation] _sre = True [translation] _testing = True [translation] _weakref = True [translation] cStringIO = True [translation] errno = True [translation] gc = True [translation] itertools = True [translation] marshal = True [translation] math = True [translation] md5 = True [translation] operator = True [translation] parser = True [translation] posix = True [translation] rbench = True [translation] sha = True [translation] symbol = True [translation] time = True [translation] token = True As you can see, _socket is missing. It's surely possible to write a version of it that works for the jvm, but as usual it takes some time and nobody cared enough to implement it. ciao, Anto

2 1

Problems translating PyPy to Java
by Jan Wedel 08 Oct '09

08 Oct '09

Ok, I tried to translate PyPy to Java using ./translate.py --backend=jvm targetpypystandalone.py But it tried to call the MS Compiler cl.exe which I don’t have. I’m wondering why? This results in several errors (see below). What’s happening? Thanks, Jan C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dateien\repo\trunk\rialto\tools\ pypy-1.1.0\pypy\translator\goal>translate.py --backend=jvm targetpypystandalone. py [translation:info] Translating target as defined by targetpypystandalone [platform:error] Could not find a Microsoft Compiler [platform:msg] Setting platform to 'host' cc=None [translation] PyPy config object: [translation] [objspace] [translation] compiler = ast [translation] nofaking = True [translation] [opcodes] [translation] CALL_LIKELY_BUILTIN = True [translation] CALL_METHOD = True [translation] [std] [translation] builtinshortcut = True [translation] getattributeshortcut = True [translation] multimethods = doubledispatch [translation] optimized_list_getitem = True [translation] withmethodcache = True [translation] withmultidict = True [translation] withprebuiltchar = True [translation] withrangelist = True [translation] withshadowtracking = True [translation] withtypeversion = True [translation] [usemodules] [translation] __pypy__ = True [translation] _codecs = True [translation] _pickle_support = True [translation] _random = True [translation] _sre = True [translation] _weakref = True [translation] cStringIO = True [translation] errno = True [translation] gc = True [translation] itertools = True [translation] marshal = True [translation] math = True [translation] md5 = True [translation] operator = True [translation] posix = True [translation] rbench = True [translation] recparser = True [translation] sha = True [translation] symbol = True [translation] time = True [platform:execute] cl.exe /nologo /c /MD /O2 /IC:\Python26\PC c:\windows\temp\us ession-2\gcctest.c /Foc:\windows\temp\usession-2\gcctest.obj [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\translator\goal\translate.p y", line 257, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\translator\driver.py", line 723, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 202, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 209, in get_entry_po int [translation:ERROR] space = make_objspace(config) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\tool\option.py", line 48, in make_o bjspace [translation:ERROR] space = Space(config) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\interpreter\baseobjspace.py ", line 265, in __init__ [translation:ERROR] self.initialize() [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\objspace\std\objspace.py", line 73, in initialize [translation:ERROR] self.model = StdTypeModel(self.config) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\objspace\std\model.py", line 88, in __init__ [translation:ERROR] import pypy.objspace.std.marshal_impl # install marshal multimethods [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\objspace\std\marshal_impl.p y", line 40, in <module> [translation:ERROR] from pypy.module.marshal.interp_marshal import register [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\module\marshal\interp_marsh al.py", line 4, in <module> [translation:ERROR] from pypy.module._file.interp_file import W_File [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\module\_file\interp_file.py ", line 3, in <module> [translation:ERROR] from pypy.rlib import streamio [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rlib\streamio.py", line 167, in <mo dule> [translation:ERROR] from pypy.rlib import rwin32 [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rlib\rwin32.py", line 5, in <module > [translation:ERROR] from pypy.rpython.tool import rffi_platform [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rpython\tool\rffi_platform. py", lin e 5, in <module> [translation:ERROR] from pypy.rpython.lltypesystem import rffi [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rpython\lltypesystem\rffi.p y", line 348, in <module> [translation:ERROR] NUMBER_TYPES = setup() [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rpython\lltypesystem\rffi.p y", line 340, in setup [translation:ERROR] tp = platform.inttype(name.upper(), c_name, signed) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rpython\tool\rfficache.py", line 53 , in inttype [translation:ERROR] bits = sizeof_c_type(c_name, **kwds) * 8 [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rpython\tool\rfficache.py", line 39 , in sizeof_c_type [translation:ERROR] answer = ask_gcc(question, **kwds).split('=') [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\rpython\tool\rfficache.py", line 34 , in ask_gcc [translation:ERROR] return build_executable_cache([c_file], eci) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\tool\gcc_cache.py", line 22, in bui ld_executable_cache [translation:ERROR] result = platform.execute(platform.compile(c_files, eci) ) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\translator\platform\__init_ _.py", l ine 61, in compile [translation:ERROR] ofiles = self._compile_o_files(cfiles, eci, standalone) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\translator\platform\__init_ _.py", l ine 70, in _compile_o_files [translation:ERROR] ofiles.append(self._compile_c_file(self.cc, cfile, compi le_args)) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\translator\platform\windows .py", li ne 143, in _compile_c_file [translation:ERROR] self._execute_c_compiler(cc, args, oname) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\translator\platform\__init_ _.py", l ine 98, in _execute_c_compiler [translation:ERROR] returncode, stdout, stderr = _run_subprocess(cc, args, s elf.c_environ) [translation:ERROR] File "C:\Dokumente und Einstellungen\Jan Wedel\Eigene Dat eien\repo\trunk\rialto\tools\pypy-1.1.0\pypy\translator\platform\__init_ _.py", l ine 25, in _run_subprocess [translation:ERROR] pipe = Popen(args, stdout=PIPE, stderr=PIPE, shell=shell , env=env) [translation:ERROR] File "C:\Python26\lib\subprocess.py", line 595, in __init __ [translation:ERROR] errread, errwrite) [translation:ERROR] File "C:\Python26\lib\subprocess.py", line 804, in _execu te_child [translation:ERROR] startupinfo) [translation:ERROR] WindowsError: [Error 2] Das System kann die angegebene Date i nicht finden [translation] start debugger... > c:\python26\lib\subprocess.py(810)_execute_child() -> raise WindowsError(*e.args) (Pdb+) exit

3 10

is there incremental building for the translator?
by Philip Guo 06 Oct '09

06 Oct '09

Hi all, Sorry for another newbie question, but I'm wondering whether there is any sort of "Makefile"-like incremental building functionality for the PyPy translator. I just ran: cd pypy/translator/goal python translate.py --opt=3 targetpypystandalone.py to create pypy-c, which works perfectly, but it took about an hour on my machine. If I start hacking the PyPy source code, it seems like I have to do the full translation step each time I want to create a new pypy-c. Is there any way around this? Thanks, Philip

3 5

gcc 4.4
by Maciej Fijalkowski 06 Oct '09

06 Oct '09

I've seen various reports about gcc 4.4 misbehaving. I installed this on bigdog-vm2, so if anyone feels like giving it a shot, it's there. Cheers, fijal

1 0