Mailman 3 April 2013 - pypy-dev

Further details regarding the GSOC 2013 project idea
by Hareesan 15 Apr '13

15 Apr '13

Hi there, I'm Hareesan, An undergraduate from University of Moratuwa, Sri Lanka. I have gone through the GSOC 2013 ideas from pypy. I'm much interested in working on the project "New JitViewer infrastructure". I have cloned the source and gone through pypy jitviewer introduction paper.<http://people.cs.uct.ac.za/~tmullins/work/writeup.pdf> To start up working with the project, it would be great if anyone could direct me with further details such as what limitations exists, and what are the features to be implemented what improvements required in the existing product. Thank you -- *Hareesan R Undergraduate Department of Computer Science And Engineering University Of Morat**uwa Sri Lanka**** *

2 1

Fwd: Windows 64 bit version
by Maciej Fijalkowski 14 Apr '13

14 Apr '13

windows 64 patches that I got, can someone have a look? ---------- Forwarded message ---------- From: Roger Flores <aidembb(a)yahoo.com> Date: Wed, Apr 10, 2013 at 9:02 PM Subject: Re: Windows 64 bit version To: Maciej Fijalkowski <fijall(a)gmail.com> OK, sorry this is going to be long. The short version is I have attached a diff for C files in rpython/translator/c handling long != ptr except for three files. >It depends on the context. Essentially the approach is to run tests and figure out what's going on. Starting from rpython/translator/c OK, looking around rpython/translator/c I see that there are C files. Looking through there, I see cases where void * is being cast to (long). I started scoping out the extent of the change and realized it was a finite set, not too huge. So I decided to just make the changes and send you diff so we can all just stop talking about it. Then I realized that there were (unsigned long) cases. And then ulong. And ULONG. And ULLONG. And so on. Yes, I got sucked in. In the end I reviewed ALL cases involving "long", caseless. The approach: At first I quickly changed (long) to (long long). Then I realized that long long is 64 bits on 32 bit builds. Either we need to 1) conditionally compile each line of code based on the platform or 2) use a type that we magically change once based on the platform. Then I noticed that ptrdiff_t is an existing type that is magically sized correctly for all platforms. So now (long) is (ptrdiff_t) with no header changes. The diff covers all C files. It's not big. I don't know the command to run tests there so I didn't even check linux builds, let alone win64, which I'm extremely reluctant to set up. But what I changed should be largely obvious. In particular please review stack.c and stacklet.c which used unsigned long and that might be an issue. The three files I did not change are thread_pthread.c, dtoa.c and obmalloc.c. The changes are small but I didn't have high confidence. obmalloc makes it's own ulong for some reason. There are also python files. I examined them for Long usage but couldn't find a clear pattern. Can you point out one example of what needs changing? >it would go in rpython/jit/backend/x86 I looked around but I'm going to need more direction. I was specifically looking for the linux and win32 portions to compare and again, figure out the size of the change. All I found though were register descriptions and code to emit assembly instructions. -Roger ________________________________ From: Maciej Fijalkowski <fijall(a)gmail.com> To: Roger Flores <aidembb(a)yahoo.com> Cc: Amaury Forgeot d'Arc <amauryfa(a)gmail.com>; matti picus <matti.picus(a)gmail.com>; PyPy Developer Mailing List <pypy-dev(a)python.org> Sent: Tuesday, April 9, 2013 12:52 AM Subject: Re: Windows 64 bit version On Tue, Apr 9, 2013 at 9:47 AM, Roger Flores <aidembb(a)yahoo.com> wrote: > Maciej Fijalkowski wrote: >>It's just a bunch of work. There is nothing special or magic about it, but >> so far noone volunteered to spend enough effort there. > > Got it. I'm just trying to understand what the work is because that list > hasn't been captured anywhere yet. > > >>Essentially, the main problem is that sizeof(long) != sizeof(intptr_t), >> which is not a big problem per-se, it's just that this assumption has to be >> removed from places in the RPython source code. > > OK. Could you find one or two of those places in the RPython source code so > we can see? And what is the fix? Does long get changed to long long or > another type or ? It depends on the context. Essentially the approach is to run tests and figure out what's going on. Starting from rpython/translator/c > > >>Another one is the calling convention for the JIT, which is different than >> on linux. Again, nothing special, just more work. > > Does "calling convention for the JIT" mean a function calling convention for > 64 bit windows or something else? Where is this in the code, or maybe, > where is it for the Windows 32 code and where would the 64 bit version go? > There's no magic for exceptions, gc, or anything else? it would go in rpython/jit/backend/x86 > > > Thanks for the details, > -Roger > >

5 7

Control flow graph for method
by Evgeny Startsev 14 Apr '13

14 Apr '13

I am interesting in control-flow graph generation for python projects using python. I read http://doc.pypy.org/en/latest/translation.html#flow-model and studied how to investigate cfg for function. Is there any way to generate cfg for method using pypy?

2 2

Re: [pypy-dev] [pypy-commit] pypy default: preallocate list to hold the locations of arguments to a call
by Maciej Fijalkowski 13 Apr '13

13 Apr '13

There is an optimization that does it for you FYI (or just use newlist_hint) On Sat, Apr 13, 2013 at 6:13 PM, bivab <noreply(a)buildbot.pypy.org> wrote: > Author: David Schneider <david.schneider(a)picle.org> > Branch: > Changeset: r63312:1bd588482153 > Date: 2013-04-13 18:00 +0200 > http://bitbucket.org/pypy/pypy/changeset/1bd588482153/ > > Log: preallocate list to hold the locations of arguments to a call > > diff --git a/rpython/jit/backend/arm/regalloc.py b/rpython/jit/backend/arm/regalloc.py > --- a/rpython/jit/backend/arm/regalloc.py > +++ b/rpython/jit/backend/arm/regalloc.py > @@ -554,10 +554,9 @@ > return self._prepare_call(op) > > def _prepare_call(self, op, force_store=[], save_all_regs=False): > - args = [] > - args.append(None) > + args = [None] * (op.numargs()+1) > for i in range(op.numargs()): > - args.append(self.loc(op.getarg(i))) > + args[i+1] = self.loc(op.getarg(i)) > # spill variables that need to be saved around calls > self.vfprm.before_call(save_all_regs=save_all_regs) > if not save_all_regs: > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit

2 1

Redis «clone» using pypy stm-thread-2 (Was: Redis clone)
by Amirouche Boubekki 13 Apr '13

13 Apr '13

Héllo again, 2013/4/9 Amirouche Boubekki <amirouche.boubekki(a)gmail.com> > Héllo, > > I'm working on porting a redis clone to pypy-stm [1]. > > I did some benchmark, the interesting code is [2]. it runs 5 thread both > on sides client/server, it might be much, what do you think ? I have 4 > cores. > > (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python > test_client_threaded.py > start training > training done > start spellchecking > 1181 out of 4236 > > real 34,81s > user 12,76s > sys 0,40s > > (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python > test_client.py > start training > training done > start spellchecking > 1181 out of 4234 > > real 61,22s > user 5,69s > sys 0,87s > > I'm thinking about implementing other algorithms; no sure what yet > probably bloom filters or bit arrays but I don't have right now a usecase > in mind. > > I also started Graph database [3] which AFAIK should work and which also > has even if MVCC some atomic call to be used, the test I'm aiming with this > one is loading a wikipedia. > > How can I help you better ? > > > Thanks. > > > Amirouche > > > [1] https://github.com/amirouche/Memo/tree/pypy-stm > [2] a simple suggestion algorithm: > https://github.com/amirouche/Memo/blob/pypy-stm/server/memo/structures/sugg… > [3] http://blog-amirouche.dotcloud.com/notes/projects/graphitidb.html > I did another benchmark with 2 thread both sides, numbers are... different: (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python test_client_threaded.py start training training done start spellchecking 1181 out of 4236 real 14,66s user 5,79s sys 0,60s (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python test_client.py start training training done start spellchecking 1181 out of 4234 real 12,63s user 5,49s sys 0,66s Also even if I've written it's redis clone, it's not quite as useful as Redis but I though that it could be useful to test pypy stm, HTH, Amirouche

2 6

Re: [pypy-dev] [pypy-commit] pypy default: make looking up encodings free with the JIT
by Antonio Cuni 12 Apr '13

12 Apr '13

Hi Alex, could we have a test_pypy_c test for this please? On 10/04/13 03:53, alex_gaynor wrote: > Author: Alex Gaynor <alex.gaynor(a)gmail.com> > Branch: > Changeset: r63186:c514bbc4c086 > Date: 2013-04-09 19:53 -0700 > http://bitbucket.org/pypy/pypy/changeset/c514bbc4c086/ > > Log: make looking up encodings free with the JIT > > diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py > --- a/pypy/module/_codecs/interp_codecs.py > +++ b/pypy/module/_codecs/interp_codecs.py > @@ -1,10 +1,18 @@ > +from rpython.rlib import jit > +from rpython.rlib.objectmodel import we_are_translated > +from rpython.rlib.rstring import UnicodeBuilder > + > from pypy.interpreter.error import OperationError, operationerrfmt > from pypy.interpreter.gateway import interp2app, unwrap_spec, WrappedDefault > -from rpython.rlib.rstring import UnicodeBuilder > -from rpython.rlib.objectmodel import we_are_translated > + > + > +class VersionTag(object): > + pass > > > class CodecState(object): > + _immutable_fields_ = ["version?"] > + > def __init__(self, space): > self.codec_search_path = [] > self.codec_search_cache = {} > @@ -14,6 +22,7 @@ > self.encode_error_handler = self.make_encode_errorhandler(space) > > self.unicodedata_handler = None > + self.modified() > > def _make_errorhandler(self, space, decode): > def call_errorhandler(errors, encoding, reason, input, startpos, > @@ -86,9 +95,20 @@ > self.unicodedata_handler = UnicodeData_Handler(space, w_getcode) > return self.unicodedata_handler > > + def modified(self): > + self.version = VersionTag() > + > + def get_codec_from_cache(self, key): > + return self._get_codec_with_version(key, self.version) > + > + @jit.elidable > + def _get_codec_with_version(self, key, version): > + return self.codec_search_cache.get(key, None) > + > def _cleanup_(self): > assert not self.codec_search_path > > + > def register_codec(space, w_search_function): > """register(search_function) > > @@ -115,11 +135,12 @@ > "lookup_codec() should not be called during translation" > state = space.fromcache(CodecState) > normalized_encoding = encoding.replace(" ", "-").lower() > - w_result = state.codec_search_cache.get(normalized_encoding, None) > + w_result = state.get_codec_from_cache(normalized_encoding) > if w_result is not None: > return w_result > return _lookup_codec_loop(space, encoding, normalized_encoding) > > + > def _lookup_codec_loop(space, encoding, normalized_encoding): > state = space.fromcache(CodecState) > if state.codec_need_encodings: > @@ -143,6 +164,7 @@ > space.wrap("codec search functions must return 4-tuples")) > else: > state.codec_search_cache[normalized_encoding] = w_result > + state.modified() > return w_result > raise operationerrfmt( > space.w_LookupError, > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit >

4 5

OSError not ImportError from 'import sqlite3' on Windows
by Peter Cock 09 Apr '13

09 Apr '13

Hello all, I've downloaded and unzipped pypy-2.0-beta2-win32.zip but fairly early on in my testing ran into this puzzle: C:\>c:\pypy-2.0\pypy.exe Python 2.7.3 (3eef596df459, Apr 06 2013, 12:40:10) [PyPy 2.0.0-beta2 with MSC v.1500 32 bit] on win32 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``the zen attitude to programming: reducing the oopses in your life'' >>>> import sqlite3 Traceback (most recent call last): File "<console>", line 1, in <module> File "c:\pypy-2.0\lib-python\2.7\sqlite3\__init__.py", line 24, in <module> from dbapi2 import * File "c:\pypy-2.0\lib-python\2.7\sqlite3\dbapi2.py", line 27, in <module> from _sqlite3 import * File "c:\pypy-2.0\lib_pypy\_sqlite3.py", line 262, in <module> if _has_load_extension(): File "c:\pypy-2.0\lib_pypy\_sqlite3.py", line 259, in _has_load_extension unverified_lib = unverified_ffi.dlopen('sqlite3') File "c:\pypy-2.0\lib_pypy\cffi\api.py", line 111, in dlopen lib, function_cache = _make_ffi_library(self, name, flags) File "c:\pypy-2.0\lib_pypy\cffi\api.py", line 364, in _make_ffi_library raise OSError("library not found: %r" % (name,)) OSError: library not found: 'sqlite3' >>>> quit() While I would of course hope sqlite3 would work, the fact that I get an OSError not an ImportError is preventing my code catching this exception and handling as I would on other platforms (like Jython) lacking this module. I'm running this on Windows XP SP3 (32bit). What more information can I share about this? Thanks, Peter

4 5

Redis clone
by Amirouche Boubekki 09 Apr '13

09 Apr '13

Héllo, I'm working on porting a redis clone to pypy-stm [1]. I did some benchmark, the interesting code is [2]. it runs 5 thread both on sides client/server, it might be much, what do you think ? I have 4 cores. (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python test_client_threaded.py start training training done start spellchecking 1181 out of 4236 real 34,81s user 12,76s sys 0,40s (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python test_client.py start training training done start spellchecking 1181 out of 4234 real 61,22s user 5,69s sys 0,87s I'm thinking about implementing other algorithms; no sure what yet probably bloom filters or bit arrays but I don't have right now a usecase in mind. I also started Graph database [3] which AFAIK should work and which also has even if MVCC some atomic call to be used, the test I'm aiming with this one is loading a wikipedia. How can I help you better ? Thanks. Amirouche [1] https://github.com/amirouche/Memo/tree/pypy-stm [2] a simple suggestion algorithm: https://github.com/amirouche/Memo/blob/pypy-stm/server/memo/structures/sugg… [3] http://blog-amirouche.dotcloud.com/notes/projects/graphitidb.html

1 0

Build pypy-2.0b1
by Max Lavrenov 09 Apr '13

09 Apr '13

Hello I've tried to build pypy from repository but got some compilation error. There is a very big traceback, so i skip some parts. Maybe i could provide some additional information? I still have opened pdb console with this failed compilation. Thanks. [platform:execute] make -j 3 in /tmp/usession-release-2.0-beta2-2/testing_1 [platform:Error] data_pypy_module_cpyext_pyobject.c:118:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&_PyExc_MemoryError), /* 2.value */ [platform:Error] ^ [platform:Error] data_pypy_module_cpyext_pyobject.c:118:3: warning: (near initialization for 'pypy_g_array_916.a.items[2].d_value') [enabled by default] [platform:Error] data_pypy_module_cpyext_pyobject.c:123:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&_PyExc_NotImplementedError), /* 3.value */ [platform:Error] ^ [platform:Error] data_pypy_module_cpyext_pyobject.c:123:3: warning: (near initialization for 'pypy_g_array_916.a.items[3].d_value') [enabled by default] [platform:Error] data_pypy_module_cpyext_pyobject.c:173:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&_PyExc_RuntimeError), /* 13.value */ [platform:Error] ^ [platform:Error] data_pypy_module_cpyext_pyobject.c:173:3: warning: (near initialization for 'pypy_g_array_916.a.items[13].d_value') [enabled by default] [platform:Error] data_pypy_module_cpyext_pyobject.c:238:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&PyWrapperDescr_Type), /* 26.value */ ...... SKIPPED [platform:Error] In file included from /home/e-max/workspace/pypy/pypy/module/cpyext/include/Python.h:132:0, [platform:Error] from structseq.c:4: [platform:Error] ../pypy_decl.h:403:17: note: expected 'struct PyObject *' but argument is of type 'struct PyTupleObject *' [platform:Error] PyAPI_FUNC(int) PyTuple_SetItem(PyObject *arg0, Py_ssize_t arg1, PyObject *arg2); [platform:Error] ^ [platform:Error] In file included from ../module_cache/module_3.c:166:0: [platform:Error] /home/e-max/workspace/pypy/rpython/translator/c/src/dtoa.c:131:0: warning: "PyMem_Malloc" redefined [enabled by default] [platform:Error] #define PyMem_Malloc malloc [platform:Error] ^ [platform:Error] In file included from /home/e-max/workspace/pypy/pypy/module/cpyext/include/Python.h:117:0, [platform:Error] from ../module_cache/module_3.c:34: [platform:Error] /home/e-max/workspace/pypy/pypy/module/cpyext/include/pymem.h:8:0: note: this is the location of the previous definition [platform:Error] #define PyMem_Malloc PyMem_MALLOC [platform:Error] ^ [platform:Error] In file included from ../module_cache/module_3.c:166:0: [platform:Error] /home/e-max/workspace/pypy/rpython/translator/c/src/dtoa.c:132:0: warning: "PyMem_Free" redefined [enabled by default] [platform:Error] #define PyMem_Free free [platform:Error] ^ [platform:Error] In file included from /home/e-max/workspace/pypy/pypy/module/cpyext/include/Python.h:117:0, [platform:Error] from ../module_cache/module_3.c:34: [platform:Error] /home/e-max/workspace/pypy/pypy/module/cpyext/include/pymem.h:9:0: note: this is the location of the previous definition [platform:Error] #define PyMem_Free PyMem_FREE [platform:Error] ^ [platform:Error] Traceback (most recent call last): [platform:Error] File "app_main.py", line 52, in run_toplevel [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 2029, in <module> [platform:Error] tracker.process(f, g, filename=fn) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 1922, in process [platform:Error] tracker = parser.process_function(lines, filename) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 1437, in process_function [platform:Error] table = tracker.computegcmaptable(self.verbose) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 59, in computegcmaptable [platform:Error] self.findframesize() [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 270, in findframesize [platform:Error] self.walk_instructions_backwards(walker, insn, 0) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 355, in walk_instructions_backwards [platform:Error] for prevstate in walker(insn, state): [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 261, in walker [platform:Error] "inconsistent frame size at instruction %s" % (insn,)) [platform:Error] AssertionError: inconsistent frame size at instruction InsnCopyLocal(%rbx, %rax) --- None [platform:Error] make: *** [pypy_module__cffi_backend_ctypefunc.gcmap] Error 1 [platform:Error] make: *** Waiting for unfinished jobs.... [5dd8a] translation-task} [Timer] Timings: [Timer] annotate --- 587.2 s [Timer] rtype_lltype --- 1194.4 s [Timer] pyjitpl_lltype --- 1057.4 s [Timer] backendopt_lltype --- 215.8 s [Timer] stackcheckinsertion_lltype --- 127.2 s [Timer] database_c --- 310.0 s [Timer] source_c --- 347.3 s [Timer] compile_c --- 882.0 s [Timer] =========================================== [Timer] Total: --- 4721.3 s

2 3

2GB Limit on Windows
by Roger Flores 09 Apr '13

09 Apr '13

Is the 2GB limit being lifted on the Windows nightlies? I ask because I keep bumping, painfully, into it. I found this link: http://doc.pypy.org/en/latest/windows.html#preping-windows-for-the-large-bu… Googling more about "editbin /largeaddressaware" http://www.velocityreviews.com/forums/t492725-how-to-make-a-32bit-applicati… It appears editbin is for when you don't pass /LARGEADDRESSAWARE to the linker. Perhaps a Windows build savvy person could add the magic flag to where ever the linker is? Thanks, -Roger

6 12