Mailman 3 April 2013 - pypy-dev

Further details regarding the GSOC 2013 project idea
by Hareesan April 15, 2013

April 15, 2013

Hi there, I'm Hareesan, An undergraduate from University of Moratuwa, Sri Lanka. I have gone through the GSOC 2013 ideas from pypy. I'm much interested in working on the project "New JitViewer infrastructure". I have cloned the source and gone through pypy jitviewer introduction paper.<http://people.cs.uct.ac.za/~tmullins/work/writeup.pdf> To start up working with the project, it would be great if anyone could direct me with further details such as what limitations exists, and what are … [View More]

2 1

Fwd: Windows 64 bit version
by Maciej Fijalkowski April 14, 2013

April 14, 2013

windows 64 patches that I got, can someone have a look? ---------- Forwarded message ---------- From: Roger Flores <aidembb(a)yahoo.com> Date: Wed, Apr 10, 2013 at 9:02 PM Subject: Re: Windows 64 bit version To: Maciej Fijalkowski <fijall(a)gmail.com> OK, sorry this is going to be long. The short version is I have attached a diff for C files in rpython/translator/c handling long != ptr except for three files. >It depends on the context. Essentially the approach is to run … [View More]tests and figure out what's going on. Starting from rpython/translator/c OK, looking around rpython/translator/c I see that there are C files. Looking through there, I see cases where void * is being cast to (long). I started scoping out the extent of the change and realized it was a finite set, not too huge. So I decided to just make the changes and send you diff so we can all just stop talking about it. Then I realized that there were (unsigned long) cases. And then ulong. And ULONG. And ULLONG. And so on. Yes, I got sucked in. In the end I reviewed ALL cases involving "long", caseless. The approach: At first I quickly changed (long) to (long long). Then I realized that long long is 64 bits on 32 bit builds. Either we need to 1) conditionally compile each line of code based on the platform or 2) use a type that we magically change once based on the platform. Then I noticed that ptrdiff_t is an existing type that is magically sized correctly for all platforms. So now (long) is (ptrdiff_t) with no header changes. The diff covers all C files. It's not big. I don't know the command to run tests there so I didn't even check linux builds, let alone win64, which I'm extremely reluctant to set up. But what I changed should be largely obvious. In particular please review stack.c and stacklet.c which used unsigned long and that might be an issue. The three files I did not change are thread_pthread.c, dtoa.c and obmalloc.c. The changes are small but I didn't have high confidence. obmalloc makes it's own ulong for some reason. There are also python files. I examined them for Long usage but couldn't find a clear pattern. Can you point out one example of what needs changing? >it would go in rpython/jit/backend/x86 I looked around but I'm going to need more direction. I was specifically looking for the linux and win32 portions to compare and again, figure out the size of the change. All I found though were register descriptions and code to emit assembly instructions. -Roger ________________________________ From: Maciej Fijalkowski <fijall(a)gmail.com> To: Roger Flores <aidembb(a)yahoo.com> Cc: Amaury Forgeot d'Arc <amauryfa(a)gmail.com>; matti picus <matti.picus(a)gmail.com>; PyPy Developer Mailing List <pypy-dev(a)python.org> Sent: Tuesday, April 9, 2013 12:52 AM Subject: Re: Windows 64 bit version On Tue, Apr 9, 2013 at 9:47 AM, Roger Flores <aidembb(a)yahoo.com> wrote: > Maciej Fijalkowski wrote: >>It's just a bunch of work. There is nothing special or magic about it, but >> so far noone volunteered to spend enough effort there. > > Got it. I'm just trying to understand what the work is because that list > hasn't been captured anywhere yet. > > >>Essentially, the main problem is that sizeof(long) != sizeof(intptr_t), >> which is not a big problem per-se, it's just that this assumption has to be >> removed from places in the RPython source code. > > OK. Could you find one or two of those places in the RPython source code so > we can see? And what is the fix? Does long get changed to long long or > another type or ? It depends on the context. Essentially the approach is to run tests and figure out what's going on. Starting from rpython/translator/c > > >>Another one is the calling convention for the JIT, which is different than >> on linux. Again, nothing special, just more work. > > Does "calling convention for the JIT" mean a function calling convention for > 64 bit windows or something else? Where is this in the code, or maybe, > where is it for the Windows 32 code and where would the 64 bit version go? > There's no magic for exceptions, gc, or anything else? it would go in rpython/jit/backend/x86 > > > Thanks for the details, > -Roger > > [View Less]

5 7

Control flow graph for method
by Evgeny Startsev April 14, 2013

April 14, 2013

I am interesting in control-flow graph generation for python projects using python. I read http://doc.pypy.org/en/latest/translation.html#flow-model and studied how to investigate cfg for function. Is there any way to generate cfg for method using pypy?

2 2

Re: [pypy-dev] [pypy-commit] pypy default: preallocate list to hold the locations of arguments to a call
by Maciej Fijalkowski April 13, 2013

April 13, 2013

There is an optimization that does it for you FYI (or just use newlist_hint) On Sat, Apr 13, 2013 at 6:13 PM, bivab <noreply(a)buildbot.pypy.org> wrote: > Author: David Schneider <david.schneider(a)picle.org> > Branch: > Changeset: r63312:1bd588482153 > Date: 2013-04-13 18:00 +0200 > http://bitbucket.org/pypy/pypy/changeset/1bd588482153/ > > Log: preallocate list to hold the locations of arguments to a call > > diff --git a/rpython/jit/backend/arm/… [View More]

2 1

Redis «clone» using pypy stm-thread-2 (Was: Redis clone)
by Amirouche Boubekki April 13, 2013

April 13, 2013

Héllo again, 2013/4/9 Amirouche Boubekki <amirouche.boubekki(a)gmail.com> > Héllo, > > I'm working on porting a redis clone to pypy-stm [1]. > > I did some benchmark, the interesting code is [2]. it runs 5 thread both > on sides client/server, it might be much, what do you think ? I have 4 > cores. > > (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python > test_client_threaded.py > start training > training done > start … [View More]

2 6

Re: [pypy-dev] [pypy-commit] pypy default: make looking up encodings free with the JIT
by Antonio Cuni April 12, 2013

April 12, 2013

Hi Alex, could we have a test_pypy_c test for this please? On 10/04/13 03:53, alex_gaynor wrote: > Author: Alex Gaynor <alex.gaynor(a)gmail.com> > Branch: > Changeset: r63186:c514bbc4c086 > Date: 2013-04-09 19:53 -0700 > http://bitbucket.org/pypy/pypy/changeset/c514bbc4c086/ > > Log: make looking up encodings free with the JIT > > diff --git a/pypy/module/_codecs/interp_codecs.py b/pypy/module/_codecs/interp_codecs.py > --- a/pypy/module/_codecs/… [View More]interp_codecs.py > +++ b/pypy/module/_codecs/interp_codecs.py > @@ -1,10 +1,18 @@ > +from rpython.rlib import jit > +from rpython.rlib.objectmodel import we_are_translated > +from rpython.rlib.rstring import UnicodeBuilder > + > from pypy.interpreter.error import OperationError, operationerrfmt > from pypy.interpreter.gateway import interp2app, unwrap_spec, WrappedDefault > -from rpython.rlib.rstring import UnicodeBuilder > -from rpython.rlib.objectmodel import we_are_translated > + > + > +class VersionTag(object): > + pass > > > class CodecState(object): > + _immutable_fields_ = ["version?"] > + > def __init__(self, space): > self.codec_search_path = [] > self.codec_search_cache = {} > @@ -14,6 +22,7 @@ > self.encode_error_handler = self.make_encode_errorhandler(space) > > self.unicodedata_handler = None > + self.modified() > > def _make_errorhandler(self, space, decode): > def call_errorhandler(errors, encoding, reason, input, startpos, > @@ -86,9 +95,20 @@ > self.unicodedata_handler = UnicodeData_Handler(space, w_getcode) > return self.unicodedata_handler > > + def modified(self): > + self.version = VersionTag() > + > + def get_codec_from_cache(self, key): > + return self._get_codec_with_version(key, self.version) > + > + @jit.elidable > + def _get_codec_with_version(self, key, version): > + return self.codec_search_cache.get(key, None) > + > def _cleanup_(self): > assert not self.codec_search_path > > + > def register_codec(space, w_search_function): > """register(search_function) > > @@ -115,11 +135,12 @@ > "lookup_codec() should not be called during translation" > state = space.fromcache(CodecState) > normalized_encoding = encoding.replace(" ", "-").lower() > - w_result = state.codec_search_cache.get(normalized_encoding, None) > + w_result = state.get_codec_from_cache(normalized_encoding) > if w_result is not None: > return w_result > return _lookup_codec_loop(space, encoding, normalized_encoding) > > + > def _lookup_codec_loop(space, encoding, normalized_encoding): > state = space.fromcache(CodecState) > if state.codec_need_encodings: > @@ -143,6 +164,7 @@ > space.wrap("codec search functions must return 4-tuples")) > else: > state.codec_search_cache[normalized_encoding] = w_result > + state.modified() > return w_result > raise operationerrfmt( > space.w_LookupError, > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit > [View Less]

4 5

OSError not ImportError from 'import sqlite3' on Windows
by Peter Cock April 9, 2013

April 9, 2013

Hello all, I've downloaded and unzipped pypy-2.0-beta2-win32.zip but fairly early on in my testing ran into this puzzle: C:\>c:\pypy-2.0\pypy.exe Python 2.7.3 (3eef596df459, Apr 06 2013, 12:40:10) [PyPy 2.0.0-beta2 with MSC v.1500 32 bit] on win32 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``the zen attitude to programming: reducing the oopses in your life'' >>>> import sqlite3 Traceback (most recent call … [View More]

4 5

Redis clone
by Amirouche Boubekki April 9, 2013

April 9, 2013

Héllo, I'm working on porting a redis clone to pypy-stm [1]. I did some benchmark, the interesting code is [2]. it runs 5 thread both on sides client/server, it might be much, what do you think ? I have 4 cores. (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python test_client_threaded.py start training training done start spellchecking 1181 out of 4236 real 34,81s user 12,76s sys 0,40s (memo-stm)amirouche@funfev13 ~/src/Memo/client (pypy-stm*) $ time python … [View More]

1 0

Build pypy-2.0b1
by Max Lavrenov April 9, 2013

April 9, 2013

Hello I've tried to build pypy from repository but got some compilation error. There is a very big traceback, so i skip some parts. Maybe i could provide some additional information? I still have opened pdb console with this failed compilation. Thanks. [platform:execute] make -j 3 in /tmp/usession-release-2.0-beta2-2/testing_1 [platform:Error] data_pypy_module_cpyext_pyobject.c:118:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&… [View More]_PyExc_MemoryError), /* 2.value */ [platform:Error] ^ [platform:Error] data_pypy_module_cpyext_pyobject.c:118:3: warning: (near initialization for 'pypy_g_array_916.a.items[2].d_value') [enabled by default] [platform:Error] data_pypy_module_cpyext_pyobject.c:123:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&_PyExc_NotImplementedError), /* 3.value */ [platform:Error] ^ [platform:Error] data_pypy_module_cpyext_pyobject.c:123:3: warning: (near initialization for 'pypy_g_array_916.a.items[3].d_value') [enabled by default] [platform:Error] data_pypy_module_cpyext_pyobject.c:173:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&_PyExc_RuntimeError), /* 13.value */ [platform:Error] ^ [platform:Error] data_pypy_module_cpyext_pyobject.c:173:3: warning: (near initialization for 'pypy_g_array_916.a.items[13].d_value') [enabled by default] [platform:Error] data_pypy_module_cpyext_pyobject.c:238:3: warning: initialization from incompatible pointer type [enabled by default] [platform:Error] (&PyWrapperDescr_Type), /* 26.value */ ...... SKIPPED [platform:Error] In file included from /home/e-max/workspace/pypy/pypy/module/cpyext/include/Python.h:132:0, [platform:Error] from structseq.c:4: [platform:Error] ../pypy_decl.h:403:17: note: expected 'struct PyObject *' but argument is of type 'struct PyTupleObject *' [platform:Error] PyAPI_FUNC(int) PyTuple_SetItem(PyObject *arg0, Py_ssize_t arg1, PyObject *arg2); [platform:Error] ^ [platform:Error] In file included from ../module_cache/module_3.c:166:0: [platform:Error] /home/e-max/workspace/pypy/rpython/translator/c/src/dtoa.c:131:0: warning: "PyMem_Malloc" redefined [enabled by default] [platform:Error] #define PyMem_Malloc malloc [platform:Error] ^ [platform:Error] In file included from /home/e-max/workspace/pypy/pypy/module/cpyext/include/Python.h:117:0, [platform:Error] from ../module_cache/module_3.c:34: [platform:Error] /home/e-max/workspace/pypy/pypy/module/cpyext/include/pymem.h:8:0: note: this is the location of the previous definition [platform:Error] #define PyMem_Malloc PyMem_MALLOC [platform:Error] ^ [platform:Error] In file included from ../module_cache/module_3.c:166:0: [platform:Error] /home/e-max/workspace/pypy/rpython/translator/c/src/dtoa.c:132:0: warning: "PyMem_Free" redefined [enabled by default] [platform:Error] #define PyMem_Free free [platform:Error] ^ [platform:Error] In file included from /home/e-max/workspace/pypy/pypy/module/cpyext/include/Python.h:117:0, [platform:Error] from ../module_cache/module_3.c:34: [platform:Error] /home/e-max/workspace/pypy/pypy/module/cpyext/include/pymem.h:9:0: note: this is the location of the previous definition [platform:Error] #define PyMem_Free PyMem_FREE [platform:Error] ^ [platform:Error] Traceback (most recent call last): [platform:Error] File "app_main.py", line 52, in run_toplevel [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 2029, in <module> [platform:Error] tracker.process(f, g, filename=fn) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 1922, in process [platform:Error] tracker = parser.process_function(lines, filename) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 1437, in process_function [platform:Error] table = tracker.computegcmaptable(self.verbose) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 59, in computegcmaptable [platform:Error] self.findframesize() [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 270, in findframesize [platform:Error] self.walk_instructions_backwards(walker, insn, 0) [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 355, in walk_instructions_backwards [platform:Error] for prevstate in walker(insn, state): [platform:Error] File "/home/e-max/workspace/pypy/rpython/translator/c/gcc/trackgcroot.py", line 261, in walker [platform:Error] "inconsistent frame size at instruction %s" % (insn,)) [platform:Error] AssertionError: inconsistent frame size at instruction InsnCopyLocal(%rbx, %rax) --- None [platform:Error] make: *** [pypy_module__cffi_backend_ctypefunc.gcmap] Error 1 [platform:Error] make: *** Waiting for unfinished jobs.... [5dd8a] translation-task} [Timer] Timings: [Timer] annotate --- 587.2 s [Timer] rtype_lltype --- 1194.4 s [Timer] pyjitpl_lltype --- 1057.4 s [Timer] backendopt_lltype --- 215.8 s [Timer] stackcheckinsertion_lltype --- 127.2 s [Timer] database_c --- 310.0 s [Timer] source_c --- 347.3 s [Timer] compile_c --- 882.0 s [Timer] =========================================== [Timer] Total: --- 4721.3 s [View Less]

2 3

2GB Limit on Windows
by Roger Flores April 9, 2013

April 9, 2013

Is the 2GB limit being lifted on the Windows nightlies? I ask because I keep bumping, painfully, into it. I found this link: http://doc.pypy.org/en/latest/windows.html#preping-windows-for-the-large-bu… Googling more about "editbin /largeaddressaware" http://www.velocityreviews.com/forums/t492725-how-to-make-a-32bit-applicati… It appears editbin is for when you don't pass /LARGEADDRESSAWARE to the linker. Perhaps a Windows build savvy person could add the magic flag to … [View More]

6 12