Mailman 3 March 2013 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor March 11, 2014

March 11, 2014

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus Dec. 23, 2013

Dec. 23, 2013

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4

prevent extension module code from being executed during startup?
by wlavrijsen＠lbl.gov April 2, 2013

April 2, 2013

Hi, when starting pypy-c, I get some extension module code executed before that module is imported. The code in question is in pypy/module/cppyy/pythonify.py: gbl = make_cppnamespace(None, "::", None, False) # global C++ namespace gbl.__doc__ = "Global C++ namespace." sys.modules['cppyy.gbl'] = gbl This code is not rpython code. The module is first mentioned in the __init__.py of cppyy: appleveldefs = { 'gbl' : 'pythonify.gbl', 'load_reflection_info' : 'pythonify.load_reflection_info', 'add_pythonization' : 'pythonify.add_pythonization', } How can I prevent that code from being executed at startup, i.e. have it deferred all the way until "import cppyy"? The use case is as follows: I have a prototype where I load the Reflex library (and associated C-API) at run-time, using _cffi_backend. That way, it is no longer necessary to link pypy-c with libReflex.so. I can handle a missing .so silently, but would prefer to raise an ImportError. However, for that to work, I need to make sure nothing of the module gets touched until it's imported, so that the error is reported at the right time. Thanks, Wim -- WLavrijsen(a)lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

2 6

numpypy problem - can't find augmented assignment methods
by Kay F. Jahnke March 30, 2013

March 30, 2013

Hi group! I'm running numpypy and I have a problem with augmented assignment statements. It looks to me like the relevant methods are simply absent from my numpypy array objects. Consider for example: $ pypy Python 2.7.3 (7e4f0faa3d51, Nov 16 2012, 16:56:51) [PyPy 2.0.0-beta1 with GCC 4.4.3] on linux2 ... >>>> import numpypy as np >>>> a=np.array([1]) >>>> b = a >>>> b is a True >>>> b *= 3 >>>> b is a False >>>> b.__imul__ Traceback (most recent call last): File "<console>", line 1, in <module> AttributeError: 'ndarray' object has no attribute '__imul__' On numpy under CPython, 'b is a' evaluates to True, and np.ndarray.__imul__ is present. I suppose since the method isn't there, the system is falling back to run __mul__ and assigns the result to a new object (after all b contains correct data after the *=), but for my application I want the in-place operation. I've browsed in circles for a while to find out what's wrong or, respectively, why it is as it is; there might be a simple answer to my problem, so I decided to ask for help :) With regards Kay

2 1

Invito Esclusivo per l'IMPRENDITORE
by Smart Business Coach March 30, 2013

March 30, 2013

Caro Imprenditore, Questa mail ti raggiunge per INVITARTI ad un EVENTO ESCLUSIVO DEDICATO AI SOLI IMPRENDITORI che abbiamo organizzato a Roma il 13 Aprile 2013, DEDICATO A TE e alla TUA AZIENDA. --------- CLICCA QUI ( http://www.smartbusinesslab.com/evento ) --------- per scoprire di cosa si tratta. PS: Non ci conosci (Ancora) e ricevi questa mail perchè ABBIAMO ACQUISTATO da liste pubbliche il tuo contatto. Se vuoi, puoi disiscriverti da questa lista cliccando qui ( http://www.smartbusinesscoach.it/dem/index.php?subid=407540&option=com_acym… ); per cortesia visiona prima l'invito.

1 0

Re: [pypy-dev] [pypy-commit] pypy default: use jit.loop_unrolling_heuristic where possible
by Carl Friedrich Bolz March 29, 2013

March 29, 2013

Hi Brian, On 03/25/2013 10:42 AM, bdkearns wrote: > Author: Brian Kearns <bdkearns(a)gmail.com> > Branch: > Changeset: r62735:1f7ca87c6780 > Date: 2013-03-25 05:37 -0400 > http://bitbucket.org/pypy/pypy/changeset/1f7ca87c6780/ > > Log: use jit.loop_unrolling_heuristic where possible > > diff --git a/pypy/module/__builtin__/functional.py b/pypy/module/__builtin__/functional.py > --- a/pypy/module/__builtin__/functional.py > +++ b/pypy/module/__builtin__/functional.py > @@ -202,8 +202,8 @@ > > @specialize.arg(2) > def min_max(space, args, implementation_of): > - if not jit.we_are_jitted() or (jit.isconstant(len(args.arguments_w)) and > - len(args.arguments_w) == 2): > + if not jit.we_are_jitted() or jit.loop_unrolling_heuristic( > + args.arguments_w, len(args.arguments_w)): > return min_max_unroll(space, args, implementation_of) > else: > return min_max_normal(space, args, implementation_of) This is a very risky change. The unrolled pseudo-code for min (max is equivalent) with 5 elements is something like this: m = l0 if l1 < m: m = l1 if l2 < m m = l2 if l3 < m m = l3 if l3 < m m = l3 (the comparisons of course have to go through the object space, everything is more complex, but the control flow is essentially like this.) Every comparison can go either way, that means that there are 2**4 paths through this code, and it's completely unpredictable which of the paths is taken. Therefore it can lead to potentially too much code being generated. I think for min and max in particular the compromise of just unrolling twice is acceptable. BTW, another optimization that might be interesting for min/max is to add fast paths with listview_int/listview_str if no key and no comparison are given. Cheers, Carl Friedrich > diff --git a/pypy/objspace/std/dictmultiobject.py b/pypy/objspace/std/dictmultiobject.py > --- a/pypy/objspace/std/dictmultiobject.py > +++ b/pypy/objspace/std/dictmultiobject.py > @@ -10,8 +10,7 @@ > from rpython.rlib.debug import mark_dict_non_null > from rpython.tool.sourcetools import func_with_new_name > > -from rpython.rlib import rerased > -from rpython.rlib import jit > +from rpython.rlib import rerased, jit > > def _is_str(space, w_key): > return space.is_w(space.type(w_key), space.w_str) > @@ -29,9 +28,6 @@ > space.is_w(w_lookup_type, space.w_float) > ) > > - > -DICT_CUTOFF = 5 > - > @specialize.call_location() > def w_dict_unrolling_heuristic(w_dct): > """ In which cases iterating over dict items can be unrolled. > @@ -39,7 +35,8 @@ > an actual dict > """ > return jit.isvirtual(w_dct) or (jit.isconstant(w_dct) and > - w_dct.length() <= DICT_CUTOFF) > + w_dct.length() <= jit.UNROLL_CUTOFF) > + > > class W_DictMultiObject(W_Object): > from pypy.objspace.std.dicttype import dict_typedef as typedef > @@ -761,7 +758,8 @@ > update1_keys(space, w_dict, w_data) > > > -(a)jit.look_inside_iff(lambda space, w_dict, w_data: w_dict_unrolling_heuristic(w_data)) > +(a)jit.look_inside_iff(lambda space, w_dict, w_data: > + w_dict_unrolling_heuristic(w_data)) > def update1_dict_dict(space, w_dict, w_data): > iterator = w_data.iteritems() > while 1: > diff --git a/pypy/objspace/std/intobject.py b/pypy/objspace/std/intobject.py > --- a/pypy/objspace/std/intobject.py > +++ b/pypy/objspace/std/intobject.py > @@ -183,7 +183,8 @@ > > > # helper for pow() > -(a)jit.look_inside_iff(lambda space, iv, iw, iz: jit.isconstant(iw) and jit.isconstant(iz)) > +(a)jit.look_inside_iff(lambda space, iv, iw, iz: > + jit.isconstant(iw) and jit.isconstant(iz)) > def _impl_int_int_pow(space, iv, iw, iz): > if iw < 0: > if iz != 0: > diff --git a/pypy/objspace/std/kwargsdict.py b/pypy/objspace/std/kwargsdict.py > --- a/pypy/objspace/std/kwargsdict.py > +++ b/pypy/objspace/std/kwargsdict.py > @@ -95,7 +95,8 @@ > def getitem_str(self, w_dict, key): > return self._getitem_str_indirection(w_dict, key) > > - @jit.look_inside_iff(lambda self, w_dict, key: jit.isconstant(self.length(w_dict)) and jit.isconstant(key)) > + @jit.look_inside_iff(lambda self, w_dict, key: > + jit.isconstant(self.length(w_dict)) and jit.isconstant(key)) > def _getitem_str_indirection(self, w_dict, key): > keys, values_w = self.unerase(w_dict.dstorage) > result = [] > diff --git a/pypy/objspace/std/listobject.py b/pypy/objspace/std/listobject.py > --- a/pypy/objspace/std/listobject.py > +++ b/pypy/objspace/std/listobject.py > @@ -18,7 +18,6 @@ > from pypy.objspace.std.stdtypedef import StdTypeDef, SMM > from sys import maxint > > -UNROLL_CUTOFF = 5 > > class W_AbstractListObject(W_Object): > __slots__ = () > @@ -43,7 +42,7 @@ > return W_ListObject.from_storage_and_strategy(space, storage, strategy) > > @jit.look_inside_iff(lambda space, list_w, sizehint: > - jit.isconstant(len(list_w)) and len(list_w) < UNROLL_CUTOFF) > + jit.loop_unrolling_heuristic(list_w, len(list_w))) > def get_strategy_from_list_objects(space, list_w, sizehint): > if not list_w: > if sizehint != -1: > @@ -950,7 +949,7 @@ > raise NotImplementedError("abstract base class") > > @jit.look_inside_iff(lambda space, w_list, list_w: > - jit.isconstant(len(list_w)) and len(list_w) < UNROLL_CUTOFF) > + jit.loop_unrolling_heuristic(list_w, len(list_w))) > def init_from_list_w(self, w_list, list_w): > l = [self.unwrap(w_item) for w_item in list_w] > w_list.lstorage = self.erase(l) > @@ -999,7 +998,7 @@ > return self.wrap(r) > > @jit.look_inside_iff(lambda self, w_list: > - jit.isconstant(w_list.length()) and w_list.length() < UNROLL_CUTOFF) > + jit.loop_unrolling_heuristic(w_list, w_list.length())) > def getitems_copy(self, w_list): > return [self.wrap(item) for item in self.unerase(w_list.lstorage)] > > @@ -1008,7 +1007,7 @@ > return [self.wrap(item) for item in self.unerase(w_list.lstorage)] > > @jit.look_inside_iff(lambda self, w_list: > - jit.isconstant(w_list.length()) and w_list.length() < UNROLL_CUTOFF) > + jit.loop_unrolling_heuristic(w_list, w_list.length())) > def getitems_fixedsize(self, w_list): > return self.getitems_unroll(w_list) > > diff --git a/pypy/objspace/std/tupleobject.py b/pypy/objspace/std/tupleobject.py > --- a/pypy/objspace/std/tupleobject.py > +++ b/pypy/objspace/std/tupleobject.py > @@ -10,8 +10,6 @@ > from rpython.rlib import jit > from rpython.tool.sourcetools import func_with_new_name > > -# Tuples of known length up to UNROLL_TUPLE_LIMIT have unrolled certain methods > -UNROLL_TUPLE_LIMIT = 10 > > class W_AbstractTupleObject(W_Object): > __slots__ = () > @@ -85,15 +83,8 @@ > start, stop = normalize_simple_slice(space, length, w_start, w_stop) > return space.newtuple(w_tuple.wrappeditems[start:stop]) > > -THRESHOLD = 7 > - > -def unroll_tuple_contains(space, w_tuple, w_obj): > - if (jit.isconstant(w_tuple) or jit.isvirtual(w_tuple) and > - len(w_tuple.wrappeditems) < THRESHOLD): > - return True > - return False > - > -(a)jit.look_inside_iff(unroll_tuple_contains) > +(a)jit.look_inside_iff(lambda space, w_tuple, w_obj: > + jit.loop_unrolling_heuristic(w_tuple, len(w_tuple.wrappeditems))) > def contains__Tuple_ANY(space, w_tuple, w_obj): > for w_item in w_tuple.wrappeditems: > if space.eq_w(w_item, w_obj): > @@ -128,10 +119,8 @@ > return mul_tuple_times(space, w_tuple, w_times) > > def tuple_unroll_condition(space, w_tuple1, w_tuple2): > - lgt1 = len(w_tuple1.wrappeditems) > - lgt2 = len(w_tuple2.wrappeditems) > - return ((jit.isconstant(lgt1) and lgt1 <= UNROLL_TUPLE_LIMIT) or > - (jit.isconstant(lgt2) and lgt2 <= UNROLL_TUPLE_LIMIT)) > + return jit.loop_unrolling_heuristic(w_tuple1, len(w_tuple1.wrappeditems)) or \ > + jit.loop_unrolling_heuristic(w_tuple2, len(w_tuple2.wrappeditems)) > > @jit.look_inside_iff(tuple_unroll_condition) > def eq__Tuple_Tuple(space, w_tuple1, w_tuple2): > @@ -151,7 +140,7 @@ > def _make_tuple_comparison(name): > import operator > op = getattr(operator, name) > - # > + > @jit.look_inside_iff(tuple_unroll_condition) > def compare_tuples(space, w_tuple1, w_tuple2): > items1 = w_tuple1.wrappeditems > @@ -184,8 +173,7 @@ > return space.wrap(hash_tuple(space, w_tuple.wrappeditems)) > > @jit.look_inside_iff(lambda space, wrappeditems: > - jit.isconstant(len(wrappeditems)) and > - len(wrappeditems) < UNROLL_TUPLE_LIMIT) > + jit.loop_unrolling_heuristic(wrappeditems, len(wrappeditems))) > def hash_tuple(space, wrappeditems): > # this is the CPython 2.4 algorithm (changed from 2.3) > mult = 1000003 > diff --git a/rpython/rlib/jit.py b/rpython/rlib/jit.py > --- a/rpython/rlib/jit.py > +++ b/rpython/rlib/jit.py > @@ -206,13 +206,13 @@ > return NonConstant(False) > isvirtual._annspecialcase_ = "specialize:call_location" > > -LIST_CUTOFF = 2 > +UNROLL_CUTOFF = 5 > > @specialize.call_location() > def loop_unrolling_heuristic(lst, size): > """ In which cases iterating over items of lst can be unrolled > """ > - return isvirtual(lst) or (isconstant(size) and size <= LIST_CUTOFF) > + return isvirtual(lst) or (isconstant(size) and size <= UNROLL_CUTOFF) > > class Entry(ExtRegistryEntry): > _about_ = hint > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit >

2 2

GSOC 2013
by Alex Gaynor March 29, 2013

March 29, 2013

Hi everyone, I've started putting together an ideas page for this GSOC: https://bitbucket.org/pypy/pypy/wiki/GSOC%202013 If you've got an idea, or are interested in mentoring you should add yourself/the idea. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero

6 8

Python Execution Contexts Ch.2
by anatoly techtonik March 28, 2013

March 28, 2013

On Mon, Mar 25, 2013 at 8:13 PM, Amaury Forgeot d'Arc <amauryfa(a)gmail.com>wrote: > > 2013/3/25 anatoly techtonik <techtonik(a)gmail.com> > >> This module opened a Pandora box of Python internals. Version 0.4 still >> fails to >> trace files specified on command line, and I am lost in internal details >> of execfile >> + locals()/globals()/namespacing/scoping. Python tracker doesn't help >> here. >> > > Seriously: always always pass your own locals and globals to functions > like exec and execfile. > Good luck. I'd like to understand what's going on there, because it directly affects if I will be able to extend xtrace further and troubleshoot reported problems when more errors appear. I know that locals() is not an ordinary dictionary ( http://bugs.python.org/issue17546). And the first question - what should I pass to execfile as 'my own locals'? 1. an exact livedict that is returned by locals() function 2. my own dict, which will become live inside the execfile call 3. locals function that returns a reference to livedict The second question - why should I do this? To make the explanation easier, here is a real world problem that made me seek the answers. https://bitbucket.org/techtonik/xtrace/issue/3/unable-to-xtrace-indefpy The file that execfile() *inside* xtrace fails to execute is below. I named it indef.py, because subprocess call is inside defined function. import subprocess def ret(): print subprocess.check_output(['hg', 'id', '-nib']) ret() The problem repeats if the file is executed from console, but only in special case: Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] >>> execfile('indef.py') 0cef574f7f62+ 23+ default >>> def x(): ... execfile('indef.py') ... >>> x() 0cef574f7f62+ 23+ default Now I restart the console and do the same stuff, but without the first call: Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] >>> def x(): ... execfile('indef.py') ... >>> x() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in x File "indef.py", line 6, in <module> ret() File "indef.py", line 4, in ret print subprocess.check_output(['hg', 'id', '-nib']) NameError: global name 'subprocess' is not defined I guess the same stuff happens with my xtrace(), where execfile() is executed inside the function. Now the questions: 1. I understand that when Python starts executing code, it creates a namespace. This is the 'first_namespace' where Python starts to store all variables encountered. I intentionally do not call it global, as this term is not introduced at this point. Python continues to put and change variables there until it encounters [something], this [something] causes it to create a 'new_namespace' to store variables. I know that a function call makes Python create 'new_namespace' to store variables encountered inside this function, so "function call" is in [something]. What else is in [something], i.e. what else causes Python to create 'new_namespaces'? It will make me comfortable to see the full list described somewhere. 2. 'new_namespace's are created when function is entered and destroyed when it exits. When variable is requested, the lookup mechanism looks first inside 'new_namespace', then checks its parent (caller namespace), then parent of the parent and so on until the 'first_namespace'. Does it work like this? 3. Now the problem. When I run execfile() without parameters, I expect the code inside to be 'virtualized' - isolated from parent process, like in LXC or VirtualBox, but on Python level. I also expect that if I want to communicate with this code inside execfile(), I craft my own 'first_namespace' and pass it down. When execfile() returns, I inspect the state of this modified namespace, and maybe import some results back into my 'current_namespace'. The example with indef.py above shows the execfile() doesn't work this way. So the question is - why execfile() doesn't it work the way I described? 4. At this point I hope to reach the state there it is clear how execfile() works in reality, and the next question in this state - what was the reason to make it so complicated? Optimization? Thanks for the feedback. -- anatoly t.

5 6

Tips to Improve Your Memory
by Daniel Strauss March 28, 2013

March 28, 2013

Hi, I'm a contributor to an education improvement resource and I recently published an article providing many tips about improving your memory I think you'd be interested in. I found your site http://bugs.python.org/issue15490 during my research and would like to invite you to take a look at my published piece. Please let me know if you'd be interested in taking a look and possibly sharing with your readers. Thanks, Daniel

1 0

Python Execution Contexts
by anatoly techtonik March 26, 2013

March 26, 2013

Hi, I've created a module to dump function trace during execution of Python script. You can see session example in attachment. The format is of PHP Xdebug tool [2] just because I had some scripts from the past to analyze it. The module uses sys.settrace() to analyse frames in 'call' events with callback(frame, event, arg). Recently I've got an anonymous report that some frame misses filename information when run under IDE: https://bitbucket.org/techtonik/xtrace/issue/2/use-of-xtrace-module-within-… There is unlikely to be any more feedback from the user to make me understand and reproduce the behavior, nor there is a sufficient documentation in sys.trace description [1]. So I am on my own. But I can not read C code the CPython is written in, so I ask here. What are possible execution contexts for Python? How each execution context affects data structure that is passed to trace function (frame and arg)? Please, CC. Thanks. 1. http://docs.python.org/3/library/sys.html?highlight=sys.settrace#sys.settra… 2. Example of PHP function trace: http://xdebug.org/docs/execution_trace -- anatoly t.

2 7