Mailman 3 May 2012 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

Patches for Pypy under cygwin
by Uwe F. Mayer 15 Jun '12

15 Jun '12

I edited / hacked the sources and managed to compile PyPy under Cygwin based on Python 2.6.7 and gcc-4.5.3. If you find this of interest, the patches are available at http://www.tux.org/~mayer/cygwin/pypy/. In particular the work-around for the tm structure to get by without the tm_gmtoff and tm_zone fields may be of interest, as it allows compilation of the rctime module on systems that don't have it, such as some versions of SunOS. --Uwe

5 20

cppyy use case question
by Alex Pyattaev 10 Jun '12

10 Jun '12

Hello, I am considering trying cppyy on a large project, and I have a question. I need to be able to add a reference to Python objects into the c++ classes. In essence: #Python code: x="My python object" inst = cpp_class() inst.add_data(x) inst.do_stuff()#This does not modify x in any way z=inst.get_data(x) So, cpp_class acts as a container for python objects, and it does not need to know what kind of object is inside. I was able to make it work with pypy through basic C API, using SWIG and typemaps by mapping void* into PyObject*, but that is somewhat hackish. I am now thinking how would the same logic look with cppyy extension. PS: building cppyy is pain in the ass... You guys seriously need to think about build logs =)

3 17

update (+patch) on embedding pypy
by Roberto De Ioris 09 Jun '12

09 Jun '12

Hi everyone, i have finally managed to have a pypy plugin into uWSGI via libpypy-c. The first step was exporting a bunch of symbols in libpypy-c: "RPython_StartupCode", "pypy_g_ObjSpace_startup", "pypy_g_call_startup", "pypy_g_State_startup", "pypy_g_Module_startup", "pypy_g_pypy_module_cpyext_state_State" pypy_g_ObjSpace_startup and pypy_g_Module_startup are not strictly required, i have used them only for debugging some very specific areas. Attached there is a patch for latest tip. After that, i have defined a Py_Initialize() replacement: http://projects.unbit.it/uwsgi/browser/plugins/python/python_plugin.c#L8 and placed a bunch of #ifdef for unsupported functions (like PyFile_FromFile) Finally i have built libpypy with pypy translate.py -Ojit --shared and the uWSGI plugin with LDFLAGS="-L." pypy-c uwsgiconfig.py --plugin plugins/pypy core (LDFLAGS is used as i have libpypy-c in the current dir) Run uWSGI with the welcome app (and a bunch of processes): uwsgi --http-socket :8080 --plugin pypy --module welcome --master --processes 8 Incredibly the whole uwsgi api (that it is pretty complex, involving tons of c code) works flawlessly as well as the custom wsgi.input object (the whole upload test suite passes). Running python threads from the WSGI app works too, but mapping python threads to already available pthreads is a no go (all explodes, i will investigate on that) Importing applications as modules works normally, but importing via file requires implementing a bunch of c-api in pypy (i will address that soon) virtualenv, sys.argv and setting programname is not supported (no c api available) i can get rid of this too in the next few days. Important thing: i have tested it only on OSX -- Roberto De Ioris http://unbit.it

6 23

change of strategy for the py3k branch?
by Antonio Cuni 05 Jun '12

05 Jun '12

Hi all, after some months of work on the py3k branch, I realized that the current strategy/workflow does not scale well and thus I'd like to change it. For those who are not aware, currently we have the default branch where the main development is done, and which includes code for both the rpython translator toolchain and the python 2 interpreter. The py3k branch does not touch the translator toolchain, but modifies the python 2 interpreter to make it a py3k interpreter. These changes are actually destroying the python 2 semantics, which means that the long term goal is to never merge py3k to default, but keeping the development in parallel, and regularly merge default into py3k to make sure that py3k gets benefits of the various improvements in the JIT, GC, etc. In the past months, I ended up spending a considerable amount of time in resolving merge conflicts. This happens all the time that someone modifies something in the python 2 interpreter, for example to apply a new cool optimizations. While on the one hand it is cool to automatically have new cool optimizations on py3k, on the other hand it is blocker which stops me to work on it effectively. After a bit of discussion on IRC, I propose to solve this problem by detaching the development of the py3k interpreter from the development of the python 2 one. Pros: - faster development of py3k - lower entry barrier for new contributors, because the relationship between the various parts will be much simpler - it will be straightforward to apply the new features of the translator toolchain to the py3k branch - it will be easier to split the toolchain from the actual interpreter the day we will finally decide to do it Cons: - we will need to manually port the optimizations done in the interpreter on the default branch to py3k. Note that right now it's now really "automatic" anyway, because merging is painful. If we decide to go for this route, the next question is: where to store the code? I think there are two main solutions: 1) add a new "pypy/py3k" directory where to copy all the relevant modules. E.g. pypy/py3k/interpreter, pypy/py3k/objspace/std, "pypy/py3k/modules. 2) start a completely new repository which contains only the code for py3k. Solution (2) is better and cleaner in theory. However I fear it would soon become a mess to handle, because every change in the translator toolchain would potentially break py3k. I don't want a situation in which we say "yes, you can build py3k but only if you take revision XXX and you use revision YYY of the toolchain, unless the phase of the moon is empty". Solution (1) is more practical and it would probably lead to less problems in the short term. For now, I would still keep the code in the py3k branch, so the normal development of pypy would not be affected. Before doing it, I'd like to hear opinions and comments, in particular of people who already worked on py3k and/or are generally interested in it. Please be constructive :-). ciao, Anto

10 30

IOError with getpass() in django
by Gelonida N 04 Jun '12

04 Jun '12

Hi, I reposted as I noticed, that my inital subject line doesn't really indicate what's the issue. ------------------------------------------------------------------- I wanted to make my very first test with pypy by using django and sqlite3 on a Linux host (Ubuntu 12.04) when running ./manage.py syncdb I'm asked to enter a password and pypy aborts as soon as I press enter. > File "/home/klausf/mypypy/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 109, in handle > password = getpass.getpass() > File "/usr/lib/pypy/lib-python/2.7/getpass.py", line 74, in unix_getpass > stream.flush() # issue7208 > IOError: [Errno 29] Illegal seek: '<fdopen>' Version info: Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08) [PyPy 1.8.0 with GCC 4.6.2] Django version: 1.3.1 I even found a bug, which seems related ( https://bugs.pypy.org/issue872 ) However I don't understand the impact and what this means exactly. Should I try to install another pypy (and if yes, how can I install it next to an existing one on Ubuntu 12.04)? Thanks in advance for your help.

3 3

[Repost] PyPy 1.8 50% slower than PyPy 1.7 on my test case
by Laurent Vaucher 04 Jun '12

04 Jun '12

Sorry to repost, but does anyone have an idea about what I could do to track down the source of the slowdown? Should I run with some trace to try to compare how different parts behave? How could I do that? Thanks, Laurent. Hi and first of all, thanks for that great project. Now to my "problem". I'm doing some puzzle-solving, constraint processing with Python and on my particular program, PyPy 1.8 showed a 50% increase in running time over PyPy 1.7. I'm not doing anything fancy (no numpy, itertools, etc.). The program is here: https://github.com/slowfrog/hexiom To reproduce: - fetch hexiom2.py and level36.txt from github - run 'pypy hexiom2.py -sfirst level36.txt' On my machine (Windows 7) the timings are the following: Python 2.6.5/Windows 32 3m35s Python 2.7.1 (930f0bc4125a, Nov 27 2011, 11:58:57) [PyPy 1.7.0 with MSC v.1500 32 bit] 31s Python 2.7.2 (0e28b379d8b3, Feb 09 2012, 18:31:47) [PyPy 1.8.0 with MSC v.1500 32 bit] 48s I'm using the default options. Do you have any idea what could be causing that? Thanks, Laurent.

3 5

Re: [pypy-dev] pypy-dev Digest, Vol 13, Issue 21
by exarkun＠twistedmatrix.com 03 Jun '12

03 Jun '12

On 03:11 pm, pypy-dev-request(a)python.org wrote: > >Hi all, > >Fijal and me would like to raise interest among various groups of >people about building a better ctypes replacement for Python. > >The general background first, at least as far as we know it. People >generally agree that CPython extension modules are not extremely hard >to write, but also not the ultimate solution in the matter of >interfacing with C code. Obviously you need to know and write C code, >and learn the large CPython C API. Moreover a CPython extension >module is rather specific to CPython or even a particular version of >it. Even if you are lucky and your CPython extensions work on all 2.x >versions, it won't work unmodified on Python 3.x, and be slow (or not >work at all) on PyPy. (I'm leaving Jython and IronPython out of the >picture here but it's very far from ideal for them too.) > >The working alternatives nowadays are Cython and ctypes. For Cython >you have to learn a whole new language; we can argue infinitely about >preferences, but from my point of view it's mostly just a nicer way to >write CPython C extensions using --- instead of C --- a custom >language. About ctypes, I would say that the fact that its use >remains marginal is enough to show that its mission kind of failed. >Its problems are more than one, but most importantly the level at >which you describe the information is slightly too low - it's >ABI-level instead of API-level. It does not make a difference for some >libraries, but for some it does make a huge difference, like when you >use macros. We would like something that can be potentially used to >wrap almost all C libraries. > >So we would like to propose something (a bit more publically than the >two subprojects of PyPy attempting to do that --- did you know about >their existence?). The goal would be to come up with a >better-designed project. "Better designed" in my vocabulary implies >"simpler". I'm starting in this mailing list but the idea is to try >to move it in its own project, and to develop at the same time at >least a CPython 2.7 and 3.x and PyPy implementation. > >The simplest FFI we know of for a high-level language is LuaJIT's FFI. >If you are interested, read the four pages starting at >http://luajit.org/ext_ffi.html . A crucial design decision was to >avoid making use of tons of objects representing the C types; instead, >the user mostly specifies C types as plain strings, simplifying the >basic API a lot. Hi Armin, I don't think it's true that using strings instead of types (or other rich objects) simplifies anything. Quite the opposite, it takes all of the complexity which must exist and throws a huge wall up to prevent anyone from understanding it without undertaking a huge amount of extra work. C is C, whether you hide the representation of its structure in a string or whether you expose that representation as objects with useful, easy to use APIs. Jean-Paul

10 21

PyPy PPA
by Alex Gaynor 01 Jun '12

01 Jun '12

Hey all, I'm hoping the people behind https://launchpad.net/~pypy/+archive/ppa are hear :) I'm wondering what the status of this (or any other PPAs) are, and whether it'd be possible to get a PPA that works with Ubuntu 12.04. The people behind http://travis-ci.org/ which is an awesome CI service, would like to be able to deploy PyPy via a deb on 12.04. Thanks, Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero

3 2

AOSA book chapter
by Benjamin Peterson 31 May '12

31 May '12

Hello pypy-dev, As some of you may know, this past year I wrote a chapter about PyPy for Architecture of Open Source Applications Volume 2. It has brought to my attention that my communication with the PyPy community about it was poor. Since it deeply involved the PyPy project, I should have been more public in its writing; I should have sent it to pypy-dev for review and comments. Now that this has been pointed out to me, it's painfully obvious that I fell short. Next time such an opportunity arises, I look forward to working fully with the PyPy community on it. Please accept my apologies, Benjamin

6 5