Mailman 3 November 2011 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

Building lxml
by nitralime 07 Dec '11

07 Dec '11

Hello! I'v tried to build lxml 2.3.2 with pypy 1.7 (linux64) and got the following error: ------------------------------------------------------------------------------------ $ pypy setup.py build_ext -i pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Building lxml version 2.3.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib running build_ext building 'lxml.etree' extension creating build creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/src creating build/temp.linux-x86_64-2.7/src/lxml cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/local/pypy/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w src/lxml/lxml.etree.c: In function ‘__Pyx_PyObject_Pop’: src/lxml/lxml.etree.c:3683:16: error: ‘PyListObject’ has no member named ‘allocated’ src/lxml/lxml.etree.c:3688:30: error: ‘PySet_Type’ undeclared (first use in this function) src/lxml/lxml.etree.c:3688:30: note: each undeclared identifier is reported only once for each function it appears in src/lxml/lxml.etree.c: In function ‘__pyx_f_4lxml_5etree_18_FileReaderContext__readDtd’: src/lxml/lxml.etree.c:72226:7: error: ‘_save’ undeclared (first use in this function) .... ... error: command 'cc' failed with exit status 1 ------------------------------------------------------------------------------------ Next I have tried the following (which is probabely a "nonsense"): ------------------------------------------------------------------------------------ $ pypy setup.py build_ext -i -I/usr/include/python2.7 pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Building lxml version 2.3.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib running build_ext building 'lxml.etree' extension cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 -I/usr/local/pypy/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so building 'lxml.objectify' extension cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 -I/usr/local/pypy/include -c src/lxml/lxml.objectify.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -w cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o /home/nik/tmp/lxml-2.3.2/src/lxml/objectify.pypy-17.so ------------------------------------------------------------------------------------ $ PYTHONPATH=src pypy pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:11) [PyPy 1.7.0 with GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``happy new year'' >>>> from lxml import etree Traceback (most recent call last): File "<console>", line 1, in <module> ImportError: unable to load extension module '/home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so': /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so: undefined symbol: PyClass_Type >>>> Has anyone ever tried to build lxml with pypy? What is the correct way to build an extension module? Any feedback is very much appreciated! Regards Nik PS: By the way, what do pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) mean? Both libraries are available on my PC.

4 4

PyPy 1.6 not working on Windows XP
by Ram Rachum 05 Dec '11

05 Dec '11

Hey guys, I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on both system when running the GarlicSim test suite. It shows a Windows error dialog saying "pypy.exe has encountered a problem and needs to close. We are sorry for the inconvenience." and giving this data: AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll ModVer: 5.1.2600.5512 Offset: 00040b0d I can also open a dialog with a lot of data on the error (don't know how useful it is) but Windows won't let me Ctrl-C it. What can I do? Thanks, Ram.

8 32

Win32 build of the release 1.7
by Armin Rigo 01 Dec '11

01 Dec '11

Hi all, We have a tentative Win32 build of PyPy 1.7 including both the JIT and Stackless features (the '_continuation' primitive module and the app-level 'stackless' and 'greenlet' modules). It built and passed a good proportion of tests. If you are interested in helping, could a few people try to download it and tell us if it works, and if something is obviously missing? Thank you! http://buildbot.pypy.org/nightly/release-1.7.x/pypy-c-jit-49856-930f0bc4125… A bientôt, Armin.

7 8

Pip, Pypy, and Win7
by Yaacov Finkelman 30 Nov '11

30 Nov '11

I've been a complete fanboy to Pypy for the past couple of months. After the official Sprint started I was checking the blog every couple of hours to see what wonderful and fascinating ideas people were working on implementing. I just want to thank you all for helping with this fascinating project. I was hoping to get some help. I have been trying to follow these (http://doc.pypy.org/en/latest/getting-started.html#installing-pypy) instructions to install pip on Pypy on my Win7 computer. $ curl -O http://python-distribute.org/distribute_setup.py So I downloaded that file and saved it to my Pypy directory "C:\pypy-1.7\" $ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py So I downloaded that file and saved it to my Pypy directory "C:\pypy-1.7\" $ ./pypy-1.6/bin/pypy distribute_setup.py Run this file with Pypy. (Runs with no errors) $ ./pypy-1.6/bin/pypy get-pip.py Run that file with Pypy. (Runs with no errors) $ ./pypy-1.6/bin/pip install pygments # for example I am not sure how to translate this Instruction. There is no executable Pip in my Pypy folder. How do I actually install something with pip? Jacob

2 3

PIL, Pypy-1.7, and Win7
by Yaacov Finkelman 30 Nov '11

30 Nov '11

Now the pip is working I was trying to use it to install PIL part of long error message is below. I am using Pypy-1.7 Any help? Jacob File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\build_ext.py", line 314, in run force=self.force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\ccompiler.py", line 10 54, in new_compiler return klass(None, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", l ine 309, in __init__ CygwinCCompiler.__init__ (self, verbose, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", l ine 99, in __init__ (status, details) = check_config_h() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", l ine 383, in check_config_h fn = sysconfig.get_config_h_filename() AttributeError: 'module' object has no attribute 'get_config_h_filename' Complete output from command C:\pypy-1.7\pypy.exe -c "import setuptools;__fi le__='c:\\pypy-1.7\\bin\\build\\PIL\\setup.py';exec(compile(open(__file__).read( ).replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally -managed --record c:\users\jlap\appdata\local\temp\pip-zpbnsd-record\install-rec ord.txt: WARNING: '' not a valid package name; please use only.-separated package nam es in setup.py running install running build running build_py creating build creating build\lib.win32-2.7 copying PIL\ArgImagePlugin.py -> build\lib.win32-2.7 copying PIL\BdfFontFile.py -> build\lib.win32-2.7 copying PIL\BmpImagePlugin.py -> build\lib.win32-2.7 copying PIL\BufrStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\ContainerIO.py -> build\lib.win32-2.7 copying PIL\CurImagePlugin.py -> build\lib.win32-2.7 copying PIL\DcxImagePlugin.py -> build\lib.win32-2.7 copying PIL\EpsImagePlugin.py -> build\lib.win32-2.7 copying PIL\ExifTags.py -> build\lib.win32-2.7 copying PIL\FitsStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\FliImagePlugin.py -> build\lib.win32-2.7 copying PIL\FontFile.py -> build\lib.win32-2.7 copying PIL\FpxImagePlugin.py -> build\lib.win32-2.7 copying PIL\GbrImagePlugin.py -> build\lib.win32-2.7 copying PIL\GdImageFile.py -> build\lib.win32-2.7 copying PIL\GifImagePlugin.py -> build\lib.win32-2.7 copying PIL\GimpGradientFile.py -> build\lib.win32-2.7 copying PIL\GimpPaletteFile.py -> build\lib.win32-2.7 copying PIL\GribStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\Hdf5StubImagePlugin.py -> build\lib.win32-2.7 copying PIL\IcnsImagePlugin.py -> build\lib.win32-2.7 copying PIL\IcoImagePlugin.py -> build\lib.win32-2.7 copying PIL\Image.py -> build\lib.win32-2.7 copying PIL\ImageChops.py -> build\lib.win32-2.7 copying PIL\ImageCms.py -> build\lib.win32-2.7 copying PIL\ImageColor.py -> build\lib.win32-2.7 copying PIL\ImageDraw.py -> build\lib.win32-2.7 copying PIL\ImageDraw2.py -> build\lib.win32-2.7 copying PIL\ImageEnhance.py -> build\lib.win32-2.7 copying PIL\ImageFile.py -> build\lib.win32-2.7 copying PIL\ImageFileIO.py -> build\lib.win32-2.7 copying PIL\ImageFilter.py -> build\lib.win32-2.7 copying PIL\ImageFont.py -> build\lib.win32-2.7 copying PIL\ImageGL.py -> build\lib.win32-2.7 copying PIL\ImageGrab.py -> build\lib.win32-2.7 copying PIL\ImageMath.py -> build\lib.win32-2.7 copying PIL\ImageMode.py -> build\lib.win32-2.7 copying PIL\ImageOps.py -> build\lib.win32-2.7 copying PIL\ImagePalette.py -> build\lib.win32-2.7 copying PIL\ImagePath.py -> build\lib.win32-2.7 copying PIL\ImageQt.py -> build\lib.win32-2.7 copying PIL\ImageSequence.py -> build\lib.win32-2.7 copying PIL\ImageShow.py -> build\lib.win32-2.7 copying PIL\ImageStat.py -> build\lib.win32-2.7 copying PIL\ImageTk.py -> build\lib.win32-2.7 copying PIL\ImageTransform.py -> build\lib.win32-2.7 copying PIL\ImageWin.py -> build\lib.win32-2.7 copying PIL\ImImagePlugin.py -> build\lib.win32-2.7 copying PIL\ImtImagePlugin.py -> build\lib.win32-2.7 copying PIL\IptcImagePlugin.py -> build\lib.win32-2.7 copying PIL\JpegImagePlugin.py -> build\lib.win32-2.7 copying PIL\McIdasImagePlugin.py -> build\lib.win32-2.7 copying PIL\MicImagePlugin.py -> build\lib.win32-2.7 copying PIL\MpegImagePlugin.py -> build\lib.win32-2.7 copying PIL\MspImagePlugin.py -> build\lib.win32-2.7 copying PIL\OleFileIO.py -> build\lib.win32-2.7 copying PIL\PaletteFile.py -> build\lib.win32-2.7 copying PIL\PalmImagePlugin.py -> build\lib.win32-2.7 copying PIL\PcdImagePlugin.py -> build\lib.win32-2.7 copying PIL\PcfFontFile.py -> build\lib.win32-2.7 copying PIL\PcxImagePlugin.py -> build\lib.win32-2.7 copying PIL\PdfImagePlugin.py -> build\lib.win32-2.7 copying PIL\PixarImagePlugin.py -> build\lib.win32-2.7 copying PIL\PngImagePlugin.py -> build\lib.win32-2.7 copying PIL\PpmImagePlugin.py -> build\lib.win32-2.7 copying PIL\PsdImagePlugin.py -> build\lib.win32-2.7 copying PIL\PSDraw.py -> build\lib.win32-2.7 copying PIL\SgiImagePlugin.py -> build\lib.win32-2.7 copying PIL\SpiderImagePlugin.py -> build\lib.win32-2.7 copying PIL\SunImagePlugin.py -> build\lib.win32-2.7 copying PIL\TarIO.py -> build\lib.win32-2.7 copying PIL\TgaImagePlugin.py -> build\lib.win32-2.7 copying PIL\TiffImagePlugin.py -> build\lib.win32-2.7 copying PIL\TiffTags.py -> build\lib.win32-2.7 copying PIL\WalImageFile.py -> build\lib.win32-2.7 copying PIL\WmfImagePlugin.py -> build\lib.win32-2.7 copying PIL\XbmImagePlugin.py -> build\lib.win32-2.7 copying PIL\XpmImagePlugin.py -> build\lib.win32-2.7 copying PIL\XVThumbImagePlugin.py -> build\lib.win32-2.7 copying PIL\__init__.py -> build\lib.win32-2.7 running build_ext Traceback (most recent call last): File "app_main.py", line 51, in run_toplevel File "app_main.py", line 534, in run_it File "<string>", line 1, in <module> File "c:\pypy-1.7\bin\build\PIL\setup.py", line 486, in <module> version=VERSION, File "C:\pypy-1.7\lib-python\modified-2.7\distutils\core.py", line 152, in set up dist.run_commands() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 953, in run _commands self.run_command(cmd) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run _command cmd_obj.run() File "C:\pypy-1.7\site-packages\distribute-0.6.24-py2.7.egg\setuptools\command \install.py", line 53, in run return _install.run(self) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\install.py", line 572, in run self.run_command('build') File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cmd.py", line 326, in run_ command self.distribution.run_command(command) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run _command cmd_obj.run() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\build.py", line 12 7, in run self.run_command(cmd_name) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cmd.py", line 326, in run_ command self.distribution.run_command(command) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run _command cmd_obj.run() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\build_ext.py", lin e 314, in run force=self.force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\ccompiler.py", line 1054, in new_compiler return klass(None, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", line 309, in __init__ CygwinCCompiler.__init__ (self, verbose, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", line 99, in __init__ (status, details) = check_config_h() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", line 383, in check_config_h fn = sysconfig.get_config_h_filename() AttributeError: 'module' object has no attribute 'get_config_h_filename' ---------------------------------------- Command C:\pypy-1.7\pypy.exe -c "import setuptools;__file__='c:\\pypy-1.7\\bin\\ build\\PIL\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record c:\user s\jlap\appdata\local\temp\pip-zpbnsd-record\install-record.txt failed with error code 1 Storing complete log in C:\Users\Jlap\AppData\Roaming\pip\pip.log c:\pypy-1.7\bin>

2 2

Optimizing Append Only structures
by Timothy Baldridge 30 Nov '11

30 Nov '11

I have a dictionary (in RPython) that looks something like this: class MyDict(Obj): def add(self, k, v): newdict = self.dict.copy() newdict[k] = v self.dict = newdict def get(self, k): d = self.dict return MyDict._static_get(d, k) @staticmethod @purefunction def _static_get(d, k): if d not in k: return None return d[k] I'm trying to figure out the best way to optimize this. As you see, this structure is "append only". That is if mydict.get("foo") returns a value it will always return that value, for all time. In that sense, the function is pure. If however, the function returns None, then it could change in the future. This is for my Clojure on PyPy implementation. The idea is that types can be extended via protocols, at runtime, at any point in the program's execution. However, once a given type is extended to support a given interface (or protocol) it will never be able to be changed. That is, once we extend Foo so that it implements Foo.count() it will implement Foo.count() for all time. Any thoughts on how to optimize this further for the jit? I'd like to promote the value of get() to a constant, but I can only do that as long as it is not None. After Foo is extended, I'd like the jit to re-generate the jitted code to remove all guards, hopefully saving a few cycles. Timothy -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth)

5 4

Sandbox startup time
by Ned Batchelder 29 Nov '11

29 Nov '11

Hi all, I'm trying to use the PyPy sandbox to execute student code on a server. I've got it all working, but it takes a few seconds to start the sandbox process, so the response time from the server is poor. There are a few issues: 1) The sandbox makes a lot of system calls as part of the Python startup. I understand how to reduce these somewhat by creating dummy resources. 2) The sandbox invokes the C compiler to determine system characteristics. I got a patch from Geoff Thomas as MIT (they used the sandbox for labs in a security course). He found that this change got rid of his C compiler invocation: --- clean/pypy-pypy-release-1.6/pypy/rlib/rmarshal.py 2011-08-15 11:10:35.000000000 -0400 +++ pypy-pypy-release-1.6/pypy/rlib/rmarshal.py 2011-09-27 19:32:51.339470297 -0400 @@ -195,7 +195,7 @@ def dump_float(buf, x): buf.append(TYPE_FLOAT) - s = formatd(x, 'g', 17) + s = '%f' % (x) buf.append(chr(len(s))) buf += s add_dumper(annmodel.SomeFloat(), dump_float) I tried this change, and it reduced the number of compiler calls from three to one. Before the change: [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused /tmp/usession-default-4/platcheck_7.c -o /tmp/usession-default-4/platcheck_7.o Warning: cannot find your CPU L2 cache size in /proc/cpuinfo [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused -I/home/ned/pypy/pypy/translator/c /tmp/usession-default-4/module_cache/module_0.c -o /tmp/usession-default-4/module_cache/module_0.o [platform:execute] gcc -shared /tmp/usession-default-4/module_cache/module_0.o -pthread -lrt -Wl,--export-dynamic,--version-script=/tmp/usession-default-4/dynamic-symbols-0 -o /tmp/usession-default-4/shared_cache/externmod.so After the change: [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused /tmp/usession-default-3/platcheck_7.c -o /tmp/usession-default-3/platcheck_7.o Warning: cannot find your CPU L2 cache size in /proc/cpuinfo I think this compile is to determine the size of floats? I don't understand why the sandbox needs to determine this dynamically every time it's run, and I don't understand how to get it not to, and I don't know what bad effects the patch has. I also don't know what the remaining compile does or how to disable it. If someone could mentor me through this process, I would appreciate it. Email or IRC are fine channels for me. --Ned.

1 0

PyPy related internship at NCAR
by Davide Del Vento 28 Nov '11

28 Nov '11

Folks, NCAR has an internship program in computer science (see http://www2.cisl.ucar.edu/siparcs for details - note that interns are well paid and have many benefits). I'm mentor for a PyPy internship: http://www2.cisl.ucar.edu/siparcs/opportunities/ad The internship is open to university students enrolled an USA university. Feel free to contact me if you have any questions, or simply DO APPLY :-) Best, Davide

1 0

Real world comparison with native application & cpython
by Serhat Sevki Dincer 25 Nov '11

25 Nov '11

I wrote a tiny grep with multi-line match support, and compared its speed under pypy 1.7 with grep and CPython 2.7.1 (on ubuntu 11.04 laptop). No special algorithm/implementation is employed; it is bare re module. input: Plone 4.1.2 eggs directory, size 286mb, possible processed input size is about 75mb, processed 3958 files total commands: time mgrp -lcrN '\.py$' for . takes 1.95s time python2.7 /usr/local/bin/mgrp -lcrN '\.py$' for . takes 1.45s time grep -lcr --color=none --include='*.py' for . takes 0.6s Is the input too small to see the benefits of pypy?

4 4