Mailman 3 March 2014 - pypy-dev

Cross-post from python-ideas: Compressing the stack on the fly
by Ram Rachum 28 Jul '14

28 Jul '14

Hi guys, I made a post on the Python-ideas mailing list that I was told might be relevant to Pypy. I've reproduced the original email below. Here is the thread on Python-ideas with all the discussion.<https://groups.google.com/forum/?fromgroups#!topic/python-ideas/hteGSNTyC_4> -------------- Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be accessed, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

2 1

Regression for lxml-cffi install (?)
by Костя Лопухин 16 Apr '14

16 Apr '14

Hello! https://bugs.pypy.org does not work at the moment, so posting here. pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi works under pypy 2.1 and fails with pypy nighly. Not sure that this does really belong to pypy bugtracker, but it looks like a regression. Tested under Ubuntu 12.04 and OS X, on the same machines. In order to install lxml from source on ubuntu 12.04, I installed the following ubuntu packages: libxml2 libxslt1-dev zlib1g-dev pypy nighly (it failed on older nightlies too, and I can check on other releases if needed) Python 2.7.3 (8df8a01902f8, Feb 27 2014, 23:00:44) [PyPy 2.3.0-alpha0 with GCC 4.6.3] on linux2 Error under OS X (here error message is better than with gcc): VerificationError: CompileError: command 'cc' failed with exit status 1 Complete output from command python setup.py egg_info: src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1555:18: warning: incompatible pointer types initializing 'xmlBuffer **' (aka 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka 'struct _xmlBuf **') [-Wincompatible-pointer-types] { xmlBuffer * *tmp = &p->buffer; (void)tmp; } ^ ~~~~~~~~~~ src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1556:18: warning: incompatible pointer types initializing 'xmlBuffer **' (aka 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka 'struct _xmlBuf **') [-Wincompatible-pointer-types] { xmlBuffer * *tmp = &p->conv; (void)tmp; } ^ ~~~~~~~~ src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: cannot take the address of an rvalue of type 'const xmlChar *' (aka 'const unsigned char *') return (&XML_XML_NAMESPACE); ^~~~~~~~~~~~~~~~~~ 2 warnings and 1 error generated. Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.28 Building against libxml2/libxslt in the following directory: /opt/local/lib Full error under Ubuntu 12.04: (testlxml_nighly)[DEV] kostia@dev ~ $ pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi Obtaining lxml-cffi from git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi Updating ./testlxml_nighly/src/lxml-cffi clone (to cffi) Running setup.py (path:/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py) egg_info for package lxml-cffi src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In function '_cffi_var_XML_XML_NAMESPACE': src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: lvalue required as unary '&' operand Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "app_main.py", line 578, in run_it File "<string>", line 17, in <module> File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 220, in <module> **setup_extra_options() File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 131, in setup_extra_options STATIC_CFLAGS, STATIC_BINARIES) File "setupinfo.py", line 161, in ext_modules import lxml.etree File "src/lxml/../lxml-cffi/etree.py", line 7, in <module> from .apihelpers import _assertValidNode, _assertValidDoc File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in <module> from .includes.etree_defs import _isString, _getNs, _isElement File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", line 1, in <module> from . import tree File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line 490, in <module> libraries=['xml2']) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", line 339, in verify lib = self.verifier.load_library() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 74, in load_library self._compile_module() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 139, in _compile_module outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 25, in compile outputfilename = _build(tmpdir, ext) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 50, in _build raise VerificationError('%s: %s' % (e.__class__.__name__, e)) VerificationError: CompileError: command 'cc' failed with exit status 1 Complete output from command python setup.py egg_info: src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In function '_cffi_var_XML_XML_NAMESPACE': src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: lvalue required as unary '&' operand Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "app_main.py", line 578, in run_it File "<string>", line 17, in <module> File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 220, in <module> **setup_extra_options() File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 131, in setup_extra_options STATIC_CFLAGS, STATIC_BINARIES) File "setupinfo.py", line 161, in ext_modules import lxml.etree File "src/lxml/../lxml-cffi/etree.py", line 7, in <module> from .apihelpers import _assertValidNode, _assertValidDoc File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in <module> from .includes.etree_defs import _isString, _getNs, _isElement File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", line 1, in <module> from . import tree File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line 490, in <module> libraries=['xml2']) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", line 339, in verify lib = self.verifier.load_library() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 74, in load_library self._compile_module() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 139, in _compile_module outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 25, in compile outputfilename = _build(tmpdir, ext) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 50, in _build raise VerificationError('%s: %s' % (e.__class__.__name__, e)) VerificationError: CompileError: command 'cc' failed with exit status 1 ---------------------------------------- Cleaning up...

2 2

performance problems with Krakatau
by Robert Grosse 13 Apr '14

13 Apr '14

Hi, I have been trying to use Pypy to speed up the Krakatau decompiler ( https://github.com/Storyyeller/Krakatau). It is a large, pure Python application with several compute intensive parts, so I thought it would work well. Unfortunately, there is no clear speedup, and Pypy requires several times as much memory as well, making it unusual for larger inputs. For example, decompiling a quarter of ASM, I got the following results (execution time, memory usage) cpython 64 - 62.5s, 102.6kb cpython 32 - 69.2s, 54.5kb pypy 2.1.0 - 106.5s, 277.8kb pypy 2.2.1 - 109.2s, 194.6kb Sometimes, 2.2.1 is faster than 2.1.0, but they're both clearly much worse than CPython. These tests were performed on Windows 7 64bit using the prebuilt 32bit binaries of Pypy. I tested the 32bit version of CPython too, to see if the problem was a lack of 64bit support. However, CPython 32bit also vastly outperformed Pypy. Execution time was measured using time.time(). Memory usage was measured by watching the Windows Resource Manager and recording the peak Private value. Similar patterns were seen in Working Set, etc. I thought the increased memory usage at least might be explained by constant overhead from compiled code or from it not running long enough to trigger full garbage collection. However, Pypy continues to use several times as much memory on much larger examples. Does anyone know what could be going on here? Pypy isn't normally slower than CPython. Is there a way for me to tell what the problem is?

2 11

Why CFFI is not useful - need direct ABI access 4 humans
by anatoly techtonik 03 Apr '14

03 Apr '14

I know what C in CFFI stands for C way of doing things, so I hope people won't try to defend that position and instead try to think about what if we have to re-engineer ABI access from scratch, for explicit and obvious debug binary interface. CFFI is not useful for Python programmers and here is why. The primary reason is that it requires you to know C. And knowing C requires you to know about OS architecture. And knowing about OS architecture requires you to know about ABI, which is: http://stackoverflow.com/a/3784697 This is how the compiler builds an application. It defines things (but is not limited too): How parameters are passed to functions (registers/stack). Who cleans parameters from the stack (caller/callee). Where the return value is placed for return. How exceptions propagate. The problematic part of it is that you need to think of OS ABI in terms of unusual C abstractions. Coming through several levels of them. Suppose you know OS ABI and you know that you need direct physical memory access to set bytes for a certain call in this way: 0024: 00 00 00 6C 33 33 74 00 How would you do this in Python? The most obvious way is with byte string - \x00\x00\x00\x6c\x33\x33\x74\x00 but that's not how you prepare the data for the call if, for example, 00 6C means anything to you. What is the Python way to convert 00 6C to convenient Python data structure and back and is it Pythonic (user friendly and intuitive)? import struct struct.unpack('wtf?', '\x00\x6C') If you try to lookup the magic string in struct docs: http://docs.python.org/2/library/struct.html#format-characters You'll notice that there is the mapping between possible combinations of these 2 bytes to some Python type is very mystic. First it requires you to choose either "short" or "unsigned short", but that's not enough for parsing binary data - you need to figure out the proper "endianness" and make up a magic string for it. This is just for two bytes. Imagine a definition for a binary protocol with variable message size and nested data structures. You won't be able to understand it by reading Python code. More than that - Python *by default* uses platform specific "endianness", it is uncertain (implicit) about it, so not only you should care about "endianness", but also be an expert to find out which is the correct metrics for you. Look at this: 0024: 00 00 00 6C 33 33 74 00 Where is "endianness", "alignment", "size" from this doc http://docs.python.org/2/library/struct.html#byte-order-size-and-alignment People need to *start* with this base and this concept and that's why it is harmful. CFFI proposes to provide a better interface to skip this complexity by getting back to roots and use C level. That's a pretty nice hack for C guys, I am sure it makes them completely happy, but for academic side of PyPy project, for Python interpreter and other projects build over RPython it is important to have a tool that allows to experiment with binary interfaces in convenient, readable and direct way, makes it easier for humans to understand (by reading Python code) how Python instructions are translated by JIT into binary pieces in computer memory, pieces that will be processed by operating system as a system function call on ABI level. But let's not digress, and get back to the point that struct module doesn't allow to work with structured data. In Python the only alternative standard way to define binary structure is ctypes. ctypes documentation is no better for binary guy: http://docs.python.org/2/library/ctypes.html#fundamental-data-types See how that binary guy suffered to map binary data to Python structures through ctypes: https://bitbucket.org/techtonik/discovery/src/eacd864e6542f14039c9b31eecf94… And I am saying that this is the best way available from standard library. It is pretty close to Django models, but for binary data. ctypes still is worse that struct in one thing - looking into docs, there are no size specifiers for any kind of C type, so no guarantee that 2 bytes are read as 4 bytes or worse. By looking at the ctypes code it is hard to figure out size of structure and when it may change. I can't hardly name ctypes mapping process as user friendly and resulting code as intuitive. Probably nobody could, and that's why CFFI was born. But CFFI took a different route - instead of trying to map C types to binary data (ABI level), it decided to go onto API level. While it exposes many better tool, it basically means you are dealing with C interface again - not with Pythonic interface for binary data. I am not saying that CFFI is bad - I am saying that it is good, but not enough, and that it can be fixed with cleanroom engineering approach for a broader scope of modern usage pattern for binary data than just calling OS API in C way. Why we need it? I frankly think that Stackless way of doing thing without C stack is the future, and the problem with not that not many people can see how it works, builds alternative system without classic C stack with (R)Python. Can CFFI help this? I doubt that. So, that am I proposing. Just an idea. Given the fact that I am mentally incapable of filling 100 sheet requirement to get funding under H2020, the fact that no existing commercial body could be interested to support the development as an open source project and the fact that hacking on it alone might become boring, giving this idea is the least I can do. Cleanroom engineering. http://en.wikipedia.org/wiki/Cleanroom_software_engineering "The focus of the Cleanroom process is on defect prevention, rather than defect removal." When we talk about Pythonic way of doing thing, how can we define "a defect"? Basically, we talking about user experience - the emotions that user experiences when he uses Python for the given task. What is the task at hand? For me - it is working with binary data in Python - not just parsing save games, but creating binary commands such as OS systems calls that are executed by certain CPU, GPU or whatever is on the receiver end of whatever communication interface is used. This is hardware independent and platform neutral way of doing things. So, the UX is the key, but properties of engineered product are not limited single task. The cleanroom approach allows to concentrate on the defect - when user experience will start to suffer because of the conflicts between tasks that users are trying to accomplish. For PyPy project I see the value in library for compositing of binary structures in that these operations can be pipelined and optimized at run-time in a highly effective fashion. I think that convenient binary tool is the missing brick in the basement of academic PyPy infrastructure to enable universal interoperability from (R)Python with other digital systems by providing a direct interface to the binary world. I think that 1973 year views on "high level" and "low level" systems are a little bit outdated now that we have Python, Ruby, Erlang and etc. Now C is just not a very good intermediary for "low level" access. But frankly, I do not think that with advent of networking, binary can be called a low level anymore. It is just another data format that can be as readable for humans as program structure written in Python. P.S. I have some design ideas how to make an attractive gameplay out of binary data by "coloring" regions and adding "multi-level context" to hex dumps. This falls out of scope of this issue, and requires more drawing that texting, but if somebody wants to help me with sharing the vision - I would not object. It will help to make binary world more accessible, especially for new people, who start coding with JavaScript and Python. -- anatoly t.

10 18

Re: [pypy-dev] Question about extension support
by Laura Creighton 29 Mar '14

29 Mar '14

Ok -- the lesson I took from this is 'ref counting hurts performance'. Why was that the wrong inference to make? Laura

2 1

Re: [pypy-dev] Question about extension support
by Laura Creighton 29 Mar '14

29 Mar '14

Your C-extensions come all bundled up with a whole lot of gorp which is designed to make them play nicely in a ref-counting environment. Ref counting is a very slow way to do GC. Sometimes -- really, really, really hideously slow. You are sometimes _way_ better off writing python code instead -- pypy with the jit turned off outperforms CPython purely on the benefits of not doing ref-counting, and pypy really needs the jit to be fast. There is a bit of conceptual confusion here -- on the one hand, because C extensions often were written for reasons of performance when compared to CPython, there is a tendency to believe that C-extensions are, pretty much by definition fast. And the other thing is a sort of reflexive belief that 'if it is in C (or C++) then it has to be fast'. Both of these ideas are wrong. A whole lot of C extensions are actually really, really slow. They are just faster than CPython -- or perhaps 'faster than CPython was when I wrote this thing' which isn't, after all, that hard a target to meet. When PyPy finds a C extension which is working very hard to pretend it is a set of Python objects that can be refcounted, it isn't brilliant enough to be able to throw away all the ref-counting fakery, intuit what the code here really is trying to do here, and just run that bit. That's too hard. Instead it decides to play along with the ref-counting faking. So we are at 'watch the elephant tap-dance' time ... it doesn't have to do a very good (read fast) job at this, it is amazing that it does it at all. Laura

3 3

Re: [pypy-dev] Question about extension support
by Laura Creighton 29 Mar '14

29 Mar '14

In a message of Fri, 28 Mar 2014 09:51:42 +0100, Armin Rigo writes: >Hi Laura, <why I am not thinking properly> Ah, thank you. I actually thought we had found the odd example where a better gc beat CPython performance even when the jit was off. I am completely wrong about this? Or is it just that it is so rare it doesn't matter? Thank you for teaching me this, in any case. Sorry for the misinformation. Laura

2 1

ICOOOLPS 2014 call for papers
by Carl Friedrich Bolz 28 Mar '14

28 Mar '14

======================================================================== 9th ICOOOLPS Workshop Implementation, Compilation, Optimization of OO Languages, Programs and Systems July 28th 2014, Uppsala, Sweden Colocated with ECOOP http://soft-dev.org/events/icooolps14/ ======================================================================== Overview The ICOOOLPS workshop series brings together researchers and practitioners working in the field of OO languages implementation and optimization. ICOOOLPS key goal is to identify current and emerging issues relating to the efficient implementation, compilation and optimization of such languages, and outlining future challenges and research directions. Topics of interest for ICOOOLPS include, but are not limited to: implementation of fundamental OO and OO-like features (e.g. inheritance, parametric types, memory management, objects, prototypes), runtime systems (e.g. compilers, linkers, virtual machines, garbage collectors), optimizations (e.g. static or dynamic analyses, adaptive virtual machines), resource constraints (e.g. time for real-time systems, space or low-power for embedded systems) and relevant choices and tradeoffs (e.g. constant time vs. non-constant time mechanisms, separate compilation vs. global compilation, dynamic loading vs. global linking, dynamic checking vs. proof-carrying code...). Submissions ICOOOLPS is not a mini-conference; it is a workshop designed to facilitate discussion and the exchange of ideas between peers. ICOOOLPS therefore welcomes both position (1—4 pages) and research (max. 10 pages) papers. Position papers should outline interesting or unconventional ideas, which need not be fully fleshed out. Research papers are expected to contain more complete ideas, but these need not necessarily be fully complete as with a traditional conference. Authors will be given the option to publish their papers (short or long) in the ACM Digital Library if they wish. Submissions must be written in English, formatted according to ACM SIG Proceedings style. Please submit via EasyChair (link on the ICOOOLPS website). Important dates Submission: May 5th 2014 (FIRM DEADLINE) Notification: May 26th 2014 Workshop: July 28th 2014 Programme chairs Laurence Tratt, King's College London, UK Olivier Zendra, INRIA Nancy, France e-mail: icooolps14(a)easychair.org Programme committee Carl Friedrich Bolz, King's College London, UK Eric Jul, University of Copenhagen, DK José Manuel Redondo López, Universidad de Oviedo, ES Stefan Marr, INRIA Lille, FR Floréal Morandat, Labri, FR Todd Mytkowicz, Microsoft, US Tobias Pape, Hasso-Plattner-Institut Potsdam, DE Ian Rogers, Google, US Jeremy Singer, University of Glasgow, UK Jan Vitek, Purdue University, US Mario Wolczko, Oracle Labs, US

1 0

Question about extension support
by Kevin Modzelewski 28 Mar '14

28 Mar '14

Hi all, I've been trying to learn about how PyPy supports (unmodified) Python extensions, and one thing I've heard is that it's much slower than cPython, and/or uses more memory. I tried finding some documentation about why, and all I could find is this, from 2010: https://bitbucket.org/pypy/compatibility/wiki/c-api Sorry if this should be obvious, but is there more up-to-date information about this stuff? And secondly, assuming the info that I linked to is still valid, is there a reason you guys settled on this method of bridging the refcount/tracing divide, as opposed to other possibilities (can you pin the objects in the GC)? I'm curious, since I've heard a number of people mention that extension modules are the primary reason that PyPy is slower than cPython for their code; definitely an improvement over "PyPy doesn't run my code at all", but it's made me curious about whether or not / why it has to be that way. kmod

4 8

Question about extension support
by John Camara 25 Mar '14

25 Mar '14

Hi Kevin, More up to date information can be found on the FAQ page http://doc.pypy.org/en/latest/faq.html#do-cpython-extension-modules-work-wi… The best approach for PyPy is either use a pure Python module if possible or use a cffi wrapped extension instead of an extension that uses the CPython CAPI. Often CPython CAPI extensions are wrapping some c library. Creating a cffi wrapper for the library is actually much simpler than writing a CPython CAPI wrapper. Quite a few CPython CAPI extensions have already been wrapped for cffi so make sure to search for one before creating your own wrapper. If you need to create a wrapper, refer to the cffi documentation at http://cffi.readthedocs.org/en/release-0.8/ Extensions wrapped with cffi are compatible with both CPython and PyPy. On CPython the performance is similar to what you would get if you used ctypes. How every, under PyPy, the performance is much closer to a native C call plus the overhead for releasing and acquiring the gil. John

1 1

pypy-dev March 2014