Mailman 3 April 2014 - pypy-dev

Regression for lxml-cffi install (?)
by Костя Лопухин 16 Apr '14

16 Apr '14

Hello! https://bugs.pypy.org does not work at the moment, so posting here. pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi works under pypy 2.1 and fails with pypy nighly. Not sure that this does really belong to pypy bugtracker, but it looks like a regression. Tested under Ubuntu 12.04 and OS X, on the same machines. In order to install lxml from source on ubuntu 12.04, I installed the following ubuntu packages: libxml2 libxslt1-dev zlib1g-dev pypy nighly (it failed on older nightlies too, and I can check on other releases if needed) Python 2.7.3 (8df8a01902f8, Feb 27 2014, 23:00:44) [PyPy 2.3.0-alpha0 with GCC 4.6.3] on linux2 Error under OS X (here error message is better than with gcc): VerificationError: CompileError: command 'cc' failed with exit status 1 Complete output from command python setup.py egg_info: src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1555:18: warning: incompatible pointer types initializing 'xmlBuffer **' (aka 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka 'struct _xmlBuf **') [-Wincompatible-pointer-types] { xmlBuffer * *tmp = &p->buffer; (void)tmp; } ^ ~~~~~~~~~~ src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1556:18: warning: incompatible pointer types initializing 'xmlBuffer **' (aka 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka 'struct _xmlBuf **') [-Wincompatible-pointer-types] { xmlBuffer * *tmp = &p->conv; (void)tmp; } ^ ~~~~~~~~ src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: cannot take the address of an rvalue of type 'const xmlChar *' (aka 'const unsigned char *') return (&XML_XML_NAMESPACE); ^~~~~~~~~~~~~~~~~~ 2 warnings and 1 error generated. Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.28 Building against libxml2/libxslt in the following directory: /opt/local/lib Full error under Ubuntu 12.04: (testlxml_nighly)[DEV] kostia@dev ~ $ pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi Obtaining lxml-cffi from git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi Updating ./testlxml_nighly/src/lxml-cffi clone (to cffi) Running setup.py (path:/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py) egg_info for package lxml-cffi src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In function '_cffi_var_XML_XML_NAMESPACE': src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: lvalue required as unary '&' operand Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "app_main.py", line 578, in run_it File "<string>", line 17, in <module> File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 220, in <module> **setup_extra_options() File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 131, in setup_extra_options STATIC_CFLAGS, STATIC_BINARIES) File "setupinfo.py", line 161, in ext_modules import lxml.etree File "src/lxml/../lxml-cffi/etree.py", line 7, in <module> from .apihelpers import _assertValidNode, _assertValidDoc File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in <module> from .includes.etree_defs import _isString, _getNs, _isElement File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", line 1, in <module> from . import tree File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line 490, in <module> libraries=['xml2']) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", line 339, in verify lib = self.verifier.load_library() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 74, in load_library self._compile_module() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 139, in _compile_module outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 25, in compile outputfilename = _build(tmpdir, ext) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 50, in _build raise VerificationError('%s: %s' % (e.__class__.__name__, e)) VerificationError: CompileError: command 'cc' failed with exit status 1 Complete output from command python setup.py egg_info: src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In function '_cffi_var_XML_XML_NAMESPACE': src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: lvalue required as unary '&' operand Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "app_main.py", line 578, in run_it File "<string>", line 17, in <module> File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 220, in <module> **setup_extra_options() File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 131, in setup_extra_options STATIC_CFLAGS, STATIC_BINARIES) File "setupinfo.py", line 161, in ext_modules import lxml.etree File "src/lxml/../lxml-cffi/etree.py", line 7, in <module> from .apihelpers import _assertValidNode, _assertValidDoc File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in <module> from .includes.etree_defs import _isString, _getNs, _isElement File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", line 1, in <module> from . import tree File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line 490, in <module> libraries=['xml2']) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", line 339, in verify lib = self.verifier.load_library() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 74, in load_library self._compile_module() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 139, in _compile_module outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 25, in compile outputfilename = _build(tmpdir, ext) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 50, in _build raise VerificationError('%s: %s' % (e.__class__.__name__, e)) VerificationError: CompileError: command 'cc' failed with exit status 1 ---------------------------------------- Cleaning up...

2 2

RPython as a separate package
by Bogdan Opanchuk 16 Apr '14

16 Apr '14

Hello, I would like to use the RPython toolchain in my project. The problem is, RPython is currently hidden inside PyPy and therefore not readily available. I found this thread https://mail.python.org/pipermail/pypy-dev/2012-October/010602.html which goes as far as to state "Note that the fact of splitting this is not up to discussion, however, how we go about it is.". So, what is the status of the splitting? Was this idea eventually abandoned?

5 6

OS/X
by Armin Rigo 16 Apr '14

16 Apr '14

Hi OS/X'ers, I see in rpython/translator/platform/darwin.py that we use only "clang" as the compiler on OS/X. There was some discussion in #pypy, as well as a proposal from Andrew Dalke to upgrade the gcc on the OS/X buildbot, which go along the lines of "clang is not necessarily always better than gcc on OS/X". So I would like that the RPython toolchain supports both choices. Please don't discuss clang-vs-gcc endlessly here, I'm sure there are people who think that clang is better than gcc and others that have the opposite view and both camps have strong opinions -- that's not the point and I don't honestly care. This e-mail is *only* about supporting both choices. Could someone on OS/X check that we can use gcc to translate pypy's? This can be checked by trying out a smaller example, like rpython/translator/goal/targetnopstandalone.py . What fixes do you need in darwin.py? Or simply pass some environment variable? Thanks! A bientôt, Armin.

2 4

PyPy support for PPC processors
by Mike Mezeul 15 Apr '14

15 Apr '14

Hello; Does anyone have any insight as to why the support for Power PC processors has stalled? I'm specifically interested in knowing if it was technical hurdles or just lack of interest. If technical, could someone please elaborate on what the issues are? Thanks, Mike Mezeul Michael Mezeul Senior Director R&D ADVA Optical Networking 2301 North Greenville Avenue, Suite 300 Richardson, TX 75082, USA Phone: +1.972.759.1216 Fax: +1.972.759.1201 Mobile: +1.214.448.9239 e-mail: mmezeul(a)advaoptical.com<mailto:mmezeul@advaoptical.com> www.advaoptical.com<http://www.advaoptical.com/> Let's ADVANCE ADVA Optical Networking SE is a European stock corporation ("Societas Europaea") with registered offices at Maerzenquelle 1-3, D-98617 Meiningen, Germany * CEO: Brian L. Protiva, Chief Officers: Dr. Christoph Glingener, Christian Unterberger, Jaswir Singh * Chairman of the Supervisory Board: Anthony Maher * AG Jena HRB 508155 * VAT No. DE 175 446 349

3 3

performance problems with Krakatau
by Robert Grosse 13 Apr '14

13 Apr '14

Hi, I have been trying to use Pypy to speed up the Krakatau decompiler ( https://github.com/Storyyeller/Krakatau). It is a large, pure Python application with several compute intensive parts, so I thought it would work well. Unfortunately, there is no clear speedup, and Pypy requires several times as much memory as well, making it unusual for larger inputs. For example, decompiling a quarter of ASM, I got the following results (execution time, memory usage) cpython 64 - 62.5s, 102.6kb cpython 32 - 69.2s, 54.5kb pypy 2.1.0 - 106.5s, 277.8kb pypy 2.2.1 - 109.2s, 194.6kb Sometimes, 2.2.1 is faster than 2.1.0, but they're both clearly much worse than CPython. These tests were performed on Windows 7 64bit using the prebuilt 32bit binaries of Pypy. I tested the 32bit version of CPython too, to see if the problem was a lack of 64bit support. However, CPython 32bit also vastly outperformed Pypy. Execution time was measured using time.time(). Memory usage was measured by watching the Windows Resource Manager and recording the peak Private value. Similar patterns were seen in Working Set, etc. I thought the increased memory usage at least might be explained by constant overhead from compiled code or from it not running long enough to trigger full garbage collection. However, Pypy continues to use several times as much memory on much larger examples. Does anyone know what could be going on here? Pypy isn't normally slower than CPython. Is there a way for me to tell what the problem is?

2 11

CPython compatibility in the fractions module
by Naftali Harris 10 Apr '14

10 Apr '14

Hi everyone, I'd like to report a compatibility issue between CPython and PyPy in the fractions module. In short, CPython coerces Fraction objects into int's in strings formatted with "%d", but PyPy throws a TypeError. Here's an example reproducing this: ~/Downloads/pypy-2.2.1-linux64/bin$ ./pypy Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 18:48:13) [PyPy 2.2.1 with GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``<arigato> fijal: I'm sure there is tons of unspecified context information that I should ideally also be aware of'' >>>> from fractions import Fraction >>>> print "%d" % Fraction(1, 1) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type for int(): 'Fraction' >>>> ~/Downloads/pypy-2.2.1-linux64/bin$ python Python 2.7.3 (default, Feb 27 2014, 19:37:34) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from fractions import Fraction >>> print "%d" % Fraction(1, 1) 1 >>> Thanks for the great work you all do! Naftali

2 1

msvc and floating point compiler flags
by matti picus 09 Apr '14

09 Apr '14

looking at this test failure http://buildbot.pypy.org/summary/longrepr?testname=test_math&builder=own-wi… which tests that atan2(0.23/1.23) == 1.3.... via OP_FLOAT_EQ(double value, double value) OP_FLOAT_EQ is a compiler define to (a == b), which is tricky with double values. Here is a nice discussion of the issue on MSVC http://www.gamasutra.com/view/news/167402/Indepth_Intermediate_floatingpoin… The link even has a flow chart for choosing msvc or intel compiler flags. The test passes if I add /fp:fast to the msvc compile flag Would anyone object if I add this to the default MSVC flags? FWIW, the "fast" is misleading, apparently it can be slower than the default for some cases. Matti

3 2

should os.fdopen(fd-of-a-directory, "r") succeed?
by Dima Tisnek 09 Apr '14

09 Apr '14

it fails on py 2.7 (with a bug), fails correctly in py 3.3 and succeeds in pypy 2.2: >>>> import os >>>> os.open("/", os.O_RDONLY) 3 >>>> os.fdopen(3, "r") <open file '<fdopen>', mode 'r' at 0x0000000105b71830> >>>> f = _ >>>> f.read(1) Traceback (most recent call last): File "<stdin>", line 1, in <module> IOError: [Errno 21] Is a directory: '<fdopen>' >>>>

3 3

"aborts" in unroll
by Carl Friedrich Bolz 09 Apr '14

09 Apr '14

Hi all, I found the reason why unroll often produces uncounted "aborts". It's the constructor here: class InvalidLoop(JitException): """Raised when the optimize*.py detect that the loop that we are trying to build cannot possibly make sense as a long-running loop (e.g. it cannot run 2 complete iterations).""" def __init__(self, msg='?'): debug_start("jit-abort") debug_print(msg) debug_stop("jit-abort") self.msg = msg unroll uses InvalidLoop internally, which does not propagate up. I am fixing that on my branch. Cheers, Carl Friedrich

1 0

Python Execution Contexts Ch.3 - High Level
by anatoly techtonik 05 Apr '14

05 Apr '14

Hi, I am proud to present the picture that shows various places where Python code can be executed. 51k picture that worths 1k of words. https://docs.google.com/drawings/d/1B3ktR-htmql9xYeFeqTU6L1yefI9gqLiQ7uIr7g… Trying to figure out different entrypoints for executing Python code and context available for Python code in these entrypoints. Would be nice to get some feedback on this diagram. I hope it is still better than text. I didn't compare __builtins__ - that's the task better done with HTML table generator, but there are two questions about Python/PyPy console context. pypy-2.2.1 >>> dir() ['__builtins__', '__doc__', '__name__'] python-2.7.6 >>> dir() ['__builtins__', '__doc__', '__name__', '__package__'] 1. Why there is __package__ that can not be anything but None? I consider this to be a bug in Python 2.x, because I don't see how this should be used in console, so I don't mind that PyPy doesn't implement it. But then comes another question. 2. If PyPy doesn't implement useless console context properties like __package__, why there is __doc__ entry that can not be anything but None?

2 1