Mailman 3 November 2013 - pypy-dev

List of finished NumPy functions // Progressbar
by anatoly techtonik 16 Nov '13

16 Nov '13

Hi, Please CC. I am looking for that table of supported and not supported NumPy functions in PyPy. I expected to find it here, but it is missing - http://pypy.org/numpydonate.html It is also interesting to see NumPy status as a progress bar that consists from the area of all highlighted blocks with function names that are scattered over the page. It will be possible to add some complexity indicators to these blocks and they will directly affect which area of the progress bar is covered by this block. The biggest blocks can then be decomposed into smaller ones, which may not carry only function names, but rather task/research steps to be done to make the stuff operational. If this visual decomposition is implemented, it will enable focused parallel development, and include more people, as more people will be able to see where can they help. -- anatoly t.

3 3

Missing PyUnicode_Compat
by Michal Vyskocil 15 Nov '13

15 Nov '13

Hi all, during a build of pypy-numpy from https://bitbucket.org/pypy/numpy.git I've got a following error cc: numpy/core/src/dummymodule.c In file included from numpy/core/src/private/npy_pycompat.h:4:0, from numpy/core/src/dummymodule.c:11: numpy/core/include/numpy/npy_3kcompat.h: In function 'PyUnicode_ConcatAndDel': numpy/core/include/numpy/npy_3kcompat.h:125:5: warning: implicit declaration of function 'PyUnicode_Concat' [-Wimplicit-function-declaration] newobj = PyUnicode_Concat(*left, right); So first of all, is dummymodule.c imporant or errors can be ignored there? Guessing from its name, I'd not say so. But during a grepping I found that even if it is somehow declared in cpyext/stubs.py, the symbol is missing from pypy itself, where some others PyUnicode_ things are in there. Is that correct and cpyext/stub.py is enough, or not? Regards Michal Vyskocil

2 1

Preparing for the release of PyPy 2.2
by Armin Rigo 15 Nov '13

15 Nov '13

Hi all, Preparing for the next release. The Linux binaries of PyPy 2.2, release candidate, are here: http://buildbot.pypy.org/nightly/release-2.2.x/ This corresponds to the "release-2.2.x" branch in the repo. There are a few bug reports that will likely *not* get fixed for lack of manpower: we would need someone involved with Windows, OS/X, or the BSDs. https://bugs.pypy.org/issue1626 - Tkinter on Windows https://bugs.pypy.org/issue1166 - kqueue on OSX/BSD https://bugs.pypy.org/issue1441 - version of sqlite3.dll on Win. Also, we need again someone to make the OS/X binary (Alex?); we can't use our buildbot because its old compiler builds slightly bogus binaries. A bientôt, Armin.

7 19

PyPy 2.2
by Armin Rigo 14 Nov '13

14 Nov '13

======================================= PyPy 2.2 - Incrementalism ======================================= We're pleased to announce PyPy 2.2, which targets version 2.7.3 of the Python language. This release main highlight is the introduction of the incremental garbage collector, sponsored by the `Raspberry Pi Foundation`_. This release also contains several bugfixes and performance improvements. You can download the PyPy 2.2 release here: http://pypy.org/download.html Note that the Win32 download is delayed as we fix a few more things. We would like to thank our donors for the continued support of the PyPy project. We showed quite a bit of progress on all three projects (see below) and we're slowly running out of funds. Please consider donating more so we can finish those projects! The three projects are: * Py3k (supporting Python 3.x): the release PyPy3 2.2 is imminent. * STM (software transactional memory): a preview will be released very soon, as soon as we fix a few bugs * NumPy: the work done is included in the PyPy 2.2 release. More details below. .. _`Raspberry Pi Foundation`: http://www.raspberrypi.org What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (`pypy 2.2 and cpython 2.7.2`_ performance comparison) due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows 32, or ARM (ARMv6 or ARMv7, with VFPv3). Work on the native Windows 64 is still stalling, we would welcome a volunteer to handle that. .. _`pypy 2.2 and cpython 2.7.2`: http://speed.pypy.org Highlights ========== * Our Garbage Collector is now "incremental". It should avoid almost all pauses due to a major collection taking place. Previously, it would pause the program (rarely) to walk all live objects, which could take arbitrarily long if your process is using a whole lot of RAM. Now the same work is done in steps. This should make PyPy more responsive, e.g. in games. There are still other pauses, from the GC and the JIT, but they should be on the order of 5 milliseconds each. * The JIT counters for hot code were never reset, which meant that a process running for long enough would eventually JIT-compile more and more rarely executed code. Not only is it useless to compile such code, but as more compiled code means more memory used, this gives the impression of a memory leak. This has been tentatively fixed by decreasing the counters from time to time. * NumPy has been split: now PyPy only contains the core module, called ``_numpypy``. The ``numpy`` module itself has been moved to ``https://bitbucket.org/pypy/numpy`` and ``numpypy`` disappeared. You need to install NumPy separately with a virtualenv: ``pip install git+https://bitbucket.org/pypy/numpy.git``; or by directly doing ``git clone https://bitbucket.org/pypy/numpy.git``, ``cd numpy``, ``python setup.py install``. * non-inlined calls have less overhead * Things that use ``sys.set_trace`` are now JITted (like coverage) * JSON encoding used to be very fast, now decoding is as well * various buffer copying methods experience speedups (like list-of-ints to ``int[]`` buffer from cffi) * We finally wrote (hopefully) all the missing ``os.xxx()`` functions, including ``os.startfile()`` on Windows and a handful of rare ones on Posix. * numpy has a rudimentary C API that cooperates with ``cpyext`` Cheers, Armin Rigo and Maciej Fijalkowski

1 0

FreeBSD buildslaves: change proposal
by Tobias Oberstein 12 Nov '13

12 Nov '13

Hi, we (Tavendo) currently operate 2 Twisted/FreeBSD buildslaves: a) FreeBSD 8.2, i386 with 1 builder (CPy 2.7) => part of the list of "officially supported" builders b) FreeBSD 9.1, amd64 with 2 builders (CPy 2.7 and PyPy) => only listed under "unsupported" We would like to change that into the following: c) one FreeBSD amd64 host with 3 builders c1) CPy 2.7 (supported) c2) PyPy 2.2 (supported) c3) PyPy Nightly (unsupported) Here are some reasons: - FreeBSD 8.2 is very old (read, unsupported) now, FreeBSD 10 is around the corner, and 9.2 the current official stable release - i386 is .. getting obscure. Does _anybody_ run FreeBSD i386 anymore? - there are ongoing workarounds to support that thing .. time that could be spent better - we want to run a PyPy slave for the PyPy project on that same host (this allows to produce the PyPy/Nightlies .. on a daily basis .. and immediately run Twisted trials against that) - someone needs to run a PyPy slave anyway .. the current one is broken - the current Twisted 13.2 has run 100% clean on above b) (CPy) : http://buildbot.twistedmatrix.com/builders/freebsd-9.1-amd64-python2.7/buil… So, I'd like to ask Twisted developers if above makes sense, and you would support this way forward. I've been in contact with PyPy developers, and there seems to be clear interest in improving support for both Twisted and FreeBSD (hence the cross-post). In general, Tavendo is committed to contribute time & material making Twisted + CPy|PyPy + FreeBSD a combination fully (even better) supported and actually first-class in terms of features, compatibility and performance. We think < Twisted + PyPy + FreeBSD > in particular has great potential and can provide an extremely powerful stack: "The _complete_ power to serve." .. stealing from FreeBSD's motto here - heh, and sorry the the marketing blurb .. I just couldn't resist;) Thanks! /Tobias

1 0

PyPy acting slow
by Ryan Gonzalez 10 Nov '13

10 Nov '13

For some reason, my PyPy installation seems very slow; both the version from the Ubuntu repos and a custom build I did(WITH the JIT enabled). I ran example2.py from the RPython tutorial using CPython and PyPy. Even after the JIT(visibly) kicked in, it was still much slower than CPython. While CPython took 0.5 seconds, PyPy took 10.8! Why is this happening? -- Ryan

2 1

cffi python long
by Nathan Hurst 08 Nov '13

08 Nov '13

Is there a nicer way to pass python long ints (bigint) into C efficiently? I'm currently cutting the value up into 64 bit chunks in python and passing in as an unsigned long*: cdef("int bigInt(int n, unsigned long* x);") x = sum(1 << i for i in [100,200,123]) xs = [] while x > 0: xs.append(x & ((1 << 64) - 1)) x >>= 64 print lib.bigInt(len(xs), xs) but this is quite slow, and it seems like this data must already be lurking somewhere in exactly the right form. njh

3 2

fail about resumable hashlib code under pypy
by KaShining 08 Nov '13

08 Nov '13

Here is my test.py about resumable-hashlib: from ctypes import * import hashlib PyObject_HEAD = [ ('ob_refcnt', c_size_t), ('ob_type', c_void_p), ] class EVP_MD(Structure): _fields_ = [ ('type', c_int), ('pkey_type', c_int), ('md_size', c_int), ('flags', c_ulong), ('init', c_void_p), ('update', c_void_p), ('final', c_void_p), ('copy', c_void_p), ('cleanup', c_void_p), ('sign', c_void_p), ('verify', c_void_p), ('required_pkey_type', c_int*5), ('block_size', c_int), ('ctx_size', c_int), ] class EVP_MD_CTX(Structure): _fields_ = [ ('digest', POINTER(EVP_MD)), ('engine', c_void_p), ('flags', c_ulong), ('md_data', POINTER(c_char)), ] class EVPobject(Structure): _fields_ = PyObject_HEAD + [ ('name', py_object), ('ctx', EVP_MD_CTX), ] #import string # str4096 = random_str(4096) str4096 = 'sssss' hash2 = hashlib.sha1() c_evp_obj = cast(c_void_p(id(hash2)), POINTER(EVPobject)).contents ctx = c_evp_obj.ctx digest = ctx.digest.contents state = ctx.md_data[:digest.ctx_size] #print digest.ctx_size Pass in cpython while fail in pypy: Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "t.py", line 49, in <module> digest = ctx.digest.contents File "/opt/pypy2.1/lib_pypy/_ctypes/pointer.py", line 83, in getcontents raise ValueError("NULL pointer access") ValueError: NULL pointer access

2 5

error of gunicorn+gevent under pypy
by KaShining 06 Nov '13

06 Nov '13

set up gevent by http://smetj.net/installing-pypy-with-gevent-and-virtualenv-on-fedora.html and run my test.py by gunicorn+gevent, err below: Dose anything else I need to do??

1 1

May we delete the pypy/tkinter repository?
by Matti Picus 04 Nov '13

04 Nov '13

3 4