Mailman 3 April 2012 - pypy-dev

STM proposal funding
by Armin Rigo 17 Apr '12

17 Apr '12

Hi all, For those of you who missed it: the proposal on Software Transactional Memory is launched ("remove the GIL but give a better way to write multi-threaded applications"). It is available at http://pypy.org/tmdonate.html . I plan to ask the PSF for some participation, following the push towards doing so at the PyCon PSF meeting. See http://mail.python.org/pipermail/python-dev/2012-March/117680.html , "Funding from the Python Software Foundation". A bientôt, Armin.

5 14

STM/AME for CPython?
by Armin Rigo 16 Apr '12

16 Apr '12

Hi all, Countrary to what is written in the STM/AME proposal, since last week I believe it might be (reasonably) possible to apply the same techniques to CPython. For now I am experimenting with applying them in a simple CPython-like interpreter. If it works, it might end up as a patch to the core parts of CPython. The interesting property is that it would still be able to run unmodified C extension modules --- the Python code gets the benefits of multi-core STM/AME only if it involves only the patched parts of the C code, but in all cases it still works correctly. (Moreover the performance hit is well below 2x, more like 20%.) I did not try to hack CPython so far, but only a custom interpreter for a Lisp language, whose implementation should be immediately familiar to anyone who knows CPython C code: https://bitbucket.org/arigo/duhton . The non-standard built-in function is "transaction", which schedules a transaction to run later. The code contains the necessary tweaks to reference counting, and seems to work on all examples, but leaks some of the objects so far. Fixing this directly might be possible, but I'm not sure yet (it might require interaction with the cycle-detecting GC of CPython). If anyone is interested, I could create a different mailing list in order to discuss this in more details, as it's only related to PyPy on the original idea level. From experience I would think that this has the potential to become a Psyco-like experiment, but unlike 10 years ago, today I'm not ready any more to dive completely alone into a project of that scale :-) A bientôt, Armin.

3 4

Cython-CEP: Native dispatch through Python callables
by Stefan Behnel 15 Apr '12

15 Apr '12

Hi, I already mentioned this possibility a couple of times on this list, but now the idea has attracted some more general interest. The Cython project would like to specify a general way of unpacking wrapped native functions for Python implementations. This is interesting for CPython because it would allow NumPy and other extensions to unpack functions implemented by other extensions before calling them in loops. PyPy could let its JIT do the same at runtime. Cython and low-level wrapper generators would be the obvious way to implement this, but it would also provide a generic way for Cython to unpack a function pointer from wrapped functions exported by other modules, at least if the native signature is known (or likely to be known) at compile time. Dag Seljebotn has started writing a CEP (Cython Enhancement Proposal) about it and opened the discussion on the Cython core developers mailing list. http://wiki.cython.org/enhancements/cep1000 http://thread.gmane.org/gmane.comp.python.cython.devel/13416/focus=13417 http://mail.python.org/mailman/listinfo/cython-devel Additionally, there is PEP 362 which aims to provide a Signature object for Python functions. I think we should build native signatures on top of that. http://www.python.org/dev/peps/pep-0362/ Note that a wrapper function may offer more than one native signature, e.g. when wrapping overloaded C++ functions or when using Cython fused functions. We may eventually end up with a PEP instead of a CEP for this, but I think it's still some way before we get there. Please participate in the discussion if you are interested. Stefan

3 10

Communications of the European Neurological Society - April 2012
by ENS Newsletter 14 Apr '12

14 Apr '12

1 0

PyPy as part of a larger, bundled project?
by Leo Trottier 12 Apr '12

12 Apr '12

Hi, A number of Python applications (e.g. http://calibre-ebook.com/, http://www.psychopy.org/ ... http://en.wikipedia.org/wiki/List_of_Python_software#Applications) are deployed together with the libraries and interpreter that they will use. Often, these applications are larger, and can end up performing operations that are computationally intensive. In the case of Calibre, e.g., large batch conversions from one book format to another can take more than an hour (for sufficiently large batches). This motivates a couple questions: 1) How difficult might it be, currently, to swap in PyPy[1] as the interpreter for, say, Calibre (http://calibre-ebook.com/download_linux)? (I am in the process of trying to do so presently, but the question is meant to be a bit more general) 2) Might these larger, interpreter-bundled applications be good targets for "high impact" deployments of PyPy? PyPy could, here, provide a tangible benefit without requiring any extra work by end-users, thereby potentially serving as a useful demonstration platform while quickly increasing PyPy usage. Leo [1] https://bugs.launchpad.net/calibre/+bug/977453

3 6

__exit__ not called in certain cases
by Hong Minhee 12 Apr '12

12 Apr '12

I reported https://bugs.pypy.org/issue1126 which shows that __exit__ is not called in certain cases when I think it should be. The code to reproduce the bug is there. The code works on CPython. It was closed as invalid saying this is a GC issue, but I don't think it is. This is about __exit__, not __del__. __exit__ should be always called if __enter__ succeeds.

2 1

Re: [pypy-dev] [pypy-commit] pypy win32-stdlib: fix close() and tests for file closing
by Antonio Cuni 11 Apr '12

11 Apr '12

Hi Matti, On 04/11/2012 01:24 PM, mattip wrote: > Author: Matti Picus<matti.picus(a)gmail.com> > Branch: win32-stdlib > Changeset: r54283:2a8e4f56269f > Date: 2012-04-11 13:26 +0300 > http://bitbucket.org/pypy/pypy/changeset/2a8e4f56269f/ > > Log: fix close() and tests for file closing > > diff --git a/lib-python/2.7/mailbox.py b/lib-python/2.7/mailbox.py > deleted file mode 100644 the idea for 2.7 vs modified-2.7 is to keep 2.7 intact, so you should not remove the file from there. What we usually do is: $ hg cp 2.7/foo.py modified-2.7/ $ hg ci -m "make a copy" $ emacs modified-2.7/foo.py ... ciao, Anto

1 0

changing the stdlib tracking
by Ronny Pfannschmidt 11 Apr '12

11 Apr '12

hi, since its kind of troublesome to deal with stdlib vs modified-stdlib, i'd like to propose an new model for tracking our changes to the stdlib the unmodified stdlib would be in a branch, say vendor/stdlib, while our modifications will get inlined in the default branch that way we get easier merging and easier diffing + we might be able to supply patches to cpython more easily the same approach can also be applied to pylib/pytest so i propose 2 vendor branches for tracking unmodified versions of libs we use * vendor/stdlib * vendor/pytest and of course the inlining of modified-stdlib unless there are complaints i would implement the changes in a few days in a branch -- Ronny

2 2

multiple versions of Python.h
by Matti Picus 10 Apr '12

10 Apr '12

Two copies of Python.h exist: pypy\include\Python.h pypy\module\cpyext\include\Python.h When translating targetpypystandalone on win32, the created Makefile prefers the cpyext\include. It appears that the CConfig class used to create the Makefile is the one in module/cpyext/api.py (since I see the flag Py_BUILD_CORE which occurs only there), and the include_dirs there does not pull in the first path. Two questions: 1. Do we really want two copies of this header in our tree 2. If so, shouldn't the first one be given precedence?

1 1

Findings in the Cython test suite
by Stefan Behnel 08 Apr '12

08 Apr '12

Hi, the new exception handling functions enabled a lot more tests in Cython's test suite, with still lots and lots of failures. I created tickets for a couple of them where it's clear that the problem is in PyPy/cpyext. The test results are collected here: https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-sco… Here's the general log (which I find easier to read in this case): https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-sco… For C compiler errors ("CompileError" exceptions) you'll have to look in the log at the place where the test was executed, but those are getting few. The test results are also recorded in the log, at the end (which, given the amount of failures, starts at about one third through the log). Some failures are due to wrong expectations in test code, e.g. we sometimes use the fact that small integers are cached in CPython. Others fail due to different exception messages in the doctests. Both can be handled on Cython side, given that CPython's own error messages aren't exactly carved in stone either. There is one major problem that accounts for the bulk of the test failures, somehow related to frame handling. You can tell by the huge amount of long traceback sequences that run into StackOverflowErrors and equivalent RuntimeErrors. When you look closer (e.g. repeatedly searching the log for the "classkwonlyargs" test), you will notice that the traceback refers to more than one test, i.e. the next doctest execution somehow picks up the frame of a previous test and continues from it. Funny enough, the frame leaking tests that have run (and failed) before the current one appear *below* the current test in the stack trace. This makes it likely that this is due to the way exception stack frames are constructed in Cython. They are only instantiated when an exception is being propagated, and then registered using PyTraceBack_Here(). It looks similarly likely that this is a bug in cpyext as it being due to problematic code in Cython. In case it's PyPy's fault, maybe there's something like an off-by-one when cleaning up the traceback frames somewhere? Another minor thing I noticed, PyPy seems to check the object type of keyword argument names at the caller side, whereas CPython does it as part of the function argument unpacking. Since Cython also does it on the function side and therefore expects its own error messages in the tests (which mimic CPython's, including the name of the function), these tests fail now (look for "keywords must be strings" in the test log). Not sure if it's worth doing anything about this - checking the type at call time isn't really wrong, just different. However, this is one of the cases where it would be nice if PyPy simply included the function name in the error message. The same applies to the "got multiple values for keyword argument" error. Apart from the frame handling bug, the remaining problems look minor enough to say that we are really close to a point where Cython on PyPy becomes usable. Stefan

4 10