Mailman 3 March 2011 - pypy-dev

Thinking about the GIL
by Laura Creighton March 14, 2011

March 14, 2011

Robert Hancock hosted a bof at pycon about concurrency and multiprocessing. I went there looking to find out how other people were doing things, especially looking for information about how other languages handled things. It would be nice to kill the GIL, if only we knew of a brilliant way to do this. Unfortunately, I was one year to late for this discussion. This is what Robert Hancock David Beazley, Peter Portante and others discussed at Pycon _last year_. So I asked Robert Hancock for … [View More]the notes he took then. (I continue after this forwarded message) ------- Forwarded Message Return-Path: hancock.robert(a)gmail.com Delivery-Date: Mon Mar 14 17:09:51 2011 Return-Path: <hancock.robert(a)gmail.com> Subject: Re: please send me the notes you took last year To: Laura Creighton <lac(a)openend.se> These are the books that I mentioned: Machine Learning: An Algorithmic Perspective http://www.amazon.com/gp/product/1420067184 <http://www.amazon.com/gp/product/1420067184> I found this more approachable than the Bishop and a number of examples are in Python. Introduction to Data Mining http://www.amazon.com/gp/product/0321321367 <http://www.amazon.com/gp/product/0321321367> I've only started this, but it is nice with David Mease's Google Tech Talk series. http://www.youtube.com/watch?v=zRsMEl6PHhM 1. Make all IO non-blocking and mediate the processes like greenlets. This does not allow you take advantage of the OS level thread scheduler which is far more sophisticated than greenlets. See the Linux kernel specifications for the details of the multi-level feedback queue. 2. Construct a multi-level feedback queue within Python. This is extraordinarily complicated and complex to implement. Why duplicate what already exists? 3. Do we need to maintain compatibility with being able to call out to C functions? The primary complaint about the GIL is that it does not efficiently handle CPU bound processes and multi-cores. Running sequential processes in threads on multi-cores can actually slow down the processes. 4. Who has already solved this problem as part of the language? - Erlang (No one knew the nitty gritty details.) - Go - based on Tony Hoare's CSP and the work done on Plan 9 at Bell Labs. Uses the system scheduler and creates its own mini-threads (4k). Need to investiagate the source code on line. Goroutines do not have OS thread affinity; they can multiplex over multiple threads. - Java - Early on Java used several versions of Greenlets, but now uses system threads. The JVM punts to the OS. Conclusions -------------- 1. Do not reinvent the wheel! Many people have worked decades on this problem. Leverage thier expertise. 2. Coroutines are frequently better than threads, but do scale and each coroutine must me restarted in the thread where it was spawned. See greenlet.c. Greenlets are also chained and have mutual dependencies. The order of execution is arbitrary with not method for priorities. 3. Investigate if there is an alternative to the current method of calling external C objects. 4. Dave did a POC on priorities: http://dabeaz.blogspot.com/2010/02/revisiting-thread-priorities-and-new.html 5. Everyone agreed that some type of priority mechanism is a good idea, but wanted to see what Unladen Swallow does. (As of March 2011 Google is no longer actively developing this project.) References - ----------- Dave Beazley - GIL Wars Dave Beazley - Yieldable Threads http://www.dabeaz.com/blog.html Linux Kernel http://goo.gl/RkxVs Erlang Go - golang.org CSP - Tony Hoare http://www.usingcsp.com/cspbook.pdf I spoke with Peter Portante yesterday, and he would be very interested in participating even though he has very little free time. Peter works at HP and worked on their OS threading model. Also, see his Pycon 2010 talk on non-blocking IO and the 2011 talk on co-routines. Let me know if you have any questions. Bob Hancock Blog - www.bobhancock.org Twitter - bob_hancock and nycgtug ------- End of Forwarded Message And, indeed, Peter Portante is very interested in thinking about doing without the GIL. He's already sent me this: Date: Sun, 13 Mar 2011 16:42:00 -0400 To: Laura Creighton <lac(a)openend.se> From: Peter Portante <peter.a.portante(a)gmail.com> Subject: Re: [pypy-dev] possibly of use for our documentation Return-Path: peter.a.portante(a)gmail.com Delivery-Date: Sun Mar 13 21:42:21 2011 Return-Path: <peter.a.portante(a)gmail.com> Hi Laura, Just left pycon and heard about talks of pypy removing the gil. I work on tru64 unix's thread library for 8 years. If there is any thing I can do to help with this effort, please let me know. Thanks, -peter ------------------------- Note: I have never promised anybody anything. This was a 'please educate me appeal'. But Bob Hancock is coming back this afternoon to talk with us. Anybody got any questions they want to make sure I ask him? Laura [View Less]

4 3

problem after merging of jit-virtual_state
by Antonio Cuni March 14, 2011

March 14, 2011

Hi Hakan, hi all, as you might have noticed, there is test_pypy_c_new.test_f1 which is currently failing: http://buildbot.pypy.org/summary/longrepr?testname=TestPyPyCNew.%28%29.test… buildbot did not show it earlier because before yesterday it did not run the test at all. As you can see from the traceback, the problem is that running f1 makes three loops instead of one as expected. I did a bit of manual binary search and discovered that the test started failing between r42273:b8bb5d085684 … [View More]

2 8

possibly of use for our documentation
by Laura Creighton March 13, 2011

March 13, 2011

I went and saw the pybookbuilder project's poster. https://launchpad.net/pybookbuilder This is Jeff Elkner and his extremely smart high school students. They have a neat plugin for doing Latex inside sphinx, and to combine it with ReST to make, well, books of documentation. I don't want to lose this idea. Laura

1 0

Re: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
by Laura Creighton March 13, 2011

March 13, 2011

Note: I told python-dev about this behaviour http://mail.python.org/pipermail/python-dev/2011-March/109130.html thinking to get a 'NotPortableWarning'. Now python-dev is hot for making this a bug in Cpython as well. Laura

2 1

wrong precedence of __radd__ vs list __iadd__
by Greg Price March 13, 2011

March 13, 2011

The following program works in CPython, but fails in PyPy: class C(object): def __radd__(self, other): other.append(1) return other z = [] z += C() print z # should be [1] In PyPy, this fails with "TypeError: 'C' object is not iterable". The issue is that PyPy is allowing the list's inplace_add behavior to take precedence over the C object's __radd__ method, while CPython does the reverse. A similar issue occurs in the following program: class C(object): … [View More]def __rmul__(self, other): other *= 2 return other def __index__(self): return 3 print [1] * C() # should be [1,1] where PyPy instead prints [1,1,1]. In Python, if the LHS of a foo-augmented assignment has an in-place foo method defined, then that method is given the first opportunity to perform the operation, followed by the foo method and the RHS's reverse-foo methods. http://docs.python.org/reference/datamodel.html#object.__iadd__ But: CPython's 'list' type does not have any numeric methods defined at the C level, in-place or otherwise. See PyList_Type in listobject.c, where tp_as_number is null. So an in-place addition falls through to the RHS's nb_add, if present, and for a class with metaclass 'type' and an __radd__() method this is slot_nb_add() from typeobject.c, which calls __radd__(). So when "z += [1,2]" runs in CPython, it works via a further wrinkle. The meat of the implementation of INPLACE_ADD is PyNumber_InPlaceAdd() in abstract.c. When all numeric methods fail to handle the addition, this function falls back to sequence methods, looking for sq_inplace_concat or sq_concat methods on the LHS. 'list' has these methods, so its sq_inplace_concat method handles the operation. Similarly, PyNumber_InPlaceMultiply() tries all numeric methods first, before falling back to sq_inplace_repeat or sq_repeat. In PyPy, by contrast, it doesn't look like there's any logic for falling back to a concatenate method if numeric methods fail. Instead, if I'm reading correctly, the inplace_add__List_ANY() method in pypy.objspace.std.listobject runs *before* any numeric methods on the RHS are tried. For the narrow case at hand, a sufficient hack for e.g. addition would be to teach inplace_add__List_ANY() to look for an __radd__() first. At a quick grep through CPython, full compatibility by that approach would require similar hacks for bytes, array.array, collections.deque, str, unicode, buffer, and bytearray; plus the users of structseq.c, including type(sys.float_info), pwd.struct_passwd, grp.struct_group, posix.stat_result, time.struct_time, and several others. Those types in CPython all have sq_concat and not nb_add or nb_inplace_add, so they will all permit an RHS's __radd__ to take precedence, but then fall back on concatenation if no such method exists. A more comprehensive approach would teach the generic dispatch code about sequence methods falling after numeric methods in the dispatch sequence. Then each sequence type would need to identify its sequence methods for that code. Perhaps a hybrid approach is best. Assume that no type with a sq_concat also has a nb_add, and no type with a sq_repeat also has a nb_multiply. (I believe this is true for all built-in, stdlib, and pure-Python types.) Then whenever we define a method {inplace_,}{add,mul}__Foo_ANY, for a sequence type Foo, it's enough for that method to check for __r{add,mul}__ on the RHS. So we can write a generic helper function and use that in each such method. Does that last approach sound reasonable? I'm happy to go and implement it, but I'm open to other suggestions. I've posted tests (which fail) at https://bitbucket.org/price/pypy-queue/changeset/9dd9c2a5116a Greg [View Less]

5 7

Re: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
by Laura Creighton March 13, 2011

March 13, 2011

In a message of Sat, 12 Mar 2011 17:21:01 EST, Philip Jenvey writes: > >On Mar 10, 2011, at 3:53 PM, Armin Rigo wrote: > >> Hi Greg, >> >> On Wed, Mar 9, 2011 at 7:13 PM, Greg Price <greg(a)quora.com> wrote: >>> The following program works in CPython, but fails in PyPy: >> >> This is (if we are positive) an internal implementation detail and (if >> we are negative) a bug in CPython. > >Jython passes the first example. Shouldn'… [View More]

1 0

translation with --stackless option
by Hervé Coatanhay March 13, 2011

March 13, 2011

After a clone of the repository, whereas i can do the following translation with just some warnings: python translate.py --opt=jit targetpypystandalone.py I got an exception with that one: python translate.py --stackless targetpypystandalone.py As the --stackless option is not detailed in documentation maybe I should had something along with it ? I tried this on both Mac OS X and a CentOS Linux box, the attach file contains the traceback of the translation. Any idea on what i am doing wrong … [View More]

2 2

[PATCH] Rework borrowed-ref bookkeeping, to fix a crash
by Greg Price March 12, 2011

March 12, 2011

(I've sent this also as a Bitbucket pull request. Sending it here too because I'm not sure whose attention that reaches, and also for the sake of developing in the open.) We were increfing an object on the first time we borrow a reference to it, then decrefing it on the first time we destroy a borrowed-from container. So if we borrow from two containers, then destroy one, we'd take the refcount back where it started -- possibly zero -- even though the C code still has a legitimate reference. … [View More]This is demonstrated by test_borrow_destroy in test_borrow.py, added last night. Change the logic to incref the object the first time we borrow it from each container, and decref when any borrowed-from container is destroyed. Also fix a NameError in some code that runs when DEBUG_REFCOUNT is set. pypy/module/cpyext/pyobject.py | 32 +++++++++----------------------- pypy/module/cpyext/test/test_borrow.py | 17 ++++++++++++++++- 2 files changed, 25 insertions(+), 24 deletions(-) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py index 99aabfb..9e42153 100644 --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -144,14 +144,11 @@ class RefcountState: # { w_container -> { w_containee -> None } } # the None entry manages references borrowed during a call to # generic_cpy_call() - self.borrowed_objects = {} - # { addr of containee -> None } # For tests self.non_heaptypes_w = [] def _freeze_(self): - assert not self.borrowed_objects assert self.borrow_mapping == {None: {}} self.py_objects_r2w.clear() # is not valid anymore after translation return False @@ -187,22 +184,19 @@ class RefcountState: """ ref = make_ref(self.space, w_borrowed) obj_ptr = rffi.cast(ADDR, ref) - if obj_ptr not in self.borrowed_objects: - # borrowed_objects owns the reference - self.borrowed_objects[obj_ptr] = None - else: - Py_DecRef(self.space, ref) # already in borrowed list borrowees = self.borrow_mapping.setdefault(w_container, {}) - borrowees[w_borrowed] = None + if w_borrowed in borrowees: + Py_DecRef(self.space, w_borrowed) # cancel incref from make_ref() + else: + borrowees[w_borrowed] = None + return ref def reset_borrowed_references(self): "Used in tests" - while self.borrowed_objects: - addr, _ = self.borrowed_objects.popitem() - w_obj = self.py_objects_r2w[addr] - Py_DecRef(self.space, w_obj) + for w_container, w_borrowed in self.borrow_mapping.items(): + Py_DecRef(self.space, w_borrowed) self.borrow_mapping = {None: {}} def delete_borrower(self, w_obj): @@ -232,17 +226,10 @@ class RefcountState: ref = self.py_objects_w2r.get(w_obj, lltype.nullptr(PyObject.TO)) if not ref: if DEBUG_REFCOUNT: - print >>sys.stderr, "Borrowed object is already gone:", \ - hex(containee) + print >>sys.stderr, "Borrowed object is already gone!" return - containee_ptr = rffi.cast(ADDR, ref) - try: - del self.borrowed_objects[containee_ptr] - except KeyError: - pass - else: - Py_DecRef(self.space, ref) + Py_DecRef(self.space, ref) class InvalidPointerException(Exception): pass @@ -290,7 +277,6 @@ def track_reference(space, py_obj, w_obj, replace=False): if not replace: assert w_obj not in state.py_objects_w2r assert ptr not in state.py_objects_r2w - assert ptr not in state.borrowed_objects state.py_objects_w2r[w_obj] = py_obj if ptr: # init_typeobject() bootstraps with NULL references state.py_objects_r2w[ptr] = w_obj diff --git a/pypy/module/cpyext/test/test_borrow.py b/pypy/module/cpyext/test/test_borrow.py index 25f5101..c903271 100644 --- a/pypy/module/cpyext/test/test_borrow.py +++ b/pypy/module/cpyext/test/test_borrow.py @@ -39,7 +39,6 @@ class AppTestBorrow(AppTestCpythonExtensionBase): assert module.test_borrowing() # the test should not leak def test_borrow_destroy(self): - skip("FIXME") module = self.import_extension('foo', [ ("test_borrow_destroy", "METH_NOARGS", """ @@ -59,3 +58,19 @@ class AppTestBorrow(AppTestCpythonExtensionBase): """), ]) assert module.test_borrow_destroy() == 42 + + def test_double_borrow(self): + module = self.import_extension('foo', [ + ("run", "METH_NOARGS", + """ + PyObject *t = PyTuple_New(1); + PyObject *i = PyInt_FromLong(42); + PyTuple_SetItem(t, 0, i); + i = PyTuple_GetItem(t, 0); + PyTuple_GetItem(t, 0); + Py_DECREF(t); + return PyInt_FromLong(PyInt_AsLong(i)); + """), + ]) + # i.e., there was an InvalidPointerException + assert module.run() == 0 [View Less]

2 2

tooling/sandboxing (remarks from Ikai Lan)
by holger krekel March 11, 2011

March 11, 2011

Hi all, especially devs at Pycon, FYI i just saw saw some tweets from Ikan Lan (http://twitter.com/#!/ikai ) who works with the Google App Engine time, oldest to newest: At the Python VM panel. I want to believe alternate VMs are about more than just benchmarks. Question do not agree with me #pycon For instance, I want to see the tooling that can be built around alternative VMs. Profilers, heap analyzers, better debuggers #pycon Python VM implementors think … [View More]

3 3

Regex for RPython
by Timothy Baldridge March 10, 2011

March 10, 2011

I know rlib has a regex module, but it seems to be limited to only recognizing characters and not returning groups. Is there a full featured regex parser somewhere that is supported by rpython? Thanks, Timothy -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth)

3 2