Mailman 3 January 2011 - pypy-dev

"OSError: [Errno 10] No child processes"
by Arnd Rechenburg Jan. 13, 2011

Jan. 13, 2011

Hi, Could someone help me to avoid the following exception? Exception "OSError: [Errno 10] No child processes" in method __del__ of <closed file '<fdopen>', mode 'r' at 0x00002b58059f05f8> ignored Thanks in advance, Arnd

5 10

Re: [pypy-dev] Continuations and sandboxing
by Laura Creighton Jan. 11, 2011

Jan. 11, 2011

When I was last in the USA, I met Alessandro Warth, who now works for the Viewpoints Research Institute and whose Phd thesis which you can get on this page: http://www.tinlizzie.org/~awarth/ is about experiments in computer languages to do the sorts of things that you are interested in. He had a neat implementation of something he called 'worlds' to show off, and gave a talk about it. It's chapter 4 of his thesis. It was neat, but I got the distinct impression that the price he was paying for having unlimited cheap undo was that it was going to be quite difficult when users in different worlds, all of whom 'sprouted' from a common, but very distant ancestor and who didn't sprout from each other wanted to share information. Many other people in the room had pressing needs for such things now, and some of them were interested in PyPy. They concluded that what they would really like is a programming language which multiple heaps -- in the sense of chunks of memory that you get when you malloc -- guaranteed to be separate from each other. I couldn't tell them how hard this would be to implement in PyPy. Aside from needing to write a custom garbage collector, what else would be needed? Laura

4 3

Continuations and sandboxing
by Nathanael D. Jones Jan. 11, 2011

Jan. 11, 2011

Hi folks, I've got a challenging set of requirements that I think PyPy may be able to meet, but I'd like to get some feedback before I jump in. I'm developing a collaboratively edited game. It's not a MUD, but similar in concept, and also text based in nature. HogwartsLive.com and lotgd.ne <http://www.lotgd.net/home.php?>t are examples of the genre I'm going for. I believe if a game of that genre makes user modification simple and rewarding enough, it could be self-sustaining and grow quickly enough to keep users interested perpetually, like Wikipedia. It's also a fascinating computer science challenge, because it requires a LOT from a computer language. 1) Sandboxing. To allow users to make changes to the game without being able to "cheat" at it or escalate privileges. 2) Serializeable continuations. With gameplay being based on good plot and story flow, continuations are critical to allow straightforward implementation of 'workflows' that depend on user choice at every turn. 3) Tail-call elimination. By nature, players will accumulate a very large call stack. While this isn't terribly bad a first glance, the following issue combines with it to cause a very big problem: When code changes underneath a continuation, we need to determine how to resume flow. One option is annotating a checkpoint method in each code 'file'. However, if a user's call stack includes every file in the system, each change will cause them to restart. Tail-call elimination would help eliminate unneeded stack frames and minimize re-spawning. 3) Dynamic, strong typing and metaprogramming are key for keeping the API simple. 4) Dynamic code loading. Users will be able to 'branch' their own version of the world and share it with others. There may be thousands of versions of a class, and they need to be able to execute in separate sandboxes at the same time. Source code will be pulled from a Git repository or some kind of versioning database. I'm interested in knowing which of these PyPy already does, and which of them I can make it do. I appreciate your help! Nathanael Jones http://nathanaeljones.com/

7 10

Calling into PyPy from multi-threaded C++
by Duncan Grisby Jan. 11, 2011

Jan. 11, 2011

Hi, I've been experimenting with using PyPy in a big application, and the initial results are very encouraging. One part of the application is a fairly complex compiler, and that goes 2.5 times faster with PyPy than with Python 2.6, which is a very positive result. The compiler is the one CPU-intensive bit of the system that I could easily isolate to test with PyPy. All the rest of it consists of a number of servers connected together using omniORB (which I maintain). omniORBpy integrates the C++ core of omniORB with Python, via the C API. omniORB is multi-threaded, and needs to call into Python from threads created in the C++ world. With CPython that's possible, although it takes some care to behave properly with thread states and the interpreter lock and so on. As far as I can see, PyPy's C API implementation doesn't support the thread state functions like PyGILState_Ensure and PyThreadState_Get, which isn't a huge surprise given how different PyPy is. Are these functions likely to be implemented at any point? Alternatively, is there any other way for C/C++ code to call into PyPy from a thread created outside PyPy? If there isn't a mechanism to do it at the moment, is it possible at all? How much effort would it be to create one? Thanks, Duncan. -- -- Duncan Grisby -- -- duncan(a)grisby.org -- -- http://www.grisby.org --

3 6

good by swap of death
by Daniel Poelzleithner Jan. 11, 2011

Jan. 11, 2011

hi, as I'm surly not the only one having sometimes memory problems when translating pypy. Slowly getting bitten by the swap of death, not registering anything until the desktop stops and rescuing is a very slow process (yes, i have a special root login with /bin/sh shell and memlockd...). I was just able to fix the problem with ulatencyd [1] and as I think translating pypy is a real usecase I would be glad having some feedback. I was quite surprised that after a simple memleak emulator got isolated successfully, pypy translation works like a charm, even when i put additional pressure on the system :-) [1] https://github.com/poelzi/ulatencyd kind regards poelzi

2 1

Re: [pypy-dev] [pypy-svn] pypy jit-longlong: Intermediate check-in, breaks everything by exposing cases of
by Maciej Fijalkowski Jan. 10, 2011

Jan. 10, 2011

For what is worth I'm happy with making "rething regalloc API" a sprint topic. So far I failed to come up with something more reasonable. Cheers, fijal On Mon, Jan 10, 2011 at 12:33 AM, arigo <commits-noreply(a)bitbucket.org> wrote: > Author: Armin Rigo <arigo(a)tunes.org> > Branch: jit-longlong > Changeset: r40545:9a472ddf9893 > Date: 2011-01-09 23:33 +0100 > http://bitbucket.org/pypy/pypy/changeset/9a472ddf9893/ > > Log: Intermediate check-in, breaks everything by exposing cases of > constant arguments that are not correctly handled so far. /me is > again annoyed by the API of regalloc.py, which seems to be very > reasonable, until we hit these cases. > > diff --git a/pypy/jit/backend/x86/assembler.py b/pypy/jit/backend/x86/assembler.py > --- a/pypy/jit/backend/x86/assembler.py > +++ b/pypy/jit/backend/x86/assembler.py > @@ -1174,6 +1174,13 @@ > self.mc.SBB_rr(resloc.value, resloc.value) > self.mc.NEG_r(resloc.value) > > + def genop_llong_lt(self, op, arglocs, resloc): > + # XXX just a special case for now: "x < 0" > + loc1, = arglocs > + self.mc.PMOVMSKB_rx(resloc.value, loc1.value) > + self.mc.SHR_ri(resloc.value, 7) > + self.mc.AND_ri(resloc.value, 1) > + > def genop_new_with_vtable(self, op, arglocs, result_loc): > assert result_loc is eax > loc_vtable = arglocs[-1] > @@ -1784,6 +1791,7 @@ > if isinstance(op.getdescr(), LongLongCallDescr): > self.mc.MOV_br(resloc.value, eax.value) # long long > self.mc.MOV_br(resloc.value + 4, edx.value) > + # XXX should ideally not move the result on the stack > else: > self.mc.FSTP_b(resloc.value) # float return > elif size == WORD: > > diff --git a/pypy/jit/codewriter/jtransform.py b/pypy/jit/codewriter/jtransform.py > --- a/pypy/jit/codewriter/jtransform.py > +++ b/pypy/jit/codewriter/jtransform.py > @@ -894,7 +894,8 @@ > args = op.args > op1 = self.prepare_builtin_call(op, "llong_%s", args) > op2 = self._handle_oopspec_call(op1, args, > - EffectInfo.OS_LLONG_%s) > + EffectInfo.OS_LLONG_%s, > + EffectInfo.EF_PURE) > return op2 > ''' % (_op, _oopspec.lower(), _oopspec)).compile() > > @@ -1273,7 +1274,7 @@ > > def _handle_oopspec_call(self, op, args, oopspecindex, extraeffect=None): > calldescr = self.callcontrol.getcalldescr(op, oopspecindex) > - if extraeffect: > + if extraeffect is not None: > calldescr.get_extra_info().extraeffect = extraeffect > if isinstance(op.args[0].value, str): > pass # for tests only > > diff --git a/pypy/jit/backend/x86/regalloc.py b/pypy/jit/backend/x86/regalloc.py > --- a/pypy/jit/backend/x86/regalloc.py > +++ b/pypy/jit/backend/x86/regalloc.py > @@ -653,6 +653,22 @@ > self.PerformLLong(op, [loc1, loc2, loc3], loc0) > self.xrm.possibly_free_vars(args) > > + def _maybe_consider_llong_lt(self, op): > + # XXX just a special case for now > + from pypy.rlib.longlong2float import longlong2float > + box = op.getarg(2) > + if not isinstance(box, ConstFloat): > + return False > + if not (box.value == longlong2float(r_longlong(0))): > + return False > + # "x < 0" > + box = op.getarg(1) > + loc1 = self.xrm.make_sure_var_in_reg(box, imm_fine=False) > + loc0 = self.rm.force_allocate_reg(op.result) > + self.PerformLLong(op, [loc1], loc0) > + self.xrm.possibly_free_var(box) > + return True > + > def _consider_llong_to_int(self, op): > # accept an argument in a xmm register or in the stack > loc1 = self.xrm.loc(op.getarg(1)) > @@ -769,6 +785,9 @@ > if (oopspecindex == EffectInfo.OS_LLONG_EQ or > oopspecindex == EffectInfo.OS_LLONG_NE): > return self._consider_llong_cmp_xx(op) > + if oopspecindex == EffectInfo.OS_LLONG_LT: > + if self._maybe_consider_llong_lt(op): > + return > # > self._consider_call(op) > > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn >

1 0

psycopg2 on PyPy
by Alex Gaynor Jan. 6, 2011

Jan. 6, 2011

Hi all, For the past week and a half I've been working on an RPython implementation of psycopg2 (the most popular database adapter for PostgreSQL). I'm happy to announce that at this point it is passing all the DB-API2.0 tests, and all the psycopg2 tests I converted into AppTests (most of them I think), it also passes the entire DJango test suite (which makes extensive use of the database). The downside is that running the full Django test suite on top of pypy with it is still about 20% slower than CPython. So we have some work to do. The code is all at: http://bitbucket.org/alex_gaynor/pypy-postgresql, I don't plan on merging this back into the main dev tree, as it doesn't seem appropriate for pypy-core to have a PostgreSQL adapter, rather I'd like to spend some time looking at seperate compilation, so we can have external RPython modules. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me

3 2

pypy GC on large objects Re: funding/popularity?
by René Dudfield Jan. 4, 2011

Jan. 4, 2011

Hello, I think this is a case where the object returned by ctypes.create_string_buffer() could use a correct __sizeof__ method return value. If pypy supported that, then the GC's could support extensions, and 'opaque' data structures in C too a little more nicely. I think ctypes was started before __sizeof__ became available... so it seems many of it's methods are not updated yet. __sizeof__ is not mandatory, so many extensions have not been updated to support it yet. cheers, On Thu, Dec 23, 2010 at 11:14 AM, Armin Rigo <arigo(a)tunes.org> wrote: > Hi Dima, > > On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek <dimaqq(a)gmail.com> wrote: >> --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky(a)mit.edu> 2010-12-22 >> 13:43:23 PST --- >> So what I see this page do, in horizontal mode, is create 17 canvases each of >> which is width="816" height="3587". That means that each of them has a backing >> store of 816*3587*4 = 11,707,968 bytes. So that's about 200MB of memory usage >> right there. >> >> I have no idea why they feel a need for 17 huge canvases, but if they want >> them, that's how much memory they'll take... > > That looks very similar to an issue with PyPy's own GC, in which > ctypes.create_string_buffer() returns objects which tend to be GC'ed > late. That's because the string buffer object in ctypes appears (to > PyPy's GC) to be just a small object, even though it actually > references a potentially large piece of raw memory. Similarly, my > vague guess about the above is that the 17*11MB of memory are hold by > 17 small objects which firefox's GC think don't need to be collected > particularly aggressively. > > > A bientôt, > > Armin. > _______________________________________________ > pypy-dev(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev >

5 6

PyPy is now available in Fedora
by David Malcolm Jan. 4, 2011

Jan. 4, 2011

I've packaged pypy in RPM form for the Fedora distribution [1] - RPM packages are now built in the development branch targeting the next major release (Fedora 15). So it should now be possible for Fedora users to type # yum install pypy and obtain a precompiled /usr/bin/pypy executable via an rpm package, consisting of the JIT-enabled pypy, with all standard settings, without needing to do the full build themselves. I'll do my best to keep these "downstream" packages promptly updated as further "upstream" releases of PyPy occur. Many thanks to everyone who helped with this, especially fijal, for all his great feedback on the packaging review [2] (Caveat: actually, the build won't be available yet via "yum" until a cronjob runs tonight; for now, the build can be downloaded from: http://koji.fedoraproject.org/koji/buildinfo?buildID=212473 ) Some other links, for the curious, can be seen here: https://admin.fedoraproject.org/pkgdb/acls/name/pypy e.g. to the rpm packaging sources. Hope this is helpful Dave [1] http://fedoraproject.org/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=588941

2 1

Wing Sit invited you to Dropbox
by Dropbox Jan. 3, 2011

Jan. 3, 2011

Wing Sit wants you to use Dropbox to sync and share files online and across computers. Get started here: http://www.dropbox.com/link/20.eHuq1oJQCM/NjU2NTA1ODEwNw?src=referrals_bulk2 - The Dropbox Team ____________________________________________________ To stop receiving invites from Dropbox, please go to http://www.dropbox.com/bl/4b1a4ffd25d4/pypy-dev%40codespeak.net

1 0