Mailman 3 February 2013 - pypy-dev

Re: [pypy-dev] [pypy-commit] pypy jitframe-on-heap: start fixing call_assembler for ARM
by Maciej Fijalkowski Feb. 14, 2013

Feb. 14, 2013

Hi David. I started working on this, but also this is again a direct copy of x86 code. I understand where it comes from but please refrain from doing it in the future. I will work out the call_assembler and friends, but I'll backout your commit. On Thu, Feb 14, 2013 at 5:44 PM, bivab <noreply(a)buildbot.pypy.org> wrote: > Author: David Schneider <david.schneider(a)picle.org> > Branch: jitframe-on-heap > Changeset: r61238:1dd0aa6c631a > Date: 2013-02-14 16:43 +0100 > http://bitbucket.org/pypy/pypy/changeset/1dd0aa6c631a/ > > Log: start fixing call_assembler for ARM > > floats do not work correctly yet > > diff --git a/rpython/jit/backend/arm/opassembler.py b/rpython/jit/backend/arm/opassembler.py > --- a/rpython/jit/backend/arm/opassembler.py > +++ b/rpython/jit/backend/arm/opassembler.py > @@ -1104,59 +1104,52 @@ > # XXX Split into some helper methods > def emit_guard_call_assembler(self, op, guard_op, arglocs, regalloc, > fcond): > - tmploc = arglocs[1] > - resloc = arglocs[2] > - callargs = arglocs[3:] > - > self._store_force_index(guard_op) > descr = op.getdescr() > assert isinstance(descr, JitCellToken) > - # check value > - assert tmploc is r.r0 > - self._emit_call(imm(descr._ll_function_addr), > - callargs, fcond, resloc=tmploc) > + if len(arglocs) == 4: > + [frame_loc, vloc, result_loc, argloc] = arglocs > + else: > + [frame_loc, result_loc, argloc] = arglocs > + vloc = imm(0) > + > + # > + # Write a call to the target assembler > + # we need to allocate the frame, keep in sync with runner's > + # execute_token > + jd = descr.outermost_jitdriver_sd > + base_ofs = self.cpu.get_baseofs_of_frame_field() > + self._emit_call(imm(descr._ll_function_addr), [argloc], fcond) > if op.result is None: > - value = self.cpu.done_with_this_frame_void_v > + assert result_loc is None > + value = self.cpu.done_with_this_frame_descr_void > else: > kind = op.result.type > if kind == INT: > - value = self.cpu.done_with_this_frame_int_v > + assert result_loc is r.r0 > + value = self.cpu.done_with_this_frame_descr_int > elif kind == REF: > - value = self.cpu.done_with_this_frame_ref_v > + assert result_loc is r.r0 > + value = self.cpu.done_with_this_frame_descr_ref > elif kind == FLOAT: > - value = self.cpu.done_with_this_frame_float_v > + value = self.cpu.done_with_this_frame_descr_float > else: > raise AssertionError(kind) > - from rpython.jit.backend.llsupport.descr import unpack_fielddescr > - from rpython.jit.backend.llsupport.descr import unpack_interiorfielddescr > - descrs = self.cpu.gc_ll_descr.getframedescrs(self.cpu) > - _offset, _size, _ = unpack_fielddescr(descrs.jf_descr) > - fail_descr = self.cpu.get_fail_descr_from_number(value) > - value = fail_descr.hide(self.cpu) > - rgc._make_sure_does_not_move(value) > - value = rffi.cast(lltype.Signed, value) > > - if check_imm_arg(_offset): > - self.mc.LDR_ri(r.ip.value, tmploc.value, imm=_offset) > - else: > - self.mc.gen_load_int(r.ip.value, _offset) > - self.mc.LDR_rr(r.ip.value, tmploc.value, r.ip.value) > + > + gcref = cast_instance_to_gcref(value) > + rgc._make_sure_does_not_move(gcref) > + value = rffi.cast(lltype.Signed, gcref) > + ofs = self.cpu.get_ofs_of_frame_field('jf_descr') > + assert check_imm_arg(ofs) > + self.mc.LDR_ri(r.ip.value, r.r0.value, imm=ofs) > + > if check_imm_arg(value): > self.mc.CMP_ri(r.ip.value, imm=value) > else: > self.mc.gen_load_int(r.lr.value, value) > self.mc.CMP_rr(r.lr.value, r.ip.value) > - > - > - #if values are equal we take the fast path > - # Slow path, calling helper > - # jump to merge point > - > - jd = descr.outermost_jitdriver_sd > - assert jd is not None > - > - # Path A: load return value and reset token > - # Fast Path using result boxes > + # Path 1: Fast Path > > fast_path_cond = c.EQ > # Reset the vable token --- XXX really too much special logic here:-( > @@ -1164,45 +1157,40 @@ > from rpython.jit.backend.llsupport.descr import FieldDescr > fielddescr = jd.vable_token_descr > assert isinstance(fielddescr, FieldDescr) > - ofs = fielddescr.offset > - tmploc = regalloc.get_scratch_reg(INT) > - self.mov_loc_loc(arglocs[0], r.ip, cond=fast_path_cond) > - self.mc.MOV_ri(tmploc.value, 0, cond=fast_path_cond) > - self.mc.STR_ri(tmploc.value, r.ip.value, ofs, cond=fast_path_cond) > + assert isinstance(fielddescr, FieldDescr) > + vtoken_ofs = fielddescr.offset > + assert check_imm_arg(vtoken_ofs) > + self.mov_loc_loc(vloc, r.ip, cond=fast_path_cond) > + self.mc.MOV_ri(r.lr.value, 0, cond=fast_path_cond) > + self.mc.STR_ri(tmploc.value, r.ip.value, vtoken_ofs, cond=fast_path_cond) > + # in the line above, TOKEN_NONE = 0 > > if op.result is not None: > - # load the return value from fail_boxes_xxx[0] > + # load the return value from the dead frame's value index 0 > kind = op.result.type > if kind == FLOAT: > - t = unpack_interiorfielddescr(descrs.as_float)[0] > - if not check_imm_arg(t): > - self.mc.gen_load_int(r.ip.value, t, cond=fast_path_cond) > + descr = self.cpu.getarraydescr_for_frame(kind) > + ofs = self.cpu.unpack_arraydescr(descr) > + if not check_imm_arg(ofs): > + self.mc.gen_load_int(r.ip.value, ofs, cond=fast_path_cond) > self.mc.ADD_rr(r.ip.value, r.r0.value, r.ip.value, > cond=fast_path_cond) > - t = 0 > + ofs = 0 > base = r.ip > else: > base = r.r0 > - self.mc.VLDR(resloc.value, base.value, imm=t, > - cond=fast_path_cond) > + self.mc.VLDR(result_loc.value, base.value, imm=ofs, > + cond=fast_path_cond) > else: > - assert resloc is r.r0 > - if kind == INT: > - t = unpack_interiorfielddescr(descrs.as_int)[0] > - else: > - t = unpack_interiorfielddescr(descrs.as_ref)[0] > - if not check_imm_arg(t): > - self.mc.gen_load_int(r.ip.value, t, cond=fast_path_cond) > - self.mc.LDR_rr(resloc.value, resloc.value, r.ip.value, > - cond=fast_path_cond) > - else: > - self.mc.LDR_ri(resloc.value, resloc.value, imm=t, > - cond=fast_path_cond) > + assert result_loc is r.r0 > + descr = self.cpu.getarraydescr_for_frame(kind) > + ofs = self.cpu.unpack_arraydescr(descr) > + self.load_reg(self.mc, r.r0, r.r0, ofs, cond=fast_path_cond) > # jump to merge point > jmp_pos = self.mc.currpos() > self.mc.BKPT() > > - # Path B: use assembler helper > + # Path 2: use assembler helper > asm_helper_adr = self.cpu.cast_adr_to_int(jd.assembler_helper_adr) > if self.cpu.supports_floats: > floats = r.caller_vfp_resp > @@ -1213,28 +1201,24 @@ > # the result > core = r.caller_resp > if op.result: > - if resloc.is_vfp_reg(): > + if result_loc.is_vfp_reg(): > floats = r.caller_vfp_resp[1:] > else: > core = r.caller_resp[1:] + [r.ip] # keep alignment > with saved_registers(self.mc, core, floats): > # result of previous call is in r0 > - self.mov_loc_loc(arglocs[0], r.r1) > + self.mov_loc_loc(vloc, r.r1) > self.mc.BL(asm_helper_adr) > - if not self.cpu.use_hf_abi and op.result and resloc.is_vfp_reg(): > + if not self.cpu.use_hf_abi and op.result and result_loc.is_vfp_reg(): > # move result to the allocated register > - self.mov_to_vfp_loc(r.r0, r.r1, resloc) > + self.mov_to_vfp_loc(r.r0, r.r1, result_loc) > > # merge point > currpos = self.mc.currpos() > pmc = OverwritingBuilder(self.mc, jmp_pos, WORD) > pmc.B_offs(currpos, fast_path_cond) > > - self.mc.LDR_ri(r.ip.value, r.fp.value) > - self.mc.CMP_ri(r.ip.value, 0) > - > - self._emit_guard(guard_op, regalloc._prepare_guard(guard_op), > - c.GE, save_exc=True) > + self._emit_guard_may_force(guard_op, arglocs, op.numargs()) > return fcond > > # ../x86/assembler.py:668 > diff --git a/rpython/jit/backend/arm/regalloc.py b/rpython/jit/backend/arm/regalloc.py > --- a/rpython/jit/backend/arm/regalloc.py > +++ b/rpython/jit/backend/arm/regalloc.py > @@ -1101,20 +1101,18 @@ > prepare_guard_call_release_gil = prepare_guard_call_may_force > > def prepare_guard_call_assembler(self, op, guard_op, fcond): > + > descr = op.getdescr() > assert isinstance(descr, JitCellToken) > - jd = descr.outermost_jitdriver_sd > - assert jd is not None > - vable_index = jd.index_of_virtualizable > - if vable_index >= 0: > - self._sync_var(op.getarg(vable_index)) > - vable = self.frame_manager.loc(op.getarg(vable_index)) > + arglist = op.getarglist() > + self.rm._sync_var(arglist[0]) > + frame_loc = self.fm.loc(op.getarg(0)) > + if len(arglist) == 2: > + self.rm._sync_var(arglist[1]) > + locs = [frame_loc, self.fm.loc(arglist[1])] > else: > - vable = imm(0) > - # make sure the call result location is free > - tmploc = self.get_scratch_reg(INT, selected_reg=r.r0) > - self.possibly_free_vars(guard_op.getfailargs()) > - return [vable, tmploc] + self._prepare_call(op, save_all_regs=True) > + locs = [frame_loc] > + return locs + self._prepare_call(op, save_all_regs=True) > > def _prepare_args_for_new_op(self, new_args): > gc_ll_descr = self.cpu.gc_ll_descr > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit

1 0

efficient string concatenation (yep, from 2004)
by Christian Tismer Feb. 14, 2013

Feb. 14, 2013

Hi friends, _efficient string concatenation_ has been a topic in 2004. Armin Rigo proposed a patch with the name of the subject, more precisely: /[Patches] [ python-Patches-980695 ] efficient string concatenation// //on sourceforge.net, on 2004-06-28.// / This patch was finally added to Python 2.4 on 2004-11-30. Some people might remember the larger discussion if such a patch should be accepted at all, because it changes the programming style for many of us from "don't do that, stupid" to "well, you may do it in CPython", which has quite some impact on other implementations (is it fast on Jython, now?). It changed for instance my programming and teaching style a lot, of course! But I think nobody but people heavily involved in PyPy expected this: Now, more than eight years after that patch appeared and made it into 2.4, PyPy (!) still does _not_ have it! Obviously I was mislead by other optimizations, and the fact that this patch was from a/the major author of PyPy who invented the initial patch for CPython. That this would be in PyPy as well sooner or later was without question for me. Wrong... ;-) Yes, I agree that for PyPy it is much harder to implement without the refcounting trick, and probably even more difficult in case of the JIT. But nevertheless, I tried to find any reference to this missing crucial optimization, with no success after an hour (*). And I guess many other people are stepping in the same trap. So I can imagine that PyPy looses some of its speed in many programs, because Armin's great hack did not make it into PyPy, and this is not loudly declared somewhere. I believe the efficiency of string concatenation is something that people assume by default and add it to the vague CPython compatibility claim, if not explicitly told otherwise. ---- Some silly proof, using python 2.7.3 vs PyPy 1.9: > $ cat strconc.py > #!env python > > from timeit import default_timer as timer > > tim = timer() > > s = '' > for i in xrange(100000): > s += 'X' > > tim = timer() - tim > > print 'time for {} concats = {:0.3f}'.format(len(s), tim) > $ python strconc.py > time for 100000 concats = 0.028 > $ pypy strconc.py > time for 100000 concats = 0.804 Something is needed - a patch for PyPy or for the documentation I guess. This is not just some unoptimized function in some module, but it is used all over the place and became a very common pattern since introduced. /How ironic that a foreseen problem occurs _now_, and _there_ :-)// / cheers -- chris (*) http://pypy.readthedocs.org/en/latest/cpython_differences.html http://pypy.org/compat.html http://pypy.org/performance.html -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

7 10

Great experience with PyPy
by Marko Tasic Feb. 13, 2013

Feb. 13, 2013

Hi, I would like to share short story with you and share what we have accomplished with PyPy and its friends so far. Company that I have worked for last 7 months (intentionally unnamed) gave me absolute permission to pick up technologies on which we based our solution. What we do is: crawl for PDFs and newspapers articles, download, translate them if needed, OCR if needed, do extensive analysis of downloaded PDFs and articles, store them in more organized structures for faster querying, search for them and generate bunch of complex reports. >From very beginning I decided to go with PyPy no matter what. What we picked is following: * Flask for web framework, and few of its extensions such as Flask-Login, Flask-Principal, Flask-WTF, Flask-Mail, etc. * Cassandra as database because of its features and great experience with it. PyCassa is used as client to talk to Cassandra server. * ElasticSearch as distributed search engine, and its client library pyes. * Whoosh as search engine, but with some modifications to support Cassandra as storage and distributed locking. * Redis, and its client library redis-py, for caching and to speed up common auto-completion patterns. * ZooKeeper, and its client library Kazoo, for distributed locking which plays essential role in system for transaction-like behavior over many services at once. * Celery in conjunction with RabbitMQ for task distribution. * Sentry for error logging. What we have developed on our own are wrappers and clients for: * Moses which is language translator * Tesseract which is OCR engine * Cassandra store for Whoosh * wkhtmltopdf and wkhtmltoimage which are used for conversion of HTML to PDF/Image * etc Now when product is finished and in final testing phase, I can say that we did not regret because we used PyPy and stack around it. Typical speed improvement is 2x-3x over CPython in our case, but anyway we are mostly IO and memory bound, expect for Celery workers where we do analysis which are again many small CPU intensive tasks that are exchanged via RabbitMQ. Another reason why we don't see speedup us is that we are dependent on external software (servers) written in Erlang and Java. I'm already planing to do Cassandra (distributed key/value only database without index features), ZooKeeper, Redis and ElasticSearch ports in Python for next projects, and hopefully opensource them. Regards, Marko Tasic

8 9

gcc warnings / errors in translation
by Matti Picus Feb. 13, 2013

Feb. 13, 2013

warning: the stdio output of a translate is a very large webpage. I wondered why the jit-benchmark-linux-x86-64 tests were failing to translate, where the pypy-c-jit-linux-x86-64 passed for instance the failed build on tannit64 http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-64/builds/581/ste… versus the successful build on allegro64 http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/1257/steps… Two things struck me from the failure of the make command: - Could the difference be environmental gcc flags making warnings into errors on tannit64? - There are alot of warnings, some of them seem important. In looking back into history, we seem to have gotten worse with warnings. I went back a bit, say to one of the release-2.0-beta1 builds, there are still tons of gcc warnings, but fewer http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/1115/steps… Anyone feel like taking a look? Matti

5 7

Generator leaks
by Maciej Fijalkowski Feb. 12, 2013

Feb. 12, 2013

Hi pypy-dev (and hi armin :) Quick question - do we make https://bugs.pypy.org/issue1282 a release blocker? As far as I understand this is a chain-of-destructors scenario. Can we do better than wait N gc.collects? If not, can we fix generators? Cheers, fijal

1 0

Pypy's parser
by Yellow Sq Feb. 8, 2013

Feb. 8, 2013

Hi. Short question: It says at http://doc.pypy.org/en/latest/parser.html that Pypy's parser is a recursive descent one. But following the content on that page it actually seems that the parser is a table-based LL. Is this perhaps out-dated and did Pypy had a different parser at some point? If so, what were the reasons that triggered the change? Is there a list of projects (either from the industry or the academy) using Pypy? Thx.

2 1

Parallella open hardware platform
by John Camara Feb. 7, 2013

Feb. 7, 2013

A couple of days ago I heard about the Parallella [1] project which is an open hardware platform similar to the Raspberry Pi but with much higher capabilities. It has a Zynq Z-7010 which has both a dual core ARM A9 (800 MHz) processor and a Artix-7 FPGA, a 16 core Epiphany multicore accelerator, 1GB ram (see [2] for more info) and currently boots up in Ubuntu. The goal of the Parallella project is to develop an open parallel hardware platform and development tools. Recently they announced support for Python with Mark Dewing [3] leading the effort. I had asked Mark if he considered PyPy but at this time he doesn't have time for this investigation and he reposted my comment on the forum [4] with a couple of question. Maybe one of you could answer them. Working with the Parallella project maybe a good opportunity for the PyPy project from both a PR perspective and as well as the technical challenges it would present. On the technical side it would give the opportunity to test STM on a reasonable number of cores while also dealing with cores from different architectures (ARM and Epiphany). I could see all the JITting occurring on the ARM cores with it producing output for both architectures based on which type of core STM decides to use for a chunk of work to execute on. Of course there is also the challenge of bridging between the 2 architectures. Maybe even some of the more expensive STM operations could be offloaded to the FPGA or even a limited amount of very hot sections of code could be JITted to the FPGA (although this might be more work than its worth). >From a PR perspective PyPy needs to excel at some niche market so that the PyPy platform can take off. When PyPy started concentrating on the scientific market with increasing support for Numpy I thought this would be the niche market that would get PyPy to take off. But there have been a couple of issue with this approach. There is a tremendous amount of work that needs to be done so that PyPy can look attractive to this niche market. It requires supporting both NumPy and SciPy and their was an expectation that if PyPy supports NumPy others would come to help out with the SciPy support. The problem is that there doesn't seam to be many who are eager to pitch in for the SciPy effort and there also has not been a whole lot willing to help will the ongoing NumPy work. I think in general the ratio of people who use NumPy and SciPy to those willing to contribute is quite small. So the idea of going after this market was a good idea and can definitely have the opportunity to showing the strength of PyPy project it hasn't done much to improve the image of the PyPy project. It also doesn't help that there is some commercial interests that have popped up recently that have decided to play hard ball against PyPy by spreading FUD. Unlike the Raspberry Pi hardware which can only support hobbyist the Parallella hardware can support both hobbyists and commercial interests. They cost $100 which is more than the $35 for Raspberry Pi but still within reach of most hobbyists and they didn't cut out the many features that are needed for commercial interests. The Parallella project raised nearly $0.9 million on kickstarter [5] for the project with nearly 5000 backers. Since many who will use the Parallella hardware also have experience on embedded systems they and are more likely used to writing low level code in assembly, FPGAs, and even lots of C code and I'm sure have hit many issues with programming in parallel/multithreaded and would welcome a better developer experience. I bet many of them would be willing to contribute both financially and time to supporting such an effort. I believe the Architecture of PyPy could lend it self to becoming the core of such a development system and would allow Python to be used in this space. This could provide a lot of good PR for the PyPy project. Now I'm not saying PyPy shouldn't devote any more time to supporting NumPy as I'm sure when PyPy has very good support for both NumPy and SciPy it's going to be a very good day for all Python supporters. I just think that the PyPy team needs to think about a strategy that in the end will help its PR and gain support from a much larger community. This project is doing a lot of good things technically and now it just needs to get the attention of the development community at large. Now I can't predict if working with the Parallella project would be the break though in PR that PyPy needs but it's at least an option that's out there. BTW I don't have any commercial interests in the Parallella project. If some time in the future I use their hardware it would likely be as a hobbyist and it would be nice to program it in Python. My real objective of this post to see the PyPy project gain wider interest as it would be a good thing for Python. [1] - http://www.parallella.org/ [2] - http://www.parallella.org/board/ [3] - http://forums.parallella.org/memberlist.php?mode=viewprofile&u=3344 [4] - http://forums.parallella.org/viewtopic.php?f=26&t=139 [5] - http://www.kickstarter.com/projects/adapteva/parallella-a-supercomputer-for… John

4 9

Cheetah on PyPy?
by Skip Montanaro Feb. 7, 2013

Feb. 7, 2013

I'm slowly working through some little tests which don't require and of our libraries at work. My current test is a script which uses the Cheetah template engine. I see that it is buried somewhere in PyPy's tests (inside genshi?), but my straightforward install of Cheetah 2.4.4 doesn't work because the str() of a compiled template doesn't do any of the required template expansion. It just returns what looks suspiciously like the repr() of the object. Is there a modified version of Cheetah somewhere (or patch) which coaxes it to work with PyPy? Failing that, where is the test suite code? I see no references to it as a standalone download, and am currently working with a binary build of 1.9. Thx, Skip Montanaro

3 4

NumPyPy effort
by Esteban Gínez Feb. 7, 2013

Feb. 7, 2013

Hi there! I am currently looking to help out with PyPy and it seems like a good place to put some effort is in the NumPy. If someone can give pointers and resource on where/how to get started I would appreciated a ton. Thanks a bunch E.

2 1

Should jitviewer come with a warning?
by John Camara Feb. 4, 2013

Feb. 4, 2013

I have been noticing a pattern where many who are writing Python code to run on PyPy are relying more and more on using the jitviewer to help them write faster code. Unfortunately, many of them who do so don't look at improving the design of their code as a way to improve the speed at which it will run under PyPy but instead start writing obscure Python code that happens to run faster under PyPy. I know that at least with the PyPy core developers they would like to see every one just create good clean Python code and that often code that has been made into obscure Python was don so to try to optimize it for CPython which in many cases causes it to run slower on PyPy than it would run it the code just followed typical Python idioms. I feel that a normal developer should be using tools like cProfiler and runsnakerun and cleaning up design issues way before they should even consider using jitviewer. In a recent case where I saw someone using the jitviewer who likely doesn't need to use it. At least they don't need to use it considering the current design of the code I said the following "The jitviewer should be mainly used by PyPy core developers and those building PyPy VMs. A normal developer writing Python code to run on PyPy shouldn’t have a need to use it. They can use it to point out an inefficiency that PyPy has to the core developers but it should not be used as a way to get you to write Python code in a way that has a better chance of being optimized under PyPy except for very rare occasions and even then it should only be made by those who follow closely and understand PyPy’s development." Do others here share this same opinion and should some warning be added to the jitviewer? John

4 14