Mailman 3 December 2016 - pypy-dev

Object pinning
by Kunshan Wang 23 Dec '16

23 Dec '16

Hi folks, I have a question regarding object pinning in RPython. Consider the following snippet from rpython/rtyper/lltypesystem/rstr.py @jit.oopspec('stroruni.copy_contents(src, dst, srcstart, dststart, length)') @signature(types.any(), types.any(), types.int(), types.int(), types.int(), returns=types.none()) def copy_string_contents(src, dst, srcstart, dststart, length): """Copies 'length' characters from the 'src' string to the 'dst' string, starting at position 'srcstart' and 'dststart'.""" # xxx Warning: don't try to do this at home. It relies on a lot # of details to be sure that it works correctly in all cases. # Notably: no GC operation at all from the first cast_ptr_to_adr() # because it might move the strings. The keepalive_until_here() # are obscurely essential to make sure that the strings stay alive # longer than the raw_memcopy(). assert length >= 0 ll_assert(srcstart >= 0, "copystrc: negative srcstart") ll_assert(srcstart + length <= len(src.chars), "copystrc: src ovf") ll_assert(dststart >= 0, "copystrc: negative dststart") ll_assert(dststart + length <= len(dst.chars), "copystrc: dst ovf") # from here, no GC operations can happen asrc = _get_raw_buf(SRC_TP, src, srcstart) adst = _get_raw_buf(DST_TP, dst, dststart) llmemory.raw_memcopy(asrc, adst, llmemory.sizeof(CHAR_TP) * length) # end of "no GC" section keepalive_until_here(src) keepalive_until_here(dst) copy_string_contents._always_inline_ = True copy_string_contents = func_with_new_name(copy_string_contents, 'copy_%s_contents' % name) There is a region where heap objects in the RPython heap is accessed externally by native programs. I understand that GC must neither recycle the object nor move it in the memory. But I have two questions about how object pinning is done in RPython: (1) From the perspective of the RPython user (e.g. high-level language implementer, interpreter writer, library writer, ...), what is the "protocol" to follow when interacting with native programs (such as passing a buffer to the `read` syscall)? I have seen idiomatic use of `cast_ptr_to_adr` followed by `keepalive_until_here`. But there is also `pin` and `unpin` functions in the rpython/rlib/rgc.py module. What is the expected way for *the user* to pin objects for native access? (2) From the perspective of the RPython developer (those who develop the translation from RTyped CFGs to C source code, assembly code and machine code), how does the C backend actually enforce the no-GC policy between "from here, no GC operations can happen" and "end of 'no GC' section"? As I observed, keepalive_until_here essentially generates a no-op inline assembly that "uses" the variable so that the C compiler keeps that variable alive. But what is preventing GC from happening? Some background: We are building a new backend for RPython on the Mu micro virtual machine (https://gitlab.anu.edu.au/mu/mu-client-pypy). This VM has built-in GC and exception handling, so they don't need to be injected after RTyper. But the micro VM also keeps the representation of object references opaque. The only way to "cast" references to addresses is using the "object pinning" operation which returns its address. The idiom in Mu when passing a buffer to native programs is that you "pin" the object, give the pointer to the native functions (such as `memcpy`, `read` and `write`), and then "unpin" it (nested pinning is allowed, and it needs to be unpinned as many times as it was pinned). The GC will neither reclaim the object nor move it when it is pinned, but object pinning does not prevent GC from happening: GC can still move other objects, but not the pinned ones. So the crux is how to translate the RPython primitives into the Mu counterparts. If `cast_ptr_to_adr` and `keep_alive_until_here` is a well-obeyed idiom, we can simply translate them to `pin` and `unpin`, respectively. The problem is, it only works for non-GC types. Mu cannot copy references in this way. There are several problems: (1) Mu does not want to expose the byte-by-byte representation of references. So references may not really be addresses, and may not be copied naively as a word (it could contain bit flags, too). (2) Mu does not prevent the movement of other object. So if an array of references is pinned, the objects its elements point *to* may still be moved. I guess the reason why the snippet above prevents *all* kinds of GC activity is because it may be used to copy GC-managed object references, too. If this is the case, we have to find alternative solutions. Regards, Kunshan Wang School of Computer Science Australian National University

3 2

access to PyFrame within x86 backend
by Shubha Ramani 21 Dec '16

21 Dec '16

First of all let me say that all I've done for the past several days is read, read read.I have read the rpython and pypy documentation. I've read bolz-tracing-jit-final.pdf as well as antocuni-phd-thesis.pdf. I also feel that I comprehended most of what I read. In studying the vmprof implementation there is still a gap in my comprehension. How do I get access to PyFrame within assembly.py ? VmProf dumps everything into a specially formatted binary file, embedded with variousmarkers and the function code objects among many other stuff, per PyFrame which gets executed. This file later gets post-processed by the vmprof tool. Suppose I do something similar to vmprof in order to obtain access to function code objects, i.e. see below. But how do I get access to PyFrame in the x86 backend code at run time ? _get_code = lambda frame, w_inputvalue, operr: frame.pycode _decorator = rvmprof.vmprof_execute_code("pypy", _get_code, W_Root) my_execute_frame = _decorator(PyFrame.execute_frame) class __extend__(PyFrame): def execute_frame(self, w_inputvalue=None, operr=None): # indirection for the optional arguments return my_execute_frame(self, w_inputvalue, operr)

1 2

which API returns Python stack in codemap.c given pc ?
by Shubha Ramani 20 Dec '16

20 Dec '16

I have studied rpython/jit/backend/llsupport/src/codemap.c.I am nt finding the API which returns the Python stack in codemap.c.

2 3

Pypy Contribution
by George Joseph 20 Dec '16

20 Dec '16

Hi, I'm an undergrad Computer Science student from India in my final year. I'm interested in participating in GSoC 2017 for python, but I'm new to open source. Would You kindly tell he how I can start contributing? I'm familiar with C, C++, Java, Python, MATLAB, Verilog MySQL. Thanks ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ *George Joseph* *M.Sc (Hons.) Physics + B.E. (Hons.) Computer Science* *Birla Institute of Technology and Science, Pilani* *K. K. Birla Goa Campus * *+91 9503519766 | georgejoseph1(a)protonmail.com <georgejoseph1(a)protonmail.com>* ᐧ

2 1

Adding init/variables to W_Root
by Frank Wang 19 Dec '16

19 Dec '16

Hi, I don't know if this is the proper list to ask this question. I am trying to add a dictionary to W_Root. The reason for this is that I want all interpreter objects to contain this dictionary. This is for a research project to implement some form of information flow control in PyPy. Right now, I know W_Root has no __init__ function, and when I try to add one. I run into all sorts of problems because TypeDef calls W_Root.__new__ with some parameters, but "new" function seems to be called nowhere else. Any thoughts on this would be helpful. Thanks, Frank

3 6

Why does pypy include my code three times in the JIT-capable executable?
by hgualandi＠inf.puc-rio.br 18 Dec '16

18 Dec '16

Hi there, I have a question about pypy's compilation process when producing an executable from an RPython source program. Apparently when I compile my program with --opt=JIT, my code gets included 3 times in the resulting executable. I made a little experiment where I included a magic number in my RPython code and counted how often it showed up in the generated executable. With ahead of time compilation (--opt=3) my magic number appeared once: $ PERLIO=:raw perl -nE '$c++ while m/\xef\xbe\xad\xde/g; END{say $c}' ./rpython-deadbeef.aot 1 And with --opt=JIT it appeared 3 times: $ PERLIO=:raw perl -nE '$c++ while m/\xef\xbe\xad\xde/g; END{say $c}' ./rpython-deadbeef.jit 3 When I investigated, it turns out that the JIT-capable executable has one copy of my program in bytecode form and two copies in machine code form. To understand the machine-code copies I inspected the intermediate "implement.c" file that pypy generated and found out that my interpreter code appears in two functions, one named pypy_g_mainloop_1 and one named pypy_g_portal. These two functions were completely identical, other than the function names that show up in the debugging messages. I understand why pypy would create a separate bytecode version of my program but why is it generating two seemingly-identical machine code versions of it? Thanks in advance, Hugo

2 1

no module named 'backports' error when vmprof executed with pypy
by Shubha Ramani 17 Dec '16

17 Dec '16

shubha@shubha-Z170X-UD5:/opt/vtune_pypy$ pypy-v -m vmprof fib.py/usr/bin/pypy-v: No module named backports; 'vmprof' is a package and cannot be directly executed However when I do python -m vmprof fib.py it works. Based on a quick Google search there doesn't seem to be a package called 'backports' which caneasily be installed via pip or easy_install. I'm on Ubuntu 16.04. Any clues ? Thanks, Shubha

1 1

Dontbug: A reversible debugger for PHP (similar in concept to RevDB for Python/PyPy)
by Sidharth Kshatriya 17 Dec '16

17 Dec '16

Dear All, There have been some interesting blogs about RevDB a reversible debugger for Python on the PyPy blog. I'd like to tell you about Dontbug, a reversible debugger for PHP that I recently released. Like RevDB, it allows you to debug forwards and backwards -- but in PHP. See: https://github.com/sidkshatriya/dontbug For a short (1m35s) demo video: https://www.youtube.com/watch?v=DA76z77KtY0 Why am I talking about this in a PyPy mailing list :-) ? Firstly, because I think reverse debuggers for dynamic languages are relatively rare -- so its a good idea that we know about each other! Secondly, the fact that there are more and more reversible debuggers for various languages every year means that reverse debugging is definitely entering the mainstream. We could be at an inflexion point here! Hope you guys find Dontbug interesting! Thanks, Sidharth

3 4

cling-support
by Armin Rigo 15 Dec '16

15 Dec '16

Hi Wim, Great that cling-support is merged! Could you (1) document the branch inside pypy/doc/whatsnew-head.rst, and (2) look at the failure on 32-bit? It seems that running py.test module/cppyy crashes with exit code 3, according to http://buildbot.pypy.org/summary/longrepr?testname=&builder=own-linux-x86-3… . Thanks! A bientôt, Armin.

2 3

PYPY:why withprebuiltint option is off default? The speed is slower after I open it.
by 张耀 14 Dec '16

14 Dec '16

Hi, every experts: Recently I'm trying to move my old project from python to pypy. In the past, I have do some optimizations for python interpreter. One Important thing is modify python default integer object cache range(default is from -5 to 257), my config is -100 to 1024 * 1024, this range covers most of the my app logics. The benchmark shows that it saves a lot of memory , and speedup the program(mainly benefits from avoiding large amount of creating intobject and releasing intobject) . BUT, in pypy, I found the withprebuiltint option is set off default. While I tried open this option and add translateoption : --prebuiltintfrom -5 --prebuiltintfrom 1048576, the benchmark shows that it saves memory(OK, it's expected), BUT the speed is SLOWER. I check the pypy sourcecode, but I have not find out the problem so far. So, my doubt is: why withprebuiltint option is off default? Maybe the performance is a problem as known? ############################################################################################# #test_speed4_prebuit_int.py def foo(): j = 0 for i in xrange(1024 * 1024): j = i -1 j = i + 1 import timeit print timeit.repeat(foo, repeat = 3, number= 100) user:~/pypy_run$ official-pypy/bin/pypy-c -B test_speed4_prebuit_int.py #default pypy [0.21457695960998535, 0.21348810195922852, 0.21274185180664062] user:~/pypy_run$ pypy-build-by-pypy-test/bin/pypy-c -B test_speed4_prebuit_int.py #--prebuiltintfrom -5 --prebuiltintfrom 1048576 [0.7308619022369385, 0.7308559417724609, 0.7256178855895996] #almost 3 times slower ############################################################################################# #test_mem4_prebuilt_int.py listlen = 1024 itemlen = 1024 * 100 x = 1 a = [None] * listlen def init_data(): for i in xrange(listlen): a[i] = [None] * itemlen def create_data(): for i in xrange(listlen): for j in xrange(itemlen - 1): a[i][j] = x + 0 init_data() create_data() ###mem test is expected, no problem Thanks regards! shadow 2016-12-14

3 2