Mailman 3 March 2015 - pypy-dev

How to help PyPy 3 ?
by Ludovic Gasc March 23, 2015

March 23, 2015

Hi, I want to try to help PyPy 3, especially to run AsyncIO on PyPy 3. For now, I've tried to compile PyPy from 3.3 branch, and install AsyncIO. I'd an issue with time.monotonic() and time.get_clock_info() in AsyncIO, because it isn't implemented in PyPy 3. I've replaced that by time.time() and hardcoded value of time.get_clock_info() return, only to see until AsyncIO should work on PyPy 3. I've launched several examples from AsyncIO documentation, it works, except when I launch a TCP client: https://docs.python.org/3/library/asyncio-protocol.html#tcp-echo-client-pro… I've the same issue with aiotest: https://bitbucket.org/haypo/aiotest All tests pass, except: $ ./pypy-c -Wd ../tulip/run_aiotest.py Run tests in debug mode ...............E ====================================================================== ERROR: test_tcp_hello (aiotest.test_network.NetworkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/lg/Documents/IDEA/pypy/site-packages/aiotest/test_network.py", line 78, in test_tcp_hello self.loop.run_until_complete(coro) File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/base_events.py", line 317, in run_until_complete return future.result() File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/futures.py", line 275, in result raise self._exception File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/tasks.py", line 238, in _step result = next(coro) File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/coroutines.py", line 79, in __next__ return next(self.gen) File "/home/lg/Documents/IDEA/pypy/site-packages/asyncio/base_events.py", line 644, in create_connection sock, protocol_factory, ssl, server_hostname) TypeError: '_SelectorSocketTransport' object is not iterable ---------------------------------------------------------------------- Ran 16 tests in 0.337s FAILED (errors=1) Finally, I've tested to launch the aiohttp server example: https://github.com/KeepSafe/aiohttp/blob/master/examples/srv.py The servers starts, but when I launch a HTTP request, I've a strange stackstrace: Error handling request Traceback (most recent call last): File "/home/lg/Documents/IDEA/pypy/site-packages/aiohttp/server.py", line 240, in start yield from handler File "/home/lg/tmp/asyncio_examples/7_aiohttp_server.py", line 25, in handle_request message.method, message.path, message.version)) AttributeError: 'str' object has no attribute 'method' Traceback (most recent call last): File "/home/lg/Documents/IDEA/pypy/site-packages/aiohttp/server.py", line 240, in start yield from handler File "/home/lg/tmp/asyncio_examples/7_aiohttp_server.py", line 25, in handle_request message.method, message.path, message.version)) AttributeError: 'str' object has no attribute 'method' After debugging a little bit with pdb, I see that aiohttp.HttpRequestParser parses correctly the request, but "message = yield from httpstream.read()" in aiohttp.server at line 226 returns "GET" string instead of a RawRequestMessage object. I'm a total newbie about PyPy internals, implement a monotonic timer seems to be over-complicated to me. But, I should maybe help with tests and fix small issues like with aiohttp.server if I've a clue to fix that. Are you interested if I create issues on PyPy project for each problem, or I will add only noise in PyPy project ? BTW, to help PyPy project not only with my brain time, I've did a small recurring donation for PyPy 3. Regards. -- Ludovic Gasc (GMLudo)

2 1

GSoC 2015
by w0mTea March 23, 2015

March 23, 2015

Dear developers, I'm a student interested in the idea of copy-on-write list slicing. I noticed that on the PSF's GSoC wiki, students are suggested to fix a beginner-friendly bug, but after some searching, I eventually failed in finding some appropriate ones. May you help me about this? Another question is about building pypy. I'm novice in pypy's implementation, so I decide to modify it's source code to help me understand it clearly. But it's really slow to build pypy. According to the pypy doc, I use this command to build it: pypy rpython/bin/rpython --opt=jit pypy/goal/targetpypystandalone.py It takes about an hour to complete on my computer. Then I modify a file, adding only one line. But rebuilding through this command also takes an hour. Is there any faster way to rebuild pypy with little modification? Your answer and advice will be highly appreciated. Thanks, Yule Zhao

2 2

Specialization for app level types
by Timothy Baldridge March 22, 2015

March 22, 2015

I'd like to add some optimization to app level types in Pixie. What I'm thinking of is something like this (in app level PyPy code): class Foo(object): def __init__(self, some_val): self._some_val = some_val def set_value(self, val): self._some_val = val In a perfect world the JIT should be able to recognize that ._some_val is only ever an int, and therefore store it unboxed in the instance of the type, hopefully this would decrease pressure on the GC if ._some_val is modified often. Also in a perfect world, the value of _some_val should be auto promoted to an object if someone ever decides to set it to something besides an int. How would I go about coding this up in RPython? I can't seem to figure out a way to do this without either bloating each instance of the type with an array of object, an array of ints and an array of floats. Currently app level objects in Pixie are just a wrapper around a object array. The type then holds the lookups for name->slot_idx. Thanks in advance. Timothy Baldridge

4 4

GSoC ideas -- AArch64 JIT backend?
by Manuel Jacob March 21, 2015

March 21, 2015

Hi, I decided that this year is a good time to apply for GSoC. Because I never worked on anything JIT-related, this could be a chance to get started with it. There was some discussion about possible improvements for the ARM backend. Also, some newer ARM processors support a 64-bit execution mode called AArch64. I think it makes sense to implement a JIT backend for AArch64 sooner or later. There are just a few affordable AArch64 development boards available at the moment, but I was able to cross-compile a non-jitted version of PyPy and run it in QEMU. Do you think that implementing an AArch64 JIT backend is a good GSoC project? If not, can you think of other JIT-related projects that fit better in the scope of GSoC? -Manuel

5 4

Compiling PyPy interpreter without GC
by John Zhang March 21, 2015

March 21, 2015

Hi all, I'm working on developing a MicroVM backend for PyPy. It's a virtual machine under active research and development by my colleagues in ANU. It aims to capture GC, threading and JIT in the virtual machine, and frees up the burden of the language implementers. Since MicroVM provides GC, I need to remove GC from the PyPy interpreter. As I was trying to compile it with the following command: pypy $PYPY/rpython/bin/rpython \ -O0 \ --gc=none \ --no-translation-rweakref \ --annotate \ --rtype \ --translation-backendopt-none \ $PYPY/pypy/goal/targetpypystandalone.py It gives off an error during annotation stage, saying that it's not able to find a module called '_rweakref'. Does anyone know what the problem might be, and how one might go and solve it? Appreciate greatly, John Zhang

4 4

Re: [pypy-dev] Compiling PyPy interpreter without GC
by John Zhang March 19, 2015

March 19, 2015

Hi Carl, Great! It worked! So the option disables all modules, and IO as well? Cheers, John Zhang > On 19 Mar 2015, at 4:18 am, Carl Friedrich Bolz <cfbolz(a)gmx.de> wrote: > > On 18/03/15 01:01, John Zhang wrote: >> Hi all, >> I'm working on developing a MicroVM backend for PyPy. It's a >> virtual machine under active research and development by my colleagues >> in ANU. It aims to capture GC, threading and JIT in the virtual machine, >> and frees up the burden of the language implementers. >> >> Since MicroVM provides GC, I need to remove GC from the PyPy >> interpreter. As I was trying to compile it with the following command: >> pypy $PYPY/rpython/bin/rpython \ >> -O0 \ >> --gc=none \ >> --no-translation-rweakref \ >> --annotate \ >> --rtype \ >> --translation-backendopt-none \ >> $PYPY/pypy/goal/targetpypystandalone.py > > Hey John, > > Try the following: > rpython -O0 --gc=none --no-translation-rweakref --annotate --rtype --translation-backendopt-none targetpypystandalone.py --no-allworkingmodules --withoutmod-_io > > Cheers, > > Carl

2 1

Some summary and questions about the 'small function' problem
by 黄若尘 March 18, 2015

March 18, 2015

Hi Fijal, This is Ruochen Huang, I want to begin to write my proposal and I think actually there is not so much time left. I tried to make a summary of what I have understood until now and the questions I want to know. Please feel free to point out any incorrect things in my summary, and for the questions, if you think the question is meaningless, you can just skip it, or provide some possible document link or source code path if you think it will take too much time to explain it. As far as I understood, The ‘small function’ problem occurred when one trace try to call another trace. In source code level, it should be the situation that, inside one loop, there is a function invocation to a function which has another loop. Let me take the small example we discussed before, function g() tried to call function f(a,b,c,d) in a big loop, and there is another loop inside f(a,b,c,d). So in current version of PyPy, the result is that, two traces were generated: the trace for the loop in g(), let me call it T1, actually, g() tried to inline f(a,b,c,d), but since there is a loop in f, so the result is that T1 will inline only the first iteration of the loop in f, let’s say f was taken apart into f1(for the first iteration) and f’(for the rest iterations), so what T1 exactly does is start the loop in g -> do f1 -> do some allocations of PyFrame (in order to call f’) -> call_assembler for f’. the trace for the loop in f’. let me call it T2. T2 firstly unpack the PyFrame prepared by T1, then do a preamble work, which means f’ is once again taken apart into f2 (for the 1st iteration in f’, and it actually is also the 2nd iteration in original f), and f’’(the iterations from 3rd iteration to the last iteration), for f2 and f’’, there is a label in the head of them, respectively. So finally we can say T2 consist of 3 parts: T2a (for PyFrame unpacking), T2b(with label1, do f2), T2c(with label2, do f’’). As mentioned above, we have T1 -> T2a -> T2b -> T3c, from the viewpoint of the loop in f, f is distributed into: T1(f1) -> T2a -> T2b(f2) -> T2c(f’’), which means the loop in f was peeled twice, so T2b might be not needed, further more, the work for PyFrame before call_assembler in T1, and the unpacking work in T2a is a waste. I can’t understand why it’s a waste very well, but I guess it’s because T2c(f’') actually do the similar thing as f1 in T1, (or, T2c is already *inside* the loop) Anyway, T2b is also not needed, so we want to have T1 -> T2c, and since the work in PyFrame in T2a is eliminated, the allocation for PyFrame in T1 can also be eliminated. So ideally we want to have T1’ (without PyFrame allocation) -> T2c. Some questions until now: What’s the bridge you mentioned? To be honest I have only a very slight understand of bridge, I know it is executed when some guard failed, but as far as I knew, in normal trace JIT compiler, only one path of a loop will be traced, any guard failure will make the execution escape from the native code and return to VM, but I guess the bridge is a special kind of trace (native code), is it right? Could you please explain more about why T2b is not needed? I guess the answer may be related to the “virtualizable” optimization for PyFrame, so what if PyFrame is not virtualizable? I mean, if in that situation, does the problem disappear? or become easier to solve? What’s the difficulties in solving this problem? I’m sorry I’m not so familiar with the details of RPython JIT, but in my opinion, we need just to make the JIT know that, when tries to inline a function, and encounter a loop so the inline work has to stop, it’s time to do optimization O. what O does is to delete the allocation instructions about PyFrame before call_assembler, and them tell call_assembler to jump to 2rd label of target trace. (In our example is T2c). So It may seem not so difficult to solve. Best Regards, Ruochen Huang

1 0

GSoC 2015: Implement copy-on-write list slicing
by Tushar Tiwari March 18, 2015

March 18, 2015

Hi All, I am a masters student at the University of Southern California and was hoping to participate in this year’s Google Summer of Code on Implementing copy-on-write list slicing. I would love some documentation on the issue and also, I noticed there is no mentor for the projects so that info would be nice. Thank You. — Tushar Tiwari tushart91(a)gmail.com http://www.github.com/tushart91 University of Southern California

2 1

Language Parser Ontology
by anatoly techtonik March 16, 2015

March 16, 2015

I'll start from afar, so that it will be easier to understand what I am thinking about.. CFFI uses pycparser, which parses C files, but! uses C compiler to strip comments from C files and process defines, but almost all .c files contain comments, so pycparser is basically useless as a parser, but maybe it has a good API for working with AST. Anyway, I tried to see if I can teach pycparser to strip comments itself, and in c_lexer.py I found a list of tokens, among which there were no token representing the comment start. Stripped list: ## ## All the tokens recognized by the lexer ## tokens = keywords + ( # Identifiers 'ID', # Type identifiers (identifiers previously defined as # types with typedef) 'TYPEID', # constants 'INT_CONST_DEC', 'INT_CONST_OCT', 'INT_CONST_HEX', 'FLOAT_CONST', 'HEX_FLOAT_CONST', 'CHAR_CONST', 'WCHAR_CONST', . ... So I thought that I need to add a name for a token corresponding to comments start //, /* and end */ and it will be better if the token name would be somewhat common among parsers, so that people looking at token could immediately recognize that it is a comment related. Apparently, properly naming is a little bit ambiguous for a automated processing. Editors like Spyder could also benefit information about token and their meaning in different programming languages. The processing of text comments that can be catched from the parsing stream is same for any language and could be IDE independent. Right now you can't just reuse the language definitions (such as ASDL) to just feed the IDE so that it can automatically figure out, what parts of text it can attach its functions to. I read the ontologies is way to express relations between object in this automatic was as triples. Like; COMMENTSTART is a TOKEN COMMENTSTART starts a COMMENT And I wonder, have anybody tried to apply this ontology stuff to designing and analysing computer languages? If yes, maybe there are some databases with such information about parsers. I would like to query names of all tokens that represent a program comment. -- anatoly t.

5 6

culling non-functional build slaves
by Matti Picus March 14, 2015

March 14, 2015

I propose we remove the non-functioning build slaves. I have been trying to contact the owners, here is what I have so far: http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-64-single-run speed-python-64: offline since June http://buildbot.pypy.org/builders/own-indiana-x86-32 jcea-openindiana-32: no owner listed http://buildbot.pypy.org/builders/own-win-x86-64 snakepit64: offline since at least October http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-8-x86-64 ananke: offline since at least Nov http://buildbot.pypy.org/builders/pypy-c-jit-freebsd-7-x86-64 headless: The buildslave is connected but non-functional. I contacted the owner, they cannot continue to run the buildslave at this time [0] http://buildbot.pypy.org/builders/pypy-c-jit-macosx-x86-64 xerxes: The buildslave is connected but non-functional. The owner cannot diagnose the build failures, but is willing to work through the problems if anyone versed in an old version of MacOS can help him. In the mean time I have paused the slave as we have an alternative slave available I suggest we remove speed-python-64, jcea-openindiana-32, snakepit64, anake, and headless, and leave xerxes in "Paused" state. Any objections? Matti [0] here is what the headless buildslave owner replied, I encouraged him to join us again in the future: Hi Matti, I still have the FreeBSD machine, but at some point the memory requirements for a 64 bits build went over 4GB and the machine couldn’t complete the builds in a reasonable time. Unfortunately, the machine is getting old and I cannot add more memory to it. I remember trying the use of the following environment variables to attempt reducing the build time memory usage without success: PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300 I initially provided this buildslave as I had some time to fix PyPy on FreeBSD a few years ago for my master degree, I haven’t had the chance to work with Python for a long time. Regards, Gabriel

4 3