Mailman 3 August 2012 - pypy-dev

numpypy array expressions
by mark florisson 28 Aug '12

28 Aug '12

Hey, For this year's summer of code, and for my master dissertation, we created a project to compile array expressions efficiently (Dag was my mentor, CCed), which can be found here: https://github.com/markflorisson88/minivect , the thesis is under subdirectory 'thesis'. It's currently integrated in Cython, and can significantly outperform open source and commercial Fortran compilers for certain benchmarks (see the bench/graphs subdirectory on github, or the dissertation). For other benchmarks, it's mostly close, accounting for some runtime overhead. The purpose of the project is to be reusable, it's used by Cython, it's likely going to be integrated into Theano, it's partially integrated into numba at the moment. It can generate C code for static compilers, and uses llvmpy to do just-in-time specialization for runtime compilers (likely combined with some form of lazy evaluation). However, I think pypy has its own approach, and I think it relies on the JIT to evaluate the expressions (feel free to point me to documentation or source code)? But I don't imagine it optimizes for the cache hierarchy? So the question is, would there be an interest to use this project in PyPy? It's currently meant to be used as a git submodule, or to be included verbatim, but I also intend to make it distributable and installable. The project is designed so that adding new code generators should require minimal effort. But in the event this type of code translation is not compatible with pypy's approach, you could have a look at the techniques for optimizations, such as tiling for spatial locality and several of the optimizations described in the thesis and prototyped in code on github, such as SIMD transposes when mixing C and Fortran contiguous arrays, a tractable way to prove data independence for NumPy arrays to avoid array temporaries, optimizing broadcasting through loop-invariant code motion, and so forth. Anyway, it doesn't have all functionality yet, for instance it doesn't support reductions yet. But let us know if pypy would want to use it, and if so, how we could collaborate and make minivect compatible for the purposes of pypy. Mark

2 2

wsgi
by Stepanov Aleksandr 28 Aug '12

28 Aug '12

Are there any wsgi server for latest or latest stable pypy? uwsgi needs to be built with pypy-c and shared libpypy , but i failed to build pypy as described on uwsgi wiki after 5 hours of compiling. i have not find precompiled uwsgi_pypy. uwsgi_python does not works with pypy. gunicorn starts, but workers fall without traceback. have somebody recipe to join nginx and pypy? -- Best regards, Aleksandr.

5 4

[ANN] global nonlinear optimization constrained solver for PyPy
by Dmitrey 28 Aug '12

28 Aug '12

Hi all, FYI global nonlinear optimization (GLP <http://openopt.org/GLP>) constrained solver *de* (differential evolution) from free OpenOpt framework ( http://openopt.org ) now works with PyPy, for more details see http://forum.openopt.org/viewtopic.php?id=615 ----------------- Regards, D. http://openopt.org/Dmitrey

2 1

PPC64 backend compile_framework_7_interior failure
by David Edelsohn 25 Aug '12

25 Aug '12

Armin, et al, compile_framework_7_interior in test_zrpy_gc fails for the PPC64 backend in a strange way. Identical problems happen on the public system and my private system, but GDB on the public system is not permitting hardware watchpoints, so it is more difficult to catch the cause of the failure. The testcase segfaults in a load following the instructions generated for getinteriorfield_gc: 0xfffb74e1958: li r3,0 0xfffb74e195c: li r0,24 0xfffb74e1960: mullw r0,r3,r0 0xfffb74e1964: addic r0,r0,16 0xfffb74e1968: ldx r3,r30,r0 ; $r30 + $r0 = 0xfffb75ea030 => 0xfffb74e196c: ld r3,8(r3) ; $r3 = 904 The ldx is the load from getinteriorfield_gc and the subsequent load fails. "904" appears to correspond to "n + i * 100 + ..." in the testcase for i = 9. If I set a watchpoint on the address, it initially is written during memcpy of a minor collection: #0 0x00000fffb7d0daa0 in .memcpy () from /lib64/power7/libc.so.6 #1 0x000000001000fb0c in .pypy_g_trace___trace_drag_out () #2 0x0000000010014288 in .pypy_g_MiniMarkGC_collect_oldrefs_to_nursery () #3 0x0000000010015358 in .pypy_g_MiniMarkGC_minor_collection.part.0 () #4 0x000000001001ce04 in .pypy_g_MiniMarkGC_collect () at which point (gdb) x/gx 0xfffb75ea030 0xfffb75ea030: 0x00000fffb776c170 (gdb) x/gd 0x00000fffb776c170 0xfffb776c170: 904 In other words, the location contains a pointer to another location containing the value "904". The value at that location later is overwritten in jitted code: 0xfffb74e0dec: li r28,0 0xfffb74e0df0: li r0,24 0xfffb74e0df4: mullw r0,r28,r0 0xfffb74e0df8: addic r0,r0,16 0xfffb74e0dfc: stdx r3,r30,r0 0xfffb74e0e14: li r4,16 0xfffb74e0e18: add r4,r3,r4 0xfffb74e0e1c: li r3,904 0xfffb74e0e20: std r3,0(r4) ; $r4 = 0xfffb75ea030 (gdb) print/x $r4 $6 = 0xfffb75ea030 $r3 is stored as part of a setinteriorfield_gc sequence. Subsequently $r3+16 is used as an address to store "904". So now that location instead of containing a pointer to the value "904" has been overwritten with the value "904", which fails when it later is accessed. I don't know if this flattening is intentional and the code reading the value was not updated to compensate for some sort of unboxing or one level of indirection has been lost. Any guidance to help debug this would be appreciated. Thanks, David

3 4

graph not in self.fixed_graphs
by Timothy Baldridge 25 Aug '12

25 Aug '12

I'm back ;-) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 176, in setup [translation:ERROR] self.add_source_for_attribute(name, source) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 193, in add_source_for_attribute [translation:ERROR] attrdef.mutated(cdef) # reflow from all read positions [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 109, in mutated [translation:ERROR] self.bookkeeper.annotator.reflowfromposition(position) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/annrpython.py", line 370, in reflowfromposition [translation:ERROR] self.reflowpendingblock(graph, block) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/annrpython.py", line 432, in reflowpendingblock [translation:ERROR] assert graph not in self.fixed_graphs [translation:ERROR] AssertionError The best I can infer is that perhaps a function is returning two types somewhere in my rpython code? If that were the case though, I'd assume that would blow up with a "SomeObject" error. I've never seen this error before, and it has me a bit baffled. I was able to debug it enough to find out where the code is that it is happening, but the code itself is nothing special: def first(self): return rt.first.invoke1(self) Of course, I have python tests for all this, and if I run my rpython code through CPython it runs without a hitch. What should I be doing to debug this sort of thing more on my own? Thanks for all the help so far... Timothy

2 3

Re: [pypy-dev] [pypy-commit] pypy ppc-jit-backend: For getinteriorfield and setinteriorfield, if immediate offset is too
by Maciej Fijalkowski 25 Aug '12

25 Aug '12

Such fixes should come with tests in test_runner On Fri, Aug 24, 2012 at 8:56 PM, edelsohn <noreply(a)buildbot.pypy.org> wrote: > Author: edelsohn > Branch: ppc-jit-backend > Changeset: r56844:a4efe6ca0483 > Date: 2012-08-24 14:56 -0400 > http://bitbucket.org/pypy/pypy/changeset/a4efe6ca0483/ > > Log: For getinteriorfield and setinteriorfield, if immediate offset is > too large for instruction, load it into reg. > > diff --git a/pypy/jit/backend/ppc/regalloc.py b/pypy/jit/backend/ppc/regalloc.py > --- a/pypy/jit/backend/ppc/regalloc.py > +++ b/pypy/jit/backend/ppc/regalloc.py > @@ -751,7 +751,8 @@ > if _check_imm_arg(ofs): > ofs_loc = imm(ofs) > else: > - ofs_loc = self._ensure_value_is_boxed(ConstInt(ofs), args) > + ofs_loc = self.get_scratch_reg(INT, args) > + self.assembler.load(ofs_loc, imm(ofs)) > self.possibly_free_vars_for_op(op) > self.free_temp_vars() > result_loc = self.force_allocate_reg(op.result) > @@ -770,7 +771,8 @@ > if _check_imm_arg(ofs): > ofs_loc = imm(ofs) > else: > - ofs_loc = self._ensure_value_is_boxed(ConstInt(ofs), args) > + ofs_loc = self.get_scratch_reg(INT, args) > + self.assembler.load(ofs_loc, imm(ofs)) > return [base_loc, index_loc, value_loc, ofs_loc, imm(ofs), > imm(itemsize), imm(fieldsize)] > prepare_setinteriorfield_raw = prepare_setinteriorfield_gc > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit

1 0

"Don't know how to convert" error
by Timothy Baldridge 25 Aug '12

25 Aug '12

I hit this bug this evening while translating my code: [translation:ERROR] return self.call('simple_call', hop) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/lltypesystem/rpbc.py", line 245, in call [translation:ERROR] return hop.llops.convertvar(v_result, rresult, hop.r_result) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 933, in convertvar [translation:ERROR] (r_from, r_to)) [translation:ERROR] TyperError: don't know how to convert from <InstanceRepr for system.core.Object> to <VoidRepr Void> [translation:ERROR] .. (system.rt:60)PolymorphicFunc.invoke3 [translation:ERROR] .. block@48 with 1 exits [translation:ERROR] .. v14 = simple_call(v13, a_0, b_0, c_0) What could cause something like this? I can seem to figure out how to even go debugging this. I dug into the stack trace a bit and poked around in the locals, but I can't seem to find any more information. Any ideas? Timothy

2 1

spectacular fail while upgrading tannit
by Maciej Fijalkowski 24 Aug '12

24 Aug '12

Hi I tried to upgrade tannit to a new Ubuntu LTS server. Unfortunately ubuntu forces people to upgrade grub to grub2, which also does not work in the partition setup that we have on tannit. The machine is lost (unbootable) and someone with physical access have to go there and fix it. Cheers, fijal

2 2

Re: [pypy-dev] win64
by jason ye 24 Aug '12

24 Aug '12

Hi Does anybody know the progress of the WIN64 port? I have a python program which require >4GB memory, expecting WIN64 PYPY Thanks Jason

2 1

Hello World in Windows with MinGW
by Welch, Matt 23 Aug '12

23 Aug '12

Sorry to bother you with a newbie question. I'm trying to send a simple Hello World program through translate. It's failing to compile platcheck_1.c. I'm getting: [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c: In function `dump_section_PROCESS_QUERY_LIMITED_INFORMATION': [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:383: error: `PROCESS_QUERY_LIMITED_INFORMATION' undeclared (first use in this function) [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:383: error: (Each undeclared identifier is reported only once [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:383: error: for each function it appears in.) [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c: In function `dump_section_PROCESS_SUSPEND_RESUME': [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:403: error: `PROCESS_SUSPEND_RESUME' undeclared (first use in this function) I tried to figure out where the values in platcheck_1.c were getting defined in the first place so I could determine what I was doing wrong, but couldn't quite get a handle on it. I'm on Windows, running MinGW through Msys. I launched translate like this: pypy-1.9/pypy.exe pypy-pypy-341e1e3821ff/pypy/translator/goal/translate.py --cc=gcc --batch hello.py The compile line that is failing looks like this: [platform:execute] gcc -c -O3 c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\gcctest.c -o c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\gcctest.o Any help would be appreciated. Do I need a different environment? Is it not MinGW/Msys friendly?

2 1