Mailman 3 August 2012 - pypy-dev

_marshal EOF error?
by Jonathan Slenders Aug. 9, 2012

Aug. 9, 2012

Hi everyone, I have almost a sandlib.py library ready for the Twisted Matrix framework. (The sandlib which is shipped with PyPy is not usable within Twisted.) However, the deserialisation of marshalled messages fails when the sandbox Python process does very big write system calls. Actually, the problem is that the write calls arrive in multiple chunks in the outReceived of the ProcessProtocol (Twisted code). Marshal.loads thinks the first chunk is valid on its own while it isn't yet. Long story short, there is a difference between the Python built-in and the pure python _marshall.py from PyPy which should be used for the sandbox hypervisor. It should throw EOF when a string-object is truncated, while it doesn't. >>> import _marshal # lib_pypy/_marshal.py >>> _marshal.loads('(\x02\x00\x00\x00i\x03\x00\x00\x00sB\xf9\x00\x00\n<stations version="1.1" ') (3, '\n<stations version="1.1" ') >>> >>> import marshal # Python built-in >>> marshal.loads('(\x02\x00\x00\x00i\x03\x00\x00\x00sB\xf9\x00\x00\n<stations version="1.1" ') Traceback (most recent call last): File "<stdin>", line 1, in <module> EOFError: EOF read where object expected The patch, just add this one line to lib_pypy/_marshal.py (line 433) def _read(self, n): pos = self.bufpos newpos = pos + n * if newpos > len(self.bufstr): raise EOFError* ret = self.bufstr[pos : newpos] self.bufpos = newpos return ret This is the first bug that I found. Should I create a bug report or patch file for this? Thanks, Jonathan Slenders

2 2

Questions Regarding Scipy in Pypy (Not a Feature Request)
by Steven Jackson Aug. 9, 2012

Aug. 9, 2012

I know there is no current plan to implement scipy in pypy, but searching the PyPy website, I was not able to find the reason. If it is not to be included as a feature due to lack of interest or developer time, I am offering to begin rewriting scipy in pure python (I have quite a bit of free time to do this). If it has been discussed and actively excluded from pypy, I would like to know that before I waste too much time rewriting it. I have an active interest in being able to use some modules which depend on scipy while using the PyPy interpreter (especially pyBrain). I have seen the hack on the pypy blog: http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.ht… I could simply use the hack myself, but unless there's a reason not too, I think it'd be nice to have scipy available as a pure python module (perhaps called scipypy in the same manner as numpy). -- Steven Jackson

6 9

Dependence Graphs in Pypy?
by Tim Henderson Aug. 9, 2012

Aug. 9, 2012

Hi Pypy devs! I am a graduate student working on mining software repositories, specifically mining programming patterns (or rules) from code bases.[1] I would like to do some work mining patterns from Python programs and I could jump start that work if I could generate dependence graphs from python source files (or from the bytecode). So I was wondering if Pypy does this already. Actually it would be a big help even if pypy only generated a control flow graph. From my quick skim of some of the source code it seems like it does this for RPython but I wasn't sure about regular python. If I could get a CFG I could then annotate it with the data dependence edges. Any pointers to where I should look in the Pypy code would be appreciated! Additionally, if you feel having python pDG's would be helpful in Pypy's interpreter perhaps any work I do towards that end could be a starting point. Thanks for the excellent work on Pypy! [1] Here is some of my research group's previous work on pattern extraction http://doi.wiley.com/10.1002/smr.532 (if you can't get the paper and would like it let me know) Cheers --- Tim Henderson mail me: tim.tadh(a)gmail.com mail me: timothy.henderson(a)case.edu github: github.com/timtadh

4 4

Help (again)
by Léonard de Haro Aug. 8, 2012

Aug. 8, 2012

Hi everyone! First of all, thaks for your answers last time. Second of all, I need your help again. I'm coding an imperative version of my interpret and to do so, I coded continuations. I used a similat technique to the one used in the Prolog interpret you advised me to take a look at last time. I define a serie of class all inheriting from a simulated-abstract Contk class. Every class has a method apply which contains instruction for what's next to do. All these methods take the same arguments and (except in one case) return the same kind of tuple. The interpret function uses 4 variables that are updated: expr, env, cont, val. Thus, the result of .apply(_,_,_,_) is the tuple for update. class Contk(object): def __init__(self,*arg): raise NotImplementedError("For abstract class") def apply(self,expr,env,val): raise NotImplementedError("For abstract class") class Endk(Contk): def __init__(self,val): self.val = val def apply(self,expr, env, val): return self.val class Idk(Contk): def __init__(self): pass def apply(self, expr, env, val): return expr, env, Endk(val), val There is a class for each continuation needed by the interpret. The trouble is, I get this error: [translation:ERROR] AnnotatorError': annotation of 'union' degenerated to SomeObject() [translation:ERROR] Simple call of incompatible family: [translation:ERROR] (KeyError getting at the binding!) [translation:ERROR] [translation:ERROR] In <FunctionGraph of (RPinterpretImperative:27)Idk.apply at 0x1b31be8>: [translation:ERROR] Happened at file RPinterpretImperative.py line 28 [translation:ERROR] [translation:ERROR] ==> return expr, env, Endk(val), val [translation:ERROR] [translation:ERROR] Previous annotation: [translation:ERROR] (none) [translation:ERROR] .. v0 = simple_call((classobj Endk), val_0) [translation:ERROR] .. '(RPinterpretImperative:27)Idk.apply' [translation:ERROR] Processing block: [translation:ERROR] block@12 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'> [translation:ERROR] in (RPinterpretImperative:27)Idk.apply [translation:ERROR] containing the following operations: [translation:ERROR] v0 = simple_call((classobj Endk), val_0) [translation:ERROR] v1 = newtuple(expr_0, env_0, v0, val_0) [translation:ERROR] --end-- So at first I thought it was due to tuple that would not accept newly created objects, but after a few tests, it appears that not. Then I tought the annotator couldn't realize by himself that returning an Endk object is the same as returning a Contk, so I annotated the whole file with assertion to force him to realize that my variable cont should be a Contk not any other kind of subclass, but it didn't work either. So I don't know what to do now. I've even tried to implement apply outside of the classes, or to make a function for each parameters returned, but nothing worked. You can find the source here : https://github.com/zebign1/RPython-internship/tree/master/Interpreters/ifF1… (treeClass.py and parser.py as tools, the bugged file is RPinterpretImperative.py) Thanks again. Leonard

3 3

Fatal RPython error: MemoryError
by Roger Flores Aug. 8, 2012

Aug. 8, 2012

I'm still getting MemoryError exceptions in RPython, as I mentioned a few months back. I only knew of one way to reproduce it back then, but now I'm seeing different ways. So, inspired, I wrote a short program to quickly exhaust memory in the hopes of reproducing the problem. It seems to work! Fast! If you run the code pasted at the bottom with CPython, it should end printing 'test passed'. If you run the code with pypy 1.9 (I tried Windows), you should get: RPython traceback: File "translator_goal_targetpypystandalone.c", line 1048, in entry_point File "interpreter_function.c", line 929, in funccall__star_1 File "interpreter_function.c", line 958, in funccall__star_1 File "rpython_memory_gc_minimark.c", line 2518, in MiniMarkGC_collect_and_rese rve File "rpython_memory_gc_minimark.c", line 2220, in MiniMarkGC_minor_collection File "rpython_memory_gc_minimark.c", line 4258, in MiniMarkGC_collect_oldrefs_ to_nursery File "rpython_memory_gc_base.c", line 1799, in trace___trace_drag_out File "rpython_memory_gc_minimark.c", line 6667, in MiniMarkGC__malloc_out_of_n ursery_nonsmall Fatal RPython error: MemoryError Can someone else at least confirm this? Thanks, -Roger Flores exhaust-mem.py: """ Code to exhaust memory and generate a MemoryError exception """ size = 1 # find the largest memory allocation that succeeds try: while True: new_size = size * 2 d = [0] * size size = new_size except MemoryError as msg: pass l = [] # exhaust memory while True: try: # add the allocations to a list so references to them remain. # otherwise the garbage collector will clean them out! l.append([0] * size) print '{:,}'.format(size) except MemoryError as msg: if size > 1: # can't allocate this much memory anymore. Try a smaller size size = size / 2 else: print 'test passed' break

6 11