Mailman 3 July 2005 - pypy-dev

Hildesheim sprint progress report
by Carl Friedrich Bolz July 26, 2005

July 26, 2005

Just some quick (and probably confused) notes about went on at the Hildesheim sprint day 1. Corrections welcome. As in Gothenburg we tried to write an evolving file that listed all the tasks: http://codespeak.net/svn/pypy/extradoc/sprintinfo/hildesheim2-planning.txt somehow we didn't get very far and decided that we just would all work on rtyper issues. The following people are here: Hogler, Armin, Samuele, Christian, Richard, Carl Friedrich. Holger and Christian paired and started … [View More]

1 1

translation and tests
by hpk＠trillke.net July 26, 2005

July 26, 2005

Hello pypy-dev! today at the hildesheim2 sprint we decided that we'd like to annotate and translate from the trunk during the sprint. After the sprint we are probably changing back to work from a snapshot. This is more convenient than porting changes to the snapshost all the time as we are making a lot of changes. Also, please try to avoid checking in changes with failing tests. All of us are doing it sometimes, for example, when we change computers or locations and want to continue … [View More]

1 0

Re: [pypy-svn] r15069 - pypy/dist/pypy/translator/llvm2/tool
by Christian Tismer July 25, 2005

July 25, 2005

ericvrp(a)codespeak.net wrote: > Wrote simple tool to get an overview of the suggested_primitive functions that > still need to be implemented in the llvm backend. > The tool should be run from it's own directory for now! > This is the current output: (tail is what we want to know) Nice tool, very useful! Just a hint for further development: PyPy tends to avoid source code analysis wherever possible. And in this tool's case it would bhe quite easy to use introspecction: > + … [View More]

1 0

next pypy sprints / some infos
by hpk＠trillke.net July 20, 2005

July 20, 2005

Hi folks, first some quick notes regarding upcoming PyPy sprints. After the EuroPython 2005 sprint we decided that we are going for a sprint that i am organizing at my living and working place, Trillke-Gut in Hildesheim, Germany. It is going to start 25th July and last until 31st July 2005. Especially already scheduled participants please look at http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/Hildesheim2-sprint.… for some more information (and correct wrong dates!). The … [View More]

1 0

Re: rtyper hook ++
by amaury.forgeotdarc＠ubitrade.com July 19, 2005

July 19, 2005

Hello, Chritian Tismer wrote: > If we teach the specializer to not crash if something > does not work out, but to add the problem blocks to a list, > then we could have one report with all pending problems > and work through this in a more general way. > We also would get an estimate how much has still to be done. That's what I have done to follow the fantastic progress you make on the translation process. - In rtyper.py, there is already a crash_on_first_typeerror variable. … [View More]Turn it to False, this will catch all TyperErrors. - There are some assertions and other exceptions that will stop the process. I changed all of them into TyperErrors. I suppose that some tests are not done at the right place, but they do the job... And the specialization goes to the end! There are 81 TyperErrors left for 20206 blocks. They were 230 last week. Good job! Then the "Back-end optimizations" fail, probably because of the incompletely specialized blocks. I hope this helps! Here are my diffs: Index: pypy/rpython/rstr.py =================================================================== --- pypy/rpython/rstr.py (revision 14755) +++ pypy/rpython/rstr.py (working copy) @@ -264,7 +264,8 @@ if curstr: r.append(curstr) curstr = '' - assert f in 'xdsrf' + if f not in 'xdsrf': + raise TyperError("String format not supported: %r" % (f,)) r.append((f,)) else: @@ -295,6 +296,9 @@ vitem, r_arg = argsiter.next() rep = inputconst(Void, r_arg) if code == 's' or (code == 'r' and isinstance(r_arg, InstanceRepr)): + if not hasattr(r_arg,'ll_str'): + raise TyperError("AttributeError: %s instance has no attribute 'll_str'" + % r_arg) vchunk = hop.gendirectcall(r_arg.ll_str, vitem, rep) elif code == 'd': assert isinstance(r_arg, IntegerRepr) Index: pypy/rpython/callparse.py =================================================================== --- pypy/rpython/callparse.py (revision 14755) +++ pypy/rpython/callparse.py (working copy) @@ -2,6 +2,7 @@ from pypy.interpreter.argument import Arguments, ArgErr from pypy.annotation import model as annmodel from pypy.rpython import rtuple +from pypy.rpython.rmodel import TyperError class CallPatternTooComplex(Exception): pass @@ -47,7 +48,7 @@ try: holders = arguments.match_signature(signature, defs_h) except ArgErr, e: - raise TypeError, "signature mismatch: %s" % e.getmsg(arguments, func.__name__) + raise TyperError, "signature mismatch: %s" % e.getmsg(arguments, func.__name__) assert len(holders) == len(rinputs), "argument parsing mismatch" vlist = [] Index: pypy/rpython/rbuiltin.py =================================================================== --- pypy/rpython/rbuiltin.py (revision 14755) +++ pypy/rpython/rbuiltin.py (working copy) @@ -107,9 +107,15 @@ def rtype_builtin_unichr(hop): assert hop.nb_args == 1 + if not hasattr(hop.args_r[0],'rtype_unichr'): + raise TyperError("AttributeError: %s instance has no attribute 'rtype_unichr'" + % hop.args_r[0]) return hop.args_r[0].rtype_unichr(hop) def rtype_builtin_list(hop): + if not hasattr(hop.args_r[0],'rtype_bltn_list'): + raise TyperError("AttributeError: %s instance has no attribute 'rtype_bltn_list'" + % hop.args_r[0]) return hop.args_r[0].rtype_bltn_list(hop) def rtype_builtin_isinstance(hop): Index: pypy/rpython/rtyper.py =================================================================== --- pypy/rpython/rtyper.py (revision 14755) +++ pypy/rpython/rtyper.py (working copy) @@ -31,7 +31,7 @@ log = py.log.Producer("rtyper") py.log.setconsumer("rtyper", None) -crash_on_first_typeerror = True +crash_on_first_typeerror = False class RPythonTyper: @@ -45,6 +45,7 @@ self.pbc_reprs = {} self.class_pbc_attributes = {} self.typererror = None + self.nb_typeerrors = 0 # make the primitive_to_repr constant mapping self.primitive_to_repr = {} for s_primitive, lltype in annmodel.annotation_to_ll_map: @@ -138,8 +139,8 @@ n = len(self.already_seen) if n % 100 == 0: total = len(self.annotator.annotated) - print 'specializing: %d / %d blocks (%d%%)' % ( - n, total, 100 * n // total) + print 'specializing: %d / %d blocks (%d%%, %d errors)' % ( + n, total, 100 * n // total, self.nb_typeerrors) # make sure all reprs so far have had their setup() called self.call_all_setups() @@ -200,7 +201,11 @@ def specialize_block(self, block): # give the best possible types to the input args - self.setup_block_entry(block) + try: + self.setup_block_entry(block) + except TyperError, e: + self.gottypererror(e, block, "block entry", None) + return # cannot continue this block # specialize all the operations, as far as possible if block.operations == (): # return or except block @@ -269,8 +274,9 @@ new_a1 = newops.convertvar(a1, r_a1, r_a2) except TyperError, e: self.gottypererror(e, block, link, newops) - if new_a1 != a1: - newlinkargs[i] = new_a1 + else: + if new_a1 != a1: + newlinkargs[i] = new_a1 if newops: if can_insert_here: @@ -343,13 +349,16 @@ """Record a TyperError without crashing immediately. Put a 'TyperError' operation in the graph instead. """ + self.nb_typeerrors += 1 e.where = (block, position) if crash_on_first_typeerror: raise + print "*** TyperError:",e if self.typererror is None: self.typererror = sys.exc_info() c1 = inputconst(Void, Exception.__str__(e)) - llops.genop('TYPER ERROR', [c1], resulttype=Void) + if llops: + llops.genop('TYPER ERROR', [c1], resulttype=Void) # __________ regular operations __________ @@ -377,7 +386,12 @@ def translate_op_contains(self, hop): r_arg1 = hop.args_r[0] r_arg2 = hop.args_r[1] - return pair(r_arg1, r_arg2).rtype_contains(hop) + p = pair(r_arg1, r_arg2) + try: + f = p.rtype_contains + except AttributeError,e: + raise TyperError(str(e)) + return f(hop) # __________ irregular operations __________ -- Amaury Forgeot d'Arc Ubix Development www.ubitrade.com [View Less]

2 1

rtyper hook ++
by Christian Tismer July 19, 2005

July 19, 2005

Hi Samuele, you are not online, therefore using email. """ pedronis: stakkars_: btw my new hook allows to get back to a given module or function, or to skip things we know specialize fine """ I like the idea. Can't we go even a bit further? If we teach the specializer to not crash if something does not work out, but to add the problem blocks to a list, then we could have one report with all pending problems and work through this in a more general way. We also would get an estimate how much … [View More]

1 0

Re: [pypy-dev] Questions from a lurker
by Arthur Peters July 18, 2005

July 18, 2005

I accidently replied to this off list, so the off-list emails (3 of them) are appended for postarity. Also I fixed the problem with test_all.py: I had created a __init__.py file directly in pypy-dist. I don't remember why I created it. Result was sure strange. Anyway it was my fault. Thanks all. -Arthur ---------------------------------------------------- On Sun, 2005-07-17 at 00:25 +0200, Carl Friedrich Bolz wrote: > Hi Arthur, > > some more notes -- again I might not be the … [View More]perfect person to answer > this, corrections welcome. > > Arthur Peters wrote: > [snip] > > Makes sense, but it seems to me that there will be a lot of overlap > > between the translator and the JIT compiler. Couldn't large parts of > > the translator be reused? > > Yes and no. The translator (and especially the type inference which we > call annotation) works on special assumptions that don't apply to a > regular programs: For example that the objects bound to a name in a > certain place always have the same type. > > > > > BTW, I actually haven't heard for certain that there will be a JIT > > compiler. Will there be? I'd really like to see one. Python is a great > > language but runs slow sometimes and I think a JIT could really help. > > Though one of the main slow things about it is the function call > > overhead and for that to be reduced JIT compilation would have to happen > > at a level higher than functions and would need type inference and all > > that. However it would be really cool to have things like inlining of > > functions (it would require a decorator to mark the function as > > invariant so that the compiler knows that it cannot be changed at > > runtime like normal python functions can). I come from a C++ background > > so I'm used to having simple functions be as fast as macros. > > Yes, there will be a JIT compiler although probably a bit different from > other JIT compilers. The problem is that Python is so very dynamic that > you cannot really know /anything/ in advance. Which means that even > "just in time" -- e.g. right before the program is run -- is too early. > (This does not apply to the code our translator translates, because that > code, mainly PyPy's interpreter, was written with a lot of limitation > especially regarding the dynamism.) The first time you really know > anything for sure about a Python function is when you run it for the > first time (because then you have fixed values with fixed types to work > with). This is (heavily simplified) the idea driving Psyco: Psyco gets > information about a function when this function is being executed. > > I'm not totally sure whether the thing that makes Python slow is really > the function calls. One of the slower things might be dispatching (which > function will be executed for the code a + b) which is a thing that > Psyco addresses. And, ideally, the user would not need to take care of > telling the JIT which function to inline, the JIT should rather be > clever enough to realize that a function is worth inlining. I even think > that the JIT could do this better than the programmer. > > > > > Also it seems sad to have such a cool and difficult to create things as > > the translator end up only being used on one program. It seems like it > > could be useful elsewhere. > > > > Yesish. The problem is that it's rather hard to get the translator > working for a given Python program (or rather, get the program working > for the translator :-) ). At the moment you'd have to adhere to a lot of > restrictions which are sometimes not easy to explain. This will problem > only change to a certain degree. > > Regards, > > Carl Friedrich ------------------------------------------------- On Sat, 2005-07-16 at 17:40 -0500, Arthur Peters wrote: > On 7/16/2005, "Carl Friedrich Bolz" <cfbolz(a)gmx.de> wrote: > >Yes, there will be a JIT compiler although probably a bit different from > >other JIT compilers. The problem is that Python is so very dynamic that > >you cannot really know /anything/ in advance. Which means that even > >"just in time" -- e.g. right before the program is run -- is too early. > >(This does not apply to the code our translator translates, because that > >code, mainly PyPy's interpreter, was written with a lot of limitation > >especially regarding the dynamism.) The first time you really know > >anything for sure about a Python function is when you run it for the > >first time (because then you have fixed values with fixed types to work > >with). This is (heavily simplified) the idea driving Psyco: Psyco gets > >information about a function when this function is being executed. > > That makes sense. > > > > >I'm not totally sure whether the thing that makes Python slow is really > >the function calls. One of the slower things might be dispatching (which > >function will be executed for the code a + b) which is a thing that > >Psyco addresses. And, ideally, the user would not need to take care of > >telling the JIT which function to inline, the JIT should rather be > >clever enough to realize that a function is worth inlining. I even think > >that the JIT could do this better than the programmer. > > I agree. > > What I ment was a decorator that said that the method (or perhaps I > should say method name) would not be rebound. This would allow the > function to be embeded in the caller without fear of rebinding. This > would be an issue because this is valid: > > class A: > def f(self): > pass > > def call_f(a): > a.f() > > a = A() > call_f(a) # does nothing > > def new_f(): > print "Hi" > > a.f = new_f > > call_f(a) # prints "Hi" > > If A.f were inlined into call_f then the second call to call_f would > either run the wrong code or would have to trigger a recompile. > > -Arthur ---------------------------------------------------- On Sat, 2005-07-16 at 10:40 -0500, Arthur Peters wrote: > Answer interlaced. > > On 7/16/2005, "Carl Friedrich Bolz" <cfbolz(a)gmx.de> wrote: > >> - I read what some discussion of the GIL as it relate to pypy and I agree > >> that the GIL need to be implemented and that a different thread model > >> might be a good way to go. However I thought of the following: Would it > >> be possible to detect when a section of code only uses local variables > >> and unlock the GIL? This seems possible because local variables will > >> cannot be shared between threads anyway. In addition, local variables > >> likely be translated into more basic types than a python object (native > >> ints and floats for instance), that would not require any interaction > >> with the object-space to manipulate (not sure about that usage of the > >> term "object-space"). Thoughts? > > > >First a comment about the usage of the term "object space": I think you > >are mixing levels here (and I might misunderstand you). The PyPy > >interpreter (meaning the bytecode interpreter plus the standard > >objectspace) gets translated to low level code. This "interpreter level > >code" deals with the standard object space as a regular class instance > >that is in principle in no way different than any other class. The > >classes that appear on interpreter level are all translated to more > >basic types (what would be the alternative, there is no layer below that > >could deal with anything else), probably to something like a struct. > >Thus the operations of objects at this level never need to be done via > >the object space -- the object space is rather a regular object in itself. > > > >The object space /is/ used, if you interpret a Python program with > >PyPy's interpreter. The bytecode interpreter does not now how to deal > >with /any/ object (except for True and False), it has to delegate to the > >object space for every single operation. Even for basic types like ints > >and such -- at this level ("application level") there isn't any type > >inference or anything like that! > > Makes sense, but it seems to me that there will be a lot of overlap > between the translator and the JIT compiler. Couldn't large parts of > the translator be reused? > > BTW, I actually haven't heard for certain that there will be a JIT > compiler. Will there be? I'd really like to see one. Python is a great > language but runs slow sometimes and I think a JIT could really help. > Though one of the main slow things about it is the function call > overhead and for that to be reduced JIT compilation would have to happen > at a level higher than functions and would need type inference and all > that. However it would be really cool to have things like inlining of > functions (it would require a decorator to mark the function as > invariant so that the compiler knows that it cannot be changed at > runtime like normal python functions can). I come from a C++ background > so I'm used to having simple functions be as fast as macros. > > Also it seems sad to have such a cool and difficult to create things as > the translator end up only being used on one program. It seems like it > could be useful elsewhere. > > >Hope that helped a bit and wasn't too confused :-). > > I got most of it. ;-) Thanks a bunch. > > -Arthur [View Less]

1 0

reconstructing structured flow graphs: pointers
by Samuele Pedroni July 17, 2005

July 17, 2005

Our flowgraphs can be rendered easely in languages that support gotos. For some potential targets in would be nice* to be able to reconstruct structured graphs that can be rendered in terms of ifs/whiles etc. For (future) reference these are pointers to literature touching the subject and containing also further pointers to more literature: Assembly to High-Level Language Translation http://citeseer.ist.psu.edu/cifuentes98assembly.html Structuring Assembly Programs http://www.cs.uq.edu.au/~… [View More]

1 0

Questions from a lurker
by Arthur Peters July 16, 2005

July 16, 2005

Hello, I am Arthur. I've been lurking on the list for a month or so and I've had a good bit of fun playing with the code. My questions are: - Why did you start a second LLVM backend? Was the first one badly miss designed? - So I take it the initial goal will be to translate pypy into C (or LLVM) and create a binary executable. At that point will it be possible to translate any given program or will the translator still only work with a subset of the python language? Will the final goal be a … [View More]JIT VM for python or will I at some point be able to statically compile python to machine code? - Will there be support for multiple specialized versions of functions like in psyco? I know this is a long way down the road but I'm curious what people think. - I read what some discussion of the GIL as it relate to pypy and I agree that the GIL need to be implemented and that a different thread model might be a good way to go. However I thought of the following: Would it be possible to detect when a section of code only uses local variables and unlock the GIL? This seems possible because local variables will cannot be shared between threads anyway. In addition, local variables likely be translated into more basic types than a python object (native ints and floats for instance), that would not require any interaction with the object-space to manipulate (not sure about that usage of the term "object-space"). Thoughts? - I have tried to run test_all.py on my desktop (a dual PIII running ubuntu breazy (development)) but I get the following exception: Traceback (most recent call last): File "pypy/test_all.py", line 6, in ? py.test.cmdline.main() File "/home/amp/source/pypy-dist/py/test/cmdline.py", line 8, in main config, args = py.test.Config.parse(py.std.sys.argv[1:]) File "/home/amp/source/pypy-dist/py/test/config.py", line 61, in parse config = bootstrapconfig(configpaths) File "/home/amp/source/pypy-dist/py/test/config.py", line 168, in bootstrapconfig config.loadconfig(defaultconfig).adddefaultoptions() File "/home/amp/source/pypy-dist/py/test/config.py", line 91, in loadconfig mod = importconfig(configpath) File "/home/amp/source/pypy-dist/py/test/config.py", line 208, in importconfig return configpath.pyimport() File "/home/amp/source/pypy-dist/py/path/local/local.py", line 381, in pyimport return __import__(modname, None, None, ['__doc__']) ImportError: No module named defaultconftest The cause of the problem seems to be a failure at /home/amp/source/pypy-dist/py/initpkg.py(179)__getattr__() while trying to get the attribute "__path__". This happens on both 2.3 and 2.4. This does not happen at all on my Ubuntu Hoary (stable) laptop. Any ideas? Thanks a bunch. I am very impressed by what pypy can do and I'm looking forward to seeing it progress. -Arthur PS: I am the amep that left a message on IRC. [View Less]

2 1

llvm2 update
by Richard Emslie July 15, 2005

July 15, 2005

Hi all, I've just added a dummy OpaqueType implementation and refactored the database wrt to pbc nodes. So pbcs are working for classes too now. Here is a brief overview of things that (I think) still need to me done. todo (probably in this order): * exception flow of control * any missing operations implementations (bit shifting for instance). These can be added when they are needed and should be straight forward. * review and update external function implementations for new way of … [View More]

3 3