Mailman 3 January 2003 - pypy-dev

Notes on compiler package
by logistix 24 Jan '03

24 Jan '03

I'm finally starting to get my head wrapped around the compiler package, and thought I'd pass some notes on to the group since it's a beast: 1) Other than the original AST generation, it looks like it's all pure python (not tying into the C source). The original AST comes from the parser module, which is a pure wrapper module (although I've already posted python code to build ast's here) 2) Internal CPython ASTs are transformed into "compiler"'s own ASTs. These nodes are smarter than the internal ones; they attach appropriate info to "attributes". CPython AST nodes just have a list of children, this module identifies and breaks out important data (for example, the arguments in a function def become an "args" attribute instead of guessing it's the first child) 3) The CodeGenerators walk the ASTs and emit code into a code object. 4) Speed seems reasonable (which makes me think I'm missing some calls into the C side of things) 4) The generated bytecode looks reasonably good and execs fine, but there are still some diffences from what CPython generates. Here's what I've seen so far: SET_LINENO doesn't always work the same (this is how python knows what line in a file threw an exception) Doesn't have one of the few optimizations the CPython compiler does have. CPython will throw out misplaced "docstrings"... strings in the source that aren't assigned to anything. This throws off array indexes in the code objects co_const attribute. In general, the compiler package is in alot better shape than I expected. If anyone is interested in poking around, here's a quick script that does basic comparison and diff on the bytecode generated by the builtin compile and compiler's equivilent function. I'll probably expand this to compare the whole code objects in the near future. =============== CompilerTest.py =============== import sys import compiler import dis def opcodeTuples(source): """ Makes Opcode tuples in the form of: OFFSET, NAME, [OPTIONAL PARAM] """ retVal = [] a = iter(source) offset = 0 def getByte(next=a.next): return ord(a.next()) def getWord(next=a.next): return ord(a.next()) + ord(a.next()) * 256 # Little-endian while 1: try: opcode = getByte() opname = dis.opname[opcode] if opcode < 90: retVal.append( (offset, dis.opname[opcode]) ) offset += 1 else: retVal.append( (offset, dis.opname[opcode], getWord())) offset += 3 except StopIteration: break return retVal def opcodeDiff(ops1, ops2): """ Does a simple DIFF of two sets of opcodes. Can only check one skipped line. Ignores param for now since they don't match """ opcode1, opcode2 = opcodeTuples(ops1), opcodeTuples(ops2) a,b = 0,0 print "%30s%30s" % ("FIRST", "SECOND") print "%30s%30s" % ("====================" ,"====================") while 1: if opcode1[a][1:2] == opcode2[b][1:2]: print "%30s%30s" % (opcode1[a], opcode2[b] ), if opcode1[a][2:] != opcode2[b][2:]: print " ARG MISMATCH" else: print a += 1 b += 1 elif opcode1[a+1][1:2] == opcode2[b][1:2]: print "%30s%30s" % (opcode1[a], "<not here>") a += 1 elif opcode1[a][1:2] == opcode2[b+1][1:2]: print "%30s%30s" % ("<not here>", opcode2[b]) b += 1 else: print "NONTRIVIAL DIFF%25s%25s" % (opcode1[a], opcode2[b]) break if a >= len(opcode1) and b >= len(opcode2): break elif a >= len(opcode1) or b >= len(opcode2): print "UNEXPECTED END OF OPCODES" break def compareCompiles(filename, nativeCompile=compile, pythonCompile=compiler.compile): """ Compares a bytecode compile between the native python compiler and the one written in python """ source = file(filename).read() native = nativeCompile(source, filename, "exec") python = pythonCompile(source, filename, "exec") if native.co_code == python.co_code: print "compiles matched" else: print "compiles didn't match" opcodeDiff(native.co_code, python.co_code) if __name__ == "__main__": compareCompiles("c:\\python23\\lib\\code.py") =============== End CompilerTest.py =============== --------------------------------------- Get your free e-mail address @zworg.com

9 11

RE: Interfacing Python Code with Autogenerated C code
by roccomoretti＠netscape.net 24 Jan '03

24 Jan '03

"Gerald S. Williams" <gsw(a)agere.com> wrote: >Rocco Moretti wrote: >> How would the interface between the two work if the hand translated sections differ >> from the C version? > >I can give general answers from past experience. > >> Simplest example is basic types. A Python version would presumably be represented as a >> class with member functions. But the C code uses calls to accessor functions. How could >> we redirect the automatically generated code to call member functions as opposed to >> free floating functions? > >One answer: provide such functions, which call the member >functions as needed. > >> A more difficult problem lies when we we start to reorganize the object/structure >> internals. Good case of this is the frame object. The C version has an array with >> multiple pointers to internal objects. Once you reorganize the internals, how do >> you tell the C->Python translator that f_stacktop isn't a PyObject pointer anymore, >> but an index off of another list? > >If the class was providing public attributes, you may not >want to change their meaning. f_stacktop could provide an >accessor for the old representation. The new representation >would be in _f_stacktop or something. > >-Jerry In short, if we use C->Python autogeneration, we need to maintain legacy interfaces in any C code we hand translate to Python. - Sounds fair enough. One thought I had since posting is that it may be possible to build intellegence into the translator, such that it can massage calls from one function into another (perhaps multiple) call(s) to another function, reorganize functions into members, and map attribute accesses into member function calls, as the case may be. I'd anticipate you would need to feed the translator a file describing the mapping of the [C language interface of the hand translated code] onto the Python language interface. Something to the effect of: guido(a,b,c) --> n = guido_1(a,b); guido_2(n,c) tim(d,e) --> e._tim(d) F_bot.g = X --> F_bot.set_g(X) You may even be able to use this technique to map the C standard library functions onto their Python equivalents. -Rocco (Avoiding f**, b*r and b*z) __________________________________________________________________ The NEW Netscape 7.0 browser is now available. Upgrade now! http://channels.netscape.com/ns/browsers/download.jsp Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/

1 0

RE: 300,000 lines of C
by Hickey,Thom 23 Jan '03

23 Jan '03

Does anyone else have experience with TeX? Knuth wrote it in Pascal (I was there when he was considering whether Pascal or C would be the right choice), but most implementations run as C. Someone wrote a set of support routines plus a translator for the very restricted Pascal that Knuth used. Local modifications to the code are almost all handled by substituting new Pascal code which is incorporated by a preprocessor before being translated into C. Really works pretty well, and some of the same techniques should carry over to pypy. I once started a project to translate TeX to Java, but found the translation to Java more difficult than that to C and ended up abandoning it. --Th

2 1

Re: [pypy-dev] Re: Notes on compiler package
by logistix 23 Jan '03

23 Jan '03

Michael Hudson <mwh(a)python.net> wrote: > > > In general, the compiler package is in alot better shape than I expected. > > What were you expecting? > > Cheers, > M. > I don't know. I guess in general I expect nothing, and this looks like something ;) BTW, there's a note in compiler.ast that says it's autogenerated. Is this still true? If so, how?

2 1

Miasma: x86 machine code generation in Python
by Darius Bacon 23 Jan '03

23 Jan '03

I've just written a Python module to emit x86 machine code using assembler-like function calls, plus another module in C to invoke the generated code: http://accesscom.com/~darius/software/miasma/ It comes with a little test program that compiles RPN expressions. I realize nobody's talking about compiling direct to native code yet, but this was just so easy to hack up using code already lying around that I couldn't resist. Perhaps some of the hackers on this list will enjoy playing with it. Darius

2 2

[ann] Sprint to Minimal Python
by holger krekel 23 Jan '03

23 Jan '03

Hi, please find enclosed the Minimal Python Sprint announcement. I post on pypy-dev first to get some feedback before sending it off to other lists... ------------------------------------------------------ Code Sprint towards Minimal Python 17th-23rd Feb. 2003 ------------------------------------------------------ Everybody is invited to join our first Minimal Python Marathon Or was it PyPython? Nevermind. We will have one or - if needed - two big rooms, beamers for presentations (and movies?), a kitchen, internet and a piano. There is a big park and some forest in case you need some fresh air. Short Ad-Hoc presentations about your area of interest, project or plain code will certainly be appreciated. ------------------------------------------------------ Goals of the first Minimal PyPython Marathon ------------------------------------------------------ - codify some ideas that were recently discussed on the pypy-dev codespeak list. - port your favorite C-module to Python (and maintain it :-) - build & enhance infrastructure (python build system, webapps, email, subversion/cvs, automated testing, ...) - rebuild the CPython-Python interpreter in simple Python, running on a minimal "transitive closure" of/in CPython. - try to write platform dependent Assembler/C-code that enables doing C/Machine-level calls from Python without extra C-bindings. - settle on concepts - focus on having some usable results at the end - have a lot of fun meeting like minded people. ------------------------------------------------------ Current Basic Agreements ------------------------------------------------------ Please note, that we have reached some agreement on a number of basic points: a) the Python language is not to be modified. b) simplicity wins. especially against optimization. c) if in doubt we follow the CPython/PEP lead. d) CPython core abstractions are to be cleanly formulated in simple Python. e) Macro-techniques may be used at some levels if there really is no good way to avoid them. But a) to c) still hold. f) our Pythonic core "specification" is intended to be used for all kinds of generational steps e.g to C, Bytecode, Objective-C, assembler and last-but-not-least PSYCO-Specialization. g) if in doubt we follow Armins and Christians lead regarding the right abstraction levels. And any other python people taking responsibility :-) h) there are a lot of wishes, code, thoughts, suggestions and ... names :-) ------------------------------------------------------ How to proceed if you are interested ------------------------------------------------------ If you are interested to come - even parttime - then please subscribe at http://codespeak.net/mailman/listinfo/pypy-sprint where organisational stuff will be communicated. Code- related discussions still take place on the pypy-dev list. For people who don't want or can't spend the money for external rooms i can probably arrange something but it will not be luxurious. btw, there are already five people who will come among them - you guessed it - Christian Tismer and Armin Rigo. ------------------------------------------------------ Disclaimer ------------------------------------------------------ There is no explicit commercial intention behind the organisation of the sprint. On the implicit hand socializing with like-minded people tends to bring up future opportunities. Especially if it's free codespeak ... ups :-) cheers, holger

2 2

Re: [pypy-dev] Notes on compiler package
by logistix 23 Jan '03

23 Jan '03

> > > P.S. If anyone can point me to some sort of definition of what the > > "Extended" in ELL(1) stands for I'd appreciate it. Python grammar is > > not LL(1) as written. > > Are you trying to use a parser generator? > > ciao - chris > Nah, that's what I already wrote one in the above stuff. But I had to backtrack to get it to work on parts of the grammar like: varargslist: (fpdef ['=' test] ',')* ('*' NAME [',' '**' NAME] | '**' NAME) | fpdef ['=' test] (',' fpdef ['=' test])* [','] The backtracking wasn't a big deal, but it shouldn't be necessary in an LL(1) grammar. The source claims that the grammar is ELL(1). I was wondering if the ELL definition addressed my backtracking issues, so that I could confirm that my assumptions about where I throw my Syntax Errors were valid. To use XML terminology, my current assumptions are only well-formed ;-)

1 0

Re: [pypy-dev] Notes on compiler package
by logistix 23 Jan '03

23 Jan '03

Christian Tismer <tismer(a)tismer.com> wrote: > > logistix wrote: > > I'm finally starting to get my head wrapped around the compiler package, > > and thought I'd pass some notes on to the group since it's a beast: > > > > 1) Other than the original AST generation, it looks like it's all pure > > python (not tying into the C source). The original AST comes from the > > parser module, which is a pure wrapper module (although I've already > > posted python code to build ast's here) > > Oh, is that true? > Last time, when I analysed what it would take to > build a compiler for Python in Python, I ended > up in the parser module being called, finally, > which called back into the internals stuff. > This was for Python 2.2.2; maybe this changed now? > > ciao - chris > > Here's the recursive decent parser I wrote to build ASTs about six months ago for 2.2 (just tested on 2.3a): http://members.bellatlantic.net/~olsongt/concrete.zip I haven't written an extensive test-suite yet (I was just screwing around when I originally wrote it), but it did successfully build a tree that could get my version of the old nines problem to compile. It definitely needs a some rewriting (stack hog from hell), but it's a proof of concept. It should replace the parser module functionality. Then if you tie that into compiler package, it looks like pure python compilation to bytecode! I was suprised to see how much the compiler module did when I finally looked at it. To test, unzip the files somewhere, adjust the hardcoded reference to "c:\python22\lib\site-packages\concrete\misc\nines.py" as appropriate, and run "python llparser.py". You'll see the results of several trees I build getting compiled and exec'ed (using the CPython compiler) I too have trouble believing it could be this easy. A quick grep of compiler shows that it only uses parser to create ast Tuples in the early phases of generation. It doesn't use the astcompile functions on the back end. If anyone more familiar with the compiler internals would like to chime in on binary dependancies I'd appreciate it. It's getting a little late tonight, but tomorrow I'll just delete the parser pyd from my 2.3 install and see if I can plug my parser into compiler. I'm also noticing some more minor problems with the compiler bytecode generation, but the more I look the more I think the framework has the first 90% of the work done. -logistix P.S. If anyone can point me to some sort of definition of what the "Extended" in ELL(1) stands for I'd appreciate it. Python grammar is not LL(1) as written. --------------------------------------- Get your free e-mail address @zworg.com

2 1

Porting Python C-Code to Python
by Thomas Fanslau 23 Jan '03

23 Jan '03

I'm reading the list for some time now ... and although I understand what you want to achieve I may not really understand what is involved in reaching that goal ... But lately I read about porting that C-Code to Python and I have a question/suggestion that may be a serious candidate for the most stupid idea :) Shouldn't it be possible to write a (small) backend to gcc to generate python-bytecode? So you can translate the C-Part of Python (or any other C-Code :) to python-bytecode and run it. That should be easier then all the other suggestions and allows you to replace the C-Code Module for Module ... --tf

2 1

focus: CPython <-> PyPython
by holger krekel 23 Jan '03

23 Jan '03

Hi folks, There have been quite some discussions about how to relate to the ever-progressing CPython development. To recap i try to formulate some agreements: a) the Python language is not to be modified b) if in doubt we follow the CPython lead wherever possible b) CPython core abstraction levels should be cleanly formulated in Python. c) Macros may be used at some levels if there really is no good way to avoid them. But a) and b) still hold. d) our Pythonic core specification is intended to be used for all kinds of generational steps (to C, Bytecode, Object-C, assembler, for PSYCO-Specilization whatever, etc). e) if in doubt i follow Armin's and Christian's lead regarding the right abstraction levels :-) e) there is all kinds of nice stuff to try f) there are a lot of wishes, thoughts, suggestions and ... names :-) There is some disagreement about how to cope with the CPython codebase and python-dev's ongoing C-development. Which isn't really problematic for the time beeing. We might have a model and some working code before PyCon (the python developers conference in Washington in March). It should be an ideal opportunity to discuss stuff. Any strong objections or agreements on this recap? regards, holger

3 2