Mailman 3 January 2003 - pypy-dev

Notes on compiler package
by logistix Jan. 24, 2003

Jan. 24, 2003

I'm finally starting to get my head wrapped around the compiler package, and thought I'd pass some notes on to the group since it's a beast: 1) Other than the original AST generation, it looks like it's all pure python (not tying into the C source). The original AST comes from the parser module, which is a pure wrapper module (although I've already posted python code to build ast's here) 2) Internal CPython ASTs are transformed into "compiler"'s own ASTs. These nodes are smarter than the … [View More]internal ones; they attach appropriate info to "attributes". CPython AST nodes just have a list of children, this module identifies and breaks out important data (for example, the arguments in a function def become an "args" attribute instead of guessing it's the first child) 3) The CodeGenerators walk the ASTs and emit code into a code object. 4) Speed seems reasonable (which makes me think I'm missing some calls into the C side of things) 4) The generated bytecode looks reasonably good and execs fine, but there are still some diffences from what CPython generates. Here's what I've seen so far: SET_LINENO doesn't always work the same (this is how python knows what line in a file threw an exception) Doesn't have one of the few optimizations the CPython compiler does have. CPython will throw out misplaced "docstrings"... strings in the source that aren't assigned to anything. This throws off array indexes in the code objects co_const attribute. In general, the compiler package is in alot better shape than I expected. If anyone is interested in poking around, here's a quick script that does basic comparison and diff on the bytecode generated by the builtin compile and compiler's equivilent function. I'll probably expand this to compare the whole code objects in the near future. =============== CompilerTest.py =============== import sys import compiler import dis def opcodeTuples(source): """ Makes Opcode tuples in the form of: OFFSET, NAME, [OPTIONAL PARAM] """ retVal = [] a = iter(source) offset = 0 def getByte(next=a.next): return ord(a.next()) def getWord(next=a.next): return ord(a.next()) + ord(a.next()) * 256 # Little-endian while 1: try: opcode = getByte() opname = dis.opname[opcode] if opcode < 90: retVal.append( (offset, dis.opname[opcode]) ) offset += 1 else: retVal.append( (offset, dis.opname[opcode], getWord())) offset += 3 except StopIteration: break return retVal def opcodeDiff(ops1, ops2): """ Does a simple DIFF of two sets of opcodes. Can only check one skipped line. Ignores param for now since they don't match """ opcode1, opcode2 = opcodeTuples(ops1), opcodeTuples(ops2) a,b = 0,0 print "%30s%30s" % ("FIRST", "SECOND") print "%30s%30s" % ("====================" ,"====================") while 1: if opcode1[a][1:2] == opcode2[b][1:2]: print "%30s%30s" % (opcode1[a], opcode2[b] ), if opcode1[a][2:] != opcode2[b][2:]: print " ARG MISMATCH" else: print a += 1 b += 1 elif opcode1[a+1][1:2] == opcode2[b][1:2]: print "%30s%30s" % (opcode1[a], "<not here>") a += 1 elif opcode1[a][1:2] == opcode2[b+1][1:2]: print "%30s%30s" % ("<not here>", opcode2[b]) b += 1 else: print "NONTRIVIAL DIFF%25s%25s" % (opcode1[a], opcode2[b]) break if a >= len(opcode1) and b >= len(opcode2): break elif a >= len(opcode1) or b >= len(opcode2): print "UNEXPECTED END OF OPCODES" break def compareCompiles(filename, nativeCompile=compile, pythonCompile=compiler.compile): """ Compares a bytecode compile between the native python compiler and the one written in python """ source = file(filename).read() native = nativeCompile(source, filename, "exec") python = pythonCompile(source, filename, "exec") if native.co_code == python.co_code: print "compiles matched" else: print "compiles didn't match" opcodeDiff(native.co_code, python.co_code) if __name__ == "__main__": compareCompiles("c:\\python23\\lib\\code.py") =============== End CompilerTest.py =============== --------------------------------------- Get your free e-mail address @zworg.com [View Less]

9 11

RE: Interfacing Python Code with Autogenerated C code
by roccomoretti＠netscape.net Jan. 24, 2003

Jan. 24, 2003

"Gerald S. Williams" <gsw(a)agere.com> wrote: >Rocco Moretti wrote: >> How would the interface between the two work if the hand translated sections differ >> from the C version? > >I can give general answers from past experience. > >> Simplest example is basic types. A Python version would presumably be represented as a >> class with member functions. But the C code uses calls to accessor functions. How could >> we redirect the automatically … [View More]

1 0

RE: 300,000 lines of C
by Hickey,Thom Jan. 23, 2003

Jan. 23, 2003

Does anyone else have experience with TeX? Knuth wrote it in Pascal (I was there when he was considering whether Pascal or C would be the right choice), but most implementations run as C. Someone wrote a set of support routines plus a translator for the very restricted Pascal that Knuth used. Local modifications to the code are almost all handled by substituting new Pascal code which is incorporated by a preprocessor before being translated into C. Really works pretty well, and some of the … [View More]

2 1

Re: [pypy-dev] Re: Notes on compiler package
by logistix Jan. 23, 2003

Jan. 23, 2003

Michael Hudson <mwh(a)python.net> wrote: > > > In general, the compiler package is in alot better shape than I expected. > > What were you expecting? > > Cheers, > M. > I don't know. I guess in general I expect nothing, and this looks like something ;) BTW, there's a note in compiler.ast that says it's autogenerated. Is this still true? If so, how?

2 1

Miasma: x86 machine code generation in Python
by Darius Bacon Jan. 23, 2003

Jan. 23, 2003

I've just written a Python module to emit x86 machine code using assembler-like function calls, plus another module in C to invoke the generated code: http://accesscom.com/~darius/software/miasma/ It comes with a little test program that compiles RPN expressions. I realize nobody's talking about compiling direct to native code yet, but this was just so easy to hack up using code already lying around that I couldn't resist. Perhaps some of the hackers on this list will enjoy playing with it. Darius

2 2

[ann] Sprint to Minimal Python
by holger krekel Jan. 23, 2003

Jan. 23, 2003

Hi, please find enclosed the Minimal Python Sprint announcement. I post on pypy-dev first to get some feedback before sending it off to other lists... ------------------------------------------------------ Code Sprint towards Minimal Python 17th-23rd Feb. 2003 ------------------------------------------------------ Everybody is invited to join our first Minimal Python Marathon Or was it PyPython? Nevermind. We will have one or - if needed - two big rooms, beamers for … [View More]presentations (and movies?), a kitchen, internet and a piano. There is a big park and some forest in case you need some fresh air. Short Ad-Hoc presentations about your area of interest, project or plain code will certainly be appreciated. ------------------------------------------------------ Goals of the first Minimal PyPython Marathon ------------------------------------------------------ - codify some ideas that were recently discussed on the pypy-dev codespeak list. - port your favorite C-module to Python (and maintain it :-) - build & enhance infrastructure (python build system, webapps, email, subversion/cvs, automated testing, ...) - rebuild the CPython-Python interpreter in simple Python, running on a minimal "transitive closure" of/in CPython. - try to write platform dependent Assembler/C-code that enables doing C/Machine-level calls from Python without extra C-bindings. - settle on concepts - focus on having some usable results at the end - have a lot of fun meeting like minded people. ------------------------------------------------------ Current Basic Agreements ------------------------------------------------------ Please note, that we have reached some agreement on a number of basic points: a) the Python language is not to be modified. b) simplicity wins. especially against optimization. c) if in doubt we follow the CPython/PEP lead. d) CPython core abstractions are to be cleanly formulated in simple Python. e) Macro-techniques may be used at some levels if there really is no good way to avoid them. But a) to c) still hold. f) our Pythonic core "specification" is intended to be used for all kinds of generational steps e.g to C, Bytecode, Objective-C, assembler and last-but-not-least PSYCO-Specialization. g) if in doubt we follow Armins and Christians lead regarding the right abstraction levels. And any other python people taking responsibility :-) h) there are a lot of wishes, code, thoughts, suggestions and ... names :-) ------------------------------------------------------ How to proceed if you are interested ------------------------------------------------------ If you are interested to come - even parttime - then please subscribe at http://codespeak.net/mailman/listinfo/pypy-sprint where organisational stuff will be communicated. Code- related discussions still take place on the pypy-dev list. For people who don't want or can't spend the money for external rooms i can probably arrange something but it will not be luxurious. btw, there are already five people who will come among them - you guessed it - Christian Tismer and Armin Rigo. ------------------------------------------------------ Disclaimer ------------------------------------------------------ There is no explicit commercial intention behind the organisation of the sprint. On the implicit hand socializing with like-minded people tends to bring up future opportunities. Especially if it's free codespeak ... ups :-) cheers, holger [View Less]

2 2

Re: [pypy-dev] Notes on compiler package
by logistix Jan. 23, 2003

Jan. 23, 2003

> > > P.S. If anyone can point me to some sort of definition of what the > > "Extended" in ELL(1) stands for I'd appreciate it. Python grammar is > > not LL(1) as written. > > Are you trying to use a parser generator? > > ciao - chris > Nah, that's what I already wrote one in the above stuff. But I had to backtrack to get it to work on parts of the grammar like: varargslist: (fpdef ['=' test] ',')* ('*' NAME [',' '**' NAME] | '**' NAME) | fpdef ['=' … [View More]

1 0

Re: [pypy-dev] Notes on compiler package
by logistix Jan. 23, 2003

Jan. 23, 2003

Christian Tismer <tismer(a)tismer.com> wrote: > > logistix wrote: > > I'm finally starting to get my head wrapped around the compiler package, > > and thought I'd pass some notes on to the group since it's a beast: > > > > 1) Other than the original AST generation, it looks like it's all pure > > python (not tying into the C source). The original AST comes from the > > parser module, which is a pure wrapper module (although I've already > &… [View More]gt; posted python code to build ast's here) > > Oh, is that true? > Last time, when I analysed what it would take to > build a compiler for Python in Python, I ended > up in the parser module being called, finally, > which called back into the internals stuff. > This was for Python 2.2.2; maybe this changed now? > > ciao - chris > > Here's the recursive decent parser I wrote to build ASTs about six months ago for 2.2 (just tested on 2.3a): http://members.bellatlantic.net/~olsongt/concrete.zip I haven't written an extensive test-suite yet (I was just screwing around when I originally wrote it), but it did successfully build a tree that could get my version of the old nines problem to compile. It definitely needs a some rewriting (stack hog from hell), but it's a proof of concept. It should replace the parser module functionality. Then if you tie that into compiler package, it looks like pure python compilation to bytecode! I was suprised to see how much the compiler module did when I finally looked at it. To test, unzip the files somewhere, adjust the hardcoded reference to "c:\python22\lib\site-packages\concrete\misc\nines.py" as appropriate, and run "python llparser.py". You'll see the results of several trees I build getting compiled and exec'ed (using the CPython compiler) I too have trouble believing it could be this easy. A quick grep of compiler shows that it only uses parser to create ast Tuples in the early phases of generation. It doesn't use the astcompile functions on the back end. If anyone more familiar with the compiler internals would like to chime in on binary dependancies I'd appreciate it. It's getting a little late tonight, but tomorrow I'll just delete the parser pyd from my 2.3 install and see if I can plug my parser into compiler. I'm also noticing some more minor problems with the compiler bytecode generation, but the more I look the more I think the framework has the first 90% of the work done. -logistix P.S. If anyone can point me to some sort of definition of what the "Extended" in ELL(1) stands for I'd appreciate it. Python grammar is not LL(1) as written. --------------------------------------- Get your free e-mail address @zworg.com [View Less]

2 1

Porting Python C-Code to Python
by Thomas Fanslau Jan. 22, 2003

Jan. 22, 2003

I'm reading the list for some time now ... and although I understand what you want to achieve I may not really understand what is involved in reaching that goal ... But lately I read about porting that C-Code to Python and I have a question/suggestion that may be a serious candidate for the most stupid idea :) Shouldn't it be possible to write a (small) backend to gcc to generate python-bytecode? So you can translate the C-Part of Python (or any other C-Code :) to python-bytecode and … [View More]

2 1

focus: CPython <-> PyPython
by holger krekel Jan. 22, 2003

Jan. 22, 2003

Hi folks, There have been quite some discussions about how to relate to the ever-progressing CPython development. To recap i try to formulate some agreements: a) the Python language is not to be modified b) if in doubt we follow the CPython lead wherever possible b) CPython core abstraction levels should be cleanly formulated in Python. c) Macros may be used at some levels if there really is no good way to avoid them. But a) and b) still hold.… [View More]

3 2