From jeremy@zope.com Tue Mar 19 05:43:50 2002 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 19 Mar 2002 00:43:50 -0500 Subject: [Compiler-sig] test Message-ID: From jeremy@zope.com Tue Mar 19 05:45:28 2002 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 19 Mar 2002 00:45:28 -0500 Subject: [Compiler-sig] compiler-sig project for Python 2.3: new AST Message-ID: These are notes copied from a Wiki at http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/PythonAST They are a bit rough, but they set out some of the work I'd like to do for Python 2.3. Some of the work belongs in a PEP, but I'm not ready to write it yet. Please reply to compiler-sig@python.org. Jeremy A New AST for Python -------------------- I have proposed some namespace optimizations for Python 2.3. These optimizations require more analysis by the compiler, which is quite difficult given the current parse tree representation. As a first step towards those optimizations, I plan to introduce a new AST that makes analysis easier. This step is a major undertaking by itself. The primary goal of the new AST is to provide better intermediate representation(s) so that the compiler is easier to maintain and extend. One benefit is to enable optimizations by providing better tools for analyzing programs during bytecode compiling. I expect the new AST will be based largely on the one in the compiler package (part of the std library in 2.2), which was originally done by Greg Stein and Bill Tutt. Rough plan of action -------------------- -- Define the AST. Settle on C and Python (and Java?) implementations of AST. I like the look of the C code generated by asdlGen, but haven't had a chance to get the tool working. See Dan Wang's DSL 97 paper: The Zephyr Abstract Syntax Description Language. Could always write new asdlGen tool for targetted languages -- Write converter from current parse tree to AST. Basic functionality of compiler/transformer.py. Replace parser module. WingIDE? folks have implemented something like this in C. parsetools -- Expose AST to Python programs Replace parser module and parts of compiler package. asdlGen has a notion of AST pickles. Would that be sufficient? -- An AST visitor in C? The visitor pattern is very useful for manipulating ASTs? in Python. Could it be translated to C somehow? -- Reimplement compile.c with new AST Break it up into several passes: future stmts, doc strings, symbol table, code gen. Not sure how much of the code in the compiler package may be useful here. The codegen part is relatively good, the pyassem part is pretty ugly. Testing ------- I overhauled the compiler for Python 2.1 to implement nested scopes. I spent a lot of time debugging the various compiler passes, usually because I forgot to handle a particular kind of grammar production. The new AST should make this a lot easier, but I would also like to see a thorough test suite for the new code. I don't know anything about best practices for testing compilers. I imagine that a system to generate sample inputs from the language grammar would be helpful for debugging the C code that walks the AST. From skip@pobox.com Tue Mar 19 06:16:26 2002 From: skip@pobox.com (Skip Montanaro) Date: Tue, 19 Mar 2002 00:16:26 -0600 Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST In-Reply-To: References: Message-ID: <15510.55226.773263.367118@12-248-41-177.client.attbi.com> Naive question, I'm sure, but why do all this in C? Why not do the compiler completely in Python and only implement pieces in C that need speeding up? You already have a byte code compiler written in Python, correct? How bad is it performance-wise? Given the fact that .py[co] files are there to cache the results, exec and eval are seldom used, and the response time of interactive sessions is dominated by the user's typing speed, I would think a compiler written in Python would be fast enough. All you need is the bootstrap, and that can probably be handled by distributing .pyc files for the compiler with the standard distribution. Skip From jeremy@zope.com Wed Mar 20 23:27:05 2002 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 20 Mar 2002 18:27:05 -0500 Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST Message-ID: > Naive question, I'm sure, but why do all this in C? Why > not do the compiler completely in Python and only > implement pieces in C that need speeding up? You already > have a byte code compiler written in Python, correct? > How bad is it performance-wise? Really bad . I haven't done a benchmark in a long time, but I recall that the compiler-in-Python is at least 100x slower than compiler-in-C. This makes starting up the interpreter to execute a small script quite slow. IIRC the Python .NET compiler is implemented in Python and its slowness was cited as a serious drawback in the summary paper available from the AS web site. Jeremy From skip@pobox.com Thu Mar 21 01:14:18 2002 From: skip@pobox.com (Skip Montanaro) Date: Wed, 20 Mar 2002 19:14:18 -0600 Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST In-Reply-To: References: Message-ID: <15513.13291.13094.885698@12-248-41-177.client.attbi.com> >> Naive question, I'm sure, but why do all this in C? Why not do the >> compiler completely in Python and only implement pieces in C that >> need speeding up? You already have a byte code compiler written in >> Python, correct? How bad is it performance-wise? Jeremy> Really bad . Jeremy> I haven't done a benchmark in a long time, but I recall that the Jeremy> compiler-in-Python is at least 100x slower than compiler-in-C. Jeremy> This makes starting up the interpreter to execute a small script Jeremy> quite slow. Okay, so let's see what hotshot says... Is it obvious how I run the Python-based compiler (e.g., is there a main program script somewhere)? Skip From jeremy@zope.com Thu Mar 21 05:37:32 2002 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 21 Mar 2002 00:37:32 -0500 Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST In-Reply-To: <15513.13291.13094.885698@12-248-41-177.client.attbi.com> Message-ID: On Wed, 20 Mar 2002 19:14:18 -0600 Skip Montanaro wrote: > Jeremy> I haven't done a benchmark in a long time, > but I recall that the > Jeremy> compiler-in-Python is at least 100x slower > than compiler-in-C. > Jeremy> This makes starting up the interpreter to > execute a small script > Jeremy> quite slow. > > Okay, so let's see what hotshot says... Is it obvious > how I run the > Python-based compiler (e.g., is there a main program > script somewhere)? Try "pydoc compiler" for the details. You could compare the performance of compiler.compile() against builtin compile(). I seem to recall that about 1/3 of the time was spent transforming the parser module output into the AST. Jeremy From neal@metaslash.com Fri Mar 22 23:42:07 2002 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 22 Mar 2002 18:42:07 -0500 Subject: [Compiler-sig] Moving Forward Message-ID: <3C9BC14F.B57DF702@metaslash.com> I think Jeremy's approach http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/PythonAST is a good starting point. So what is the next step? Defining an AST? It would be nice if we can make the AST the same for CPython and Jython. Neal From jeremy@zope.com Fri Mar 22 23:53:58 2002 From: jeremy@zope.com (Jeremy Hylton) Date: Fri, 22 Mar 2002 18:53:58 -0500 Subject: [Compiler-sig] Moving Forward In-Reply-To: <3C9BC14F.B57DF702@metaslash.com> References: <3C9BC14F.B57DF702@metaslash.com> Message-ID: <15515.50198.957387.571759@slothrop.zope.com> >>>>> "NN" == Neal Norwitz writes: NN> I think Jeremy's approach NN> http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/PythonAST NN> is a good starting point. Let's start there, then ;-). NN> So what is the next step? Defining an AST? Yes. I'd like to try to convert the compiler package's AST into something in the declaration language of ASDL. I tried to download and build the ASDL tool, but failed. Not sure if something was wrong with my sml/nj install or what. I've been meaning to get back to Dan Wang with questions, but have been busy with other things. The language for ASDL is probably good to use, even if we don't use asdlGen. We could write our own tool to generate C, Java, Python, etc. from the definitions. Or translate by hand... NN> It would be nice if we can make the AST the same for CPython and NN> Jython. Yes. Jeremy From neal@metaslash.com Sat Mar 23 00:04:25 2002 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 22 Mar 2002 19:04:25 -0500 Subject: [Compiler-sig] Moving Forward References: <3C9BC14F.B57DF702@metaslash.com> <15515.50198.957387.571759@slothrop.zope.com> Message-ID: <3C9BC689.BEA65C9B@metaslash.com> Jeremy Hylton wrote: > > >>>>> "NN" == Neal Norwitz writes: > > NN> So what is the next step? Defining an AST? > > Yes. I'd like to try to convert the compiler package's AST into > something in the declaration language of ASDL. > > I tried to download and build the ASDL tool, but failed. Not sure if > something was wrong with my sml/nj install or what. I've been meaning > to get back to Dan Wang with questions, but have been busy with other > things. Is this the asdl on SF? Looks like everything is pretty old. No actual releases. If you can provide a link, I'll try to see if I can get it to build. Neal From jeremy@zope.com Sat Mar 23 00:06:26 2002 From: jeremy@zope.com (Jeremy Hylton) Date: Fri, 22 Mar 2002 19:06:26 -0500 Subject: [Compiler-sig] Moving Forward In-Reply-To: <3C9BC689.BEA65C9B@metaslash.com> References: <3C9BC14F.B57DF702@metaslash.com> <15515.50198.957387.571759@slothrop.zope.com> <3C9BC689.BEA65C9B@metaslash.com> Message-ID: <15515.50946.822588.557653@slothrop.zope.com> >>>>> "NN" == Neal Norwitz writes: NN> Is this the asdl on SF? Looks like everything is pretty old. NN> No actual releases. If you can provide a link, I'll try to see NN> if I can get it to build. That's the one. Dan said he thought it should work. Jeremy From neal@metaslash.com Sat Mar 23 00:25:42 2002 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 22 Mar 2002 19:25:42 -0500 Subject: [Compiler-sig] Moving Forward References: <3C9BC14F.B57DF702@metaslash.com> <15515.50198.957387.571759@slothrop.zope.com> <3C9BC689.BEA65C9B@metaslash.com> <15515.50946.822588.557653@slothrop.zope.com> Message-ID: <3C9BCB86.8ED7979B@metaslash.com> Jeremy Hylton wrote: > > >>>>> "NN" == Neal Norwitz writes: > > NN> Is this the asdl on SF? Looks like everything is pretty old. > NN> No actual releases. If you can provide a link, I'll try to see > NN> if I can get it to build. > > That's the one. Dan said he thought it should work. I got SML (rpm for linux): ftp://ftp.research.bell-labs.com/dist/smlnj/release/110/RPMS/smlnj-110.0.7-4.i386.rpm Got CVS version cd asdlGen ./configure make ...and now I'm clueless. :-) But it built no problem (redhat 7.2). Maybe later this weekend I'll have time to really look into using it... Neal