From jeremy@zope.com Tue Mar 19 05:43:50 2002
From: jeremy@zope.com (Jeremy Hylton)
Date: Tue, 19 Mar 2002 00:43:50 -0500
Subject: [Compiler-sig] test
Message-ID: <web-3776031@digicool.com>
From jeremy@zope.com Tue Mar 19 05:45:28 2002
From: jeremy@zope.com (Jeremy Hylton)
Date: Tue, 19 Mar 2002 00:45:28 -0500
Subject: [Compiler-sig] compiler-sig project for Python 2.3: new AST
Message-ID: <web-3776144@digicool.com>
These are notes copied from a Wiki at
http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/PythonAST
They are a bit rough, but they set out some of the work I'd
like to do
for Python 2.3. Some of the work belongs in a PEP, but I'm
not ready
to write it yet.
Please reply to compiler-sig@python.org.
Jeremy
A New AST for Python
--------------------
I have proposed some namespace optimizations for Python 2.3.
These
optimizations require more analysis by the compiler, which
is quite
difficult given the current parse tree representation. As a
first step
towards those optimizations, I plan to introduce a new AST
that makes
analysis easier. This step is a major undertaking by itself.
The primary goal of the new AST is to provide better
intermediate
representation(s) so that the compiler is easier to maintain
and
extend. One benefit is to enable optimizations by providing
better
tools for analyzing programs during bytecode compiling.
I expect the new AST will be based largely on the one in the
compiler
package (part of the std library in 2.2), which was
originally done by
Greg Stein and Bill Tutt.
Rough plan of action
--------------------
-- Define the AST.
Settle on C and Python (and Java?) implementations of AST.
I like the look of the C code generated by asdlGen, but
haven't had a
chance to get the tool working. See Dan Wang's DSL 97 paper:
The
Zephyr Abstract Syntax Description Language.
Could always write new asdlGen tool for targetted languages
-- Write converter from current parse tree to AST.
Basic functionality of compiler/transformer.py.
Replace parser module.
WingIDE? folks have implemented something like this in C.
parsetools
-- Expose AST to Python programs
Replace parser module and parts of compiler package.
asdlGen has a notion of AST pickles. Would that be
sufficient?
-- An AST visitor in C?
The visitor pattern is very useful for manipulating ASTs? in
Python. Could it be translated to C somehow?
-- Reimplement compile.c with new AST
Break it up into several passes: future stmts, doc strings,
symbol
table, code gen.
Not sure how much of the code in the compiler package may be
useful
here. The codegen part is relatively good, the pyassem part
is pretty
ugly.
Testing
-------
I overhauled the compiler for Python 2.1 to implement nested
scopes. I
spent a lot of time debugging the various compiler passes,
usually
because I forgot to handle a particular kind of grammar
production. The new AST should make this a lot easier, but I
would
also like to see a thorough test suite for the new code.
I don't know anything about best practices for testing
compilers. I
imagine that a system to generate sample inputs from the
language
grammar would be helpful for debugging the C code that walks
the AST.
From skip@pobox.com Tue Mar 19 06:16:26 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 19 Mar 2002 00:16:26 -0600
Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST
In-Reply-To: <web-3776144@digicool.com>
References: <web-3776144@digicool.com>
Message-ID: <15510.55226.773263.367118@12-248-41-177.client.attbi.com>
Naive question, I'm sure, but why do all this in C? Why not do the compiler
completely in Python and only implement pieces in C that need speeding up?
You already have a byte code compiler written in Python, correct? How bad
is it performance-wise? Given the fact that .py[co] files are there to
cache the results, exec and eval are seldom used, and the response time of
interactive sessions is dominated by the user's typing speed, I would think
a compiler written in Python would be fast enough. All you need is the
bootstrap, and that can probably be handled by distributing .pyc files for
the compiler with the standard distribution.
Skip
From jeremy@zope.com Wed Mar 20 23:27:05 2002
From: jeremy@zope.com (Jeremy Hylton)
Date: Wed, 20 Mar 2002 18:27:05 -0500
Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST
Message-ID: <web-3789622@digicool.com>
> Naive question, I'm sure, but why do all this in C? Why
> not do the compiler completely in Python and only
> implement pieces in C that need speeding up? You already
> have a byte code compiler written in Python, correct?
> How bad is it performance-wise?
Really bad <wink>.
I haven't done a benchmark in a long time, but I recall that
the compiler-in-Python is at least 100x slower than
compiler-in-C. This makes starting up the interpreter to
execute a small script quite slow.
IIRC the Python .NET compiler is implemented in Python and
its slowness was cited as a serious drawback in the summary
paper available from the AS web site.
Jeremy
From skip@pobox.com Thu Mar 21 01:14:18 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 20 Mar 2002 19:14:18 -0600
Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST
In-Reply-To: <web-3789622@digicool.com>
References: <web-3789622@digicool.com>
Message-ID: <15513.13291.13094.885698@12-248-41-177.client.attbi.com>
>> Naive question, I'm sure, but why do all this in C? Why not do the
>> compiler completely in Python and only implement pieces in C that
>> need speeding up? You already have a byte code compiler written in
>> Python, correct? How bad is it performance-wise?
Jeremy> Really bad <wink>.
Jeremy> I haven't done a benchmark in a long time, but I recall that the
Jeremy> compiler-in-Python is at least 100x slower than compiler-in-C.
Jeremy> This makes starting up the interpreter to execute a small script
Jeremy> quite slow.
Okay, so let's see what hotshot says... Is it obvious how I run the
Python-based compiler (e.g., is there a main program script somewhere)?
Skip
From jeremy@zope.com Thu Mar 21 05:37:32 2002
From: jeremy@zope.com (Jeremy Hylton)
Date: Thu, 21 Mar 2002 00:37:32 -0500
Subject: [Compiler-sig] Re: [Python-Dev] compiler-sig project for Python 2.3: new AST
In-Reply-To: <15513.13291.13094.885698@12-248-41-177.client.attbi.com>
Message-ID: <web-3791342@digicool.com>
On Wed, 20 Mar 2002 19:14:18 -0600
Skip Montanaro <skip@pobox.com> wrote:
> Jeremy> I haven't done a benchmark in a long time,
> but I recall that the
> Jeremy> compiler-in-Python is at least 100x slower
> than compiler-in-C.
> Jeremy> This makes starting up the interpreter to
> execute a small script
> Jeremy> quite slow.
>
> Okay, so let's see what hotshot says... Is it obvious
> how I run the
> Python-based compiler (e.g., is there a main program
> script somewhere)?
Try "pydoc compiler" for the details. You could compare the
performance of compiler.compile() against builtin compile().
I seem to recall that about 1/3 of the time was spent
transforming the parser module output into the AST.
Jeremy
From neal@metaslash.com Fri Mar 22 23:42:07 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Fri, 22 Mar 2002 18:42:07 -0500
Subject: [Compiler-sig] Moving Forward
Message-ID: <3C9BC14F.B57DF702@metaslash.com>
I think Jeremy's approach
http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/PythonAST
is a good starting point.
So what is the next step? Defining an AST?
It would be nice if we can make the AST the same for CPython and Jython.
Neal
From jeremy@zope.com Fri Mar 22 23:53:58 2002
From: jeremy@zope.com (Jeremy Hylton)
Date: Fri, 22 Mar 2002 18:53:58 -0500
Subject: [Compiler-sig] Moving Forward
In-Reply-To: <3C9BC14F.B57DF702@metaslash.com>
References: <3C9BC14F.B57DF702@metaslash.com>
Message-ID: <15515.50198.957387.571759@slothrop.zope.com>
>>>>> "NN" == Neal Norwitz <neal@metaslash.com> writes:
NN> I think Jeremy's approach
NN> http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/PythonAST
NN> is a good starting point.
Let's start there, then ;-).
NN> So what is the next step? Defining an AST?
Yes. I'd like to try to convert the compiler package's AST into
something in the declaration language of ASDL.
I tried to download and build the ASDL tool, but failed. Not sure if
something was wrong with my sml/nj install or what. I've been meaning
to get back to Dan Wang with questions, but have been busy with other
things.
The language for ASDL is probably good to use, even if we don't use
asdlGen. We could write our own tool to generate C, Java, Python,
etc. from the definitions. Or translate by hand...
NN> It would be nice if we can make the AST the same for CPython and
NN> Jython.
Yes.
Jeremy
From neal@metaslash.com Sat Mar 23 00:04:25 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Fri, 22 Mar 2002 19:04:25 -0500
Subject: [Compiler-sig] Moving Forward
References: <3C9BC14F.B57DF702@metaslash.com> <15515.50198.957387.571759@slothrop.zope.com>
Message-ID: <3C9BC689.BEA65C9B@metaslash.com>
Jeremy Hylton wrote:
>
> >>>>> "NN" == Neal Norwitz <neal@metaslash.com> writes:
>
> NN> So what is the next step? Defining an AST?
>
> Yes. I'd like to try to convert the compiler package's AST into
> something in the declaration language of ASDL.
>
> I tried to download and build the ASDL tool, but failed. Not sure if
> something was wrong with my sml/nj install or what. I've been meaning
> to get back to Dan Wang with questions, but have been busy with other
> things.
Is this the asdl on SF? Looks like everything is pretty old.
No actual releases. If you can provide a link, I'll try to
see if I can get it to build.
Neal
From jeremy@zope.com Sat Mar 23 00:06:26 2002
From: jeremy@zope.com (Jeremy Hylton)
Date: Fri, 22 Mar 2002 19:06:26 -0500
Subject: [Compiler-sig] Moving Forward
In-Reply-To: <3C9BC689.BEA65C9B@metaslash.com>
References: <3C9BC14F.B57DF702@metaslash.com>
<15515.50198.957387.571759@slothrop.zope.com>
<3C9BC689.BEA65C9B@metaslash.com>
Message-ID: <15515.50946.822588.557653@slothrop.zope.com>
>>>>> "NN" == Neal Norwitz <neal@metaslash.com> writes:
NN> Is this the asdl on SF? Looks like everything is pretty old.
NN> No actual releases. If you can provide a link, I'll try to see
NN> if I can get it to build.
That's the one. Dan said he thought it should work.
Jeremy
From neal@metaslash.com Sat Mar 23 00:25:42 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Fri, 22 Mar 2002 19:25:42 -0500
Subject: [Compiler-sig] Moving Forward
References: <3C9BC14F.B57DF702@metaslash.com>
<15515.50198.957387.571759@slothrop.zope.com>
<3C9BC689.BEA65C9B@metaslash.com> <15515.50946.822588.557653@slothrop.zope.com>
Message-ID: <3C9BCB86.8ED7979B@metaslash.com>
Jeremy Hylton wrote:
>
> >>>>> "NN" == Neal Norwitz <neal@metaslash.com> writes:
>
> NN> Is this the asdl on SF? Looks like everything is pretty old.
> NN> No actual releases. If you can provide a link, I'll try to see
> NN> if I can get it to build.
>
> That's the one. Dan said he thought it should work.
I got SML (rpm for linux): ftp://ftp.research.bell-labs.com/dist/smlnj/release/110/RPMS/smlnj-110.0.7-4.i386.rpm
Got CVS version
cd asdlGen
./configure
make
...and now I'm clueless. :-)
But it built no problem (redhat 7.2).
Maybe later this weekend I'll have time to really look into using it...
Neal