On Mon, Aug 13, 2012 at 1:05 PM, fwierzbicki@gmail.com wrote:

On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon wrote:

...

...

I see nothing about ast possibly being CPython only. Should there be?

Time to ask the other VMs what they are currently doing (the ast module came into existence in Python 2.6 so all the VMs should be answer the question since Jython is in alpha for 2.7 compatibility).

[Jython]

2.5+ contains an ast.py that I obsessively compared to CPython's 2.5 ast.py.

But CPython's ast.py contains very little code -- it's all done in ast.c. Still, I'm glad you are actually considering this a cross-language feature, and I will gladly retract my warning. (Still, I don't know if it is subject to the usual backward compatibility constraints.)

I haven't applied the same obsessiveness to 2.7, but I do intend to look closely at Jython's ast.py results compared to CPython's in the 3.x effort. Also I plan to allow some backwards compatibility compromises between early point releases of our 2.7 series, as I want to apply what I learn in our 3.x effort to 2.7 point releases, so we should be able to keep up with most simple ast.py changes. I'm not so sure that the current discussion are going to be "simple though" :) -- if it's pure python we should hopefully be alright.

It might be pure python for Jython, but it's not for CPython. -- --Guido van Rossum (python.org/~guido)

On Aug 13, 2012 5:22 PM, "fwierzbicki@gmail.com" wrote:

On Mon, Aug 13, 2012 at 1:46 PM, Guido van Rossum

wrote:

...

On Mon, Aug 13, 2012 at 1:05 PM, fwierzbicki@gmail.com wrote:

...

On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon wrote:

...

...

I see nothing about ast possibly being CPython only. Should there be?

Time to ask the other VMs what they are currently doing (the ast module came into existence in Python 2.6 so all the VMs should be answer the question since Jython is in alpha for 2.7 compatibility).

[Jython]

...

2.5+ contains an ast.py that I obsessively compared to CPython's 2.5 ast.py.

But CPython's ast.py contains very little code -- it's all done in ast.c. What I did was dump a pretty print of the ast from Jython and CPython for every file in Lib/* and diff the results with a script. I got the differences down to a small number of minor variations.

...

Still, I'm glad you are actually considering this a cross-language feature, and I will gladly retract my warning. (Still, I don't know if it is subject to the usual backward compatibility constraints.) I don't know if IronPython does the same though... we might want to wait for them to respond.

...

It might be pure python for Jython, but it's not for CPython. It's actually Java for us :) -- in fact the internal AST uses the exact Java that is exposed from our _ast.py - which I've come to regard as a mistake (though it was useful at the time). I want to do the same obsessive diff game with 3.x but then probably separate out our internal ast implementation (possibly making ast.py pure Python).

BTW - is Python's internal AST exactly exposed by ast.py or is there a separate internal AST implementation?

Direct. There is an AST grammar file that gets compiled into C and Python objects which are used by the compiler (c version) or exposed to users (Python version).

-Frank

On Mon, Aug 13, 2012 at 6:35 PM, fwierzbicki@gmail.com < fwierzbicki@gmail.com> wrote:

On Mon, Aug 13, 2012 at 3:10 PM, Brett Cannon wrote:

...

Direct. There is an AST grammar file that gets compiled into C and Python objects which are used by the compiler (c version) or exposed to users (Python version). At the risk of making you repeat yourself, and just to be sure I understand: There are C objects used by the compiler and Python objects that are exposed to the users (written in C though) that are generated by the AST grammar.

Both sets of objects are generated from the grammar. It's wrapping some C structs (the C version of the AST) in an extension module where the fields of the struct and the names of the types are all the same (the Python version) no matter if it is C or Python. Converting between the two is just a matter of allocating memory and copying data from one struct to another.

That at least sounds like they are different.

Are you asking if we pass the objects through transparently, or if they are just the same API? The are the same API since the AST nodes used by the compiler just have an extension exposing them that has the same names, fields, etc. But to expose the API to Python code the C-level objects are taken, pulled apart, and used to populate and exact API copy of them as Python object (i.e. the ast2obj_* functions defined in Python/Python-ast.c). To try to make this really clear, consider the Assign node type. At the C level it's just a struct:: struct { asdl_seq *targets; expr_ty value; } Assign; An asdl_seq is just an array of AST nodes. So converting to Python code is just a matter of allocating the equivalent Assign_type (which is a PythonTypeObject), and then populating its 'targets' attribute with a list of the AST nodes and its expr instance for its 'value' type. It's all very mechanical since it is all code-generated.

The last I checked the grammar was Python.asdl and the translater was asdl_c.py resulting in /Python/Python-ast.c which looks like it is the implementation of _ast.py

There is no _ast.py, only Lib/ast.py which just provides helper code for working with the AST (e.g. a NodeVisitor class). The builtin _ast module comes from Python/Python-ast.c.

Are the AST objects from Python-ast.c used by the compiler? And what is the relationship between Python-ast.c and /Python/ast.c? And what about the CST mentioned at the top of /Python/ast.c?

http://docs.python.org/devguide/compiler.html explains it all.

I ask all of this because I want to be sure that separating the internal AST in Jython from the one exposed in ast.py is really a good idea. If CPython does not make this distinction that will be a strike against the idea.

As I said, depends if you mean API or actual objects. The compiler itself uses C objects which are nothing more than structs and unions. The AST exposed by the _ast module uses the same names, fields, etc., but are actual Python objects instead of structs and unions. The separation allows the compiler to save on memory costs by only using structs instead of a complete PyObject struct which would have tons of stuff that the compiler doesn't need (e.g. the AST has no methods so why waste memory on PyObject allocation for method slots that will never be set?).

On 8/13/2012 4:46 PM, Guido van Rossum wrote:

On Mon, Aug 13, 2012 at 1:05 PM,fwierzbicki@gmail.com wrote:

...

...

On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon wrote:

...

...

...

>I see nothing about ast possibly being CPython only. Should there be?

Time to ask the other VMs what they are currently doing (the ast module came into existence in Python 2.6 so all the VMs should be answer the question since Jython is in alpha for 2.7 compatibility). [Jython] 2.5+ contains an ast.py that I obsessively compared to CPython's 2.5 ast.py. But CPython's ast.py contains very little code -- it's all done in ast.c.

Still, I'm glad you are actually considering this a cross-language feature, and I will gladly retract my warning. (Still, I don't know if it is subject to the usual backward compatibility constraints.)

I should have quoted a bit more. After the first sentence "The ast module helps Python applications to process trees of the Python abstract syntax grammar." the next sentence is "The abstract syntax itself might change with each Python release; this module helps to find out programmatically what the current grammar looks like." The 'current grammar' is given in 30.2.2. Abstract Grammar. -- Terry Jan Reedy