[Python-Dev] Two laments about CPython's AST Nodes

[Frank Wierzbicki]

On Wed, Aug 19, 2009 at 5:00 PM, "Martin v. Löwis"<martin at v.loewis.de> wrote:
>> Now on to the complaints: Though I recently added support for this in
>> Jython, I don't like that nodes can be defined without required
>> attributes, for example:
>>
>> node = ast.Assign()
>
> I think we disagree in two points in our evaluation of this behavior:
>
> a) ast.Assign is *not* a node of the CPython AST. The CPython AST is
>   a set of C structures in Include/Python-ast.h. ast.Assign is merely
>   a mirror structure of that.
Ah -- that is different from Jython's current design (The nodes that
are constructed by ast.Assign() in Jython actually are the exact nodes
that are used in real parsing)

> b) it is, IMO, not reasonable to require users who create AST trees
>   out of nothing to have correct trees at all times. I.e. it must be
>   possible to represent incorrect trees.
That does seem reasonable.  Luckily it was easy to implement for me :)

> c) the AST is *not* part of the Python language or library. It may
>   change at any time without notice, and Jython is not required to
>   duplicate its behavior exactly.
Sure, I'm really just talking about ast.py (though for Jython ATM they
are the same thing).  So that I understand better: when this call is
made in Python:

x = compile("def foo():pass", "foo", "exec", _ast.PyCF_ONLY_AST)

Does x contain real AST nodes or does it contain mirror structures
(feel free to just tell me: don't be lazy, go read the code). If it
contains real nodes, this is where I have some implementation trouble.
 If a tree of real nodes is then manipulated so that you end up with a
mix, I don't want to walk the entire thing over again to find the
mirror objects (that might be incomplete) and replace them with real
nodes.  If this creates a tree of mirror nodes, then I may want to
consider doing the same thing on the Jython side (it makes sense, now
that I understand CPython better I realize that the cost I am
incurring is probably due to having the real and mirror AST as the
same beast).

> [so that's three items - as there should be in any good list of
> two items :-]
:)

> What precisely is it that makes this difficult to implement. If you
> would follow CPython's implementation strategy (i.e. generate glue
> code out of ASDL), I feel that it should be straight-forward to provide
> exactly the same behavior in Jython.
I do use the ASDL to generate this stuff, but again, the real and the
mirror nodes are not separated ATM, and that is what makes it
difficult.

>> BTW -- I *am* volunteering to attempt to implement these things in
>> CPython if there is support :)
>
> I'm not sure I can support such a change. Giving the child nodes at
> creation time, optionally, would be fine with me. Requiring the
> tree to conform to the grammar at all times is unreasonable, IMO
> (you can't simultaneously create all nodes in the tree and glue
> them together, so you have to create the tree in steps - which means
> that it will be intermittently incorrect).
That is quite reasonable, I'll withdraw gripe #1 -- in fact the reason
I have already implemented this in Jython is that there is already
real world use out there.  I still need to understand gripe #2 a
little better before I back down on that one.

-Frank