[Python-Dev] Two laments about CPython's AST Nodes

[Brett Cannon]

On Thu, Aug 20, 2009 at 06:20, Frank Wierzbicki<fwierzbicki at gmail.com> wrote:
> On Wed, Aug 19, 2009 at 5:00 PM, "Martin v. Löwis"<martin at v.loewis.de> wrote:
>>> Now on to the complaints: Though I recently added support for this in
>>> Jython, I don't like that nodes can be defined without required
>>> attributes, for example:
>>>
>>> node = ast.Assign()
>>
>> I think we disagree in two points in our evaluation of this behavior:
>>
>> a) ast.Assign is *not* a node of the CPython AST. The CPython AST is
>>   a set of C structures in Include/Python-ast.h. ast.Assign is merely
>>   a mirror structure of that.
> Ah -- that is different from Jython's current design (The nodes that
> are constructed by ast.Assign() in Jython actually are the exact nodes
> that are used in real parsing)
>
>> b) it is, IMO, not reasonable to require users who create AST trees
>>   out of nothing to have correct trees at all times. I.e. it must be
>>   possible to represent incorrect trees.
> That does seem reasonable.  Luckily it was easy to implement for me :)
>
>> c) the AST is *not* part of the Python language or library. It may
>>   change at any time without notice, and Jython is not required to
>>   duplicate its behavior exactly.
> Sure, I'm really just talking about ast.py (though for Jython ATM they
> are the same thing).  So that I understand better: when this call is
> made in Python:
>
> x = compile("def foo():pass", "foo", "exec", _ast.PyCF_ONLY_AST)
>
> Does x contain real AST nodes or does it contain mirror structures
> (feel free to just tell me: don't be lazy, go read the code). If it
> contains real nodes, this is where I have some implementation trouble.
>  If a tree of real nodes is then manipulated so that you end up with a
> mix, I don't want to walk the entire thing over again to find the
> mirror objects (that might be incomplete) and replace them with real
> nodes.  If this creates a tree of mirror nodes, then I may want to
> consider doing the same thing on the Jython side (it makes sense, now
> that I understand CPython better I realize that the cost I am
> incurring is probably due to having the real and mirror AST as the
> same beast).
>
I am using PEP 339 to help me with this.

Looking at Python/bltinmodule.c:builtin_compile() you will notice that
when you take an AST and compile it the call goes to
Python/Python-ast.c:PyAST_obj2mod(). That function calls obj2ast_mod()
which turns a ast.Module object into an mod_ty (defined in
Python/Python-ast.c). The reverse is done with ast2obj_mod() when you
get the AST out. And looking at those conversion functions it seems
that the PyObject values convert as needed or reuse constants like
ints and strings (see obj2ast_unaryop() to see a conversion from
object AST to internal AST).

But I would double-check me on all of this. =)

-Brett

>> [so that's three items - as there should be in any good list of
>> two items :-]
> :)
>
>> What precisely is it that makes this difficult to implement. If you
>> would follow CPython's implementation strategy (i.e. generate glue
>> code out of ASDL), I feel that it should be straight-forward to provide
>> exactly the same behavior in Jython.
> I do use the ASDL to generate this stuff, but again, the real and the
> mirror nodes are not separated ATM, and that is what makes it
> difficult.
>
>>> BTW -- I *am* volunteering to attempt to implement these things in
>>> CPython if there is support :)
>>
>> I'm not sure I can support such a change. Giving the child nodes at
>> creation time, optionally, would be fine with me. Requiring the
>> tree to conform to the grammar at all times is unreasonable, IMO
>> (you can't simultaneously create all nodes in the tree and glue
>> them together, so you have to create the tree in steps - which means
>> that it will be intermittently incorrect).
> That is quite reasonable, I'll withdraw gripe #1 -- in fact the reason
> I have already implemented this in Jython is that there is already
> real world use out there.  I still need to understand gripe #2 a
> little better before I back down on that one.
>
> -Frank
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>