Hi personally I have the following concerns about PEP 269: - if it's purpose is to offer a framework for small languages support, there are already modules around that support that (SPARK, PLY ...), the only advantage of PEP 269 being speed wrt to the pure python solutions, because of the use of the internal CPython parser, OTOH the other solutions are more flexible... - or if's purpose is to help experimenting with the grammar unless support for adding keywords is added is a quite unfinished tool. Further the PEP propose to use the actual AST format of parser module as output format. To be honest that format is quite awful, especially for general purpose use. It should be considered that Jython does not contain a parser similar to CPython one. Because of this jython does not offer parser module support. So implementing the PEP for Jython would require writing a Java or pure python equivalent of the CPython parser. My plans for resolving the lack of parser module support were to to implement an higher compatibility layer based on the AST format of tools/compiler, a more nicer format. PEP 269 adds issues to this open problem, which I would like to see addressed by future revisions and by further discussions. I can live with PEP 269 implemented only for CPython, for a lack of resources on Jython side, if is to be used for rare experimenting with the grammar. But it seems, as it is, a rather half-cooked solution to offer a module for mini language support in the standard library. regards, Samuele Pedroni.
From: Jonathan Riehl
To: Martin von Loewis cc: , MIME-Version: 1.0 Subject: [Types-sig] Re: PEP 269 X-BeenThere: types-sig@python.org X-Mailman-Version: 2.0.6 (101270) List-Help: mailto:types-sig-request@python.org?subject=help List-Post: mailto:types-sig@python.org List-Subscribe: http://mail.python.org/mailman/listinfo/types-sig, mailto:types-sig-request@python.org?subject=subscribe List-Id: Special Interest Group on the Python type system List-Unsubscribe: http://mail.python.org/mailman/listinfo/types-sig, mailto:types-sig-request@python.org?subject=unsubscribe List-Archive: http://mail.python.org/pipermail/types-sig/ Date: Thu, 13 Sep 2001 14:49:32 -0500 (CDT) Howdy all, I'm afraid Martin's attention to the PEP list has outted me before I was able to post about this myself. Anyway, for those interested, I wrote a PEP for the exposure of pgen to the Python interpreter. You may view it at:
http://python.sourceforge.net/peps/pep-0269.html
I am looking for comments on this PEP, and below, I address some interesting issues raised by Martin. Furthermore, I already have a parially functioning reference implementation, and should be pestered to make it available shortly.
Thanks, -Jon
On Tue, 11 Sep 2001, Martin von Loewis wrote:
Hi Jonathan,
With interest I noticed your proposal to include Pgen into the standard library. I'm not sure about the scope of the proposed change: Do you view pgen as a candidate for a general-purpose parser toolkit, or do you "just" contemplate using that for variations of the Python grammar?
I am thinking of going for the low hanging fruit first (a Python centric pgen module), and then adding more functionality for later releases of Python (see below.)
If the former, I think there should be a strategy already how to expose pgen to the application; the proposed API seems inappropriate. In particular:
- how would I integrate an alternative tokenizer? - how could I integrate semantic actions into the parse process, instead of creating the canonical AST?
The current change proposed is somewhat restrained by the Python 2.2 release schedule, and will initially only address building parsers that use the Python tokenizer. If the module misses 2.2 release, I'd like to make it more functional and provide the ability to override the Python tokenizer. I may also add methods to export all the data found in the DFA structure.
I am unsure what the purpose of integration of semantics into the parse process buys us besides lower memory overhead. In C/C++ such coupling is needed because of the TYPEDEF/IDENTIFIER tokenization problem, but I don't see Python and future Python-like, LL(1), languages needing such hacks. Finally, I am prone to enforce the separation of the backend actions from the AST. This allows the AST to be used for a variety of purposes, rather than those intended by the initial parser developer.
Of course, these questions are less interesting if the scope is to parse Python: in that case, Python tokenization is fine, and everybody is used to getting the Python AST.
An interesting note to make about this is that the since the nonterminal integer values are generated by pgen, pgen AST's are not currently compatible with the parser module AST's. Perhaps such unification may be slated for future work (I know Fred left room in the parser AST datatype for identification of the grammar that generated the AST using an integer value, but using this would be questionable in a "rapid parser development" environment.)
On the specific API, I think you should drop the File functions (parseGrammarFile, parseFile). Perhaps you can also drop the String functions, and provide only functions that expect file-like objects.
I am open to further discussion on this, but I would note that filename information is used (and useful) when reporting syntax errors. I think that the "streaming" approach to parsing is another hold over from days where memory constraints ruled (much like binding semantics to the parser itself.)
On the naming of the API functions: I propose to use an underscore style instead of the mixedCaps style, or perhaps to leave out any structure (parsegrammar, buildparser, parse, symbol2string, string2symbolmap). That would be more in line with the parser module.
I would like to hear more about this from the Pythonati. I am currently following the naming conventions I use at work, which of course is most natural for me at home. :)
Regards, Martin
_______________________________________________ Types-SIG mailing list Types-SIG@python.org http://mail.python.org/mailman/listinfo/types-sig
participants (1)
-
Samuele Pedroni