[Python-ideas] Hooking between lexer and parser

[Neil Girdhar]

On Sun, Jun 7, 2015 at 10:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 8 June 2015 at 12:37, Neil Girdhar <mistersheik at gmail.com> wrote:
> > Ultimately, my problem with "token transformers" is, if I'm understanding
> > correctly, that we want to change Python so that not only will 3.5 have
> > Token transformers, but every Python after that has to support this.
> This
> > risks constraining the development of the elegant solution.  And for what
> > major reason do we even need token transformers so soon?  For a toy
> example
> > on python ideas about automatic Decimal instances?  Why can't a user
> define
> > a one character function "d(x)" to do the conversion everywhere?  I
> prefer
> > to push for the better design even if it means waiting a year.
>
> Neil, you're the only one proposing major structural changes to the
> code generation pipeline, and nobody at all is proposing anything for
> Python 3.5 (the feature freeze deadline for that has already passed).
>
> Andrew is essentially only proposing relatively minor tweaks to the
> API of the existing tokenizer module to make it more iterable based
> and less file based (while still preserving the file based APIs).
> Eugene Toder's and Dave Malcolm's patches from a few years ago make
> the existing AST -> bytecode section of the toolchain easier to modify
> and experiment with (and are ideas worth exploring for 3.6 if anyone
> is willing and able to invest the time to bring them back up to date).
>
> However, if you're specifically wanting to work on an "ideal parser
> API", then the reference interpreter for a 24 year old established
> language *isn't* the place to do it - the compromises necessitated by
> the need to align with an extensive existing ecosystem will actively
> work against your goal for a clean, minimalist structure. That's thus
> something better developed *independently* of CPython, and then
> potentially considered at some point in the future when it's better
> established what concrete benefits it would offer over the status quo
> for both the core development team and Python's end users.
>

That's not what I'm doing.  All I'm suggesting is that changes to Python
that *preclude* the "ideal parser API" be avoided.  I'm not trying to make
the ideal API happen today.  I'm just keeping the path to that rosy future
free of obstacles.


>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150607/87d4afe5/attachment.html>