invalid-token syntax hook (was Re: Hack request: rational numbers)

[Jeff Epler]

On Sun, 28 Jan 2001 10:26:03 +0100, Alex Martelli
 <aleaxit at yahoo.com> wrote:
>"Tim Peters" <tim.one at home.com> wrote in message
>news:mailman.980590234.29036.python-list at python.org...
>    [snip]
>> That said, I believe your only hope of getting new literal syntax is to
>> embed it in a general proposal.  Some notation for rational literals is
>just
>> one of an unbounded number of similar desires.  Without a general scheme
>for
>
>What I was thinking, too.  So what about the "invalid-token syntax
>error hook" idea -- whenever the compiler is about to raise a
>SyntaxError because of an invalid token ('3r','@@x',etc), if
>sys.invalidTokenHook (or something of that ilk) is set to some
>callable, that callable is invoked, passed the invalid-token string,
>and given a chance to either return None (confirming that, yes, it
>IS a syntax error) or a tuple (possibly empty or singleton) of tokens
>to be used instead.

Why not go all the way and make the whole Python grammar extensible
in Python?  I'm currently working on a project, Mobius Python, which
offers such an extensible version of the Python grammar.  It could easily
implement the "rational number" proposal, I believe.

Mobius Python takes a lot of existing pieces (the tokenize module,
Tools/compiler, and the spark parser toolkit) and puts them together so
that you can register your extended syntax, associate it with an extension
for the source and bytecode versions, and then 'import' scripts in this
language just like normal.

http://sourceforge.net/projects/mobiuspython/

I think the CVS tree is outdated (sourceforge still doesn't have my cvs
access sorted out, but a co-developer has committed an older version of
the code) but in my local tree, the real need is for support for the
full 2.0 language, documentation on the various steps needed to write
your extended grammar, and a proper test suite.

The main drawback is that it is quite slow to perform all these steps
in Python.

Jeff
jepler at inetnebr.com