Other notes

[Bengt Richter]

On Sat, 08 Jan 2005 18:22:53 +1000, Nick Coghlan <ncoghlan at iinet.net.au> wrote:

>Bengt Richter wrote:
>> IOW, I think there is a fix: keep tokenizing greedily and tokenize floating point as
>> a sequence of integers and operators, and let <integer><dot><integer> be translated by
>> the compiler to floating point, and <integer><dotdot><integer> be translated to the
>> appropriate generator expression implementation.
>
>That would be:
>
><int-literal><dot><int-literal> -> float(<int-literal> + "." + <int-literal>)
><int-literal><dot><identifier> -> getattr(int(<int-literal>), <identifier>)
><int-literal><dot><dot><int-literal> -> xrange(<int-literal>, <int-literal>)
>
>However, the problem comes when you realise that 1e3 is also a floating point 
>literal, as is 1.1e3.
>
Ok, that requires a little more imagination ;-)

I think it can be solved, but I haven't explored it all the way through ;-)

The key seems to be to be to condition the recognition of tokens as if recognizing
an old potentially floating point number, but emitting number-relevant separate tokens
so long as there is no embedded spaces. When a number ends, emitting an end marker should
permit the compiler to deal with the various compositions.

We still aren't looking ahead more than one, but we are carrying context, just as we do
to accumulate digits of an integer or characters of a name, but the context may continue
and determine what further tokens are emitted. E.g. the 'e' in the embedded numeric context,
becomes <fexp> rather than a name. In the following, <eon> :== end of number token

   1.1 -> <1><dot><1><eon>
   1 .1 -> <1><eon><dot><1><eon>
   1.e1 -> <1><dot><fexp><1><eon> 
   1 .e1 -> <1><eon><dot><e1>
   1.2e3 -> <1><dot><2><fexp><3><eon>
   1..2 -> <1><eon><doubledot><1><eon>
   1. .2 -> <1><dot><eon><dot><2><eon> (syntax error)
   1 ..2 -> <1><eon><doubledot><1><eon>

I just have the feeling that there is a solution, whatever the hiccups ;-)

Regards,
Bengt Richter