Tokenizer inconsistency wrt to new lines in comments

[George Sakkis]

On Apr 4, 3:18 pm, Kay Schluehr <kay.schlu... at gmx.net> wrote:

> I guess it's just an artifact of handling line continuations within
> expressions where a different rule is applied. For compilation
> purposes both the newlines within expressions as well as the comments
> are irrelevant. There are even two different token namely NEWLINE and
> NL which are produced for newlines. NL and COMMENT will be ignored.
> NEWLINE is relevant for the parser.
>
> If it was a bug it has to violate a functional requirement. I can't
> see which one.

Perhaps it's not a functional requirement but it came up as a real
problem on a source colorizer I use. I count on newlines generating
token.NEWLINE or tokenize.NL tokens in order to produce <br> tags. It
took me some time and head scratching to find out why some comments
were joined together with the following line. Now I have to check
whether a comment ends in new line and if it does output an extra <br>
tag.. it works but it's a kludge.

George