[Python-Dev] Small tweak to tokenize.py?

[Guido van Rossum]

On 11/30/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
> Guido van Rossum wrote:
>
> > Are you opposed changing tokenize? If so, why (apart from
> > compatibility)? ISTM that it would be a good thing if it reported
> > everything except horizontal whitespace.
>
> it would be a good thing if it could, optionally, be made to report
> horizontal whitespace as well.

It's remarkably easy to get this out of the existing API; keep track
of the end position returned by the previous call, and if it's
different from the start position returned by the next call, slice the
line text from the column positions, assuming the line numbers are the
same. If the line numbers differ, something has been eating \n tokens;
this shouldn't happen any more with my patch.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)