On 11/30/06, Fredrik Lundh
Guido van Rossum wrote:
Are you opposed changing tokenize? If so, why (apart from compatibility)? ISTM that it would be a good thing if it reported everything except horizontal whitespace.
it would be a good thing if it could, optionally, be made to report horizontal whitespace as well.
It's remarkably easy to get this out of the existing API; keep track of the end position returned by the previous call, and if it's different from the start position returned by the next call, slice the line text from the column positions, assuming the line numbers are the same. If the line numbers differ, something has been eating \n tokens; this shouldn't happen any more with my patch. -- --Guido van Rossum (home page: http://www.python.org/~guido/)