Lexing in Python 2

[Paul Prescod]

Tim Peters wrote:
> 
> Yes, but that's really a different topic.  The Python world has no good
> approach to that now, paying attention to the "fast" part, and where "good"
> means "enough like Flex and Bison that you don't feel you've been stranded
> on some strange alien planet" <wink>.

At the last XML conference I told someone that the reason that re
doesn't take a stream instead of string parameter was because anyone
sane working on a large file would use a proper tokenizer. Shouldn't
such a tokenizer come with Python? With all due respect, what the hell
is shlex and how did it get into the standard distribution? 

I mean the standard distribution alone must contain half a dozen
hand-coded lexers and in a few places, the weirdness you need to apply
regular expressions to streams. Surely we can do better for Python 2?

It is my unconsidered, uneducated opinion that lexers do not vary as
widely as parsers (LL(1), LR(1), LR(N) etc.) so we could just choose one
at random and start building modules around it.

All in favor? Opposed? Carried.
-- 
 Paul Prescod  - ISOGEN Consulting Engineer speaking for himself
Earth will soon support only survivor species -- dandelions, roaches, 
lizards, thistles, crows, rats. Not to mention 10 billion humans.
	- Planet of the Weeds, Harper's Magazine, October 1998