[Eric S. Raymond]
... Lexers are painful in Python.
This is so.
They hit the language in a weak spot created by the immutability of strings.
But you lost me here -- I don't see a connection between immutability and either ease of writing lexers or speed of lexers. Indeed, lexers are (IME) more-or-less exactly as painful and slow written in Perl, where strings are mutable. Seems more to me that lexing is convenient and fast only when expressed in a language specifically designed for writing lexers, and backed by a specialized execution engine that knows a great deal about fast state-machine implementation. Like, say, Flex. Lexing is also clumsy and slow in SNOBOL4 and Icon, despite that they excel at higher-level pattern-matching tasks. IOW, lexing is in a world by itself, almost nothing is good at it, and the few things that shine at it don't do anything else. lexing-is-the-floating-point-execution-unit-of-the-language-world-ly y'rs - tim