Where regexs listed for Python language's tokenizer/lexer?

[Miles Kaufmann]

On Sep 11, 2009, at 11:10 PM, Chris Seberino wrote:

> Where regexs listed for Python language's tokenizer/lexer?
>
> If I'm not mistaken, the grammar is not sufficient to specify the
> language....
> you also need to specify the regexs that define the tokens
> right?..where is that?

The Python tokenization process is described here:

http://docs.python.org/reference/lexical_analysis.html

The tokenizer can't be expressed in terms of regular expressions,  
because it's non-regular (thanks to things like matching nested braces  
and keeping track of the indentation level).

-Miles