tokenize module

Tim Roberts timr at
Sun Jun 28 09:46:05 CEST 2009

Jim <cook_jim at> wrote:

>I'm trying to understand the output of the tokenize.generate_tokens()
>generator.  The token types returned seem to be more general than I'd
>expect.  For example, when fed the following line of code:
>def func_a():
>It seems to me that the token '(' should be identified as 'LPAR' and
>')' as 'RPAR', as found in the dictionary token.tok_name.  What am I
>missing here?

Did you read the module?  Right at the top, it says:

It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
Tim Roberts, timr at
Providenza & Boekelheide, Inc.

More information about the Python-list mailing list