tokenize module

[Tim Roberts]

Jim <cook_jim at yahoo.com> wrote:

>I'm trying to understand the output of the tokenize.generate_tokens()
>generator.  The token types returned seem to be more general than I'd
>expect.  For example, when fed the following line of code:
>
>def func_a():
>...
>It seems to me that the token '(' should be identified as 'LPAR' and
>')' as 'RPAR', as found in the dictionary token.tok_name.  What am I
>missing here?

Did you read the module?  Right at the top, it says:

<quote>
It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators
</quote>
-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.