tokenize module

[bootkey]

On Jun 28, 1:46 am, Tim Roberts <t... at probo.com> wrote:
> Jim <cook_... at yahoo.com> wrote:
> >I'm trying to understand the output of the tokenize.generate_tokens()
> >generator.  The token types returned seem to be more general than I'd
> >expect.  For example, when fed the following line of code:
>
> >def func_a():
> >...
> >It seems to me that the token '(' should be identified as 'LPAR' and
> >')' as 'RPAR', as found in the dictionary token.tok_name.  What am I
> >missing here?
>
> Did you read the module?  Right at the top, it says:
>
> <quote>
> It is designed to match the working of the Python tokenizer exactly, except
> that it produces COMMENT tokens for comments and gives type OP for all
> operators
> </quote>
> --
> Tim Roberts, t... at probo.com
> Providenza & Boekelheide, Inc.

Thanks for the reply.  I wonder why the tokenizer classifies all
operators simply as OP, instead of the various operators listed in the
tok_name dictionary.

Jim Cook