tokenize module

Jim cook_jim at
Sat Jun 27 18:14:16 EDT 2009

I'm trying to understand the output of the tokenize.generate_tokens()
generator.  The token types returned seem to be more general than I'd
expect.  For example, when fed the following line of code:

def func_a():

the (abbreviated) returned token tuples are as follows:

(NAME,        def,             ...,  def func_a():)
(NAME ,       func_a,        ...,  def func_a():)
(OP,            (,                 ...,  def func_a():)
(OP,            ),                 ...,  def func_a():)
(OP,            :,                 ...,  def func_a():)
(NEWLINE,  NEWLINE,   ...,  def func_a():)

It seems to me that the token '(' should be identified as 'LPAR' and
')' as 'RPAR', as found in the dictionary token.tok_name.  What am I
missing here?

More information about the Python-list mailing list