[docs] [issue2134] Add new attribute to TokenInfo to report specific token IDs

Nick Coghlan report at bugs.python.org
Thu Dec 15 06:48:32 CET 2011

Nick Coghlan <ncoghlan at gmail.com> added the comment:

Ah, I didn't read it as suggested documentation at all - you moved seamlessly from personal commentary to a docs suggestion without separating the two, so it appeared to be a complete non sequitur to me.

As for the docs suggestion, I think it works as the explanation of which tokens are affected once the concept of the token stream simplification is introduced:
To simplify token stream handling, all literal tokens (':', '{', etc) are returned using the generic 'OP' token type. This allows them to be simply handled using common code paths (e.g. for literal transcription directly from input to output). Specific tokens can be distinguished by checking the "string" attribute of OP tokens for a match with the expected character sequence.

The affected tokens are all symbols and symbol combinations that are syntactically significant in expressions (as listed in the token module). Anything which is not an independent token (i.e. '#' which marks comments, '_' which is just part of a name, '\' which is used for line continuations, the contents of string literals and any symbols which are not a defined part of Python's syntax) is completely unaffected by this difference in behaviour.

If "exact_type" is introduced in 3.3, then the first paragraph can be adjusted accordingly.


Python tracker <report at bugs.python.org>

More information about the docs mailing list