[New-bugs-announce] [issue42187] Deprecating / removing token.ISTERMINAL/ISNONTERMINAL

[Batuhan Taskaya]

New submission from Batuhan Taskaya <isidentical at gmail.com>:

Since the parser & symbol modules are removed, there is no real use case for ISTERMINAL/ISNONTERMINAL since the only input that contains grammar symbols is the input of the tokenize, which all its outputs are terminals. Which restricts the practicality of token.ISTERMINAL/ISNONTERMINAL. I'd propose we either deprecate them or remove them completely.

Search over the most populer PyPI packages (a set of 4k) showed there are only 2 projects that uses these functions, which all cases coupled with the parser module.
pyglet-1.5.7/pyglet/text/formats/attributed.py
133:        if token.ISNONTERMINAL(node[0]):
gdata-2.0.18/tests/coverage.py
328:        if token.ISNONTERMINAL(tree[0]):

If we are going through a deprecation period, I will also propose to remove ISEOF, since it can be easily replaced and very under used. The only project that uses token.ISEOF in my dataset is asttokens, which I would say all cases there can be easily replaced tok.type == token.EOF

asttokens-2.0.4/asttokens/mark_tokens.py
105:           not token.ISEOF(t.type)):

Also, these functions are originally reflections of C macros used in the past, and since they are not used right now, we can also remove those macros.

I have no opinions about the token.NT_OFFSET though, but there are a couple of occurrences so we might keep that? 
Paste-3.4.4/paste/util/PySourceColor.py
205:ARGS = token.NT_OFFSET + 1
206:DOUBLECOMMENT = token.NT_OFFSET + 2

ipython-7.18.1/IPython/utils/PyColorize.py
54:_KEYWORD = token.NT_OFFSET + 1
55:_TEXT    = token.NT_OFFSET + 2

----------
keywords: 3.4regression
messages: 379844
nosy: BTaskaya, gvanrossum, lys.nikolaou, pablogsal
priority: normal
severity: normal
status: open
title: Deprecating / removing token.ISTERMINAL/ISNONTERMINAL

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42187>
_______________________________________