[Python-Dev] Parser module in the stdlib

[Victor Stinner]

Le jeu. 16 mai 2019 à 23:15, Pablo Galindo Salgado
<pablogsal at gmail.com> a écrit :
> The parser module has been "deprecated" (technically we recommend to prefer the ast module instead) since Python2.5 but is still in the standard library.

Importing it doesn't emit a DeprecationgWarning. It's only deprecated
in the documentation:
https://docs.python.org/dev/library/parser.html

I searched for "import parser" in Python on GitHub. I looked at the 10
pages of results: I cannot find any code which looks to import
"parser" from the Python standard library. Only "import parser" in a
test suite of PyPy, something unrelated.


> >>> parser.suite("def f(x,y,z): pass").tolist()
> [257, [269, [295, [263, [1, 'def'], [1, 'f'], [264, [7, '('], [265, [266, [1, 'x']], [12, ','], [266, [1, 'y']], [12, ','], [266, [1, 'z']]], [8, ')']], [11, ':'], [304, [270, [271, [277, [1, 'pass']]], [4, '']]]]]], [4, ''], [0, '']]

I don't understand how anyone can read or use Concrete Syntax Tree
(CST) directly :-( I would expect that you need another module on top
of that to use a CST. But I'm not aware of anything in the stdlib, so
I understand that... nobody uses this "parser" module.

I never heard about the "parser" module in the stdlib before this email :-)


> Several 3rd party packages (black, fissix...) are using it directly or have their own forks due to the fact that it can get outdated with respect to the Python3
> grammar as it was originally used only for Python2 to Python3 migration. It has the ability to consume LL1 grammars in EBNF form and produces an LL1 parser for them (by creating parser tables
> that the same module can consume). Many people use the module currently to support or parse supersets of Python (like Python2/3 compatible grammars, cython-like changes...etc).

* Did someone review the pgen2 implementation?
* Does it have a good API?
* Does it come with documentation?
* Who is maintaining it? Who will maintain it?


> I propose to remove finally the parser module as it has been "deprecated" for a long time, is almost clear that nobody uses it and has very limited usability and replace it (maybe with a different name)
> with pgen2 (maybe with a more generic interface that is detached to lib2to3 particularities).

I'm not sure that these two things have to be done at the same time.
Maybe we can start by issuing a DeprecationWarning on "import parser"
and announce the future removal in What's New in Python 3.8, just in
case, and wait one more release before removing it.

But I'm also fine with removing it right now. As I wrote, I never
heard about this module previously...

If your proposal is to reuse "parser" name, I'm not sure that it's a
good idea, since it existed for years and had a different API. Why not
keeping "pgen2" name, you said that it's already used under this name
in the wild.

Note: One thing to consider is that https://pypi.org/project/pgen2/
name is already used. It might conflict if the "new" pgen2 (pgen3?
:-D) would have a little bit different API.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.