On Thu, Apr 2, 2020 at 12:43 PM Paul Moore wrote:

On Thu, 2 Apr 2020 at 19:20, Guido van Rossum wrote:

...

Since last fall's core sprint in London, Pablo Galindo Salgado,

Lysandros Nikolaou and myself have been working on a new parser for CPython. We are now far enough along that we present a PEP we've written:

...

https://www.python.org/dev/peps/pep-0617/

Hopefully the PEP speaks for itself. We are hoping for a speedy

resolution so we can land the code we've written before 3.9 beta 1.

Excellent news! One question - will there be any user-visible change as a result of this PEP other than the removal of the "parser" module? From my quick reading of the PEP, I didn't see anything, so I assume the answer is "no".

I suppose it depends on how deep you dig, but the intention is that the returned AST is identical in each case. (We've "cheated" a bit by making a few small changes to the code that produces an AST for the old parser, mostly bugs related to line/column numbers.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

Just to clarify, this means that 3.9 will ship with the PEG parser as default, right? If so, this would be a new feature, post beta. Since that is counter to our general policy, we would need to get explicit RM approval for such a change.

The idea is merging it *before beta* and make it the default *from the beta* (maybe merging that commit making it the default the day before the beta for instance). Pablo

On Thu, Apr 2, 2020 at 1:21 PM Barry Warsaw wrote:

On Apr 2, 2020, at 13:07, Pablo Galindo Salgado wrote:

...

...

Just to clarify, this means that 3.9 will ship with the PEG parser as

default,

...

...

right? If so, this would be a new feature, post beta. Since that is counter to our general policy, we would need to get explicit RM approval for such a change.

The idea is merging it *before beta* and make it the default *from the beta* (maybe merging that commit making it the default the day before the beta for instance).

Yes, sorry I don’t mean to be pedantic, but it would be better policy-wise if the new parser were the default in time for the first beta rather than after.

That was the intention, i.e. releasing beta 1 with the new parser being the default. The current wording in the PEP are wrong, we'll fix that. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

About the migration, can I ask who is going to (help to) fix projects which rely on the AST?

I think you misunderstood: The AST is exactly the same as the old and the new parser. The only the thing that the new parser does is not generate an immediate CST (Concrete Syntax Tree) and that is only half-exposed in the parser module.

On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith wrote:

On Thu, Apr 2, 2020 at 2:48 PM Pablo Galindo Salgado wrote:

...

...

About the migration, can I ask who is going to (help to) fix projects which rely on the AST?

I think you misunderstood: The AST is exactly the same as the old and

the new parser. The only

...

the thing that the new parser does is not generate an immediate CST (Concrete Syntax Tree) and that is only half-exposed in the parser module.

If the AST is supposed to be the same, then would it make sense to temporarily – maybe just during the alpha/beta period – always run *both* parsers and confirm that they match?

That's not a bad idea! https://github.com/we-like-parsers/cpython/issues/33 -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Thu, Apr 2, 2020 at 7:55 PM Matt Billenstein wrote:

On Thu, Apr 02, 2020 at 05:17:31PM -0700, Guido van Rossum wrote:

...

On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith wrote:

If the AST is supposed to be the same, then would it make sense to temporarily – maybe just during the alpha/beta period – always run *both* parsers and confirm that they match?

That's not a bad idea! https://github.com/we-like-parsers/cpython/issues/33

Even just running it in a dev build against the corpus of the top few thousand packages on pypi might give enough confidence -- I had a script to download the top N packages and run some script over the python files contained therein, but I can't seem to find it atm.

We got that. Check https://github.com/gvanrossum/pegen/tree/master/scripts -- look at download_pypi_packages.py and test_pypi_packages.py. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

About the migration, can I ask who is going to (help to) fix projects which rely on the AST?

Whoops, I send the latest email before finishing it by mistake. Here is the extended version of the answer: I think there is a misunderstanding here: The new parser generates the same AST as the old parser so calling ast.parse() or compile() will yield exactly the same result. We have extensive testing around that and that was a goal from the beginning. Projects using the ast module will not need to do anything special. The difference is that the new parser does not generate a CST (Concrete Syntax Tree). The concrete syntax tree is an immediate structure from where the AST is generated. This structure is only partially exposed via the "parser" module but otherwise is only used in the parser itself so it should not be a problem. On the other hand: as explained in the PEP, the lack of the CST greatly simplifies the AST generation among other advantages.

On 3/04/20 10:33 am, Victor Stinner wrote:

I also like the fact that PEG is deterministic, whereas LL(1) parsers are not.

Where do you get that LL(1) parsers are not deterministic? That's news to me! -- Greg

Le ven. 3 avr. 2020 à 02:58, Greg Ewing a écrit :

On 3/04/20 10:33 am, Victor Stinner wrote:

...

I also like the fact that PEG is deterministic, whereas LL(1) parsers are not.

Where do you get that LL(1) parsers are not deterministic? That's news to me!

On Thu, Apr 2, 2020 at 6:15 PM Victor Stinner wrote:

Sorry, I was referring to *ambiguous* grammar rules. Extract of the PEP:

"Unlike LL(1) parsers PEG-based parsers cannot be ambiguous: if a string parses, it has exactly one valid parse tree. This means that a PEG-based parser cannot suffer from the ambiguity problems described in the previous section."

Maybe we need to rephrase this a bit. It's more that the LL(1) and PEG formalisms deal very different with ambiguous *grammars*. An example of an ambiguous grammar would be: start: X | Y X: expr Y: expr expr: NAME | NAME '+' NAME There are probably better examples of ambiguous grammars (see https://en.wikipedia.org/wiki/Ambiguous_grammar) but I think this will do to explain the problem. This is a fine context-free grammar (it accepts strings like "a" and "a+b") but the LL(1) formalism will reject it because it sees an overlap in FIRST sets between X and Y -- not surprising because they have the same RHS. Also, even a more powerful formalism would have to make a choice whether to choose X or Y, which may matter if the derivation is used to build a parse tree (like Python's pgen does). OTOH a PEG parser generator will always take the X alternative -- it doesn't care that there's more than one derivation, since its '|' operator is not symmetrical: X|Y and Y|X are not the same, as they are in LL(1) and most other formalisms. (In fact, the common notation for PEG uses '/' to emphasize this, but it looks ugly to me so I changed it to '|'.) That PEG (by definition) always uses the first matching alternative is actually a blessing as well as a curse. The downside is that PEG can't tell you when have a real ambiguity in your grammar. But the upside is that it works like a programmer would write a (recursive descent) parser. Thus it "solves" the problem of ambiguous grammars by choosing the first alternative. This allows more freedom in designing a grammar. For example, it would let a language designer solve the "dangling else" problem from the Wikipedia page, by writing the form including the "else" clause first . (Python doesn't have that problem due to the use of indentation, but it might appear in another disguise.) I should probably refine this argument and include it in the PEP as one of the reasons to prefer PEG over LR or LALR (but I need to think more about that -- it was a very early choice). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On 3/04/20 2:13 pm, Victor Stinner wrote:

"Unlike LL(1) parsers PEG-based parsers cannot be ambiguous: if a string parses, it has exactly one valid parse tree. This means that a PEG-based parser cannot suffer from the ambiguity problems described in the previous section."

That paragraph seems rather confused. I think what it *might* be trying to say is that a PEG parser allows you to write productions with overlapping first sets (which would be "ambiguous" for an LL parser), but still somehow guarantees that a unique parse tree is produced. The latter suggests that the grammar as a whole still needs to be unambiguous. -- Greg

On 4/04/20 9:29 am, Brett Cannon wrote:

I think "needs" is a bit strong. It would be nice, though. Regardless, as long as this is a net improvement over the status quo I don't see this being rejected on the grounds that an LR or LALR parser would be better since we have a working PEG parser today. :)

Even if the section only says "We didn't consider any alternatives, because...", I still think it should be there. -- Greg

The only thing I'm missing from the PEP is more detail about how the cross-language nature of the parser actions are handled.

Expanded the "actions" section in the PEP here: https://github.com/python/peps/pull/1357

On 6/04/20 2:08 am, Jelle Zijlstra wrote:

...

The current CPython parser usually just produces "SyntaxError: invalid syntax" for any error, while other languages that I work with usually say something more precise like 'expected x, got y'. What will the error messages in the PEG parser look like? Making syntax errors more informative can be a nice improvement to usability.

Unfortunately they look pretty much the same. We're actually currently trying to improve the error messages for situations where the old parser produces something specialized (mostly because the LL(1) grammar can't express something and the check is done in a later pass).

On Sun, Apr 5, 2020 at 7:55 AM Greg Ewing wrote: And related to that, how precisely will it be able to pinpoint the location of the error? The backtracking worries me a bit in that regard. I can imagine it trying all possible ways to parse the input and then only being able to say "Something is wrong somewhere in this file."

There's no need to worry about this: in almost all cases the error indicator points to the same spot in the source code as with the old parser. I was worried about this too, but it really doesn't seem to be a problem -- I think this might be different with highly ambiguous grammars, but since Python's grammar is still *mostly* LL(1), it looks like we're fine. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Sun, Apr 5, 2020 at 5:16 PM Greg Ewing wrote:

On 6/04/20 4:48 am, Guido van Rossum wrote:

...

There's no need to worry about this: in almost all cases the error indicator points to the same spot in the source code as with the old parser.

I'm curious about how that works. From the description in the PEP, it seems that none of the individual parsing functions can report an error, because there might be another branch higher up that succeeds. Does it keep track of the maximum distance it got through the source or something like that?

I guess you could call it that. There is a small layer of abstraction between the actual tokenizer (which cannot go back) and the generated parser functions. This abstraction buffers tokens. When a parser function wants a token it calls into this abstraction, and that either satisfies it from its buffer, or if there is no lookahead in the buffer left, calls the actual tokenizer. When a parser function fails, it calls into the abstraction layer to back up to a previous point (which I call the "mark"). (A simplified version of this layer is shown in my blog post, https://medium.com/@gvanrossum_83706/building-a-peg-parser-d4869b5958fb -- the class Tokenizer.) When an error bubbles all the way up, we report a SyntaxError pointing to the farthest token that the abstraction has buffered (self.pos in the blog post). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Mon, Apr 6, 2020 at 5:18 AM Jeff Allen wrote:

The PEP gives a good exposition of the problem and proposed solution, thanks.

If I understand correctly, the proposal is that the PEG grammar should become the definitive grammar for Python at some point, probably for Python 3.10, so it may evolve without the LL(1) restrictions. I'd like to raise some points with respect to that, which perhaps the migration section could answer.

Thanks, you definitely have a point here.

When definitive, the grammar would not then just be for CPython, and would also appear as user documentation of the language. Whether that change leaves Python with a more useful (readable) grammar seems an important test of the idea. I'm looking at https://github.com/we-like-parsers/cpython/blob/pegen/Grammar/python.gram , and assuming that is indicative of a future definitive grammar. That may be incorrect, as it has these issues in my view:

1. It is decorated with actions in C. If a decorated grammar is offered as definitive, one with Python actions (operations on the AST) is preferable, as implementation neutral, although still hostage to AST changes that are not language changes. Maybe one stripped of actions is best.

Yes, the plan is to strip actions and a few other embellishments (types, names, cuts, and probably also lookaheads -- although the latter may be significant, we only use them for optimization). The parser generator ( https://github.com/we-like-parsers/cpython/tree/pegen/Tools/peg_generator) prints a stripped representation (though currently preserving lookaheads -- suppressing those would be a simple change to the code).

2. It's quite long, and not at first glance more readable than the LL(1) grammar. I had understood ugliness in the LL(1) grammar to result from skirting limitations that PEG eliminates. The PEG one is twice as long, but recognising about half of it is actions, let's just say that as a grammar it's no shorter.

Indeed. I believe part of this actually comes from the desire to be 100% compatible with the old parser (an important constraint is that we don't want to change the AST since we don't want to change the byte code generator). Another part of it comes from expressing in the grammar constraints that the old parser generator cannot express. For example, the old parser accepts `1 = x` as an assignment, and it is rejected in a later stage. The new parser expresses this restriction in the grammar. Note that the full grammar published in the reference manual ( https://docs.python.org/3.8/reference/grammar.html) doesn't say anything about this; the grammar used later to describe assignment_stmt does ( https://docs.python.org/3.8/reference/simple_stmts.html#grammar-token-assign...), but as a result it is not LL(1) -- those grammar sections sprinkled throughout the reference manual are all written and updated by hand (and sometimes we forget!).

3. There is some manual guidance by means of &-guards, only necessary (I think) as a speed-up or to force out meaningful syntax errors. That would be noise to the reader. (This goes away if the PEG parser generator generate guards from the first set at a simple "no backtracking" marker.)

Yeah, see above. We've thought of generating FIRST sets as a future enhancement of the generator, and then they can go away. At the moment the lookaheads we have are all carefully aimed at optimizing the time and space requirements of the parser.

4. In some places, expansive alternatives seem to be motivated by the difference between actions, for a start, wherever async pops up. Maybe it is also why the definition of lambda is so long. That could go away with different support code (e.g. is_async as an argument), but if improvements to the support change grammar rules, when the language has not changed, that's a danger sign too.

Yeah, lambda is complicated by the requirement on the generated AST. Arguably we have gone too far here (and for 'parameters', which solves almost the same problem for regular function definitions) and we should put some of the checks back in the support code. But I note that the old grammar also has some warts in the area of parameter definitions (though its lambda is definitely simpler).

All that I think means that the "operational" grammar from which you build the parser is going to be quite unlike the one with which you communicate the language. At present ~/Grammar/Grammar both generates the parser (I thought) and appears as documentation. I take it to be the ideal that we use a single, human-readable definition. For example ANTLR 4 has worked hard to facilitate a grammar in which actions are implicit, and the generation of an AST from the parse tree/events can be elsewhere. (I'm not plugging ANTLR specifically as a solution.)

Our cheaper solution is to remove the actions from the display grammar. But I don't think that Grammar/Grammar should be seen as a complete specification of the language. And I don't think it is terrible if the specification says function_def_raw: | ASYNC 'def' NAME '(' parameters? ')' ['->' annotation] ':' block | 'def' NAME '(' parameters? ')' ['->' annotation] ':' block instead of function_def_raw: [ASYNC] 'def' NAME '(' parameters? ')' ['->' annotation] ':' block -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Mon, Apr 6, 2020 at 4:03 AM Fabio Zadrozny wrote:

I think using a PEG parser is interesting, but I do have some questions related to what's to expect in the future for other people which have to follow the Python grammar, so, can you shed some light on this?

Does that mean that the grammar format currently available (which is currently specified in https://docs.python.org/3.8/reference/grammar.html) will no longer be updated/used?

The grammar format used for the PEG parser is nearly the same as the old grammar, when you remove actions and some embellishments needed for actions. The biggest difference is that the `|` operator is no longer symmetrical (since if you have alternatives `A | B`, and both match at some point in the input, PEG reports A, while the old generator would reject the grammar as being ambiguous.

Is it expected that other language implementations/parsers also have to move to a PEG parser in the future? -- which would probably be the case if the language deviates strongly off LL(1)

We don't specify how other implementations must parse the language -- in fact I have no idea how the parsers of any of the other implementations work. I'm sure there will be other ways to parse the same language. But yeah, if there are implementations that currently closely follow Python's LL(1) parser structure they may have to be changed once we start introducing new syntax that makes use of the freedom PEG gives us. (For example, I've been toying with the idea of introducing a "match" statement similar to Scala's match expression by making "match" a keyword only when followed by an expression and a colon.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Mon, Apr 6, 2020 at 11:36 AM Steven D'Aprano wrote:

On Mon, Apr 06, 2020 at 10:43:11AM -0700, Guido van Rossum wrote:

...

I've been toying with the idea of introducing a "match" statement similar to Scala's match expression by making "match" a keyword only when followed by an expression and a colon.)

Didn't we conclude from `as` that having context-sensitive keywords was a bad idea?

I'm not sure that that was the conclusion. At the time the point was that we *wanted* all keywords to be reserved everywhere, an `as` was an ugly exception to that rule, which we got rid of as soon as we could -- not because it was a bad idea but because it violated a somewhat arbitrary rule. We went through the same thing with `async` and `await`, and the experience there was worse: a lot of libraries in the very space `async def` was aimed at were using `async` as a parameter name, often in APIs, and they had to scramble to redesign their APIs and get their users to change their programs. In retrospect I wish we had just kept `async` as a context-sensitive keyword, since it was totally doable. (In an early version of the PEG parser, all keywords were context-sensitive, and there were only very few places in the grammar where this required us to insert negative lookaheads to make edge cases parse correctly. The rest was taken care by careful ordering of rules, e.g. the rule for `del_stmt` must be tried before the rule for `expression_stmt` since `del *x` would match the latter.)

Personally, I would not like to have to explain to newcomers why `match` is a keyword but you can still use it as a function or variable, but not other keywords like `raise`, `in`, `def` etc.

match expression: match = True

What kind of newcomers do you have that they even notice that, unless you were to draw attention to it? I'm serious -- from the kind of questions I've seen in user forums, most newcomers are having a hard enough time learning more fundamental concepts and abstractions than the precise rules for reserved words. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Mon, Apr 6, 2020 at 8:04 PM Guido van Rossum wrote:

On Mon, Apr 6, 2020 at 11:36 AM Steven D'Aprano wrote:

...

Personally, I would not like to have to explain to newcomers why `match` is a keyword but you can still use it as a function or variable, but not other keywords like `raise`, `in`, `def` etc.

match expression: match = True

What kind of newcomers do you have that they even notice that, unless you were to draw attention to it? I'm serious -- from the kind of questions I've seen in user forums, most newcomers are having a hard enough time learning more fundamental concepts and abstractions than the precise rules for reserved words.

Absolutely. Beginners can simply be told they are keywords. If they then come across them in other contexts, hopefully there'll be a sensible documentation page that a web search for "<keyword> keyword" would lead to an explanation that "Some Python keywords can only ever be used with that meaning. Others can be used with other meanings where the context makes it clear that the keyword interpretation does not apply. You are recommended not to use such keywords as names in your own programs. The feature was implemented to make porting existing code to future versions of Python simpler." The tutorial should contain a similar passage.

After 30 years am I not allowed to take new information into account and consider a change of heart? :-) On Mon, Apr 6, 2020 at 6:21 PM Steven D'Aprano wrote:

On Mon, Apr 06, 2020 at 11:54:54AM -0700, Guido van Rossum wrote:

...

(In an early version of the PEG parser, all keywords were context-sensitive, and there were only very few places in the grammar where this required us to insert negative lookaheads to make edge cases parse correctly. The rest was taken care by careful ordering of rules, e.g. the rule for `del_stmt` must be tried before the rule for `expression_stmt` since `del *x` would match the latter.)

I think, on first glance, I'd rather have all keywords context-sensitive than just some. But I haven't put a great deal of thought into that aspect of it, and I reserve the right to change my mind :-)

...

...

Personally, I would not like to have to explain to newcomers why `match` is a keyword but you can still use it as a function or variable, but not other keywords like `raise`, `in`, `def` etc.

match expression: match = True

What kind of newcomers do you have that they even notice that, unless you were to draw attention to it?

It didn't take me 25 years to try using "of" and "if" for "output file" and "input file", so I guess my answer to your question is ordinary newcomers :-)

"Newcomers" doesn't just including beginners to programming, it can include people experienced in one or more other language coming to Python for the first time.

But if we're talking about complete beginners, the concept of what is and isn't a keyword is not always clear. Why is the first of these legal but not the second? Both words are highlighted in my editor:

str = "Hello world" class = "wizard"

People are going to learn that `match` is a keyword, and then they are going to come across code using it as a variable or method, and while the context-sensitive rule might be obvious to us, it won't be obvious to them precisely because they are still learning the language rules.

I think that `match` would be an especially interesting case because I can easily see someone starting off with a variable `match`, that they handle in an `if` statement, and then as the code evolves they shift it to a `match` statement:

match match:

and not bother to refactor the name because they are familiar enough with is that the meaning is obvious.

On the other hand there are definitely a few keywords that collide with useful names. Apart from `if`, I have wanted to use these as variables, parameters or functions:

class, raise, in, from, except, while, lambda

(off the top of my head, there may be others). There's at least one place in the random module where a parameter is misspelled "lambd" because lambda is a keyword. So there is certainly something to be said for getting rid of keywords.

On the third hand, keywords don't just make it easier for the interpreter, they also make it easier for the human reader. You don't need to care about context, `except` is `except` wherever you see it. That makes it a dead-simple rule for anyone to learn, because there are no exceptions, pun intended.

(I guess inside strings and comments are exceptions, but they are well-understood and *simple* exceptions.)

I just can't help feeling at this point that while there are pros and cons to making things a keyword, having some keywords be context sensitive but not others is going to combine the worst of both and end up be confusing and awkward.

...

I'm serious -- from the kind of questions I've seen in user forums, most newcomers are having a hard enough time learning more fundamental concepts and abstractions than the precise rules for reserved words.

That's because the precise rules for reserved words are dead-simple to learn. You can't use them anywhere except in the correct context. If we start adding exceptions to that, that reserved words are only sometimes reserved, I think that will make them harder to learn. If it's only some reserved words but not others, that's even harder because we have three classes of words:

* words that are never reserved * words that are sometimes reserved, depending on what is around them * words that are always reserved

I had thought that "no context-sensitive keywords" was a hard rule, so I was surprised that you are now re-considering it.

-- Steven _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NNMJB7QI... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

Another point in favour of always-reserved keywords is that they make life a lot easier for syntax highlighters. -- Greg

On 7/04/20 5:43 am, Guido van Rossum wrote:

The biggest difference is that the `|` operator is no longer symmetrical (since if you have alternatives `A | B`, and both match at some point in the input, PEG reports A, while the old generator would reject the grammar as being ambiguous.

I'm still inclined to think that allowing ambiguous grammars is more of a bug than a feature. Is there some way the generator could be made to at least warn if the grammar is genuinely ambiguous (as opposed to just having overlapping first sets in alternatives)?

We don't specify how other implementations must parse the language

And this is one of the reasons. If we use a PEG grammar as the definition of the language, and aren't careful about ambiguities when we add new syntax, we might accidentally end up with something that can *only* be parsed with a PEG parser or something equally powerful.

I'm sure there will be other ways to parse the same language.

That's certainly true now, but can you be sure it will remain true if additions are made that rely on the full power of PEG? -- Greg

On Sat, Apr 18, 2020 at 9:45 PM Guido van Rossum wrote: <snip>

But I never found the time to make it an appealing parser generator tool for other languages, even though that was on my mind as a future possibility.

I simply want to +1 on this. A general purpose parser library in the stdlib would be fantastic for its many parsing needs. I'm so looking forward to that future. Thanks, Nam

Great! Please submit a PR to update the [lib]2to3 docs and CC me (@gvanrossum). While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just accepted by the Steering Council without such language, and I wouldn't want to imply that the SC agrees with everything I said. So I still think we ought to deal with lib2to3 independently (and no, it won't need its own PEP :-). A reasonable option would be to just deprecate it and recommend people use parso, LibCST or something else (I wouldn't recommend pegen in its current form yet). On Tue, Apr 21, 2020 at 6:21 PM Carl Meyer wrote:

On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum wrote:

...

Note that, while there is indeed a docs page about 2to3, the only docs

for lib2to3 in the standard library reference are a link to the source code and a single "Note: The lib2to3 API should be considered unstable and may change drastically in the future."

...

Fortunately, in order to support the 2to3 application, lib2to3 doesn't

need to change, because the syntax of Python 2 is no longer changing. :-) Choosing to remove 2to3 is an independent decision. And lib2to3 does not depend in any way on the old parser module. (It doesn't even use the standard tokenize module, but incorporates its own version that is slightly tweaked to support Python 2.)

Indeed! Thanks for clarifying, I now recall that I already knew it doesn't, but forgot.

The docs page for 2to3 does currently say "lib2to3 could also be adapted to custom applications in which Python code needs to be edited automatically." Perhaps at least this sentence should be removed, and maybe also replaced with a clearer note that lib2to3 not only has an unstable API, but also should not necessarily be expected to continue to parse future Python versions, and thus building tools on top of it should be discouraged rather than recommended. (Maybe even use the word "deprecated.") Happy to submit a PR for this if you agree it's warranted.

It still seems to me that it wouldn't hurt for PEP 617 itself to also mention this shift in lib2to3's effective status (from "available but no API stability guarantee" to "probably will not parse future Python versions") as one of its indirect effects.

...

You've mentioned a few different tools that already use different technologies: LibCST depends on parso which has a fork of pgen2, lib2to3 which has the original pgen2. I wonder if this would be an opportunity to move such parsing support out of the standard library completely. There are already two versions of pegen, but neither is in the standard library: there is the original pegen repo which is where things started, and there is a fork of that code in the CPython Tools directory (not yet in the upstream repo, but see PR 19503).

The pegen tool has two generators, one generating C code and one generating Python code. I think that the C generator is really only relevant for CPython itself: it relies on the builtin tokenizer (the one written in C, not the stdlib tokenize.py) and the generated C code depends on many internal APIs. In fact the C generator in the original pegen repo doesn't work with Python 3.9 because those internal APIs are no longer exported. (It also doesn't work with Python 3.7 or older because it makes critical use of the walrus operator. :-) Also, once we started getting serious about replacing the old parser, we worked exclusively on the C generator in the CPython Tools directory, so the version in the original pegen repo is lagging quite a bit behind (is is the Python grammar in that repo). But as I said you're not gonna need it.

On the other hand, the Python generator is designed to be flexible, and while it defaults to using the stdlib tokenize.py tokenizer, you can easily hook up your own. Putting this version in the stdlib would be a mistake, because the code is pretty immature; it is really waiting for a good home, and if parso or LibCST were to decide to incorporate a fork of it and develop it into a high quality parser generator for Python-like languages that would be great. I wouldn't worry much about the duplication of code -- the Python generator in the CPython Tools directory is only used for one purpose, and that is to produce the meta-parser (the parser for grammars) from the meta-grammar. And I would happily stop developing the original pegen once a fork is being developed.

Thanks, this is all very clarifying! I hadn't even found the original gvanrossum/pegen repo, and was just looking at the CPython PR for PEP 617. Clearly I haven't been following this work closely.

...

Another option would be to just improve the python generator in the original pegen repo to satisfy the needs of parso and LibCST. Reading the blurb for parso it looks like it really just parses *Python*, which is less ambitious than pegen. But it also seems to support error recovery, which currently isn't part of pegen. (However, we've thought about it.) Anyway, regardless of how exactly this is structured someone will probably have to take over development and support. Pegen started out as a hobby project to educate myself about PEG parsers. Then I wrote a bunch of blog posts about my approach, and finally I started working on using it to generate a replacement for the old pgen-based parser. But I never found the time to make it an appealing parser generator tool for other languages, even though that was on my mind as a future possibility. It will take some time to disentangle all this, and I'd be happy to help someone who wants to work on this.

This seems like the place to start. When we start work on Python 3.10 support for LibCST, we can start with trying to use and adapt pegen in place of the vendored fork of parso we currently use, and if that's promising enough, consider taking over maintenance of it.

Carl

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Tue, Apr 21, 2020 at 9:35 PM Gregory P. Smith wrote:

Could we go ahead and mark lib2to3 as Pending Deprecation in 3.9 so we can get it out of the stdlib by 3.11 or 3.12?

I'm going ahead and tracking the idea in https://bugs.python.org/issue40360.

lib2to3 is the basis of all sorts of general source code manipulation tooling. Its name and original reason d'etre have moved on. It is actively used to parse and rewrite Python 3 code all the time. yapf uses it, black uses a fork of it. Other Python code manipulation tooling uses it. Modernize like fixers are useful for all sorts of cleanups.

IMNSHO it would be better if lib2to3 were *not* in the stdlib anymore - Black already chose to fork lib2to3 https://github.com/psf/black/tree/master/blib2to3. So given that it is eventually not going to be able to parse future syntax, the better answer seems like deprecation, putting the final version up on PyPI and letting any descendants of it live on PyPI where they can get more active care than a stdlib module ever does.

-gps

On Tue, Apr 21, 2020 at 6:58 PM Guido van Rossum wrote:

...

Great! Please submit a PR to update the [lib]2to3 docs and CC me (@gvanrossum).

While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just accepted by the Steering Council without such language, and I wouldn't want to imply that the SC agrees with everything I said. So I still think we ought to deal with lib2to3 independently (and no, it won't need its own PEP :-). A reasonable option would be to just deprecate it and recommend people use parso, LibCST or something else (I wouldn't recommend pegen in its current form yet).

On Tue, Apr 21, 2020 at 6:21 PM Carl Meyer wrote:

...

On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum wrote:

...

Note that, while there is indeed a docs page about 2to3, the only docs

for lib2to3 in the standard library reference are a link to the source code and a single "Note: The lib2to3 API should be considered unstable and may change drastically in the future."

...

Fortunately, in order to support the 2to3 application, lib2to3

doesn't need to change, because the syntax of Python 2 is no longer changing. :-) Choosing to remove 2to3 is an independent decision. And lib2to3 does not depend in any way on the old parser module. (It doesn't even use the standard tokenize module, but incorporates its own version that is slightly tweaked to support Python 2.)

Indeed! Thanks for clarifying, I now recall that I already knew it doesn't, but forgot.

The docs page for 2to3 does currently say "lib2to3 could also be adapted to custom applications in which Python code needs to be edited automatically." Perhaps at least this sentence should be removed, and maybe also replaced with a clearer note that lib2to3 not only has an unstable API, but also should not necessarily be expected to continue to parse future Python versions, and thus building tools on top of it should be discouraged rather than recommended. (Maybe even use the word "deprecated.") Happy to submit a PR for this if you agree it's warranted.

It still seems to me that it wouldn't hurt for PEP 617 itself to also mention this shift in lib2to3's effective status (from "available but no API stability guarantee" to "probably will not parse future Python versions") as one of its indirect effects.

...

You've mentioned a few different tools that already use different technologies: LibCST depends on parso which has a fork of pgen2, lib2to3 which has the original pgen2. I wonder if this would be an opportunity to move such parsing support out of the standard library completely. There are already two versions of pegen, but neither is in the standard library: there is the original pegen repo which is where things started, and there is a fork of that code in the CPython Tools directory (not yet in the upstream repo, but see PR 19503).

The pegen tool has two generators, one generating C code and one generating Python code. I think that the C generator is really only relevant for CPython itself: it relies on the builtin tokenizer (the one written in C, not the stdlib tokenize.py) and the generated C code depends on many internal APIs. In fact the C generator in the original pegen repo doesn't work with Python 3.9 because those internal APIs are no longer exported. (It also doesn't work with Python 3.7 or older because it makes critical use of the walrus operator. :-) Also, once we started getting serious about replacing the old parser, we worked exclusively on the C generator in the CPython Tools directory, so the version in the original pegen repo is lagging quite a bit behind (is is the Python grammar in that repo). But as I said you're not gonna need it.

On the other hand, the Python generator is designed to be flexible, and while it defaults to using the stdlib tokenize.py tokenizer, you can easily hook up your own. Putting this version in the stdlib would be a mistake, because the code is pretty immature; it is really waiting for a good home, and if parso or LibCST were to decide to incorporate a fork of it and develop it into a high quality parser generator for Python-like languages that would be great. I wouldn't worry much about the duplication of code -- the Python generator in the CPython Tools directory is only used for one purpose, and that is to produce the meta-parser (the parser for grammars) from the meta-grammar. And I would happily stop developing the original pegen once a fork is being developed.

Thanks, this is all very clarifying! I hadn't even found the original gvanrossum/pegen repo, and was just looking at the CPython PR for PEP 617. Clearly I haven't been following this work closely.

...

Another option would be to just improve the python generator in the original pegen repo to satisfy the needs of parso and LibCST. Reading the blurb for parso it looks like it really just parses *Python*, which is less ambitious than pegen. But it also seems to support error recovery, which currently isn't part of pegen. (However, we've thought about it.) Anyway, regardless of how exactly this is structured someone will probably have to take over development and support. Pegen started out as a hobby project to educate myself about PEG parsers. Then I wrote a bunch of blog posts about my approach, and finally I started working on using it to generate a replacement for the old pgen-based parser. But I never found the time to make it an appealing parser generator tool for other languages, even though that was on my mind as a future possibility. It will take some time to disentangle all this, and I'd be happy to help someone who wants to work on this.

This seems like the place to start. When we start work on Python 3.10 support for LibCST, we can start with trying to use and adapt pegen in place of the vendored fork of parso we currently use, and if that's promising enough, consider taking over maintenance of it.

Carl

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c... _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RDV3GAL2... Code of Conduct: http://python.org/psf/codeofconduct/