Typing-sig May 2022

typing-sig@python.org

39 participants
26 discussions

Opened and changed typing issues week 2022-W19
by Typing Bot 16 May '22

16 May '22

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were opened last week: #1183 Annotate sync/async code via Generic | generic TypeVar / Type aliases in Generic opened by @Bobronium https://github.com/python/typing/issues/1183 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Support Algebraic Data Types with @sealed
by John Hagen 11 May '22

11 May '22

The original (superseded) PEP 622 proposed adding a @sealed decorator so that users could define algebraic data types (ADTs) that static type checkers could verify are `match`ed exhaustively: - https://peps.python.org/pep-0622/#sealed-classes-as-algebraic-data-types There is a somewhat fairly-upvoted question about ADTs in Python, indicating interest in this area: - https://stackoverflow.com/questions/16258553 @sealed would be very valuable towards making ADT usage within Python more type-safe. Is there any historical discussion about why @sealed was dropped from the final pattern matching PEP? What would be the next steps needed to push this forward? Would writing a new PEP be the next step?

14 30

Discussion about a CST parser for 3.10+ (and a separate error-recovering parser)
by S Pradeep Kumar 10 May '22

10 May '22

TL;DR: A bunch of us met informally at PyCon to discuss adding a common CST parsing library for 3.10+. We also talked about an error-recovering parser. Pablo gave us a lot of feedback and advice about a path forward. I'm sharing the meeting notes for posterity. *Attendees*: Jelle Zijlstra, Zsolt Dollenstein (libcst, etc.), Kevin Millikin (DeepMind), Matthew Rahtz (DeepMind), Shantanu Jain (mypy), Pyre (Shannon, Jia, Pradeep), Pablo Galindo Salgado (Steering Council), and any others I didn’t note down! *Notes* (May 01, 2022): - Requirements - All attendees cared about having the tool support 3.10+. - CST parsing was important for Libcst, Black, linter frameworks such as Fixit, and DeepMind. - Error recovery was important for Pyre (so it can show results in the IDE as the user types code) and for Fixit. - DeepMind: we have a custom language server that uses a parser generated by Textmapper <https://textmapper.org/>, implemented in Go. Conceivably, we should be able to get Textmapper to generate Python code as well. But we'd also like a Python parser that supported (batch) static analysis and (batch) source-to-source transformations. Their existing solution doesn’t support 3.10. - Problems mentioned by Pablo - Difficulties with PEG: Soft keywords are the most challenging part for table-based parsers. Even with an LL(k) parser, some constructs need more than k=2 lookup. - Questions for Pablo: What does it take to add a CST parser? - Extending CPython’s PEG to CST is possible, but not free. - Need to balance features and speed. - CST is more expensive than AST: The tree is bigger and needs more memory. The tokenizer needs to chunk more. The parser needs to match more. - What would it take to maintain a CST parser in the standard library? - This is quite tricky. Many people will resist it. - Maintaining it in the standard library is challenging. However, this is more doable than any other change. - Need to create another version of the parser. pegen has a Python backend. Doing this in C will be extremely costly. The Python version will be easier. - Cython could be 2x the speed. C parser would be ~100x faster. - Error-prone to change the C parser: There was a recent change that needed to throw a syntax error at the start of an unmatched paren. There were ~63 bug fixes after implementing the change. - Pablo: What about Parso? - Zsolt: The Parso developer is apparently either not going to make it work on 3.10 or is going to make it closed-source. - Pablo: Issues in building an error-recovering PEG parser - PEG is tricky. It doesn’t have a concept of error. If one rule fails, some other may succeed. - Python was built on an LL(1) parser, so the deepest-rule heuristic works well. - One interesting technique for PEG error recovery is to add one of the tokens that you expected and reparse to see if it advances. This can be extremely fast. The challenge is to mutate the token without messing with the caching. - If you use technology other than PEG - you will need to find a way to support newer features, need to handle soft keywords, need and the equivalent of infinite backtracking and infinite lookahead. - Pablo: If you build a PEG CST parser (not in Rust), I can champion adding it to the standard library. - Pablo: For Black, could adapt pegen to produce a lib2to3-compatible CST. Would be slow. - Zsolt: Would rather rewrite on top of LibCST. - Takeaways - Zsolt: I’m committing to adding an error-recovering CST parser that supports PEG. It won’t be in pure Python, but we could create a wheel for it. - Kevin: Deepmind language server is currently a non-PEG parser supporting error recoverability; unclear how to migrate. We're a language server that needs to be in Go. Would like to do more source-to-source transformation, static analysis. Don't care about error recovery. - General: Look into Pyright’s error-recovering parser to understand its strengths and limitations. - General: Also look into tree-sitter-python <https://github.com/tree-sitter/tree-sitter-python>, which seems to already support 3.10’s match statement. Calls to action: 1. If you are interested in a common CST parser, please reply on this thread so that the other interested folks are aware of you. 2. If you are interested in an error-recovering parser, please reply below. e.g., you may want to show errors or auto-complete data as people are typing code in the IDE. 3. If you already use an error-recovering parser that supports 3.10+ features, please share your solution below. (I believe Pyright rolls its own parser in TypeScript. I’m curious about other Python IDE-like tools.) -- S Pradeep Kumar

3 2

Opened and changed typing issues week 2022-W18
by Typing Bot 09 May '22

09 May '22

No issues or pull requests with the label 'topic: feature' were opened or updated last week in the typing repository on GitHub. All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Generic NamedTuples
by Jelle Zijlstra 07 May '22

07 May '22

At the moment, this will throw a runtime error: from typing import Generic, NamedTuple, TypeVar T = TypeVar("T") class NT(NamedTuple, Generic[T]): foo: int bar: T There is an open CPython PR to add runtime support for this pattern: https://github.com/python/cpython/pull/92027. I'm inclined to accept this change, because it seems like a natural extension and would work well with the rest of the type system. Mypy doesn't currently support this for internal reasons, but I'm told Pyre already supports it. Let me know if there's some problem I missed with this idea. We just discussed this at the typing summit at PyCon, and one concern was brought up about variance. I think variance should work the same as with non-NamedTuple generic classes and no special treatment is required. Relatedly, there is another open PR that allows generalized multiple inheritance on NamedTuple (https://github.com/python/cpython/pull/31781). This would allow code like this: class A: @property def prop(self) -> int: ... class B(NamedTuple, A): foo: int assert_type(B(1).prop, int) I think this is too much to ask static type checkers to support: regular classes are too different from namedtuples. But I'd be happy to hear other opinions, especially from type checker maintainers.

6 11

Opened and changed typing issues week 2022-W17
by Typing Bot 02 May '22

02 May '22

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were updated last week: #593 Buffer protocol types opened by @srittau https://github.com/python/typing/issues/593 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

2023

2022

2021

2020

2019

2018

Typing-sig May 2022