Presenting PEP 695: Type Parameter Syntax
After several rounds of debate on typing-sig, I'd like to request feedback on PEP 695: https://peps.python.org/pep-0695/ I am sponsoring this PEP, which was written by Eric Traut. The PEP attempts to solve the problem that defining generic classes, functions and type aliases currently is lacking dedicated syntax, instead using the cumbersome `T = TypeVar("T", ...)` notation to create global variables that serve as type variables. As a personal historical note, I should mention that over 22 years ago I already pondered type parameters. In an old document that I saved I found the following code snippet: ``` def f<T> (a: T) -> T: ... ``` which is eerily close to the proposal in this PEP, except that the PEP uses square brackets: ``` def f[T](a: T) -> T: ... ``` It's been a long and circuitous road! I am not quoting the entire PEP here, please follow the link: https://peps.python.org/pep-0695/ -- --Guido van Rossum (python.org/~guido)
On 12. 07. 22 6:30, Guido van Rossum wrote:
A beautifully written PEP, thank you! An extra thank you for clearly specifying compile-/run-time vs. type checker behavior! In “Type Parameter Declarations” it would be nice to better specify why this example is an error: ``` class ClassA[__T, _ClassA__S]: __T = 0 # OK __S = 0 # Syntax Error (because mangled name is _ClassA__S) ``` It's specified later in the Compiler changes section (“An active type variable symbol cannot be used for other purposes”), but it would be nice to mention it up here too – perhaps replace the specific “Type parameters for a generic function cannot overlap the name of a function parameter.” I'm not a fan of a third overload of `type`, after the “get type of” function and “default metatype” class. Would `typevar` not work? (The addition of a soft keyword itself is a heavy change, though I'll let grammar experts weigh in on that.) I wonder if we should give some thought to other cases where a name is repeated – for example, a hypothetical: namedtuple Point = ("x", "y") replacing: Point = namedtuple("Point", ("x", "y")) Is the proposed `type` potentially setting a precedent? A good one? `TypeVar` taking `covariant`, `contravariant` and `autovariance` looks inconsistent to an outsider. Why is it not `autovariant`? The Rejected ideas mention “various syntactic options for specifying type parameters that preceded def and class statements” rejected because scoping is less clear and doesn't work well with decorators. I wonder if decorator-like syntax itself was considered, e.g. something like: ``` @with type S @with type T @dec(Foo[S]) class ClassA: ... ``` And finally, I need to ask... The reference implementation doesn't include documentation. Is there any plan to document this feature outside this enhancement proposal? If not, what needs to happen to get this documented?
El mar, 12 jul 2022 a las 6:15, Petr Viktorin (<encukou@gmail.com>) escribió:
That piece of syntax is for type *aliases*, not type *variables*, which are a different concept. Using "typevar" here would be quite confusing. We could use something like "alias" or "typealias", but I think "type" is the most intuitive term, and it matches other languages like TypeScript.
We did consider variations of that. Pure decorator syntax (like your "@dec" line) wouldn't allow us to get the scoping right, since decorators can't define new names. (Unless you use a walrus operator, but that wouldn't meet the goal of providing cleaner syntax.) We also considered some ideas similar to your "@with type". It can work, but it feels more verbose than the current proposal, and it's not in line with what most other languages do.
I'm sure we'll provide detailed documentation if and when the PEP is accepted; full documentation seems a bit much to ask for in an early prototype.
I actually really like some variation on `@with type S`, or some other variation that has a keyword, because that makes it much easier for someone newly encountering one of these syntax constructs to search to figure out what it does. If you didn't already know what the square brackets did, how would you try and find out? "what do square brackets mean in Python" would probably turn up a bunch of stuff about element access, and maybe something about type generic parameters. By contrast, `@with type S` is kinda self-explanatory, and even if it's not, 'What does "with type" mean in Python' will almost certainly turn up meaningful results. An additional benefit is that I find some of these examples to be a bit visually cluttered with all the syntax: def func1[T](a: T) -> T: ... # OK class ClassA[S, T](Protocol): ... # OK Which would look less cluttered with a prefix clause: @with type T def func1(a: T) -> T: ... # OK @with type S @with type T class ClassA(Protocol): ... # OK Of the objections to this concept in the PEP <https://peps.python.org/pep-0695/#prefix-clause>, the most obvious one to me was that the scoping rules were less clear, but it is not entirely clear to me why the scope of the prefix clause couldn't be extended to include class / function decorators that follow the prefix clause; the choice of scoping seems like it was a defensible but mostly arbitrary one. I think as long as the new prefix clause is something that was syntactically forbidden prior to the introduction of PEP 695 (e.g. `@with type` or `[typevar: S]` or whatever), it will be relatively clear that this is not a normal decorator, and so "the scoping and time of execution doesn't match decorators" doesn't seem like a major concern to me relative to the benefits of using a more searchable syntax. On 7/12/22 18:09, Jelle Zijlstra wrote:
Paul Ganssle writes:
If you didn't already know what the square brackets did, how would you try and find out?
First I'd look it up in Python Essential Reference (Hi, @dabeaz! it won't be there, though ;-). Then I'd go to the Language Reference for "def" and "class". And if that failed, then I'd go buy Barry Warsaw lunch. OK, not everybody has a personal relationship with senior core devs, but is asking people to read the Language Reference really so bad?
Looks like the boomer version (square*) of C++ template variables. Of course, people learn Python to escape from C++, so maybe that's not persuasive. * telling you how old I am without telling you how ooooold I am
For me, that's absolutely awful from a readability standpoint. Put the "def" or "class" 10-20 characters in from the margin? I guess "stacked" it's no less readable than any decorator, but I also don't like overloading the well-defined decorator notation with magic. @with type T def func1(a: T) -> T: ... # OK @with type S @with type T class ClassA(Protocol): ... # OK A thought: would it be possible to actually make it a with statement? with Typevar() as T: def func1(a: T) -> T Of course there might have to be magic in Typevar, but it would be far more palatable to me than giving unary @ two kinds of magic. IMO YMMV of course.
Hi, I like this PEP but I couldn't find the motivation for using angle brackets over square braces (brackets?). The survey in Appendix A is great but lacks any conclusions. From that survey alone I would assume that angle brackets would have been chosen over square braces, given that they are the most common option and appear in (afaik) the more popular languages in that list. I think the PEP should add a section about the choice of syntax in the rejected section, which can be expanded upon in Appendix A. If you can't tell I'm in favor of angle brackets, I think the examples given in the PEP look a bit messy with so many parentheses and square braces in close proximity. Using angle brackets would make the distinction between typevars and function parameters clearer.
Yeah, we all would have liked angle brackets, but there would be problems with breaking lines between those. E.g. def foo< T: str, S: int
(arg1: T, arg2: S) -> tuple[T, S]: ...
cannot be parsed because the lexer doesn't treat angle brackets as matching pairs. In addition, we already use square brackets for *using* generics (e.g. list[int]), and most surveyed languages use the same type of brackets in declarations and uses. On Thu, Jul 14, 2022 at 1:10 PM <o.jacob.nilsson@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 7/14/2022 6:16 PM, Guido van Rossum wrote:
I do not yet use annotations, but knowing about 'list[int]', etc, I could immediately read and understand the new examples.
I presume you meant the reverse, for proposing [] over <>. I agree that giving the motivation above (and the one deleted) would be good idea. -- Terry Jan Reedy
On 13/07/2022 14:14, o.jacob.nilsson@gmail.com wrote:
Hi, I like this PEP but I couldn't find the motivation for using angle brackets over square braces (brackets?). The survey in Appendix A is great but lacks any conclusions. From that survey alone I would assume that angle brackets would have been chosen over square braces, given that they are the most common option and appear in (afaik) the more popular languages in that list. I think the PEP should add a section about the choice of syntax in the rejected section, which can be expanded upon in Appendix A.
If you can't tell I'm in favor of angle brackets, I think the examples given in the PEP look a bit messy with so many parentheses and square braces in close proximity. Using angle brackets would make the distinction between typevars and function parameters clearer.
Another reason that square brackets should be preferred over angle brackets is the difficulty in parsing: list<list<int>> is tokenised as list < list < int >> where the last token is a right shift operator, so the parser has to know that sometimes >> is used when two "angle bracket" groups are closed, instead of > >
On 12. 07. 22 6:30, Guido van Rossum wrote:
A beautifully written PEP, thank you! An extra thank you for clearly specifying compile-/run-time vs. type checker behavior! In “Type Parameter Declarations” it would be nice to better specify why this example is an error: ``` class ClassA[__T, _ClassA__S]: __T = 0 # OK __S = 0 # Syntax Error (because mangled name is _ClassA__S) ``` It's specified later in the Compiler changes section (“An active type variable symbol cannot be used for other purposes”), but it would be nice to mention it up here too – perhaps replace the specific “Type parameters for a generic function cannot overlap the name of a function parameter.” I'm not a fan of a third overload of `type`, after the “get type of” function and “default metatype” class. Would `typevar` not work? (The addition of a soft keyword itself is a heavy change, though I'll let grammar experts weigh in on that.) I wonder if we should give some thought to other cases where a name is repeated – for example, a hypothetical: namedtuple Point = ("x", "y") replacing: Point = namedtuple("Point", ("x", "y")) Is the proposed `type` potentially setting a precedent? A good one? `TypeVar` taking `covariant`, `contravariant` and `autovariance` looks inconsistent to an outsider. Why is it not `autovariant`? The Rejected ideas mention “various syntactic options for specifying type parameters that preceded def and class statements” rejected because scoping is less clear and doesn't work well with decorators. I wonder if decorator-like syntax itself was considered, e.g. something like: ``` @with type S @with type T @dec(Foo[S]) class ClassA: ... ``` And finally, I need to ask... The reference implementation doesn't include documentation. Is there any plan to document this feature outside this enhancement proposal? If not, what needs to happen to get this documented?
El mar, 12 jul 2022 a las 6:15, Petr Viktorin (<encukou@gmail.com>) escribió:
That piece of syntax is for type *aliases*, not type *variables*, which are a different concept. Using "typevar" here would be quite confusing. We could use something like "alias" or "typealias", but I think "type" is the most intuitive term, and it matches other languages like TypeScript.
We did consider variations of that. Pure decorator syntax (like your "@dec" line) wouldn't allow us to get the scoping right, since decorators can't define new names. (Unless you use a walrus operator, but that wouldn't meet the goal of providing cleaner syntax.) We also considered some ideas similar to your "@with type". It can work, but it feels more verbose than the current proposal, and it's not in line with what most other languages do.
I'm sure we'll provide detailed documentation if and when the PEP is accepted; full documentation seems a bit much to ask for in an early prototype.
I actually really like some variation on `@with type S`, or some other variation that has a keyword, because that makes it much easier for someone newly encountering one of these syntax constructs to search to figure out what it does. If you didn't already know what the square brackets did, how would you try and find out? "what do square brackets mean in Python" would probably turn up a bunch of stuff about element access, and maybe something about type generic parameters. By contrast, `@with type S` is kinda self-explanatory, and even if it's not, 'What does "with type" mean in Python' will almost certainly turn up meaningful results. An additional benefit is that I find some of these examples to be a bit visually cluttered with all the syntax: def func1[T](a: T) -> T: ... # OK class ClassA[S, T](Protocol): ... # OK Which would look less cluttered with a prefix clause: @with type T def func1(a: T) -> T: ... # OK @with type S @with type T class ClassA(Protocol): ... # OK Of the objections to this concept in the PEP <https://peps.python.org/pep-0695/#prefix-clause>, the most obvious one to me was that the scoping rules were less clear, but it is not entirely clear to me why the scope of the prefix clause couldn't be extended to include class / function decorators that follow the prefix clause; the choice of scoping seems like it was a defensible but mostly arbitrary one. I think as long as the new prefix clause is something that was syntactically forbidden prior to the introduction of PEP 695 (e.g. `@with type` or `[typevar: S]` or whatever), it will be relatively clear that this is not a normal decorator, and so "the scoping and time of execution doesn't match decorators" doesn't seem like a major concern to me relative to the benefits of using a more searchable syntax. On 7/12/22 18:09, Jelle Zijlstra wrote:
Paul Ganssle writes:
If you didn't already know what the square brackets did, how would you try and find out?
First I'd look it up in Python Essential Reference (Hi, @dabeaz! it won't be there, though ;-). Then I'd go to the Language Reference for "def" and "class". And if that failed, then I'd go buy Barry Warsaw lunch. OK, not everybody has a personal relationship with senior core devs, but is asking people to read the Language Reference really so bad?
Looks like the boomer version (square*) of C++ template variables. Of course, people learn Python to escape from C++, so maybe that's not persuasive. * telling you how old I am without telling you how ooooold I am
For me, that's absolutely awful from a readability standpoint. Put the "def" or "class" 10-20 characters in from the margin? I guess "stacked" it's no less readable than any decorator, but I also don't like overloading the well-defined decorator notation with magic. @with type T def func1(a: T) -> T: ... # OK @with type S @with type T class ClassA(Protocol): ... # OK A thought: would it be possible to actually make it a with statement? with Typevar() as T: def func1(a: T) -> T Of course there might have to be magic in Typevar, but it would be far more palatable to me than giving unary @ two kinds of magic. IMO YMMV of course.
Hi, I like this PEP but I couldn't find the motivation for using angle brackets over square braces (brackets?). The survey in Appendix A is great but lacks any conclusions. From that survey alone I would assume that angle brackets would have been chosen over square braces, given that they are the most common option and appear in (afaik) the more popular languages in that list. I think the PEP should add a section about the choice of syntax in the rejected section, which can be expanded upon in Appendix A. If you can't tell I'm in favor of angle brackets, I think the examples given in the PEP look a bit messy with so many parentheses and square braces in close proximity. Using angle brackets would make the distinction between typevars and function parameters clearer.
Yeah, we all would have liked angle brackets, but there would be problems with breaking lines between those. E.g. def foo< T: str, S: int
(arg1: T, arg2: S) -> tuple[T, S]: ...
cannot be parsed because the lexer doesn't treat angle brackets as matching pairs. In addition, we already use square brackets for *using* generics (e.g. list[int]), and most surveyed languages use the same type of brackets in declarations and uses. On Thu, Jul 14, 2022 at 1:10 PM <o.jacob.nilsson@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 7/14/2022 6:16 PM, Guido van Rossum wrote:
I do not yet use annotations, but knowing about 'list[int]', etc, I could immediately read and understand the new examples.
I presume you meant the reverse, for proposing [] over <>. I agree that giving the motivation above (and the one deleted) would be good idea. -- Terry Jan Reedy
On 13/07/2022 14:14, o.jacob.nilsson@gmail.com wrote:
Hi, I like this PEP but I couldn't find the motivation for using angle brackets over square braces (brackets?). The survey in Appendix A is great but lacks any conclusions. From that survey alone I would assume that angle brackets would have been chosen over square braces, given that they are the most common option and appear in (afaik) the more popular languages in that list. I think the PEP should add a section about the choice of syntax in the rejected section, which can be expanded upon in Appendix A.
If you can't tell I'm in favor of angle brackets, I think the examples given in the PEP look a bit messy with so many parentheses and square braces in close proximity. Using angle brackets would make the distinction between typevars and function parameters clearer.
Another reason that square brackets should be preferred over angle brackets is the difficulty in parsing: list<list<int>> is tokenised as list < list < int >> where the last token is a right shift operator, so the parser has to know that sometimes >> is used when two "angle bracket" groups are closed, instead of > >
participants (9)
-
Guido van Rossum -
Guido van Rossum -
Jelle Zijlstra -
o.jacob.nilsson＠gmail.com -
Patrick Reader -
Paul Ganssle -
Petr Viktorin -
Stephen J. Turnbull -
Terry Reedy