Discussion of PEP 505: Type Hinting Generics In Standard Collections
PEP 585: https://www.python.org/dev/peps/pep-0585/ With the help of Ethan Smith (and some code review by Serhiy) I have started a prototype of PEP585. You can see my code at https://github.com/python/cpython/pull/18239 (a PR to CPython) or directly in my branch (where I take PRs too): https://github.com/gvanrossum/cpython/tree/pep585. I am cautiously optimistic that we can get this integrated into Python 3.9. Based on this work in progress I have a bunch of questions and suggestions for PEP 585. 1. The PEP says that e.g. list[str][str] should be forbidden. But there's at least one example where we currently allow chained subscript operations. This currently works (in mypy as well as at runtime): import typing T = typing.TypeVar("T") D = typing.Dict[str, T] # Now D is a generic type x: D[int] # Now x has type dict[str, int] But if, in my branch, we replace Dict with dict, we get an error on the last line: import typing T = typing.TypeVar("T") D = dict[str, T] x: D[int] # E: TypeError: 'GenericAlias' object is not subscriptable The last line raises a TypeError. 2. The PEP says that isinstance([1, 2, 3], list[int]) should be True. I think this is wrong -- with typing.List[int] it currently raises a TypeError, and I think we should keep forbidding this. Similar with issubclass. 3. The PEP says that list == list[str]. This currently returns false in my branch (apparently this doesn't use __getattribute__ to get the __eq__ operation), and I think I like that better. It also matches the behavior of typing.List. 4. The PEP doesn't provide a name for the proxy type. I propose to name it GenericAlias and to add it to the types.py module, so it can be imported as types.GenericAlias (note: types; not typing). 5. I think the following classes should also be made generic: re.Pattern, re.Match, io.IO. 6. How far should we go with replacing everything in typing.py that shadows something that has become generic due to the PEP's changes? (collections, collections.abc, contextlib, re, io). I expect this to be somewhat controversial (Ivan may have some ideas) and difficult to keep compatible with 3.8. 7. I haven't figured out yet how to support type[x] (which should have the same meaning as typing.Type[x]) without it also enabling int[x], which we would like to prevent. (We don't want to just add object.__class_getitem__!) 8. I have a tricky bug remaining where inheriting from Tuple[something] causes problems. Take this example: class C(Tuple[int]): pass If we now examine C.__mro__, we see that it is (C, tuple, typing.Generic, object). IOW, inheriting from typing.Tuple causes builtins.tuple to be inserted into the MRO. The problem is that in my branch, tuple.__class_getitem__ overrides Generic.__class_getitem__, whereas the latter is needed to satisfy some tests for typing.py (and presumably to enable certain special behavior). I'm mostly looking for suggestions on how to make those failing tests work (see https://github.com/python/cpython/commit/35cd5f477c9da985880130d72297335e010... ). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
29.01.20 20:11, Guido van Rossum пише:
PEP 585: https://www.python.org/dev/peps/pep-0585/
With the help of Ethan Smith (and some code review by Serhiy) I have started a prototype of PEP585. You can see my code at https://github.com/python/cpython/pull/18239 (a PR to CPython) or directly in my branch (where I take PRs too): https://github.com/gvanrossum/cpython/tree/pep585. I am cautiously optimistic that we can get this integrated into Python 3.9.
It is an excellent work. It is much simpler than I expected.
Based on this work in progress I have a bunch of questions and suggestions for PEP 585.
1. The PEP says that e.g. list[str][str] should be forbidden. But there's at least one example where we currently allow chained subscript operations. This currently works (in mypy as well as at runtime):
import typing T = typing.TypeVar("T") D = typing.Dict[str, T] # Now D is a generic type x: D[int] # Now x has type dict[str, int]
But if, in my branch, we replace Dict with dict, we get an error on the last line:
import typing T = typing.TypeVar("T") D = dict[str, T] x: D[int] # E: TypeError: 'GenericAlias' object is not subscriptable
The last line raises a TypeError.
We can add `__getitem__` which replaces type vars with actual parameters. I tried to write the implementation here, but it turned out to be quite complicated, taking to account that the same type variable can be used multiple times and that they can be nested. It is not an easy problem, but neither very hard.
2. The PEP says that isinstance([1, 2, 3], list[int]) should be True. I think this is wrong -- with typing.List[int] it currently raises a TypeError, and I think we should keep forbidding this. Similar with issubclass.
Agree.
3. The PEP says that list == list[str]. This currently returns false in my branch (apparently this doesn't use __getattribute__ to get the __eq__ operation), and I think I like that better. It also matches the behavior of typing.List.
Do not have opinion.
4. The PEP doesn't provide a name for the proxy type. I propose to name it GenericAlias and to add it to the types.py module, so it can be imported as types.GenericAlias (note: types; not typing).
Great! It can be used for implementing `__class_getitem__` for many Python classes (which already have trivial `__class_getitem__`) without importing the typing module.
5. I think the following classes should also be made generic: re.Pattern, re.Match, io.IO.
But it would be nice to not make generic the BaseIO subclasses BufferedIOBase, RawIOBase and TextIOBase (if it is not too hard).
6. How far should we go with replacing everything in typing.py that shadows something that has become generic due to the PEP's changes? (collections, collections.abc, contextlib, re, io). I expect this to be somewhat controversial (Ivan may have some ideas) and difficult to keep compatible with 3.8.
Is the goal of the PEP to make typing.List an alias to list? Sorry, it's hard for me to read, so I could miss it.
7. I haven't figured out yet how to support type[x] (which should have the same meaning as typing.Type[x]) without it also enabling int[x], which we would like to prevent. (We don't want to just add object.__class_getitem__!)
We can add type.__class_getitem__ and check that "cls is type". There is a precedence in type.__call__: type('1') returns a type, but int('1') returns an int. Using the same type.__call__.
8. I have a tricky bug remaining where inheriting from Tuple[something] causes problems. Take this example:
class C(Tuple[int]): pass
If we now examine C.__mro__, we see that it is (C, tuple, typing.Generic, object). IOW, inheriting from typing.Tuple causes builtins.tuple to be inserted into the MRO. The problem is that in my branch, tuple.__class_getitem__ overrides Generic.__class_getitem__, whereas the latter is needed to satisfy some tests for typing.py (and presumably to enable certain special behavior). I'm mostly looking for suggestions on how to make those failing tests work (see https://github.com/python/cpython/commit/35cd5f477c9da985880130d72297335e010...). Interesting problem.
On Wed, Jan 29, 2020 at 10:55 AM Serhiy Storchaka
29.01.20 20:11, Guido van Rossum пише:
PEP 585: https://www.python.org/dev/peps/pep-0585/
With the help of Ethan Smith (and some code review by Serhiy) I have started a prototype of PEP585. You can see my code at https://github.com/python/cpython/pull/18239 (a PR to CPython) or directly in my branch (where I take PRs too): https://github.com/gvanrossum/cpython/tree/pep585. I am cautiously optimistic that we can get this integrated into Python 3.9.
It is an excellent work. It is much simpler than I expected.
Well, it's not over till it's over. :-)
Based on this work in progress I have a bunch of questions and suggestions for PEP 585.
1. The PEP says that e.g. list[str][str] should be forbidden. But there's at least one example where we currently allow chained subscript operations. This currently works (in mypy as well as at runtime):
import typing T = typing.TypeVar("T") D = typing.Dict[str, T] # Now D is a generic type x: D[int] # Now x has type dict[str, int]
But if, in my branch, we replace Dict with dict, we get an error on the last line:
import typing T = typing.TypeVar("T") D = dict[str, T] x: D[int] # E: TypeError: 'GenericAlias' object is not subscriptable
The last line raises a TypeError.
We can add `__getitem__` which replaces type vars with actual parameters.
I tried to write the implementation here, but it turned out to be quite complicated, taking to account that the same type variable can be used multiple times and that they can be nested. It is not an easy problem, but neither very hard.
Yeah, my main question is whether we should do this or not -- if we decide we need to do this it won't be that hard to do, but it would mean the C code is more complicated, and we couldn't change it as easily if we found problems.
2. The PEP says that isinstance([1, 2, 3], list[int]) should be True. I think this is wrong -- with typing.List[int] it currently raises a TypeError, and I think we should keep forbidding this. Similar with issubclass.
Agree.
3. The PEP says that list == list[str]. This currently returns false in my branch (apparently this doesn't use __getattribute__ to get the __eq__ operation), and I think I like that better. It also matches the behavior of typing.List.
Do not have opinion.
I hope Łukasz will either agree to change the PEP, or have a convincing argument why we should make this work.
4. The PEP doesn't provide a name for the proxy type. I propose to name it GenericAlias and to add it to the types.py module, so it can be imported as types.GenericAlias (note: types; not typing).
Great! It can be used for implementing `__class_getitem__` for many Python classes (which already have trivial `__class_getitem__`) without importing the typing module.
Right. This part is very simple.
5. I think the following classes should also be made generic: re.Pattern, re.Match, io.IO.
But it would be nice to not make generic the BaseIO subclasses BufferedIOBase, RawIOBase and TextIOBase (if it is not too hard).
I have to admit that I'm lukewarm about this because I don't know the code any more... If it's easy to do we should do this, I agree.
6. How far should we go with replacing everything in typing.py that shadows something that has become generic due to the PEP's changes? (collections, collections.abc, contextlib, re, io). I expect this to be somewhat controversial (Ivan may have some ideas) and difficult to keep compatible with 3.8.
Is the goal of the PEP to make typing.List an alias to list? Sorry, it's hard for me to read, so I could miss it.
Heh, good question. I had assumed that typing.py would be changed so that typing.List is builtins.list, but the PEP does not specify that -- it merely states that using typing.List etc. will be deprecated without generating runtime warnings (instead the static checkers should warn), and that these will be removed 5 years after 3.9.0 is released. Although the PEP doesn't *exclude* that typing.List becomes a mere alias for builtins.list -- it just doesn't require it. (If I read the PEP correctly.)
7. I haven't figured out yet how to support type[x] (which should have the same meaning as typing.Type[x]) without it also enabling int[x], which we would like to prevent. (We don't want to just add object.__class_getitem__!)
We can add type.__class_getitem__ and check that "cls is type". There is a precedence in type.__call__: type('1') returns a type, but int('1') returns an int. Using the same type.__call__.
IIRC that's done by overriding int.__new__. (Though type() does have a special case -- type(x) returns the type of x, while type(a, b, c) creates a new class with name a, bases b, and namespace c.)
8. I have a tricky bug remaining where inheriting from Tuple[something] causes problems. Take this example:
class C(Tuple[int]): pass
If we now examine C.__mro__, we see that it is (C, tuple, typing.Generic, object). IOW, inheriting from typing.Tuple causes builtins.tuple to be inserted into the MRO. The problem is that in my branch, tuple.__class_getitem__ overrides Generic.__class_getitem__, whereas the latter is needed to satisfy some tests for typing.py (and presumably to enable certain special behavior). I'm mostly looking for suggestions on how to make those failing tests work (see https://github.com/python/cpython/commit/35cd5f477c9da985880130d72297335e010... ).
Interesting problem.
There are actually several different issues that are revealed by these failures. A. In typing.py, the dunders of a generic or parameterized class are different! - `__origin__` same thing - `__args__` what PEP 585 calls `__parameters__` - `__parameters__` only parameters that are *free* (i.e. are typevars) For example: ```
import typing T = typing.TypeVar("T") D = typing.Dict[str, T] D.__origin__
D.__args__ ( , ~T) D.__parameters__ (~T,)
B. Generic classes as defined in typing.py only allow `__class_getitem__`
calls with the same number of values as there are in `__parameters__`.
C. `__orig_class__` of an instantiated generic class with parameters
(e.g. MyList[int]()) points to the generic class that was
instantiated. (Only for pure Python classes, since this is added to
the `__dict__`, and not if there's no `__dict__`.)
For example:
class C(typing.Generic[T]): pass ... x = C[int]() x.__class__
x.__orig_class__ __main__.C[int]
The assignment to `__orig_class__` is ignored if it fails (e.g. if class
`C` has `__slots__`). It would not be very hard to implement this in the C
version of GenericAlias. (Alternatively, we might be able to implement Saul
Shanabrook's feature request directly -- i.e. pass the proxy/GenericAlias
instance to class methods instead of the origin class.)
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Yeah, my main question is whether we should do this or not -- if we decide we need to do this it won't be that hard to do, but it would mean the C code is more complicated, and we couldn't change it as easily if we found problems.
If we do, it would be nice if this was accessible as an externally available function, like `replace_typevars(list[int, T], {T: str})`. I imagine this would require another PEP though? AFAIK there haven't been any designs on including a library in core that allows you to manipulate and compare types as first class entities, is that right? Along with replacing typevars, it would nice to have a builtin way of being able to walk a type tree, extract typevars (i.e. `extract_typevars(list, list[str]) == {T: str})`), and check for subtype compatibility.
On Wed, Jan 29, 2020 at 1:51 PM Saul Shanabrook via Typing-sig < typing-sig@python.org> wrote:
Yeah, my main question is whether we should do this or not -- if we decide we need to do this it won't be that hard to do, but it would mean the C code is more complicated, and we couldn't change it as easily if we found problems.
If we do, it would be nice if this was accessible as an externally available function, like `replace_typevars(list[int, T], {T: str})`. I imagine this would require another PEP though?
It's trivial to code this yourself though using the __origin__ and __parameters__/__args__ attributes. I don't think it's worth having in the stdlib. I don't actually want to encourage introspecting generics -- I begrudgingly allow it.
AFAIK there haven't been any designs on including a library in core that allows you to manipulate and compare types as first class entities, is that right? Along with replacing typevars, it would nice to have a builtin way of being able to walk a type tree, extract typevars (i.e. `extract_typevars(list, list[str]) == {T: str})`), and check for subtype compatibility.
See above for why not. :-) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
Whoops, I just realized there's a typo in the subject of this thread. I'll continue with the correct PEP number (585). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
The PEP says that list == list[str]. This currently returns false in my branch (apparently this doesn't use __getattribute__ to get the __eq__ operation), and I think I like that better. It also matches the behavior of typing.List.
I aggree, I also like this better. In a lot of my code I find I need to compare the equality of different parametrized types and it would be too bad to have to hand code this logic.
On Wed, Jan 29, 2020 at 1:25 PM Saul Shanabrook via Typing-sig < typing-sig@python.org> wrote:
The PEP says that list == list[str]. This currently returns false in my branch (apparently this doesn't use __getattribute__ to get the __eq__ operation), and I think I like that better. It also matches the behavior of typing.List.
I aggree, I also like this better. In a lot of my code I find I need to compare the equality of different parametrized types and it would be too bad to have to hand code this logic.
Well, the idea would be that list[int] == list[int] would also return False. There currently isn't a cache and I don't think it's worth implementing one -- the entire thing is currently a single memory allocation plus some increfs, which is probably cheaper than the Python implementation in typing.py with cache. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
On Wed, Jan 29, 2020 at 1:46 PM Saul Shanabrook via Typing-sig < typing-sig@python.org> wrote:
Ah, I assumed they would be compared by value, not be reference.
(I.e., list[int] vs. list[int].) I just realized that typing.py implements __eq__ on its _GenericAlias class -- maybe we should do that too. So list[int] != list, but list[int] == list[int] even if these are two distinct objects. It's easy enough (just compare __origin__ and __parameters__/__args__). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
Am 29.01.20 um 19:11 schrieb Guido van Rossum:
5. I think the following classes should also be made generic: re.Pattern, re.Match, io.IO.
+1
6. How far should we go with replacing everything in typing.py that shadows something that has become generic due to the PEP's changes? (collections, collections.abc, contextlib, re, io). I expect this to be somewhat controversial (Ivan may have some ideas) and difficult to keep compatible with 3.8.
From a user's perspective I'd say as much as possible, but it does not need to be forced, at least at the current point. Also from a user's perspective it doesn't make much sense that there are two versions of the same functionality, and that some generic types need to be imported from typing, while others can be imported directly. Furthermore it complicates adding type annotations, because suddenly the imports need to be changed. - Sebastian
participants (4)
-
Guido van Rossum -
Saul Shanabrook -
Sebastian Rittau -
Serhiy Storchaka