`__class_getitem__` returning a mix of GenericAlias and concrete classes?
Hi all, In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`. Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also. However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct. This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type. My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`. (Also, this isn't a fixed set in theory, users can create new subclasses in principle.) Hopefully this is very obvious in one direction, but unfortunately, I have no idea. Thanks and Cheers, Sebastian
This sounds like a question for the runtime checking crowd, e.g. the authors of Pydantic and similar systems. Would returning the appropriate concrete subtype in cases where it can be determined make sense for them? I don't know, we should ask Pydantic. For static checkers like mypy, Pyre, pytype, pyright I don't think the question even makes sense, right? np.dtype[whatever] is what it is, and I would hope that the concrete types you are planning to return at runtime are subtypes of those, e.g. np.types.Float64DType should be a subtype of np.dtype[np.float64]. GenericAlias doesn't exist in the static typing world. I'm a little concerned about your mention of "the typing step". What are you referring to there? The current crop of static type checkers don't produce output (other than error messages), and at runtime there is no separate typing "step" either -- there are just type annotations that are used by some tools (e.g. Pydantic). The static checkers don't run the code and don't call your `__class_getitem__` method, so whatever you do there is irrelevant for static type checkers. On Tue, Mar 14, 2023 at 7:46 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`.
Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also.
However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct.
This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type.
My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`.
(Also, this isn't a fixed set in theory, users can create new subclasses in principle.)
Hopefully this is very obvious in one direction, but unfortunately, I have no idea.
Thanks and Cheers,
Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
For my runtime typing system, that would be great. Right now, I have to do it by capturing it in a context manager and resolving it in-context: https://github.com/fondat/fondat/blob/main/fondat/types.py#L162 On Tue, 2023-03-14 at 10:41 -0700, Guido van Rossum wrote:
This sounds like a question for the runtime checking crowd, e.g. the authors of Pydantic and similar systems. Would returning the appropriate concrete subtype in cases where it can be determined make sense for them? I don't know, we should ask Pydantic.
For static checkers like mypy, Pyre, pytype, pyright I don't think the question even makes sense, right? np.dtype[whatever] is what it is, and I would hope that the concrete types you are planning to return at runtime are subtypes of those, e.g. np.types.Float64DType should be a subtype of np.dtype[np.float64]. GenericAlias doesn't exist in the static typing world.
I'm a little concerned about your mention of "the typing step". What are you referring to there? The current crop of static type checkers don't produce output (other than error messages), and at runtime there is no separate typing "step" either -- there are just type annotations that are used by some tools (e.g. Pydantic). The static checkers don't run the code and don't call your `__class_getitem__` method, so whatever you do there is irrelevant for static type checkers.
On Tue, Mar 14, 2023 at 7:46 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`.
Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also.
However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct.
This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type.
My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`.
(Also, this isn't a fixed set in theory, users can create new subclasses in principle.)
Hopefully this is very obvious in one direction, but unfortunately, I have no idea.
Thanks and Cheers,
Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
On Tue, 2023-03-14 at 11:41 -0700, Paul Bryan wrote:
For my runtime typing system, that would be great. Right now, I have to do it by capturing it in a context manager and resolving it in- context: https://github.com/fondat/fondat/blob/main/fondat/types.py#L162
Good to know that you would like it. Maybe due to my ignorance of typing, but it is still a bit unclear to me what exact setup you would expect (and whether there might be fallout if changing). Presumably: 1. `np.dtype[TypeVar/Any/?]` has to still return the GenericAlias? 2. `np.dtype[not scalar_type and not TypeVar]` would be OK to error? (I don't mind returning a weird GenericAlias, but in that case maybe not want to advertise runtime usage.) Sorry, if these are now just showing my ignorance (maybe @Paul Bryan and I can continue discussing off-list), I suppose I would then add (to the typing stubs): class Float64DType(np.dtype[np.float64]): # explicit subclass for the specialization? pass @property def finfo(self) -> Any: ... # I may want to add this eventually and: class dtype: @overload def __class_getitem__(self, item: np.float64) -> Float64DType: ... And both static and runtime checkers should be able to figure things out, e.g. that `dtype[np.float64]` provides an additional property? - Sebastian
On Tue, 2023-03-14 at 10:41 -0700, Guido van Rossum wrote:
This sounds like a question for the runtime checking crowd, e.g.
<snip>
I'm a little concerned about your mention of "the typing step". What are you referring to there? The current crop of static type checkers
Right, I think it just shows I am not a typing user myself.
don't produce output (other than error messages), and at runtime there is no separate typing "step" either -- there are just type annotations that are used by some tools (e.g. Pydantic). The static checkers don't run the code and don't call your `__class_getitem__` method, so whatever you do there is irrelevant for static type checkers.
On Tue, Mar 14, 2023 at 7:46 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`.
Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also.
However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct.
This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type.
My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`.
(Also, this isn't a fixed set in theory, users can create new subclasses in principle.)
Hopefully this is very obvious in one direction, but unfortunately, I have no idea.
Thanks and Cheers,
Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
In typing jargon, overloads are "ad hoc". They can choose different types based on the types of their arguments. Generic class instantiation like dtype[float] is *not* ad hoc. Type checkers determine the signature of the class dtype[float64] by taking the generic signature of dtype and substituting occurrences of the type parameter with the type argument float64. I think all static checkers will ignore your implementation of `__class_getitem__`. You can see that there is something weird with: ```python class Float64DType(np.dtype[np.float64]): # explicit subclass for the specialization? pass ``` If dtype[float64] was actually Float64DType then this is a class that is an immediate subclass of itself. Checkers will not know that dtype[float] has extra properties. Checkers will not know that some instantiations of dtype are runtime errors, unless you use a bound or constraints on dtype's type parameter to indicate that. On Wed, Mar 15, 2023 at 10:59 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
On Tue, 2023-03-14 at 11:41 -0700, Paul Bryan wrote:
For my runtime typing system, that would be great. Right now, I have to do it by capturing it in a context manager and resolving it in- context: https://github.com/fondat/fondat/blob/main/fondat/types.py#L162
Good to know that you would like it. Maybe due to my ignorance of typing, but it is still a bit unclear to me what exact setup you would expect (and whether there might be fallout if changing). Presumably:
1. `np.dtype[TypeVar/Any/?]` has to still return the GenericAlias?
2. `np.dtype[not scalar_type and not TypeVar]` would be OK to error? (I don't mind returning a weird GenericAlias, but in that case maybe not want to advertise runtime usage.)
Sorry, if these are now just showing my ignorance (maybe @Paul Bryan and I can continue discussing off-list), I suppose I would then add (to the typing stubs):
class Float64DType(np.dtype[np.float64]): # explicit subclass for the specialization? pass
@property def finfo(self) -> Any: ... # I may want to add this eventually
and:
class dtype: @overload def __class_getitem__(self, item: np.float64) -> Float64DType: ...
And both static and runtime checkers should be able to figure things out, e.g. that `dtype[np.float64]` provides an additional property?
- Sebastian
On Tue, 2023-03-14 at 10:41 -0700, Guido van Rossum wrote:
This sounds like a question for the runtime checking crowd, e.g.
<snip>
I'm a little concerned about your mention of "the typing step". What are you referring to there? The current crop of static type checkers
Right, I think it just shows I am not a typing user myself.
don't produce output (other than error messages), and at runtime there is no separate typing "step" either -- there are just type annotations that are used by some tools (e.g. Pydantic). The static checkers don't run the code and don't call your `__class_getitem__` method, so whatever you do there is irrelevant for static type checkers.
On Tue, Mar 14, 2023 at 7:46 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`.
Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also.
However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct.
This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type.
My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`.
(Also, this isn't a fixed set in theory, users can create new subclasses in principle.)
Hopefully this is very obvious in one direction, but unfortunately, I have no idea.
Thanks and Cheers,
Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: kmillikin@google.com
On Wed, 2023-03-15 at 11:53 +0100, Sebastian Berg wrote:
On Tue, 2023-03-14 at 11:41 -0700, Paul Bryan wrote:
For my runtime typing system, that would be great. Right now, I have to do it by capturing it in a context manager and resolving it in- context: https://github.com/fondat/fondat/blob/main/fondat/types.py#L162
Good to know that you would like it. Maybe due to my ignorance of typing, but it is still a bit unclear to me what exact setup you would expect (and whether there might be fallout if changing). Presumably:
1. `np.dtype[TypeVar/Any/?]` has to still return the GenericAlias?
If your implementation can somehow ingest a TypeVar and resolve it to a concrete type, that seems fair to me. I expect though that __class_getitem__ will lack the context to do so at runtime.
2. `np.dtype[not scalar_type and not TypeVar]` would be OK to error? (I don't mind returning a weird GenericAlias, but in that case maybe not want to advertise runtime usage.)
Yes. There is at least one class in stdlib that raises error in `__class_getitem__`.
Sorry, if these are now just showing my ignorance (maybe @Paul Bryan and I can continue discussing off-list), I suppose I would then add (to the typing stubs):
class Float64DType(np.dtype[np.float64]): # explicit subclass for the specialization? pass
@property def finfo(self) -> Any: ... # I may want to add this eventually
and:
class dtype: @overload def __class_getitem__(self, item: np.float64) -> Float64DType: ...
And both static and runtime checkers should be able to figure things out, e.g. that `dtype[np.float64]` provides an additional property?
- Sebastian
On Tue, 2023-03-14 at 10:41 -0700, Guido van Rossum wrote:
This sounds like a question for the runtime checking crowd, e.g.
<snip>
I'm a little concerned about your mention of "the typing step". What are you referring to there? The current crop of static type checkers
Right, I think it just shows I am not a typing user myself.
don't produce output (other than error messages), and at runtime there is no separate typing "step" either -- there are just type annotations that are used by some tools (e.g. Pydantic). The static checkers don't run the code and don't call your `__class_getitem__` method, so whatever you do there is irrelevant for static type checkers.
On Tue, Mar 14, 2023 at 7:46 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`.
Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also.
However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct.
This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type.
My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`.
(Also, this isn't a fixed set in theory, users can create new subclasses in principle.)
Hopefully this is very obvious in one direction, but unfortunately, I have no idea.
Thanks and Cheers,
Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
On Wed, 2023-03-15 at 10:27 -0700, Paul Bryan wrote:
On Wed, 2023-03-15 at 11:53 +0100, Sebastian Berg wrote:
On Tue, 2023-03-14 at 11:41 -0700, Paul Bryan wrote:
For my runtime typing system, that would be great. Right now, I have to do it by capturing it in a context manager and resolving it in- context: https://github.com/fondat/fondat/blob/main/fondat/types.py#L162
Good to know that you would like it. Maybe due to my ignorance of typing, but it is still a bit unclear to me what exact setup you would expect (and whether there might be fallout if changing). Presumably:
1. `np.dtype[TypeVar/Any/?]` has to still return the GenericAlias?
If your implementation can somehow ingest a TypeVar and resolve it to a concrete type, that seems fair to me. I expect though that __class_getitem__ will lack the context to do so at runtime.
Presumably one can implement anything, but the question is about what users need/expect? And whether there be dragons. Despite your enthusiasm, it is not becoming any clearer to me that returning the concrete class wouldn't lead to problems by mixing two things (maybe only for runtime type checking or user surprises). To be clear: I am not interested in spending time exploring this. I am interested whether its clearly a good thing to do or whether it is better to just shelve it for now (because the question came up again whether we cannot support this as runtime syntax). Presumably, code might say: AcceptedDTypes = np.dtype[Union[...]] guarded (or unguarded) by `if TYPE_CHECKING`. And should: ST = TypeVar("ST") DT = np.dtype[ST] actual = DT[np.float64] be expected to work at runtime (maybe `Any` doesn't matter)?
2. `np.dtype[not scalar_type and not TypeVar]` would be OK to error? (I don't mind returning a weird GenericAlias, but in that case maybe not want to advertise runtime usage.)
Yes. There is at least one class in stdlib that raises error in `__class_getitem__`.
Aha? Which class, I couldn't find anything that seemed like it does something related? My whole question could probably also be summarized as: Is there any somewhat tested/settled prior art I can steal? - Sebastian
Sorry, if these are now just showing my ignorance (maybe @Paul Bryan and I can continue discussing off-list), I suppose I would then add (to the typing stubs):
class Float64DType(np.dtype[np.float64]): # explicit subclass for the specialization? pass
@property def finfo(self) -> Any: ... # I may want to add this eventually
and:
class dtype: @overload def __class_getitem__(self, item: np.float64) -> Float64DType: ...
And both static and runtime checkers should be able to figure things out, e.g. that `dtype[np.float64]` provides an additional property?
- Sebastian
On Tue, 2023-03-14 at 10:41 -0700, Guido van Rossum wrote:
This sounds like a question for the runtime checking crowd, e.g.
<snip>
I'm a little concerned about your mention of "the typing step". What are you referring to there? The current crop of static type checkers
Right, I think it just shows I am not a typing user myself.
don't produce output (other than error messages), and at runtime there is no separate typing "step" either -- there are just type annotations that are used by some tools (e.g. Pydantic). The static checkers don't run the code and don't call your `__class_getitem__` method, so whatever you do there is irrelevant for static type checkers.
On Tue, Mar 14, 2023 at 7:46 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`.
Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also.
However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct.
This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type.
My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`.
(Also, this isn't a fixed set in theory, users can create new subclasses in principle.)
Hopefully this is very obvious in one direction, but unfortunately, I have no idea.
Thanks and Cheers,
Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
On Thu, 2023-03-16 at 10:33 +0100, Sebastian Berg wrote:
On Wed, 2023-03-15 at 10:27 -0700, Paul Bryan wrote:
On Wed, 2023-03-15 at 11:53 +0100, Sebastian Berg wrote:
On Tue, 2023-03-14 at 11:41 -0700, Paul Bryan wrote:
For my runtime typing system, that would be great. Right now, I have to do it by capturing it in a context manager and resolving it in- context: https://github.com/fondat/fondat/blob/main/fondat/types.py#L162
Good to know that you would like it. Maybe due to my ignorance of typing, but it is still a bit unclear to me what exact setup you would expect (and whether there might be fallout if changing). Presumably:
1. `np.dtype[TypeVar/Any/?]` has to still return the GenericAlias?
If your implementation can somehow ingest a TypeVar and resolve it to a concrete type, that seems fair to me. I expect though that __class_getitem__ will lack the context to do so at runtime.
Presumably one can implement anything, but the question is about what users need/expect? And whether there be dragons.
I think users probably expect that` __class_getitem__` returns what's passed in Class[...]. However, stdlib classes can deviate from their inputs. For example, deduplicating literals and unions, and combining Union and UnionType:
Literal[1, 1, 2, 3] typing.Literal[1, 2, 3] Union[int, int, str] typing.Union[int, str] int | Union[str, float] typing.Union[int, str, float]
Despite your enthusiasm, it is not becoming any clearer to me that returning the concrete class wouldn't lead to problems by mixing two things (maybe only for runtime type checking or user surprises).
The safest option would be to follow convention and just return a GenericAlias that captures whatever is passed into `__class_getitem__`.
To be clear: I am not interested in spending time exploring this. I am interested whether its clearly a good thing to do or whether it is better to just shelve it for now (because the question came up again whether we cannot support this as runtime syntax).
It is not clearly a good thing to deviate from what virtually everything else is doing today.
Presumably, code might say:
AcceptedDTypes = np.dtype[Union[...]]
guarded (or unguarded) by `if TYPE_CHECKING`. And should:
ST = TypeVar("ST") DT = np.dtype[ST] actual = DT[np.float64]
be expected to work at runtime (maybe `Any` doesn't matter)?
I don't know anyone who expects this to be resolved at runtime.
2. `np.dtype[not scalar_type and not TypeVar]` would be OK to error? (I don't mind returning a weird GenericAlias, but in that case maybe not want to advertise runtime usage.)
Yes. There is at least one class in stdlib that raises error in `__class_getitem__`.
Aha? Which class, I couldn't find anything that seemed like it does something related?
typing.Annotated[int] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.10/typing.py", line 312, in inner return func(*args, **kwds) File "/usr/lib/python3.10/typing.py", line 1698, in __class_getitem__ raise TypeError("Annotated[...] should be used " TypeError: Annotated[...] should be used with at least two arguments (a type and an annotation).
My whole question could probably also be summarized as: Is there any somewhat tested/settled prior art I can steal?
Not that I know of.
- Sebastian
Sorry, if these are now just showing my ignorance (maybe @Paul Bryan and I can continue discussing off-list), I suppose I would then add (to the typing stubs):
class Float64DType(np.dtype[np.float64]): # explicit subclass for the specialization? pass
@property def finfo(self) -> Any: ... # I may want to add this eventually
and:
class dtype: @overload def __class_getitem__(self, item: np.float64) -> Float64DType: ...
And both static and runtime checkers should be able to figure things out, e.g. that `dtype[np.float64]` provides an additional property?
- Sebastian
On Tue, 2023-03-14 at 10:41 -0700, Guido van Rossum wrote:
This sounds like a question for the runtime checking crowd, e.g.
<snip>
I'm a little concerned about your mention of "the typing step". What are you referring to there? The current crop of static type checkers
Right, I think it just shows I am not a typing user myself.
don't produce output (other than error messages), and at runtime there is no separate typing "step" either -- there are just type annotations that are used by some tools (e.g. Pydantic). The static checkers don't run the code and don't call your `__class_getitem__` method, so whatever you do there is irrelevant for static type checkers.
On Tue, Mar 14, 2023 at 7:46 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
In NumPy, we have `np.dtype` as a superclass for all DTypes. This implements `__class_getitem__` for typing to allow `np.dtype[Any]` or `np.dtype[np.float64]` returning a `GenericAlias`.
Now, `np.dtype[Any]` probably must return a GenericAlias. Maybe `np.dtype[something_that_makes_little_sense]` also.
However, `np.dtype[np.float64]` is well defined, there is a concrete runtime subclass of `np.dtype` representing exactly that construct.
This would be the same as `array.array['l']` if `array.array` had concrete subclasses implementation for each supported type.
My question is: would it be possible or even better to returning the concrete class (probably only when it exists)? Or would that just confuse the typing step, e.g. because we cannot actually type that `np.dtype[np.float64] -> np.types.Float64DType`.
(Also, this isn't a fixed set in theory, users can create new subclasses in principle.)
Hopefully this is very obvious in one direction, but unfortunately, I have no idea.
Thanks and Cheers,
Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
The safest option would be to follow convention and just return a GenericAlias that captures whatever is passed into `__class_getitem__`.
The main problem with `GenericAlias` is that it's this undocumented class where it seems like effort has been taken to keep it as a private implementation detail. Libraries like Pydantic (and it sounds like numpy?) want to keep track of what generic parameters were substituted and use that information when instantiated (or more ideally whenever there are no more type vars left and one could build a "concrete" type). But GenericAlias doesn't expose that functionality, despite internally tracking all of this. It's only interaction with the original class is to forward any calls from `GenericAlias.__call__(*args, **kwargs)` to `cls(*args, **kwargs)`. Could we devise a way for `GenericAlias` to be more useful to libraries that do runtime type checking? Or if the desire is to keep GenericAlias could we devise an API for something that can be returned from `__class_getitem__` that is compatible with being used in `List[Thing[T]][int]`? The simplest form I can think of for the latter is commenting out that line of code and allowing `__parameters__` to be collected from arbitrary objects, but I imagine there's a reason that check is there / something would break.
You should be able to use the __parameters__, __origin__ and __args__ dunders to introspect things like list[Thing[T]][int] fully. On Thu, Mar 16, 2023 at 9:09 AM Adrian Garcia Badaracco <adrian@adriangb.com> wrote:
The safest option would be to follow convention and just return a GenericAlias that captures whatever is passed into `__class_getitem__`.
The main problem with `GenericAlias` is that it's this undocumented class where it seems like effort has been taken to keep it as a private implementation detail. Libraries like Pydantic (and it sounds like numpy?) want to keep track of what generic parameters were substituted and use that information when instantiated (or more ideally whenever there are no more type vars left and one could build a "concrete" type). But GenericAlias doesn't expose that functionality, despite internally tracking all of this. It's only interaction with the original class is to forward any calls from `GenericAlias.__call__(*args, **kwargs)` to `cls(*args, **kwargs)`.
Could we devise a way for `GenericAlias` to be more useful to libraries that do runtime type checking? Or if the desire is to keep GenericAlias could we devise an API for something that can be returned from `__class_getitem__` that is compatible with being used in `List[Thing[T]][int]`? The simplest form I can think of for the latter is commenting out that line of code and allowing `__parameters__` to be collected from arbitrary objects, but I imagine there's a reason that check is there / something would break. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
The issue is the other way around: `list[Thing[T]]` won't use any of those to inspect _you_ (assuming you are the return value of Thing.__class_getitem__`) unless you are specifically an instance of `typing._GenericAlias`. But if you are a an instance of `typing._GenericAlias` then you have no control over yourself because the whole thing seems to be an implementation detail of `typing.py`.
You seem to have hijacked this discussion for a different issue altogether. I recommend treating both typing._GenericAlias and types.GenericAlias as opaque implementation details. It is true that list[X][Y] needs to introspect X so that it can tell whether it contains a type variable (if not, the ...[Y] construct is invalid). I presume this is where you would like to see a change? Can you propose specific behavior that would satisfy you? On Thu, Mar 16, 2023 at 10:31 AM Adrian Garcia Badaracco < adrian@adriangb.com> wrote:
The issue is the other way around: `list[Thing[T]]` won't use any of those to inspect _you_ (assuming you are the return value of Thing.__class_getitem__`) unless you are specifically an instance of `typing._GenericAlias`. But if you are a an instance of `typing._GenericAlias` then you have no control over yourself because the whole thing seems to be an implementation detail of `typing.py`. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
I think / hope it is the same topic really, but I am sorry if I am misunderstanding the original point that is being made. It is at the least very closely related.
It is true that list[X][Y] needs to introspect X so that it can tell whether it contains a type variable (if not, the ...[Y] construct is invalid). I presume this is where you would like to see a change? Can you propose specific behavior that would satisfy you?
I think the easiest way to make `X` introspectable is for the utility functions in `typing.py` to treat an arbitrary object that provides `__parameters__`, `__origin__` and `__args__` just like they do `GenericAlias`. Or for there to be a `Protocol` defining what the return type of `__class_getitiem__` needs to do to be introspectable by `typing.py`. It might be as easy as deleting this check: https://github.com/python/cpython/blob/70185de1abfe428049a5c43d58fcb656b46db.... I tried commenting it out and running the cpython tests, there are failures but they all seem to be checks for the specific behavior that line provides (I think).
I'll let you argue that with the people who most recently touched that code. On Thu, Mar 16, 2023 at 2:16 PM Adrian Garcia Badaracco <adrian@adriangb.com> wrote:
I think / hope it is the same topic really, but I am sorry if I am misunderstanding the original point that is being made. It is at the least very closely related.
It is true that list[X][Y] needs to introspect X so that it can tell whether it contains a type variable (if not, the ...[Y] construct is invalid). I presume this is where you would like to see a change? Can you propose specific behavior that would satisfy you?
I think the easiest way to make `X` introspectable is for the utility functions in `typing.py` to treat an arbitrary object that provides `__parameters__`, `__origin__` and `__args__` just like they do `GenericAlias`. Or for there to be a `Protocol` defining what the return type of `__class_getitiem__` needs to do to be introspectable by `typing.py`. It might be as easy as deleting this check: https://github.com/python/cpython/blob/70185de1abfe428049a5c43d58fcb656b46db.... I tried commenting it out and running the cpython tests, there are failures but they all seem to be checks for the specific behavior that line provides (I think). _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
To be clear I'm open to any solution here that allows `list[X][Y]` to not fail at runtime wheen `X` is not a `typing.GenericAlias`. It doesn't have to be changing those specific lines of code.
Maybe you can provide a unit test for the desired behavior? On Thu, Mar 16, 2023 at 16:13 Adrian Garcia Badaracco <adrian@adriangb.com> wrote:
To be clear I'm open to any solution here that allows `list[X][Y]` to not fail at runtime wheen `X` is not a `typing.GenericAlias`. It doesn't have to be changing those specific lines of code. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
`np.dtype[TypeVar/Any/?]` has to still return the GenericAlias?
One thing I've found that is quite interesting is that if you don't return a `GenericAlias` from `__class_getitem__` some things work fine: ```python MyIntTtype = MyType[int] ``` But things break down if you try to mix that with things that do use `GenericAlias`: ```python MyTypeList = List[MyType[T]] # type variables from whatever MyType.__class_get_item__ returns are not collected MyIntTypeList = MyTypeList[int] # runtime error ``` This is because regular classes are excluded from the machinery that `List.__class_getitme__/GenericAlias.__class_getitem__` uses to collect type vars: https://github.com/python/cpython/blob/70185de1abfe428049a5c43d58fcb656b46db...
participants (5)
-
Adrian Garcia Badaracco -
Guido van Rossum -
Kevin Millikin -
Paul Bryan -
Sebastian Berg