Introspecting a generic type at runtime
I'm trying to write a function that introspects the types of fields in a dataclass (this is to automatically generate SQL table creation statements). However, I have a problem when dealing with generic types. In particular, if I have a field with the type `Optional[int]`, I want to create a SQL column of type `INTEGER`, but without a `NOT NULL` constraint (so nulls are allowed). But I can't work out how to introspect the type of the field, to determine that it's based on `int`. ``` from dataclasses import dataclass, fields from typing import Optional @dataclass class Example: mandatory: int optional: Optional[int] for f in fields(Example): print(f.type) ``` This displays: ``` <class 'int'> typing.Optional[int] ``` But from `f.type`, I can't find any way to confirm it's an optional type (issubclass gives "TypeError: Subscripted generics cannot be used with class and instance checks") and I can't recover the "underlying" integer type. I understand that introspecting types at runtime is not the key use case for annotations, but with the availability of dataclasses in the standard library, it seems like this sort of usage is only going to become more common, as it's an extremely obvious way to handle this type of requirement, and the alternative, using field metadata, is very verbose: ```
``` Is there a reasonable way to do this? Or is it something that's likely to be added in the foreseeable future?
Here's my implementation... NoneType = type(None) def split_annotated(hint): """Return a tuple containing the python type and annotations.""" if not typing.get_origin(hint) is typing.Annotated: return hint, () args = typing.get_args(hint) return args[0], args[1:] def is_optional(hint): """Return if the specified type is optional (contains Union[..., None]).""" python_type, _ = split_annotated(hint) if not typing.get_origin(python_type) is Union: return python_type is NoneType for arg in typing.get_args(python_type): if is_optional(arg): return True return False Paul On Tue, 2021-06-01 at 13:36 +0000, Paul Moore wrote:
On Tue, 1 Jun 2021 at 14:44, Paul Bryan <pbryan@anode.ca> wrote:
Oh heck :-( I completely missed that there were introspection functions in the typing module. I assumed that everything in it was types (based on the fact that when skimming, that's basically all that you see). I was hunting in the inspect module, and looking at class attributes. Yes, get_origin/get_args is precisely what I need here. Thanks very much. Is there a good typing tutorial, for someone who doesn't really need much help with annotating basic stuff, but is starting to get into more complicated stuff like annotating async code, playing with introspection, etc? Paul
Hi, cattrs has some code to support use cases like this, but it's kind of messy (uses underscore imports from typing and is different depending on the version of typing/Python used). See here: https://github.com/Tinche/cattrs/blob/b079967b4e68c728654044097c36a1999fb0f8..., and then look at `__args__`. On Tue, Jun 1, 2021 at 3:40 PM Paul Moore <p.f.moore@gmail.com> wrote:
Have a look here: https://github.com/ilevkivskyi/typing_inspect Also, there are hidden gems in typing.py. Finally, beware of PEP 585 — in 3.9 you can find things like ‘int | None’, which are also introspectable. On Tue, Jun 1, 2021 at 06:40 Paul Moore <p.f.moore@gmail.com> wrote:
-- --Guido (mobile)
Adding on to the excellent stuff shared by others here: Paul, if you plan on supporting more complex annotations, I highly recommend using typing.get_type_hints (has some quirks which you can read in the docs) or inspect.get_annotations (new in 3.10, but less quirky). They support stuff like un-stringifying annotations which the `for f in fields(Example):` may not cover. get_args and get_origin are backwards compatible and support everything in 3.8 and higher. They also support 3.9's PEP 585 builtin generics like `list[int]` -- with one major exception: `collections.abc.Callable[]` 's introspection is wrong for 3.9.0 and 3.9.1. But I doubt SQL columns have any need for that type, so you should be able to safely ignore it :).
Finally, beware of PEP 585 — in 3.9 you can find things like ‘int | None’, which are also introspectable.
I think that's PEP 604 and only introspectable in 3.10. It's available (but not introspectable) in 3.9 if you utilize PEP 563's ``from __future__ import annotations``. Something cool is that PEP 585 also formalized introspection information in most builtin container types. Here's the docs for that https://docs.python.org/3.11/library/stdtypes.html#special-attributes-of-gen... . It's still recommended to use get_args and get_origin over these though, because there's some edge cases. Hope this helps :). KJ
Here's my implementation... NoneType = type(None) def split_annotated(hint): """Return a tuple containing the python type and annotations.""" if not typing.get_origin(hint) is typing.Annotated: return hint, () args = typing.get_args(hint) return args[0], args[1:] def is_optional(hint): """Return if the specified type is optional (contains Union[..., None]).""" python_type, _ = split_annotated(hint) if not typing.get_origin(python_type) is Union: return python_type is NoneType for arg in typing.get_args(python_type): if is_optional(arg): return True return False Paul On Tue, 2021-06-01 at 13:36 +0000, Paul Moore wrote:
On Tue, 1 Jun 2021 at 14:44, Paul Bryan <pbryan@anode.ca> wrote:
Oh heck :-( I completely missed that there were introspection functions in the typing module. I assumed that everything in it was types (based on the fact that when skimming, that's basically all that you see). I was hunting in the inspect module, and looking at class attributes. Yes, get_origin/get_args is precisely what I need here. Thanks very much. Is there a good typing tutorial, for someone who doesn't really need much help with annotating basic stuff, but is starting to get into more complicated stuff like annotating async code, playing with introspection, etc? Paul
Hi, cattrs has some code to support use cases like this, but it's kind of messy (uses underscore imports from typing and is different depending on the version of typing/Python used). See here: https://github.com/Tinche/cattrs/blob/b079967b4e68c728654044097c36a1999fb0f8..., and then look at `__args__`. On Tue, Jun 1, 2021 at 3:40 PM Paul Moore <p.f.moore@gmail.com> wrote:
Have a look here: https://github.com/ilevkivskyi/typing_inspect Also, there are hidden gems in typing.py. Finally, beware of PEP 585 — in 3.9 you can find things like ‘int | None’, which are also introspectable. On Tue, Jun 1, 2021 at 06:40 Paul Moore <p.f.moore@gmail.com> wrote:
-- --Guido (mobile)
Adding on to the excellent stuff shared by others here: Paul, if you plan on supporting more complex annotations, I highly recommend using typing.get_type_hints (has some quirks which you can read in the docs) or inspect.get_annotations (new in 3.10, but less quirky). They support stuff like un-stringifying annotations which the `for f in fields(Example):` may not cover. get_args and get_origin are backwards compatible and support everything in 3.8 and higher. They also support 3.9's PEP 585 builtin generics like `list[int]` -- with one major exception: `collections.abc.Callable[]` 's introspection is wrong for 3.9.0 and 3.9.1. But I doubt SQL columns have any need for that type, so you should be able to safely ignore it :).
Finally, beware of PEP 585 — in 3.9 you can find things like ‘int | None’, which are also introspectable.
I think that's PEP 604 and only introspectable in 3.10. It's available (but not introspectable) in 3.9 if you utilize PEP 563's ``from __future__ import annotations``. Something cool is that PEP 585 also formalized introspection information in most builtin container types. Here's the docs for that https://docs.python.org/3.11/library/stdtypes.html#special-attributes-of-gen... . It's still recommended to use get_args and get_origin over these though, because there's some edge cases. Hope this helps :). KJ
participants (5)
-
Guido van Rossum -
Ken Jin -
Paul Bryan -
Paul Moore -
Tin Tvrtković