On Fri, 26 Nov 2021 at 17:13, Guido van Rossum <guido@python.org> wrote:
>> Although the more I think about it, given that I believe dataclasses
>> use eval "under the hood", the less I understand *how* it manages to
>> do that without special-case knowledge of the dataclass decorator...)
>
> Static checkers special-case the @dataclass decorator. Eric Traut has a proposal to generalize this support (sorry, I'm in a rush, otherwise I'd dig up the link, but it's in the typing-sig archives).
:-( That's what I suspected, but it does mean that dataclasses has a
privilege that other libraries (like attrs, I guess?) don't get.
>> I'd like to see a clearer statement from "somewhere" about how APIs
>> should use annotations at runtime, such that Python users have a much
>> clearer intuition about APIs like the dataclass one, and library
>> designers can build their APIs based on a clear "common understanding"
>> of what to expect when annotations are used.
>
> Note that @dataclass itself is very careful not to use the annotations, it only looks for their *presence*. With one exception for ClassVar.
Understood. What I'm suggesting is that it would be good to have a
clear "common understanding" about whether libraries should be careful
like this, or whether it's OK to base runtime behaviour on type
annotations. And if it is OK, then what are good patterns of design
and behaviour? This is where the proposal to store annotations as
strings hit issues, because it appears to take the view that libraries
*shouldn't* be looking at the actual types specified by annotations
(or maybe that they should only do so via something like
`typing.get_type_hints`). There are other subtleties here (runtime
code needs to deal with the fact that int and "int" should be treated
the same) that there's no guidance on, again possibly because no-one
is really considering that use case.
You are hitting the nail on the head here. I'd say that so far the recommendation has been "use typing.get_type_hints(x) rather than x.__annotations__" -- this handles the equivalence between int and "int" (and hence forward references as well as 'from __future__ import annotations'). There's now also inspect.get_annotations() which has roughly the same functionality without quite so much bias towards typing, so perhaps we should recommend it over typing.get_type_hints() -- though before 3.10 inspect.get_annotations() didn't exist so you might have to fall back on using the other. (There are some semantic differences between the two that I'm glossing over here, because I'm not sure about what exactly they are. :-)
We now also have a document that recommends best practices (
https://docs.python.org/3/howto/annotations.html) although it's very new -- it appears Larry Hastings wrote it while he was pushing for PEP 649 (but it received buy-in from the static typing community as well).
So perhaps it isn't as bad as it seems, *if* you know where to look?
That said, I don't think the current static typing infrastructure would be prepared for an onslaught of modules that use type introspection at runtime to modify the behavior of classes a la attrs and dataclasses. I don't know enough about pydantic to say whether it also falls in this category. But it's definitely difficult to write code that makes use of type annotations at runtime that *also* passes static type checks by mypy etc.; you certainly shouldn't attempt to do so without having CI jobs to run the static checker and test the runtime-introspecting framework you're using. (It's not quite like trying to write code that's valid Python and Fortran at the same time, but it's not trivial. :-) There are also features of our static type system that tend not to be supported by the runtime-introspecting frameworks -- in particular, I'd expect generics and callable types to be hard to deal with at runtime. It's easy enough to do something at runtime with `def f(a: list[int]) -> int`. It's not so simple to handle `def f(a: Sequence[T]) -> T` or `def f(cb: (T) -> tuple[str, T], Sequence[T]) -> Mapping[str, T]`. Presumably frameworks like pydantic just don't support such things and tell the user not to do that. (Someone should look where pydantic draws the line and report back here.)
Paul
PS I've never written code myself that does runtime introspection of
type annotations - so it's quite possible that there *is* guidance
that I've just missed. But it wasn't obvious to me from a quick search
- the "introspection helpers" section of the typing module docs is
pretty basic...