Should typing.Annotated disallow certain typeforms?
Recently, we fixed Annotated so that it is allowed to wrap Final and ClassVar[1]. Since then I have found a number of other places where Annotated cannot wrap typeforms at runtime. In an issue addressing that ParamSpec P.args and P.kwargs cannot be wrapped at runtime [2], it was brought up that one might not consider these types and thus they might contravene the PEP 593 (Annotated)[3] statement "The first argument to Annotated must be a valid type" and thus we should discuss it publicly. It is my position that the PEP never intended to disallow using Annotated on top-level typeforms (like P.args and P.kwargs), and that moreover the intention either always was or should be that if an Annotated contains something that would be a valid typeform in that location, it's a valid place to use Annotated. If we look back on the typing issue [4] that discussed adding Annotated, the requirement is worded that the first Annotated argument must be a "valid `typing` type" as opposed to a type in some stricter understanding of the term. A stronger argument I think is in the impetus of the PEP itself: the PEP is intended to allow typing and non-typing users of annotation space in Python to coexist. If the PEP intended to completely disallow Annotated to be used in certain annotation spots, the PEP isn't accomplishing one of its primary stated intentions. Is there agreement that Annotated should be allowed in these locations? I will submit appropriate doc/runtime changes given agreement. [1] https://bugs.python.org/issue46491 [2] https://bugs.python.org/issue46643 [3] https://www.python.org/dev/peps/pep-0593/ [4] https://github.com/python/typing/issues/600
I agree with this idea: Annotated is meant for typing and non-typing use cases of annotations to coexist, so we should make sure that it can be used in any annotation seamlessly. El mar, 8 feb 2022 a las 12:40, Gregory Beauregard via Typing-sig (< typing-sig@python.org>) escribió:
Recently, we fixed Annotated so that it is allowed to wrap Final and ClassVar[1]. Since then I have found a number of other places where Annotated cannot wrap typeforms at runtime. In an issue addressing that ParamSpec P.args and P.kwargs cannot be wrapped at runtime [2], it was brought up that one might not consider these types and thus they might contravene the PEP 593 (Annotated)[3] statement "The first argument to Annotated must be a valid type" and thus we should discuss it publicly.
It is my position that the PEP never intended to disallow using Annotated on top-level typeforms (like P.args and P.kwargs), and that moreover the intention either always was or should be that if an Annotated contains something that would be a valid typeform in that location, it's a valid place to use Annotated. If we look back on the typing issue [4] that discussed adding Annotated, the requirement is worded that the first Annotated argument must be a "valid `typing` type" as opposed to a type in some stricter understanding of the term. A stronger argument I think is in the impetus of the PEP itself: the PEP is intended to allow typing and non-typing users of annotation space in Python to coexist. If the PEP intended to completely disallow Annotated to be used in certain annotation spots, the PEP isn't accomplishing one of its primary stated intentions.
Is there agreement that Annotated should be allowed in these locations?
I will submit appropriate doc/runtime changes given agreement.
[1] https://bugs.python.org/issue46491 [2] https://bugs.python.org/issue46643 [3] https://www.python.org/dev/peps/pep-0593/ [4] https://github.com/python/typing/issues/600 _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
I can't agree or disagree, since I've never used the Annotated type. Shouldn't we ask folks *outside* the typing community (e.g. pydantic, if they support it; or the Annotated PEP author?) what their preference would be? On Wed, Feb 9, 2022 at 5:43 PM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
I agree with this idea: Annotated is meant for typing and non-typing use cases of annotations to coexist, so we should make sure that it can be used in any annotation seamlessly.
El mar, 8 feb 2022 a las 12:40, Gregory Beauregard via Typing-sig (< typing-sig@python.org>) escribió:
Recently, we fixed Annotated so that it is allowed to wrap Final and ClassVar[1]. Since then I have found a number of other places where Annotated cannot wrap typeforms at runtime. In an issue addressing that ParamSpec P.args and P.kwargs cannot be wrapped at runtime [2], it was brought up that one might not consider these types and thus they might contravene the PEP 593 (Annotated)[3] statement "The first argument to Annotated must be a valid type" and thus we should discuss it publicly.
It is my position that the PEP never intended to disallow using Annotated on top-level typeforms (like P.args and P.kwargs), and that moreover the intention either always was or should be that if an Annotated contains something that would be a valid typeform in that location, it's a valid place to use Annotated. If we look back on the typing issue [4] that discussed adding Annotated, the requirement is worded that the first Annotated argument must be a "valid `typing` type" as opposed to a type in some stricter understanding of the term. A stronger argument I think is in the impetus of the PEP itself: the PEP is intended to allow typing and non-typing users of annotation space in Python to coexist. If the PEP intended to completely disallow Annotated to be used in certain annotation spots, the PEP isn't accomplishing one of its primary stated intentions.
Is there agreement that Annotated should be allowed in these locations?
I will submit appropriate doc/runtime changes given agreement.
[1] https://bugs.python.org/issue46491 [2] https://bugs.python.org/issue46643 [3] https://www.python.org/dev/peps/pep-0593/ [4] https://github.com/python/typing/issues/600 _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
While I use Annotated a good amount I mainly use it for adding metadata to classes to help derive parsers similar to pydantic. So for my usage of Annotated is I need it for any valid variable annotation at class level or __init__ annotation as I also allowed annotating __init__ instead of doing it dataclass style. P.args/P.kwargs is not a valid variable annotation, but it could be an annotation in an __init__. A class that wraps a function for some kind of task framework I guess. I could see usage of it to help serialize/deserialize functions with extra information. I haven't honestly looked closely at function serialization beyond simple name registration as I'm using this for config files and avoidant of having too much code live in the config. So maybe I would use this in a couple months or maybe I could continue to avoid. I also commonly attach docstring like metadata. This works with any reasonable annotation. You could use a library like docstring_parser, but well I've found it convenient and simple to attach documentation to arguments with Annotated. I think it would be reasonable usage though if I wanted alternative ways to serialize decorators. I could also see a potential use case for a library like beam/kubeflow. They are workflow orchestration libraries and they currently can use annotations to some level for describing sequence of computation tasks. If they want to support tasks generic over function signature they could use it. I'll admit honestly I'd be surprised if they did. For Annotated usage that fully ignores the type and only uses it for metadata instead of an extension, then Annotated[P.args, ...] makes sense. For libraries like pydantic/one I work on that use Annotated to extend information on top of type, it's rare to handle types like P.args or typevar T as you'll start to implement more and more type rules. Overall I think there are reasonable valid usages of Annotated[P.args, ...]. I don't expect them to be common, but generally lean towards Annotated only thing it should check is if first argument is valid as a type annotation. I rarely encounter P.args/P.kwargs in first place given there newness and mypy only recent support for them. I'll add Annotated[P.args] not working seems messier then Annotated[Final[...]] not working as I could hack around that with Final[Annotated[...]] and adjust my introspection code to check for Annotated inside any Final. Annotated[P.args] failing sounds like if I ever do need it I'd explore hacks like P.args | Annotated[Never, stuff] as I think P.args | Never is equivalent to P.args. Do all type checkers collapse Never/NoReturn unions to the other type?
Hi all, I was asked by Gregory to comment on this as the developer (and a user) of pydantic. From pydantic's point of view it would be best if Annotated could wrap all types, and indeed non-type code. In pydantic, all of the following could be used: class FooBar(BaseModel): a: Annotated[Dict[str, str], Field(...)] b: Annotated[Json[List[int]], Field(...)] = 'Bar' c: Annotated[constr(min_length=2, max_length=10), Field(...)] d: Annotated[ClassVar[int], ...]. # at least in future also with generics class MyGenericModel(GenericModel, Generic[DataT]): a: Annotated[DataT, Field(...)] b: Annotated[Optional[DataT], Field(...)] Pydantic currently wouldn't make use of Annoated in args/kwargs contexts, but IMHO there's no good reason to constrain the use of Annotated unless it makes the implementation significantly easier. I hope that covers what you were asking about Gregory, let me know if you need more details on any of it.
08.02.22 22:39, Gregory Beauregard via Typing-sig пише:
Recently, we fixed Annotated so that it is allowed to wrap Final and ClassVar[1]. Since then I have found a number of other places where Annotated cannot wrap typeforms at runtime. In an issue addressing that ParamSpec P.args and P.kwargs cannot be wrapped at runtime [2], it was brought up that one might not consider these types and thus they might contravene the PEP 593 (Annotated)[3] statement "The first argument to Annotated must be a valid type" and thus we should discuss it publicly.
It is my position that the PEP never intended to disallow using Annotated on top-level typeforms (like P.args and P.kwargs), and that moreover the intention either always was or should be that if an Annotated contains something that would be a valid typeform in that location, it's a valid place to use Annotated. If we look back on the typing issue [4] that discussed adding Annotated, the requirement is worded that the first Annotated argument must be a "valid `typing` type" as opposed to a type in some stricter understanding of the term. A stronger argument I think is in the impetus of the PEP itself: the PEP is intended to allow typing and non-typing users of annotation space in Python to coexist. If the PEP intended to completely disallow Annotated to be used in certain annotation spots, the PEP isn't accomplishing one of its primary stated intentions.
Is there agreement that Annotated should be allowed in these locations?
And if allow Annotated to wrap ParamSpecArgs and ParamSpecKwargs, should we allow it to wrap a parameters expression? For example: Callable[Annotated[P, "annotated"], int] Callable[Annotated[Concatenate[str, P], "annotated"], int] Callable[Annotated[[str, float], "annotated"], int] Callable[Annotated[..., "annotated"], int]
The goal for Annotated is to be very flexible. We intended it to be an "escape hatch" that allows clients to express things that type-checkers might not understand. I'm all for enabling it in as many places as possible. Let me know if you want me to work on a diff and/or a revision of PEP-593 to clarify any of this. Sorry I took so long to respond, Till T On Mon, 21 Feb 2022 at 12:27, Serhiy Storchaka <storchaka@gmail.com> wrote:
08.02.22 22:39, Gregory Beauregard via Typing-sig пише:
Recently, we fixed Annotated so that it is allowed to wrap Final and ClassVar[1]. Since then I have found a number of other places where Annotated cannot wrap typeforms at runtime. In an issue addressing that ParamSpec P.args and P.kwargs cannot be wrapped at runtime [2], it was brought up that one might not consider these types and thus they might contravene the PEP 593 (Annotated)[3] statement "The first argument to Annotated must be a valid type" and thus we should discuss it publicly.
It is my position that the PEP never intended to disallow using Annotated on top-level typeforms (like P.args and P.kwargs), and that moreover the intention either always was or should be that if an Annotated contains something that would be a valid typeform in that location, it's a valid place to use Annotated. If we look back on the typing issue [4] that discussed adding Annotated, the requirement is worded that the first Annotated argument must be a "valid `typing` type" as opposed to a type in some stricter understanding of the term. A stronger argument I think is in the impetus of the PEP itself: the PEP is intended to allow typing and non-typing users of annotation space in Python to coexist. If the PEP intended to completely disallow Annotated to be used in certain annotation spots, the PEP isn't accomplishing one of its primary stated intentions.
Is there agreement that Annotated should be allowed in these locations?
And if allow Annotated to wrap ParamSpecArgs and ParamSpecKwargs, should we allow it to wrap a parameters expression? For example:
Callable[Annotated[P, "annotated"], int] Callable[Annotated[Concatenate[str, P], "annotated"], int] Callable[Annotated[[str, float], "annotated"], int] Callable[Annotated[..., "annotated"], int]
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: till.varoquaux@gmail.com
It's controversial to revise an already-accepted PEP, but there's no reason we couldn't agree here on allowing Annotated in more places and implement it. I would say the first priority would be to ensure that typing.py and built-in typing notations (e.g. list[int], int|str) allow it anywhere (or at least in more places), probably by stripping it out systematically; then we can encourage type checkers to implement it. On Mon, Feb 28, 2022 at 10:56 AM Till <till.varoquaux@gmail.com> wrote:
The goal for Annotated is to be very flexible. We intended it to be an "escape hatch" that allows clients to express things that type-checkers might not understand. I'm all for enabling it in as many places as possible. Let me know if you want me to work on a diff and/or a revision of PEP-593 to clarify any of this.
Sorry I took so long to respond, Till T
On Mon, 21 Feb 2022 at 12:27, Serhiy Storchaka <storchaka@gmail.com> wrote:
08.02.22 22:39, Gregory Beauregard via Typing-sig пише:
Recently, we fixed Annotated so that it is allowed to wrap Final and ClassVar[1]. Since then I have found a number of other places where Annotated cannot wrap typeforms at runtime. In an issue addressing that ParamSpec P.args and P.kwargs cannot be wrapped at runtime [2], it was brought up that one might not consider these types and thus they might contravene the PEP 593 (Annotated)[3] statement "The first argument to Annotated must be a valid type" and thus we should discuss it publicly.
It is my position that the PEP never intended to disallow using Annotated on top-level typeforms (like P.args and P.kwargs), and that moreover the intention either always was or should be that if an Annotated contains something that would be a valid typeform in that location, it's a valid place to use Annotated. If we look back on the typing issue [4] that discussed adding Annotated, the requirement is worded that the first Annotated argument must be a "valid `typing` type" as opposed to a type in some stricter understanding of the term. A stronger argument I think is in the impetus of the PEP itself: the PEP is intended to allow typing and non-typing users of annotation space in Python to coexist. If the PEP intended to completely disallow Annotated to be used in certain annotation spots, the PEP isn't accomplishing one of its primary stated intentions.
Is there agreement that Annotated should be allowed in these locations?
And if allow Annotated to wrap ParamSpecArgs and ParamSpecKwargs, should we allow it to wrap a parameters expression? For example:
Callable[Annotated[P, "annotated"], int] Callable[Annotated[Concatenate[str, P], "annotated"], int] Callable[Annotated[[str, float], "annotated"], int] Callable[Annotated[..., "annotated"], int]
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: till.varoquaux@gmail.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
participants (7)
-
Gregory Beauregard -
Guido van Rossum -
Jelle Zijlstra -
Mehdi2277 -
Samuel Colvin -
Serhiy Storchaka -
Till