The current state of typing PEPs
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
Does PEP 563 or 649 satisfy static and dynamic typing needs? In the interest of full transparency, we want to let the Python community know that the Steering Council continues to discuss PEP 563 (Postponed Evaluation of Annotations) and PEP 649 (Deferred Evaluation Of Annotations Using Descriptors). We have concluded that we lack enough detailed information to make a decision in favor of either PEP. As we see it, adopting either PEP 563 or PEP 649 as the default would be insufficient. They will not fully resolve the existing problems these PEPs intend to fix, will break some existing code, and likely don’t address all the use cases and requirements across the static and dynamic typing constituents. We are also uncertain as to the best migration path from the current state of affairs. Defer decision on PEP 563 and 649 in 3.11 As such, at this time, the only reasonable path forward that the SC sees is to defer the decision in Python 3.11 again, essentially keeping the 3.10 status quo. We know that this is far from ideal, but it’s also the safest path since we can’t clearly make the situation better, and we don’t have confidence that either PEP solves the problems once and for all. Pragmatically, we don’t want to make the situation worse, and we really don’t want to find ourselves back here again in a couple of releases because we overlooked an important requirement for a set of users. Calling for help from static and dynamic typing users This is also a call to you, the folks who both care deeply about typing in Python, and have a solid understanding of the subject matter (or willingness to learn) to help us! There are use cases we don’t know about, and unclear requirements from both the static and dynamic typing users. We need help in defining those requirements clearly, in uncovering the use cases we’re not aware of, and most importantly, being an interface to typing enthusiasts to help build consensus with either of the proposed PEPs, or some other solution that doesn’t yet have a PEP (such as suggestions made by Łukasz in his blog post here: https://lukasz.langa.pl/61df599c-d9d8-4938-868b-36b67fdb4448/). Volunteer to be a typing PEP shepherd We’re looking for someone(s) who can be neutral as to the solution, knowledgeable enough to understand all sides of the problem, willing to serve as not quite a PEP author, not a PEP Delegate, but more than a PEP shepherd. If you are that person, please let us know! Cheers, -Barry (on behalf of the Python Steering Council)
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
Thanks to Barry and the SC for giving us this update.
<snip> I want to draw attention to this point:
all the use cases and requirements across the static and dynamic typing constituents.
TL;DR: Annotations can be, and are, used for other things than "typing". I just noticed that PEP 563 apparently deprecated those other uses (well, sort of: "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated"), but if the SC is reconsidering PEP 563, then it would be nice to be clear about whether non-typing uses of annotations are indeed deprecated. If not, then the challenge is to come up with a way forward that not only supports both static and dynamic typing, but also other potentially arbitrary use cases. And now onto more detail ... One example is a use case of mine -- I have built a hierarchical object system, built on dataclasses, in which the annotation absolutely has to be an actual type(class) object. PEP 563 will very much break this use case. Of course, there are other ways to write that code, but I'm pretty sure it would require a complete change to the API. I'm not entirely sure if PEP 646 would work for my use case -- likely it would, at least with modest modifications, rather than a new API. I have no idea how many other similar use cases are out in the wild, but we know that the Pydantic project has concerns. To be fair -- I wrote all my code after PEP 563 was accepted. And I only just now read it carefully, and found that it says: "With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated." If that's the case, then that's the case, and I'll deal, but it is disappointing. But I suspect I'm not alone in not really noticing that statement, so I think it's worth it for the SC to explicitly reiterate at this point that all non-typing uses of annotations are depreciated, if indeed, that is still the intent. Also, that statement is not entirely clear. In fact, I think my use case IS compatible with the "aforementioned PEPs", it's just not compatible with PEP 653 itself. It all depends on what is meant by "compatible". I would like to provide a bit of perspective on this. I was criticised on this list a while back for, essentially, not paying attention and then complaining later. After all, PEP 563 was accepted over four years ago. Fair enough. But the fact is that I, among others, have been a bit uncomfortable about the focus on typing in Python for years. But when issues are raised, we have been repeatedly told that typing is, and always will remain, optional. In short, it was made clear that anyone not interested in typing could safely ignore the discussions about it. And thus, a number of PEPs came and went, and those among us that did not choose to involve ourselves in the conversation did not pay attention to the details. And thus we didn't notice that buried in what seemed like a typing PEP, was, in fact, a depreciation of any non-typing uses of an existing Python feature. (also to be fair, the title of the PEP is "Postponed Evaluation of Annotations" -- so should have caught the attention of anyone interested in annotations for any use. In my personal case, I didn't notice it at the time, as I wasn't using annotations for anything at all until fairly recently. In fact, it was the introduction of dataclasses, which I think are the first use of annotations in the standard library, that drew my attention. dataclasses, as I'm sure you all know, use the presence of an annotation on a class attribute to define a field. This is a pretty nifty use of annotations as an API for defining the fields. I liked that, and decided to use it. dataclasses create a Field object, which has an attribute called "type", that gets populated with the annotation's value. PEP 563 would very much change what gets put in the "type" attribute of a Field. Eric V. Smith has indicated that he never intended that to actually be a type, but rather was intended to be whatever the annotation was, so that PEP 563 wouldn't change anything. I'm happy to take his word for it (and PEP 567 says that the actual type is not used or modified by dataclasses), but as users, all we have to go on is the documentation and the choice of names, which very much gave the impression that a Field.type would be a type -- at least if a type was set in the annotation. I also notice that PEP 567 (dataclasses) and PEP 563 were both developed and approved at about the same time. So I suspect the impact of PEP 563 was not considered when designing or documenting dataclasses. However, PEP 567 does make a number of references to typing, and was clearly written with the idea in mind that the annotations would, in fact, be used for type analysis. But again, no mention of PEP 563, or what might be in the field.type attribute -- one might wonder why it was there at all if it was not intended to be used. We need help in defining those requirements clearly, in uncovering the use
cases we’re not aware of, and most importantly, being an interface to typing enthusiasts
Again -- is it only "typing enthusiasts" that you want to engage? Or "users of annotations"? -- maybe it is, but it would be nice if that was a clear statement from the SC. - CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Sun, 21 Nov 2021 at 07:50, Christopher Barker <pythonchb@gmail.com> wrote:
TL;DR:
Annotations can be, and are, used for other things than "typing". I just noticed that PEP 563 apparently deprecated those other uses (well, sort of: "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated"), but if the SC is reconsidering PEP 563, then it would be nice to be clear about whether non-typing uses of annotations are indeed deprecated. If not, then the challenge is to come up with a way forward that not only supports both static and dynamic typing, but also other potentially arbitrary use cases.
I agree with the points made in this post. It's becoming harder and harder for people not particularly interested in static typing to simply ignore it, and any use of annotations to affect runtime behaviour is in a weird grey area. And as a library author, I'm now finding that I'm getting requests to add typing to my code "for my users" (i.e., using types is no longer just a choice I make for my project, it's an API design issue). So I too would appreciate clarity on where annotations, and more generally typing, stand as Python language features¹. Paul ¹ In particular, typing features seem to change so rapidly that supporting a wide range of Python versions is a real pain (in practical terms - not getting syntax errors is straightforward, but writing type annotations that work well across versions very definitely isn't, at least for someone who doesn't really care about static typing).
![](https://secure.gravatar.com/avatar/8da339f04438d3fcc438e898cfe73c47.jpg?s=120&d=mm&r=g)
Executive summary: The typing-suspicious crowd has a valid complaint about PEPs 563 and 649, but it's not that they weren't warned. Christopher Barker writes:
Not a PEP proponent (or even a typing user), but I thought this had been made clear long ago. My understanding is that optional, incremental type hints are and have always been considered the primary use case for annotations by the BDFL and AFAICT the SC following the BDFL. If compatibility with typing is an issue, then the burden of implementing that is on the other application. Typing *might* do something to help, but it's not obligated to do so.
If PEP 649 works for you, I expect you could work with PEP 563 by moving Larry's data descriptor concept out of the compiler and into your library code. You could also appeal to the "might do something" caveat above. Yury suggested providing both the immediately evaluated object (if there were no forward references, I guess) and the string representation, which would be backward compatible with your use case. The idea was rejected on the basis that it doesn't help with the primary use in typing, and doesn't remove the need for a __future__ import during the transition period. You could ask that that decision be revisited, since it would ensure that uses of __annotations__ like yours continue to work.
But I suspect I'm not alone in not really noticing that statement,
I'm not surprised people haven't noticed that statement. I am surprised that a lot of folks haven't noticed that this comes up occasionally and the answer every time is "we're not going to go out of our way to break other use cases, but typing *is* the primary use case and will take precedence if the question comes up". I will grant that restricting the type of compiled annotations from typing.Any to "string serializing an object's AST" could reasonably be said to be "going out of your way to break other use cases". You have a valid beef here, although it's not obvious to me how it should be resolved. So I'm not saying that you don't have an interest in the future semantics of this dunder. I am +1 on Python providing relief for your use case.
But the fact is that I, among others, have been a bit uncomfortable about the focus on typing in Python for years.
You mean y'all are uncomfortable with the popularity of typing. You wouldn't care if it wasn't used outside of hugely complex proprietary codebases you'll never see. Thing is, reasoning about programs is What We Do as programmers, and efforts to make it easier to do that for complex programs, and to provide software to help with that task, are here to stay and they will be used in mixed company. Even in Python. (And that's why, as someone who doesn't use typing, I support typing. Anything that helps *other people* to write better code, I can get behind. My code, I'm going to write as incompetently as ever.[1] ;-)
But when issues are raised, we have been repeatedly told that typing is, and always will remain, optional.
This has always been in the context of *your* source code. For example, nobody ever claimed you could keep it out of anybody else's source code (although that's clearly what a lot of those "raising issues" want!) Still, features like stub files were provided so typists could work and play nicely with non-typists if they wanted to (and many typists want that, themselves). But dunders are the property of the language (or sometimes the implementation), and they always have been. If you use them in a way that's not documented to work, you're at risk even if it happens to work now. PEP 3107 didn't document that anything would work. ;-) Footnotes: [1] Barry, do not tell Abhilash I wrote that.
![](https://secure.gravatar.com/avatar/880ac02012dcaf99f0392f69af4f8597.jpg?s=120&d=mm&r=g)
On 25/11/2021 15:15, Stephen J. Turnbull wrote:
“As you will no doubt be aware, the plans for development of the outlying regions of the Galaxy require the building of a hyperspatial express route through your star system, and regrettably your planet is one of those scheduled for demolition. The process will take slightly less than two of your Earth minutes. Thank you.” Well we were warned, so everything's fine. Rob Cliffe
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
First: At this point, I am not advocating anything in particular. I am just asking that the SC makes a clear statement about the intention at this point. From Barry's email: """ This is also a call to you, the folks who both care deeply about typing in Python, and have a solid understanding of the subject matter (or willingness to learn) to help us! There are use cases we don’t know about, and unclear requirements from both the static and dynamic typing users. """ I don't care deeply about typing, and I'm not a static or dynamic typing user. So this implies that a decision has been made that annotations are for typing, and typing only. (not that you're not allowed to use for anything else, of course you are, but that other uses won;t be taken into account when designing the new interface) But I have never seen that clearly stated anywhere. The closest is from PEP 563, where it says: """ With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated. """ Which pretty much makes the point, but it's a bit subtle -- what does "incompatible' mean? On Thu, Nov 25, 2021 at 7:15 AM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
The typing-suspicious crowd has a valid complaint about PEPs 563 and 649, but it's not that they weren't warned.
Well, "typing-suspicious" is part of it, but the real problem is folks that might be using annotations for non-typing purposes. As for warning: yes and no -- what's new here is that as far as I know, this is the first time since the py3 transition that a __future__ import would be made standard behavior. And you'd have to be paying a lot of attention to know that was going to happen. And I suspect that only folks involved in the development of typing tools were paying that much attention. In the case of the py3 transition -- it was very well known and documented that lots of things would be changing, and we had a very long transition period. But the challenge with this is that there is no way to raise a depreciation warning for not using a __future__ import. So while we all should have been testing our code with: from __future__ import annotations For the last few versions, I doubt that very many people not developing typing tools were doing that.
well no -- not "always" -- look at PEP 3107, when annotations were first introduced: "By itself, Python does not attach any particular meaning or significance to annotations." and it goes on to give non-typing examples. Since then, the shift from "nothing more than a way of associating arbitrary Python expressions with various parts of a function at compile-time" to a system for defining types has been gradual, with a number of PEPs, and, as far as i can tell, PEP 563 is the first time that a change was proposed that would affect other uses for annotations. Anyway, that is water under the bridge. But the reason I bring it up is because of all the many thousands of Python users, I pay probably more attention to changes than most -- I have followed Python-ideas and Python-dev for years. So I'm guessing that there are other folks out there that will be surprised if and when the behavior of annotations changes.
If compatibility with typing is an issue, then the burden of implementing that is on the other application.
That does indeed seem to be the current intent. And I'm happy to deal with that -- I just want it to be clear that I (and others) have to wait and see how it shakes out, and then figure out what to do, or if I should contribute to the conversation now. If non-typing use cases of annotations are still considered important, then we should encourage folks like me to be part of the conversation.
That would indeed require maybe not a change to the API, but certainly some complex code. But yeah, probably doable one way or another.
I think the "breaking other use cases" has been subtle to those not paying attention to typing.
well, yes. The issue is that, intended or not, typing is making it's way into Python culture. As an instructor of beginning python users, I am unsure at this point when to introduce type annotations. What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required. But that's getting pretty OT here.
Anything that helps
*other people* to write better code, I can get behind. My code, I'm going to write as incompetently as ever.[1] ;-)
me too -- actually, not quite. I write a lot of "scripts" that I will continue to keep simple, and I also develop complex systems -- and those require more care.
Sure -- my point with that is that if I don't want to use typing in my code, then I didn't have to pay attention to the development of the typing systems. But my point here is that what changed with PEP 563 is that while typing is still optional, this is the first time that the language itself may be changed to accommodate typing -- so it can no longer be completely ignored. But dunders are the property of the language (or sometimes the
no, but it did document that annotations were "arbitrary Python expressions" -- that is changed in PEP 563 -- intentionally, but it is changing something about the language that was documented. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Thu, 25 Nov 2021 at 21:45, Christopher Barker <pythonchb@gmail.com> wrote:
The issue is that, intended or not, typing is making it's way into Python culture. As an instructor of beginning python users, I am unsure at this point when to introduce type annotations.
What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.
Agreed this is somewhat OT, but I also think the messaging around annotations needs to be reviewed. I suspect that students will also not clearly understand the fact that annotations aren't checked at runtime. And IMO this is particularly difficult to get across in the case of non-typechecker uses of annotations. The following, for example, is extremely non-intuitive to me:
Even though the "typing is optional" message is well-understood, my instincts still lead me to expect that declaring the type of n will result in a runtime type-check, in the constructor at least. (If I check the code with mypy, it does raise an error, so that's good. Although the more I think about it, given that I believe dataclasses use eval "under the hood", the less I understand *how* it manages to do that without special-case knowledge of the dataclass decorator...) I'd like to see a clearer statement from "somewhere" about how APIs should use annotations at runtime, such that Python users have a much clearer intuition about APIs like the dataclass one, and library designers can build their APIs based on a clear "common understanding" of what to expect when annotations are used. Paul
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 1:37 AM Paul Moore <p.f.moore@gmail.com> wrote:
Static checkers special-case the @dataclass decorator. Eric Traut has a proposal to generalize this support (sorry, I'm in a rush, otherwise I'd dig up the link, but it's in the typing-sig archives).
Note that @dataclass itself is very careful not to use the annotations, it only looks for their *presence*. With one exception for ClassVar. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Fri, 26 Nov 2021 at 17:13, Guido van Rossum <guido@python.org> wrote:
:-( That's what I suspected, but it does mean that dataclasses has a privilege that other libraries (like attrs, I guess?) don't get.
Understood. What I'm suggesting is that it would be good to have a clear "common understanding" about whether libraries should be careful like this, or whether it's OK to base runtime behaviour on type annotations. And if it is OK, then what are good patterns of design and behaviour? This is where the proposal to store annotations as strings hit issues, because it appears to take the view that libraries *shouldn't* be looking at the actual types specified by annotations (or maybe that they should only do so via something like `typing.get_type_hints`). There are other subtleties here (runtime code needs to deal with the fact that int and "int" should be treated the same) that there's no guidance on, again possibly because no-one is really considering that use case. Paul PS I've never written code myself that does runtime introspection of type annotations - so it's quite possible that there *is* guidance that I've just missed. But it wasn't obvious to me from a quick search - the "introspection helpers" section of the typing module docs is pretty basic...
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 11:58 AM Paul Moore <p.f.moore@gmail.com> wrote:
Actually both are done through plugins (since what they do just doesn't fit in the PEP 484 type system) and both have the same status, at least in mypy. (In fact we have twice as many lines of code dedicated to attrs than to dataclasses. https://github.com/python/mypy/tree/master/mypy/plugins) The proposal I mentioned by Eric Traut (pyright's author) would make it easier to support many similar libraries across all static type checkers. ( https://mail.python.org/archives/list/typing-sig@python.org/thread/TXL5LEHYX... )
You are hitting the nail on the head here. I'd say that so far the recommendation has been "use typing.get_type_hints(x) rather than x.__annotations__" -- this handles the equivalence between int and "int" (and hence forward references as well as 'from __future__ import annotations'). There's now also inspect.get_annotations() which has roughly the same functionality without quite so much bias towards typing, so perhaps we should recommend it over typing.get_type_hints() -- though before 3.10 inspect.get_annotations() didn't exist so you might have to fall back on using the other. (There are some semantic differences between the two that I'm glossing over here, because I'm not sure about what exactly they are. :-) We now also have a document that recommends best practices ( https://docs.python.org/3/howto/annotations.html) although it's very new -- it appears Larry Hastings wrote it while he was pushing for PEP 649 (but it received buy-in from the static typing community as well). So perhaps it isn't as bad as it seems, *if* you know where to look? That said, I don't think the current static typing infrastructure would be prepared for an onslaught of modules that use type introspection at runtime to modify the behavior of classes a la attrs and dataclasses. I don't know enough about pydantic to say whether it also falls in this category. But it's definitely difficult to write code that makes use of type annotations at runtime that *also* passes static type checks by mypy etc.; you certainly shouldn't attempt to do so without having CI jobs to run the static checker and test the runtime-introspecting framework you're using. (It's not quite like trying to write code that's valid Python and Fortran at the same time, but it's not trivial. :-) There are also features of our static type system that tend not to be supported by the runtime-introspecting frameworks -- in particular, I'd expect generics and callable types to be hard to deal with at runtime. It's easy enough to do something at runtime with `def f(a: list[int]) -> int`. It's not so simple to handle `def f(a: Sequence[T]) -> T` or `def f(cb: (T) -> tuple[str, T], Sequence[T]) -> Mapping[str, T]`. Presumably frameworks like pydantic just don't support such things and tell the user not to do that. (Someone should look where pydantic draws the line and report back here.)
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 1:47 PM Guido van Rossum <guido@python.org> wrote frameworks like pydantic just don't support such things and tell the user not to do that. I don’t know about Pydantic, but that’s exactly my use case: the type needs to be an actual intstantiatable type. So list and tuple is fine, and Sequence[t] won’t work. But it’s fine that this use case is restricted. In fact, other than the basic core types, you need to use specialized types with this system anyway. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
So does your library work both with and without ‘from __future__ import annotations’? If it does, you shouldn’t have to worry. If it doesn’t, it would be useful if you could post some detailed examples of what goes wrong. On Fri, Nov 26, 2021 at 14:24 Christopher Barker <pythonchb@gmail.com> wrote:
-- --Guido (mobile)
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
I hope my understanding of where the SC’s debate about this currently sits isn’t a misrepresentation, but since I sent the email on behalf of the SC, let me try to clarify what I was trying to say. Aside: A little insight into how the SC works. For communications like this, after whatever debate we have in our live weekly meetings (and sometimes off-line discussions in Slack and email), one of us volunteers to write a draft of the email. We then all comment and wordsmith until we’re in unanimous agreement that it’s ready to go out. Then usually the person who wrote it will send the email. We all have slightly different styles, but I usually sign it with my name “on behalf of the Steering Council”. That’s my way of saying that the communication comes from all of us and represents the SC’s opinion, but it generally has the written style of the original drafter.
Practically speaking, I think that’s current reality: annotations are for typing, be they static type checking or dynamic runtime uses. I was inelegantly saying that even with that narrower interpretation of annotations, there are use cases, interactions, side-effects, semantics, and implications that we do not fully understand. PEP 563 and 649 have visible effects that even within that domain can have important side effects. For example, PEP 563’s loss of local scope, which even “de-stringify-ing” can’t recover. This is what we need help with.
You make a good point. I agree that while all the signs are there for “annotations are for typing”, this has never been explicitly or sufficiently codified, and I’ve been proposing that Someone write a PEP that makes this an official pronouncement. That someone may very well be a person on the SC, but given the timing of elections, it’s likely to fall to the next SC, if they still agree with that position! :-D My recollection of the history of annotations falls somewhere between Greg’s and Guido’s. Annotations as a feature were inspired by the typing use case (with no decision at the time whether those were to be static or runtime checks), but at the same time allowing for experimentation for other use cases. Over time, annotations-for-typing clearly won the mindset and became the predominant use case. Personally, I was strongly against type annotations because of my experience in other languages, but over time I was also won over. From library documentation, to complex code bases, to the transient nature of contributors, type annotations are pretty compelling, and most (but not all) of my worries really didn’t come to fruition. We can lament the non-typing use of annotations, but I think that horse is out of the barn and I don’t know how you would resolve conflicts of use for typing and non-typing annotations. It’s been a slow boil, with no definitive pronouncement, and that needs to be fixed, but I think it's just acknowledging reality. That’s my personal opinion.
well, yes. The issue is that, intended or not, typing is making it's way into Python culture. As an instructor of beginning python users, I am unsure at this point when to introduce type annotations.
What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.
This is an important observation. As much as I’m in the "type annotations are good” crowd now, I still think they should always be optional. Python’s use is so broad these days, I for one don’t want to have to add annotations to every bit of Python I write. -Barry
![](https://secure.gravatar.com/avatar/53c166c5e1f0eef9ff4eb4d0b6ec9371.jpg?s=120&d=mm&r=g)
On 11/29/21 2:56 PM, Barry Warsaw wrote:
PEP 563 and 649 have visible effects that even within that domain can have important side effects. For example, PEP 563’s loss of local scope, which even “de-stringify-ing” can’t recover. This is what we need help with.
Well, sure. If PEP 563 and 649 didn't have visible effects, there'd be no point in doing them. That said, I suggest 649 does a lovely job of avoiding /undesirable/ side-effects. Sure, 649 has observable side effects. For example, you can detect whether or not 649 is active by rebinding a name after it's used in an annotation but before examining that annotation at runtime. This seems harmless--unlikely to happen in production code, and easily remedied if someone did trip over it. A more credible side effect: if you use an undefined name in an annotation, you won't notice at compile-time. Now, this is actually 649's major feature! But there are also scenarios where this feature could cover up a bug, like if you misspell a name--you won't notice until you examine the annotation at runtime (or, more likely, until you run your static analyzer). 563 has this same behavior--and it wasn't enough to prevent 563 being accepted. So I assume this wouldn't be enough to prevent accepting 649 either. 649 has effects on memory usage and performance, but honestly I'm not worried about these. I don't think the memory usage and performance of the prototype were particularly bad. Anyway, as I've said many times: we should figure out the semantics we want first, and then we can worry about optimization. The Python core dev community has no end of smart people who love optimizing things--I'm sure if 649 was accepted and merged, the optimizations would start rolling in. Then of course there are also things 649 simply doesn't do, e.g. resolve the "if TYPE_CHECKING" situation. But it's not appropriate to call that a "side effect" per se. And that's my list. If anybody knows of other visible side effects from 649, naturally you should contact the SC. And/or me, if you think we could change 649 to mitigate it without losing its major features. Happy holidays, //arry/
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Nov 29, 2021, at 15:57, Larry Hastings <larry@hastings.org> wrote:
PEP 563 also has visible side effects, but the real implication of them wasn’t well understood at the time the PEP was approved. Or maybe it was and we underestimated the impact on certain important users and use cases. I’m particularly wary about getting into that same (um) pickle again.
A more credible side effect: if you use an undefined name in an annotation, you won't notice at compile-time. Now, this is actually 649's major feature! But there are also scenarios where this feature could cover up a bug, like if you misspell a name--you won't notice until you examine the annotation at runtime (or, more likely, until you run your static analyzer). 563 has this same behavior--and it wasn't enough to prevent 563 being accepted. So I assume this wouldn't be enough to prevent accepting 649 either.
How common is it for code to examine annotations at run time, outside of specialized libraries that have to be more defensive anyway (especially given they already have to be defensive in light of 563)?
649 has effects on memory usage and performance, but honestly I'm not worried about these. I don't think the memory usage and performance of the prototype were particularly bad. Anyway, as I've said many times: we should figure out the semantics we want first, and then we can worry about optimization. The Python core dev community has no end of smart people who love optimizing things--I'm sure if 649 was accepted and merged, the optimizations would start rolling in.
I agree about that priority (semantics first, optimization later), as long as those semantics don’t make such future optimizations impossible, or severely degrade important performance constraints such as import times.
Then of course there are also things 649 simply doesn't do, e.g. resolve the "if TYPE_CHECKING" situation. But it's not appropriate to call that a "side effect" per se.
Although, it would be nice to have a comprehensive solution to this and other situations too, even if they happen outside of PEP 563/649 discussion…
And that's my list. If anybody knows of other visible side effects from 649, naturally you should contact the SC. And/or me, if you think we could change 649 to mitigate it without losing its major features.
That list is exactly what I'm (we’re?) looking for. -Barry
![](https://secure.gravatar.com/avatar/d6b9415353e04ffa6de5a8f3aaea0553.jpg?s=120&d=mm&r=g)
On 11/29/2021 5:56 PM, Barry Warsaw wrote:
On Nov 25, 2021, at 13:41, Christopher Barker <pythonchb@gmail.com> wrote:
What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.
This is an important observation. As much as I’m in the "type annotations are good” crowd now, I still think they should always be optional. Python’s use is so broad these days, I for one don’t want to have to add annotations to every bit of Python I write.
Maybe it should be reiterated with whatever decision comes forth that
and other duck-typed code will always be legal, idiomatic, and even expected as good practice for beginner, informal, exploratory, and similar python code. -- Terry Jan Reedy
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Mon, Nov 29, 2021 at 07:56:16PM -0500, Terry Reedy wrote:
[...]
Many of the typing-related PEP comes with such a disclaimer, listed as "Non-goals". For example: https://www.python.org/dev/peps/pep-0484/#non-goals Łukasz's stringified annotations PEP has a non-goals section: https://www.python.org/dev/peps/pep-0563/#non-goals Larry's deferred evaluation PEP does not: https://www.python.org/dev/peps/pep-0649/ but I don't think we should hold it against PEP-649. It's not trying to sneakily sneak mandatory static typechecking in by the back door like some sort of sneaking sneak :-) -- Steve
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Mon, Nov 29, 2021 at 5:01 PM Terry Reedy <tjreedy@udel.edu> wrote:
Why would it need to be reiterated? Are there really people who believe that such code would become invalid? AFAIK *everybody* here agrees that this should stay valid. So who would we be reiterating it for? (Yes, several static type checkers have options that cause the checker to complain about unannotated code. But that's not the default behavior and running a static checker is a choice, like running a linter. Three-space indents or capitalized function names are never going to be disallowed either, even though PEP 8 says that's not how you ought to code.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/d6b9415353e04ffa6de5a8f3aaea0553.jpg?s=120&d=mm&r=g)
On 11/29/2021 8:16 PM, Guido van Rossum wrote:
Chris, can you say anything more about why people would get such a mis-impression?
The people that Chris B. referred to, and their 'sources'. But we need more info to know how to best counter such wrong beliefs. -- Terry Jan Reedy
![](https://secure.gravatar.com/avatar/be200d614c47b5a4dbb6be867080e835.jpg?s=120&d=mm&r=g)
On 11/30/2021 1:16 AM, Guido van Rossum wrote:
I know it sounds like a straw-man argument, but I don't think it is. I just think the non-straw-men making the argument aren't "important" enough to call out specifically - they are "random individual programmers who have gotten the wrong impression from another language and applied it to Python" rather than "influencers trying to change how we code".[1] And we can always say that the *language* does not require running the type checker, but the reality for many developers is that the projects and teams they are part of will require it. And often, they require it because it's some kind of global "Best Practice" rather than because it makes sense for their particular situation. For example, I saw a snippet fly by on Twitter the other day (again, not the fault of the individual who posted it, so I'm not naming-and-shaming) showing how to use Black to format a big directory of Jupyter notebooks. Convenient, perhaps, but the vast majority of people with "big directories of Jupyter notebooks" are going to be much happier if you just leave them alone! And yet, "The Community" is suggesting we should format all of them in bulk. THAT'S the kind of thing that also has been happening with typing, and why some of us feel the need to publicly re-state things that are all agreed upon within this group, but are struggling to be heard over the public discourse on the topics. And this kind of reiteration is easier when our official documents (PEPs, etc.) state it explicitly. Cheers, Steve [1]: Okay, I will call out the Python Bytes podcast - which I've been on a couple of times - as one that regularly opines on topics based on very little information and says things more definitively than they've been said anywhere else. They have a much bigger (and more easily influenced) audience than python-dev. I'll also say that what I see from LWN is generally very good in this respect, their writers handle things well. Though I don't tend to read or listen to either of these that often.
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 02:52, Steve Dower <steve.dower@python.org> wrote:
This is very definitely the case. There's a subtle (maybe not so subtle, actually) and increasing pressure on projects to add typing. Often with little or no justification beyond "you should", as if having typing is a sort of "obvious best practice". Sometimes "because it will make it easier for your users who use typing" is given as a justification, but while that's fair, it's also a disturbing gradual pressure for typing to extend everywhere, manifesting by making it feel like not adding typing is somehow "not caring about your users". Also, related to the question Terry raised, IMO it would be useful to have a clear statement on code that *does* use type annotations, but violates them at runtime. To be specific, is the following considered as an error?
If (as in the current interpreter) typing is optional, this isn't an error. Certainly, running mypy over it fails, but the point here is that mypy is an external tool - the Python interpreter itself (and the language definition) allows it. I think it would be useful to have a clear message that this is intended, and won't be altered lightly. Remember that the "correct" annotation here is either muladd(x: Any, y: Any, z: Any) -> Any, or muladd(x: SomeMultiplyProtocol, y: SomeAddProtocol, z: SomeAddProtocol) -> Any, neither of which is particularly helpful... Paul
![](https://secure.gravatar.com/avatar/d67ab5d94c2fed8ab6b727b62dc1b213.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 8:19 PM Paul Moore <p.f.moore@gmail.com> wrote:
My understanding is that it's precisely as wrong, or not wrong, as this: def add(x, y): """Multiply two numbers""" return x / y To my mind, annotations are machine-readable metadata, with no inherent "correctness" to them (from the language's point of view), other than syntactically. ChrisA
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 09:17:13AM +0000, Paul Moore wrote:
Isn't that just duck-typing? You've got a function that is documented as taking arguments of one type (int), but actually will work with any numeric type (and some non-numeric types) that quacks like an int. muladd(2, 'ab', 'cd') # returns 'abcdabcd' If you replaced the type annotations with a docstring that listed the parameters x, y, z as int, would duck-typing be wrong? Maybe. I don't think we can make any concrete claims here. Sometimes duck-typing is perfectly fine. Sometimes its not. It depends on the function's implementation and its semantics. Sometimes calling a function with a duck-typed value seems to work fine but the result is meaningless junk. (Garbage In, Garbage Out.) I guess all we can say is that the Python language is agnostic and neutral on this matter. -- Steve
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 09:23, Paul Moore <p.f.moore@gmail.com> wrote:
I suspect that a big source of this is editors that use typing for autocompletion etc. Speaking as a vim user this is something that has passed me by except that I have students who I see adding type hints even though I deliberately haven't taught them anything about type hints. I can only presume that some editor is doing this for them or telling them that they need to do this (students often can't tell the difference between editor warnings and actual errors). Others have mentioned the pressure on libraries to adopt typing and I've certainly noticed this with SymPy. I think type hints could be good for SymPy for internal use but it seems that a lot of users want it for external reasons that I don't always understand but that also seems to come partly from editors as well. Some people apparently want to add type hints that look completely useless to me like def f() -> Union[MyClass, Any] As I understand it this does not give any meaningful information to a type checker but it apparently makes vscode work better: https://github.com/sympy/sympy/pull/22180#discussion_r718917577 Here in SymPy you can see a suggestion that type hints are needed because otherwise vscode is slow and uses 4GB of memory just to load documentation for SymPy functions: https://github.com/sympy/sympy/issues/17945#issuecomment-928798977 Not to do with editors but also indicative is this PR whose title implies that SymPy is "incompatible" with PEP 561 (a PEP that I had never heard of before seeing the PR): https://github.com/sympy/sympy/pull/22337 There are other open "issues" like this for SymPy where the presumption is that not having type hints is now to be considered a deficiency of the library regardless of whether the hints have any benefit for internal use. I don't object to adding the hints but it's a huge amount of work that someone would have to do and I don't want to add useless/inaccurate hints temporarily (as some have suggested to do). -- Oscar
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 12:39, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
This is precisely the sort of concern I have. Although it is awfully tempting to passive-aggressively annotate everything as Any, just to shut people up :-( And to be clear, it's often very non-obvious how to annotate something - in https://github.com/pfmoore/editables I basically gave up because I couldn't work out how to write a maintainable annotation for an argument that is "a Path, or something that can be passed to the Path constructor to create a Path" (it's essentially impossible without copy/pasting the argument annotation for the Path constructor). I also spent a lot of time trying to deal with a typeshed bug. And even worse, if I put the types in a .pyi file (which I want to do, because I don't want the library to import typing at runtime because it's a significant overhead for something that's supposed to be lightweight), apparently mypy won't check them so I gain no benefit for the project itself. Anyway, we're *way* off topic now, and I doubt there's much that the SC or python-dev can do to change the community view on typing, anyway. Paul
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 02:30:18PM +0000, Paul Moore wrote:
I thought that type inference was supposed to solve that sort of problem? If the typechecker can see that an argument is passed to the Path constructor, it should be able to infer that it must be the same types as accepted by Path. Aside: I'm a little disappointed in the way the typing ecosystem has developed. What I understood was that we'd get type inference like ML or Haskell use, so we wouldn't need to annotate *everything*, only the bits needed to resolve ambiguity. But what we seem to have got is typing like C, Pascal and Java, except gradual. Am I being unreasonable to be disappointed? I'm not a heavy mypy user, I just dabble with it occasionally, so maybe I've missed something.
Heh. We could update PEP 8 to ban type annotations, then watch as the people who over-zealously apply PEP 8 to everything AND over-zealously insist on adding type annotations to everything have their heads explode. -- Steve
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano <steve@pearwood.info> wrote:
You're after https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike: `str | PathLike[str]` (if you're only accepting string paths).
I would change that "should" to "may". Python's dynamism makes inferencing really hard.
It really depends on the code base. Type checkers can make guesses based on the code they have available to them, but that only works if the usage is really clear and the dynamic nature of the code doesn't make things murky. For instance, look at open() and how whether you opened a file with `b` or not influences whether the object's methods return strings or bytes. What would you expect to be inferred in that case if you didn't annotate open() with overrides to specify how its arguments influence the returned object? It also depends on how lenient your type checker is willing to be. Compiled languages get to know all potential inputs and outputs of a function upfront, while Python it's a guess based on what you happen to have installed since you can always just `importlib.import_module()` anything at any time. But since type checkers typically prefer false-positives over false-negatives for correctness, it means having to clarify more code.
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 19:07, Brett Cannon <brett@python.org> wrote:
Well, it's not really. What I'm after, as I stated, is "anything that can be passed to the Path constructor". Yes, str | PathLike[str] is probably close enough (although why is it OK for me to prohibit bytes paths?) but that's what I mean about copying the Path constructor's annotations. If Path changes, I have to change my code. This is a very common idiom: def f(p: ???): p = Path(p) ... Why isn't it correspondingly straightforward to annotate? If PathLike[str] included str, then it would be a lot easier. It's not at all obvious to me why it doesn't (well, that's not entirely true - it's because PathLike is an ABC, not a protocol, and it's not intended to define "the type of objects that the Path constructor takes"). It would still not be documented anywhere, though.
That's fair. That's why I think it should be straightforward for the user to explicitly say "this argument should accept the same types as pathlib.Path does". If inference can't do it automatically, and the user can't (easily) let the checker know that it's OK to do it, then we're left with no easy way to express a very common pattern.
Personally, I'd be quite happy leaving open() as duck typed. I see this as what Steven was getting at - the "typing ecosystem" has moved into a situation where it's acknowledged that some typing problems are really hard, due to Python's dynamism, and yet there's still a drive to try to express such highly dynamic type constraints statically. And worse still, to insist that doing so is somehow necessary. Whatever happened to "practicality beats purity", and typing being "gradual"? Surely annotating everything except open() is practical and gradual? Paul
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
Another concern I have is the over specification of types. I have seen many examples of, e.g. func(x: int, y: float, stuff: List(int]): but very few of: func(x: SupportsInt, y: SupportsFloat, stuff: Sequence[SupportsInt]): (or even Iterable[int]) Is that even the right thing to do to get generic number types? How do I specify a type that could be any number type (float, int, Fraction, Decimal, numpy.float32) ... I just spent a few minutes perusing the typing module and MyPy docs, and didn't find a discussion of that. But I'd really love to see the community as a whole start with more generic types in examples, and be clear that the concrete types should be used only when necessary -- even the typing module docs use a lot of concrete types in the examples. I think this is an issue for two reasons: 1) Folks learn from examples more than instruction -- people are very likely to follow the first example they see that seems to do the job. 2) as mentioned in this thread, library authors are being asked to type hint their libraries. I think it will be all too common for that type hinting to be more specific than required. For the most part, that won't cause any real issues right away, but as soon as someone uses that lib in a large project that enforces static type checking, it's could get ugly. -CHB On Tue, Nov 30, 2021 at 1:37 PM Paul Moore <p.f.moore@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 2:52 PM Christopher Barker <pythonchb@gmail.com> wrote:
There is some discussion of the numeric tower in PEP 484 but the PEP says you should just use 'int', 'float' and be happy. *If* we were to change our mind on that we would let you spell it using numbers.Integer, numbers.Real etc., not SupportsInt, SupportsFloat. But the usability of any of those generic solutions is just terrible (too much typing for too little benefit) and honestly in many cases the runtime is not very good at these either (e.g. Decimal refuses to play). In practice it is just fiction that you can write your own numeric type. Regarding Iterable[int] vs. Sequence[int], there are subtle semantic differences so it depends, but these have much better support and *are* encouraged. (What helps is that List is invariant but Sequence is covariant, and people often want covariance, so there is a real incentive to use Sequence, Iterable etc.)
Maybe you are speaking with a lot of enthusiasm but not a lot of experience? Generics aren't always better, and in many cases they just aren't worth the effort to get them right (just like not every bit of code you write needs to be published on PyPI). I also have a feeling that most frameworks that use *runtime* introspection of types strongly prefer concrete types, i.e. list[int] rather than Sequence[T].
It would be nice if someone did some work and collected a list of tutorials about type annotations that exist (especially the ones that are discoverable with a simple Bing query) and ranked them by quality.
That's just the tip of the iceberg. In practice, there are quite a few reasons why the first few iterations of type hints for many popular packages fall short of expectations. Often the issue isn't just sufficiently wide types (like you speculate here) but missing types (for example, stubs containing functions with arguments but no annotations), missing methods/attributes, missing classes, and missing submodules. Not to mention signatures that are impossible to type with the current type system (you can't actually type zip() or filter(), for example). We should definitely push back on zealous new converts to typing who insist that everything should be annotated. But we should also recognize that even in their current, far from perfect state, type annotations can provide a lot of value, when used right. (Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 23:37, Guido van Rossum <guido@python.org> wrote:
We should definitely push back on zealous new converts to typing who insist that everything should be annotated. But we should also recognize that even in their current, far from perfect state, type annotations can provide a lot of value, when used right. (Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.)
I might be misunderstanding but if "hover-doc" is the same as "mouse-over" then it apparently needs many gigabytes of memory and a lot of CPU time when you put the mouse over a sympy function that you are using in your code. It has been suggested that SymPy should fix this by adding hints like Any but I don't see the point of that: SymPy has plenty of bugs but this particular bug is in VS Code. I can see the potential value that type hints can bring for an editor like VS Code but is the editor also pushing an agenda that all code should be *explicitly* typed? Does VS Code pressurise people into using type hints in their own code? I presume it does because I see the effect in my students (although I might be misattributing this to VS Code rather than something else). -- Oscar
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 4:48 PM Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
(Yes, I meant mouse-over. I had a senior moment. :-) I cannot help you much here, there's a tracker and generally speaking the team does pay attention to tracker issues, so I recommend starting a discussion there. Feel free to CC me on the issue if you decide to create one.
There are two modes in VS Code, a lenient and a strict mode. The strict mode does insist on typing, but you have to opt in to it first. Once you've opted in, it does indeed encourage you to type everything -- but that's pretty much the meaning of strict type checking (mypy has a --strict flag that also requires this). That said, your students probably copied some configuration that turns on strict mode from someone else who is thereby pushing this agenda unwittingly. Sadly most people forget where they got their initial configuration (I know I forget it as soon as it works :-). And the strict mode also provides useful error messages in certain cases. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/be200d614c47b5a4dbb6be867080e835.jpg?s=120&d=mm&r=g)
On 12/1/2021 12:47 AM, Oscar Benjamin wrote:
For the record, this is the *old* support that has since been replaced (and was based on full program analysis in the Python of ~2010 that had no annotations at all, so always struggled with programs that could not be thoroughly inferred and particularly with those that generated runtime types - Sympy was a special case in it since about 2012 to avoid those issues, but that may have been lost when it was migrated from Visual Studio to VS Code). The latest versions of VS Code have a completely new system, that does rely very heavily on programs and libraries being annotated, in exchange for being much lighter on memory use and preprocessing. But it should stay out of your way on unannotated code if you haven't enabled its more strict modes. Cheers, Steve
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 3:31 PM Guido van Rossum <guido@python.org> wrote:
There is some discussion of the numeric tower in PEP 484 but the PEP says you should just use 'int', 'float' and be happy.
Thanks -- I didn't think to look there. And this: "when an argument is annotated as having type float, an argument of type int is acceptable" makes it pretty workable. I do wonder what happens with oddball numbers, like, say numpy.float32. But maybe that's Ok, as for though most part one will be dealing with arrays of them -- the scalars are fairly rare in real code. however, IIRC, __index__ was introduced at least partially so that numpy integer types could be used as indexes. Which would mean SupportsIndex for a type that's going to be used as an index, yes?
Yes, Decimal and Fraction are pretty special purpose, really.
well, yes, for sure.
Generics aren't always better, and in many cases they just aren't worth the effort to get them right
I suspect that depends a bit on the use case. For something internal to one system, probably not worth the effort at all. But for a general purpose library, I'm (perhaps needlessly) concerned about types getting locked down more than they need to be, just because its easier, and "it works for me"
(just like not every bit of code you write needs to be published on PyPI).
Someone should tell that to all the folks publishing 0.0.0.1 version of packages :-)
This is getting into what it means to use Types at runtime -- if it's runtime type checking, then more abstract types might be fine. But if it's using the type itself (like my use case), then absolutely -- my system will only work with real concrete type objects. Though that may not play that well with type checkers :-(
yes, it would.
We should definitely push back on zealous new converts to typing who insist that everything should be annotated.
well, we got folks wanting to change PEP 8 becuase they don't want their linter to complain -- so it will be a battle.
(Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.)
Some of the folks on my team use it -- I've been meaning to check it out -- one more reason now. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/15c3eef50a943fdefa0cb50a21d11203.jpg?s=120&d=mm&r=g)
It would be nice if someone did some work and collected a list of tutorials about type annotations that exist (especially the ones that are discoverable with a simple Bing query) and ranked them by quality.
I went with Google rather than Bing but here's what I found: https://gist.github.com/stroxler/ade7977ed07e27448222a468796bc467 I did check a few Bing queries, the results are mostly a bit less helpful, I get more discussions of the `type` function and fewer about type annotations. But the Real Python tutorial still shows up on the first page most of the time.
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 1:34 PM Paul Moore <p.f.moore@gmail.com> wrote:
It's your code; do what you want. 😁 If you want to accept bytes, then you can accept bytes. I personally just don't care about the bytes edge case, so I leave it out. Plus I don't think pathlib supports bytes paths to begin with, so if you are after just the pathlib.Path case then you don't want bytes unless you're going to handle the decoding. But if you want bytes then you're after https://github.com/python/typeshed/blob/8542b3518a36313af57b13227996168e592f... which is what `open()` takes (plus ints): `str | bytes | os.PathLike[str] | os.PathLike[bytes]`.
but that's what I mean about copying the Path constructor's annotations. If Path changes, I have to change my code.
Yep. You are writing down your expectations of what is acceptable to pass in and you're choosing to be very flexible, and so you have to write down more. I don't think anyone is claiming that typing your code takes no effort. But practically speaking the constructor to pathlib.Path isn't going to change.
If we added an object to pathlib that represented the type annotation for the constructor of pathlib then it wouldn't be. It's probably not an unreasonable idea since this idiom is common enough, I just don't know if anyone has bothered to suggest it for the module.
You're essentially wrapping the constructor to `pathlib.Path` as broadly as possible, which means you want types as broad as possible while still being accurate. You could also tell users, "give me pathlib.Path objects" and your troubles go away or "an object that implements the `__fspath__` protocol" (which is what `os.PathLike` represents, hence why `str` isn't covered by it); it all depends on what sort of assumptions you want to be able to make about what you're given. But because you're wanting to accept multiple types to support both the "old" way of string paths and the "new" way of `__fspath__` you then need to put the work in to accept those types. Personally, I treat string paths vs pathlib object paths like encoding and decoding strings; get it converted immediately at the boundary of your code and then just assume pathlib everywhere else. Pathlib was added in Python 3.4 and is well-known enough at this point that I don't worry about people not being aware of it, so I let users do the glue from any string paths they get to my APIs. But to be clear, path representations != pathlib.Path type parameters != `__fspath__` protocol; there's subtlety here regardless of the types.
But how would you specify that? And what if the thing you're wrapping isn't typed itself? And what if the object you're wrapping takes multiple arguments? You could talk to the typing-sig and see if they have discussed some sort of `Parameter[pathlib.Path][0]` object to specify the type of the first argument to `pathlib.Path`.
Sure, but how would you have expected old code that isn't about to change to be typed?
Yep, hence why people can still use code you didn't type and get *some* benefit from typing still. But there are limits as flowing types through untyped code is hard, hence why your users are asking for types; t limits what *they* can check for without writing type annotations for your code (at which point they are now the ones worrying about a drifting of types, much like you are with pathlib.Path). Code like pathlib.Path that can take various types or other things that can return different types become something of a blackhole for typing because the type checkers just can't figure out what you want or it figures it out to be so broad as to be useless.
Paul
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
I know this isn't really the place for this conversation, but:
which is what `os.PathLike` represents, hence why `str` isn't covered by it);
wait, what? It seems so clear to me that "PathLike" (as a type specifier) would mean: anything that can be passed into os.fspath to give me a path. (or, of course to the stdlib functions that take paths) Isn't the entire purpose of os.fspath that you can write code like: def fun(some_kind_of_path): some_kind_of_path = os.fspath(some_kind_of_path) (or just pass it to a function you takes PathLIke) and go on your merry way -- e.g. duck typing, baby! Is there really no way to annotate that simply now? -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/3b73b776444fa777acfa37bbdcff23fe.jpg?s=120&d=mm&r=g)
On Wed, Dec 1, 2021 at 10:50 PM Christopher Barker <pythonchb@gmail.com> wrote:
Assuming you want the return value of 'fun' to be covariant with the path input, I believe you would say this: def fun(some_kind_of_path: str) -> str: ... def fun(some_kind_of_path: bytes) -> bytes: ... def fun(some_kind_of_path: os.PathLike[AnyStr]) -> AnyStr: some_kind_of_path = os.fspath(some_kind_of_path) # transform it return some_kind_of_path I would love to be shown how to do this with just a one-line declaration of 'fun', but I've given up trying to figure it out.
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Thu, Dec 2, 2021 at 9:48 AM Eric Fahlgren <ericfahlgren@gmail.com> wrote:
I just read the wikipedia page on "Covariant return type", and I either misunderstood what it means, or it's not relevant to my intended example.
Ahh I see, I wasn't intending, in my example, to return the path -- my intent was a function that would need to use the path, to, e.g. open a file. This Strikes me as a very common use case -- you want your users to be able to pass multiple different types in, as long as they can be used as a path. I have a LOT of code that does this -- for years it accepted only string paths, and now I'm updating it to take PathLike as well. My usual code is one of: def fun_that_opens_a_file(the_path): # if I'm working with an "old" library, or one that, e.g. wraps a C lib: the_path = os.fspath(the_path) # if I'm working with things I know will take a Path object: the_path = Path(the_path)
I would love to be shown how to do this with just a one-line declaration of 'fun', but I've given up trying to figure it out.
Darn. could you at least use a Union type: def fun_that_opens_a_file(the_path: Union[str, Path]): ... which isn't too bad. Personally, I think accepting bytes is the way of madness, but you could add that too. On Thu, Dec 2, 2021 at 12:32 PM Brett Cannon <brett@python.org> wrote:
in my code I just take pathlib.Path or pathlib.PurePath and I'm done as I don't want to be dealing with encoded file paths and instead with objects
Sorry, I wasn't clear. By "PathLike" (as a type specifier)" I meant using it as, well, a type specifier, not as an ABC -- often we can use an ABC as a type specifier, but not always. But you've prompted me to go re-read the PEP and I see this: """ Provide specific type hinting support There was some consideration to providing a generic typing.PathLike class which would allow for e.g. typing.PathLike[str] to specify a type hint for a path object which returned a string representation. While potentially beneficial, the usefulness was deemed too small to bother adding the type hint class. This also removed any desire to have a class in the typing module which represented the union of all acceptable path-representing types as that can be represented with typing.Union[str, bytes, os.PathLike] easily enough and the hope is users will slowly gravitate to path objects only. """ I didn't pay any attention to that at the time, as I wasn't paying attention to typing. But personally, I think I do have that desire :-) But I can see that "the hope is users will slowly gravitate to path objects only". would lead to: that represent file paths. For my part, I'm not interested in encoded paths (e.g. bytes) -- that is absolutely not the right boundary to deal with file system encoding. (that's mentioned in the PEP) But I also don't want my users (or me, with old code, scripts, etc) to have to suddenly go in and wrap a call to Path() everywhere in order to continue to use my library. String paths are ubiquitous, and I think here to stay. I really like PEP 519 -- I think it not only provides a transition, but also a future in which Path objects and string paths can continue to play well together. While Union[PathLike, str] is a pretty light lift, this is one tiny example of what we "typing skeptics" are concerned about: a transition away from full on dynamic typing to a more static style. Maybe that won't come to pass -- we'll see. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Wed, Dec 1, 2021 at 10:40 PM Christopher Barker <pythonchb@gmail.com> wrote:
That is not what the docs say: https://docs.python.org/3/library/os.html#os.PathLike. And as the creator of that ABC it's very much on purpose (see https://www.python.org/dev/peps/pep-0519/ for details).
Depends on what "your merry way" is. As the docs say, os.fspath() is about getting the file system representation. That's not something to directly manipulate in a pathlib world unless you're using os.path to manipulate that string encoding (but which PEP 519 was specifically created to avoid such encoding headaches): https://docs.python.org/3/library/os.html#os.fspath .
Is there really no way to annotate that simply now?
Once again, depends on what "simply" means to you. The examples I gave I don't find complicated, especially when we are talking about APIs that are trying to accept a broad set of types to convert into a single type. If you want to create a type variable that represents anything that is valid to `os.fspath()` or `pathlib.Path` to avoid typing out 2 or 4 types, then that can be considered for inclusion in the stdlib somewhere. But as I said previously, in my code I just take pathlib.Path or pathlib.PurePath and I'm done as I don't want to be dealing with encoded file paths and instead with objects that represent file paths.
![](https://secure.gravatar.com/avatar/392880c03b3481024eca1b762ffe3ff2.jpg?s=120&d=mm&r=g)
On 2021-11-30 09:01, Steven D'Aprano wrote:
Captain Kirk would be proud: https://memory-alpha.fandom.com/wiki/Induced_self-destruction -Mike
![](https://secure.gravatar.com/avatar/d5d40a40d75ba32b1bdff02b57201cd1.jpg?s=120&d=mm&r=g)
Am 30.11.21 um 13:39 schrieb Oscar Benjamin:
Please note that users of you library usually won't care that the library uses type hints. It's more important that there are type hints for the API, which can also be supplied using a stub file.
We have several resources that can help with questions like these: * https://github.com/python/typing/discussions is a forum for questions about typing. * The forum also has a separate section where users can ask for reviews of type annotations, for example in stub files. * https://gitter.im/python/typing is a chat for typing-related questions. We are also working on a documentation hub at https://typing.readthedocs.io/, but there is not much to see at the moment.
Please note that users of you library usually won't care that the library uses type hints. It's more important that there are type hints for the API, which can also be supplied using a stub file. This is usually quite a bit easier than typing an existing code base. Alternatively, just adding type annotations to the public API, but not using type checking yourself, can be useful for your users. - Sebastian
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Wed, 1 Dec 2021 at 12:08, Sebastian Rittau <srittau@rittau.biz> wrote:
Please note that users of you library usually won't care that the library uses type hints. It's more important that there are type hints for the API, which can also be supplied using a stub file.
I tried that route, but I was informed that if I do that, mypy will not check my stubs against the source code. Which means that there's no easy way of testing that the stubs are correct - is that accurate? Paul PS I appreciate the links you posted to various typing forums, but IMO the most critical missing resource is a central set of typing documentation that includes examples, FAQs and best practices as well as reference materials. Typically when I hit an issue with types, I want to research my own solution, not ask a question, which requires me to bother other people, as well as interrupting my flow while I wait for a response. I'd much rather see work being done on documenting what we currently have in typing, rather than yet more changes which while no doubt useful, is effectively just churn that makes maintaining a codebase that has to support multiple Python versions harder. PPS Sorry if this sounds negative. TBH, I'd quite happily not use typing if I didn't want to and stay quiet. A lot of the frustration I see being expressed here (including my own) seems to come from the fact that it's so difficult to actually take that sort of "I can ignore it if I don't use it" attitude, whether that's because of community pressure, tool requirements, or whatever.
![](https://secure.gravatar.com/avatar/d5d40a40d75ba32b1bdff02b57201cd1.jpg?s=120&d=mm&r=g)
Am 01.12.21 um 13:36 schrieb Paul Moore:
mypy includes a "stubtest" tool that compares a stub to introspected runtime data. We use it in typeshed as part of the CI process and it's a very powerful tool to help keeping stubs up-to-date and correct.
Completely agree. This is what typing.readthedocs.io is supposed to become, but time constraints mean that it's very incomplete at the moment.
I completely understand this pressure, especially for library authors. Providing high quality stubs and the best user experience is not easy. But I believe that referring people to typeshed can help. While we of course prefer high quality stubs or type annotations shipped with the package in question, typeshed can provide a fairly low barrier of entry for projects that don't have the resources to maintain type annotations themselves. It can also be used as an "incubator", where stubs are created and improved iteratively, until they are deemed ready for inclusion in an upstream package. - Sebastian
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
for library authors.
Providing high quality stubs and the best user experience is not easy. But I believe that referring people to typeshed can help.
This is actually very helpful. It provides an answer for open source projects for which do users want typing. One can say to the users that want type stubs for the library that they are encouraged to contribute those stubs to type shed themselves and once they have been tested and vetted they can be included in the project. it’s not unlike any other feature request. The developer of an open source project is under no obligation to provide any feature asked for, but a well-managed project will encourage useful contributions from users. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/7b5bbadd9baf9c6b33a053e9687ce97e.jpg?s=120&d=mm&r=g)
@Paul
... missing resource is a central set of typing documentation that includes examples, FAQs and best practices as well as reference materials
Like Sebastian, I agree, and this is something we're making progress on.
... easy way of testing that the stubs are correct
mypy ships with a tool called stubtest. It's what typeshed uses to validate itself against CPython. I recently wrote up documentation for it, see here: https://mypy.readthedocs.io/en/latest/stubtest.html
Thanks for not staying quiet and helping us make typing better. In particular, I think one takeaway from this thread is that a lot of our documentation is focussed on "how to get started with typing my code", and not much addressed at library authors with limited time to deal with typing questions (like "what is a PEP 561" and "if my type hints are incomplete does that cause problems for my users") On Thu, 2 Dec 2021 at 15:34, Rob Cliffe via Python-Dev < python-dev@python.org> wrote:
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Fri, 3 Dec 2021 at 00:10, Shantanu Jain <hauntsaninja@gmail.com> wrote:
That's great to know - it's easy for me to say "we need more docs" but I know it's hard to produce them. I'm glad it's on the radar :-)
... easy way of testing that the stubs are correct
mypy ships with a tool called stubtest. It's what typeshed uses to validate itself against CPython. I recently wrote up documentation for it, see here: https://mypy.readthedocs.io/en/latest/stubtest.html
Thanks, I wasn't aware of that. Although TBH I like the suggestion of using typeshed as a kind of staging point for interested users to develop annotations, which can then be moved back into the library once they've been proven stable. I'm still surprised that mypy doesn't check stub files though. If I write stubs for my external API, and then later decide I want to add types to my internal functions, to do static checking on my library, am I therefore required to move the annotations out of the stub file into the main code? What if I want to avoid the runtime overhead of importing the typing module? Or I want to keep the source code clear, by not making function definitions into multi-line monsters because of type declarations?
A lot of the frustration I see being expressed here (including my own) seems to come from the fact that it's so difficult to actually take that sort of "I can ignore it if I don't use it"
Thanks for not staying quiet and helping us make typing better. In particular, I think one takeaway from this thread is that a lot of our documentation is focussed on "how to get started with typing my code", and not much addressed at library authors with limited time to deal with typing questions (like "what is a PEP 561" and "if my type hints are incomplete does that cause problems for my users")
That's a good point that I hadn't spotted - thanks for picking up on it! Paul
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Thu, Dec 2, 2021 at 3:35 PM Rob Cliffe via Python-Dev < python-dev@python.org> wrote:
I assume you accidentally pressed Send prematurely.
Actually, it was my phone making the text white -- what the heck? why is it so hard to send plain text email? resent now. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Wed, 1 Dec 2021 at 12:12, Sebastian Rittau <srittau@rittau.biz> wrote:
The distinction between internal code and API is not always that clear in SymPy but in any case it is still very hard to type many of SymPy's public API functions. You are right that there is a distinction here because when I imagine that type hints could be useful for SymPy I think of adding hints in places that users definitely don't care about. Adding hints for public API is more immediately useful for users but also much harder because many public APIs are quite liberal/flexible in what they accept and return.
I'm not sure what your point is here. Are you suggesting that I use these forums or that I redirect other people there?
My previous statement still stands: it's a huge amount of work and I do not want to add bogus hints as a stopgap. If the hints are to be added then someone needs to do that hard work to make sure that they are accurate. -- Oscar
![](https://secure.gravatar.com/avatar/cdc87637918eccd37ca88e9079e73705.jpg?s=120&d=mm&r=g)
On Mon, Nov 29, 2021 at 8:17 PM Guido van Rossum <guido@python.org> wrote:
I'm certainly not alone, among people on this list, in regard to the following. But I teach a lot of people who are coders, but not necessarily senior (in Python, or otherwise). I also mentor/lead such junior programmers. I find it quite common when people who haven't known Python for 20 years see type annotations, they have trouble getting the optional and gradual nature of them. Explaining that is certainly not impossible, nor even all that difficult. However, it DOES need to be explained. Especially when less experienced Pythonistas have worked with statically typed languages, they often see code with type annotations and overgeneralize to their requirement. I don't know exactly what that means about *where* this needs to be explained. Maybe just by trainers and authors. But maybe things in official Python documentation as well, which seem more definitive. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
![](https://secure.gravatar.com/avatar/dbdddb64dc47a7853e836edfed6b1f3f.jpg?s=120&d=mm&r=g)
On 29 Nov 2021, at 23:56, Barry Warsaw wrote:
But isn't that the reason why we have `typing.Annotated`, so that annotations used for typing and annotations used for other purposes can coexist? An annotation used for typing only is: ```
An annotation used for something else is:
`typing.Annotated` gives us both:
Granted, for someone who only wants to use annotations for their own
purpose, digging out the original not typing related stuff from
`typing.Annotated` is more work, but doable.
Or am I missing something here?
> [...]
Servus,
Walter
![](https://secure.gravatar.com/avatar/72ee673975357d43d79069ac1cd6abda.jpg?s=120&d=mm&r=g)
On 26/11/21 4:15 am, Stephen J. Turnbull wrote:
I'm not sure that's true. The way I remember it, back when annotations were first introduced, the BDFL didn't say that, or at least didn't say it very clearly. It sounded more like type hints were just one of many possible uses, and he encouraged people to experiment. There were even discussions about coming up with a convention to manage conflicting uses of annotations in the same code. That wouldn't have happened if typing were considered the only supported use. -- Greg
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Thu, Nov 25, 2021 at 3:49 PM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
My memory is also hazy, but I'm quite sure that *in my mind* annotations were intended as a compromise between conflicting proposals for *typing*. We didn't have agreement on the syntax or semantics, but we did know we wanted to do something with types eventually. Some folks wanted to enforce types at runtime. Others wanted to use them to generate faster code. Yet others wanted types to be checked by the compiler. The term "gradual typing" wasn't invented (or hadn't reached our community) yet, and offline static type checking wasn't something we had thought of either (I think). But it was clear that typing would have to be optional. Fortunately the current situation is that there are few people who want to use annotations for non-typing, but there are still conflicting use cases, in particular offline static type checking on the one hand (which started with mypy and PEP 484) and some runtime use of types on the other hand ( pydantic <https://pydantic-docs.helpmanual.io/> being the main example IIUC). Tools like pydantic seem to have adopted the *notation* of PEP 484 fully, but their *interpretation* is different. This was explicitly provided for by PEP 484 (the design there carefully ensures that annotations are fully introspectable) but it wasn't anticipated to be a big use case, hence the mistaken belief at the time PEP 563 was accepted that stringifying annotations would be sufficient for the introspection use case. So I'm in agreement with Stephen, but since he left out any mention of offline static checking, his observation isn't helping much. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Fri, 26 Nov 2021 at 05:14, Guido van Rossum <guido@python.org> wrote:
My memory is also hazy, but I'm quite sure that *in my mind* annotations were intended as a compromise between conflicting proposals for *typing*. We didn't have agreement on the syntax or semantics, but we did know we wanted to do something with types eventually.
More hazy memories here, but I think the original proposal left open the possibility of annotations not being types at all - for example, being docstrings for the arguments, or option names for a "function call to CLI" tool, etc. I think that some libraries took this approach (in particular the "CLI builder" idea). At some point (probably around PEP 484) it became clear that the expectation was that annotations would be *types*, and in particular would be the expected/intended type of the annotated value. I think there was some discussion at that point about "what do we do about the CLI builders" but the conclusion was that they had relatively low adoption rates, and could adapt (or more accurately, abandon the annotation approach). That was the point, in my recollection, where the "annotations are for types" principle was established. However, I feel that many people got (or were given) the impression that static type checkers were the core use case (probably as a result of the emphasis on the message that "type annotations will never be mandatory"), for better or worse. I think that by now, it's long been understood by most people that annotations are types. This may actually be one reason why people are so uncomfortable with the idea of stringified annotations, because it violates that assumption - personally, I have the same discomfort about using explicit string annotations for forward references, it feels like I'm not declaring a "proper type". If what I say above is right, the debate here isn't about whether annotations "are for types", but rather about whether reading the types in annotations and using them to affect behaviour *at runtime* is a legitimate use of annotations. That is the use case that stringifying annotations makes more difficult, and which doesn't seem to have a strong enough voice in the direction of typing proposals. I lurk on the typing-sig, and from an outsider's perspective, the participants seem to be almost entirely designers or heavy users of static type checkers. That gives a certain emphasis to the proposals coming from that group. I'd therefore interpret Barry's plea as being for *anyone* with a use for annotations to provide their feedback (at least, anyone who accepts that annotations are types), with particular emphasis on people who want to use the types declared in annotations to affect runtime behaviour, as that's the most under-represented group at the moment (and it's not clear whether it's under-represented because there aren't many such uses, or because the users aren't being heard from). Paul
![](https://secure.gravatar.com/avatar/d91ce240d2445584e295b5406d12df70.jpg?s=120&d=mm&r=g)
Paul Moore wrote:
Absolutely. While it was clear that Guido's own use cases were about typing, annotations were explicitly not limited to typing, which is one reason why some of the later changes have felt to some people like bait and switch. Maybe it is already too late to avoid that.
... the expectation was that annotations would be *types*,
Even from the start, it was assumed that they would be objects. (Specifically types was expected to be common, but not universal.) The particular way strings are being substituted for evaluated objects has sometimes reminded me of raising a string instead of an exception class/object. It will work, but it can seem sloppy, and it can be annoying if you were assuming otherwise and suddenly have to add a bunch of evals. (That said, I haven't yet been sufficiently motivated to even tease out exactly what the problems are, let alone to propose an alternative that also satisfies the typing fans -- in part because it feels like the obvious optimization is to just not run typing, and it isn't clear what middle grounds are generally worthwhile.)
I see that as a second dispute, which I had previously missed. I think you're right, though. On the other hand, I'm not sure the solution to both isn't just a helper function that does the 2nd-pass resolution -- preferably without requiring that all the rest of typing be imported, since even the people who want to use the typing package agree that importing it is not lightweight.
At times, it sort of reminds me of OWL and "Semantic Web". There are plenty of people who will want to use annotations as a tool, but won't be willing to wade through what can feel like "How many angels can dance on the head of a pin?" discussions. That said, I'm not sure how to best reach people who just want a rough-and-ready usually-good-enough tool. -jJ
![](https://secure.gravatar.com/avatar/2828041405aa313004b6549acf918228.jpg?s=120&d=mm&r=g)
Here's a use case for runtime type annotations: dataclasses use annotations to define fields. With the exception of ClassVar, the actual type is ignored. There's code in dataclasses.py to deal with stringized annotations, specifically just looking for ClassVar. I'd like to see this special case go away, assuming 649 is accepted. And while dataclasses itself doesn't care about the types, apparently users do, and want to be able to call get_type_hints on the annotations. Many people actually want dataclasses to call this for them, but I've resisted. See https://bugs.python.org/issue36643 There are plenty of bugs around this issue, some of which have been fixed. But there are also at least a couple that haven't been fixed, and I'm not sure they will be. I'm hoping that 649 gets accepted and I don't have to worry about the problem at all. Here's an example of an open bug around get_type_hints with dataclasses: https://bugs.python.org/issue45524 Eric On 11/29/2021 6:00 PM, Barry Warsaw wrote:
![](https://secure.gravatar.com/avatar/034bb4232e7e14e4cbf258aad8ca2a57.jpg?s=120&d=mm&r=g)
So maybe this is my time to chime in. I have used annotations for runtime behavior. My primary use case is an injection library that I wrote. It allows something along the lines of: class IMyService(IService): pass @inject def call_something(arg1: str, arg2: int, svc: IMyService = None): assert svc is not None # do stuff... At runtime the `inject()` decorator will scan the function signature and insert an instance of `IMyService` based on a registry lookup. (We use zope.component utilities, but that's an implementation detail.) When testing, one can simply pass a mock version of the service. Using the mypy-zope plugin, the above also passes all mypy type checking. It is a simple pattern in terms of annotation but a very effective pattern that mimics other injection systems in Java and TypeScript (Angular). We have used it for a few years now and like it a lot. Regards, Stephan On Monday, November 29, 2021 6:00:04 PM EST Barry Warsaw wrote:
-- Stephan Richter Entrepreneur & Geek
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Thu, Dec 2, 2021 at 6:57 PM Stephan Richter <stephan.richter@gmail.com> wrote:
So maybe this is my time to chime in. I have used annotations for runtime behavior. My primary use case is an injection library that I wrote.
Does it work with __future__ annotations? -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/489cdabedd1112f98474038939668778.jpg?s=120&d=mm&r=g)
On Fri, 26 Nov 2021 at 09:14, Paul Moore <p.f.moore@gmail.com> wrote:
I don't know if this helps, but I'm a maintainer/creator of a Python library that uses type annotations at runtime. The library is used to create GraphQL APIs[1]. We took a lot of inspiration from dataclasses, our API looks like this: ```python @strawberry.type class User: name: str ``` The `strawberry.type` decorator reads the annotations and creates an object with a list of fields and types, but it doesn't try to resolve them (that happens later to prevent issues with circular dependencies and so on). I remember the discussion around PEP 563 and how it would break libraries using type annotations, in our case things are fine since the types we define should be defined in the global namespace[2] anyway. I think the only issue we have with type annotations is when you have circular references inside your python code (where you'd use `if TYPE_CHECKING: import X`), we "solved" that with a custom class called `LazyType[type, module]`, that allows to specify a type and where it comes from, so our library can resolve the proper type, even if it was not imported in the namespace of the class using that type. Sorry for the digression into some of the details of how the library works, I hope it helps a bit understanding the use case. I think there's definitely use cases for type hints being used at runtime. I personally love the fact that we are allowed to do this, in other languages (like TypeScript) you don't have access to the types at runtime and it does prevent the creation of libraries like dataclasses or mine, which is a bit of a shame (you'd need to use code generation to get a similar experience). There's also other libraries that use type hints at runtime (not for runtime type validation[3]): - Typer - http://typer.tiangolo.com - SQLModel - http://sqlmodel.tiangolo.com - Beanie - https://github.com/roman-right/beanie/ - Pydantic - http://pydantic-docs.helpmanual.io - odmantic - https://github.com/art049/odmantic/ - singledispatch - https://docs.python.org/3/library/functools.html#functools.singledispatch [1] If you're not familiar with GraphQL there's some info here: https://graphql.org [2] I did ask if there was a way to resolve types when having them encapsulated into functions, but that was a specific use case with tests and I was/am fine with using global there https://mail.python.org/archives/list/typing-sig@python.org/thread/SNKJB2U5S... [3] while I see runtime validation being something useful, I think it is worth taking a look at different use cases for type hints :) -- Patrick Arminio
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
This is great Patrick, thanks. My use case is similar to Patrick's, except it builds on dataclasses directly: It's part of a larger system, but I've just pulled it out into its own poorly documented and poorly tested package: https://github.com/PythonCHB/flexi (I think it's generally useful, so may one day make a proper package out of it) Anyway, the principle is this: The goal is to have a flexible data model that enforces the structure, but not necessarily the validity of values, or even types. This is useful because we need to store / edit / import complex hierarchy of data, and we very much don't want to have all of it to have to be valid in order to even store it. An example is a dataset we recently imported (from spreadsheets--arrgghh), where in a thousand records, they had put "too viscous to measure" instead of a value for interfacial tension. That, of course, would be helpful to a person reading the spreadsheet, but would have been a pain if we had enforced that that field had to be a float > 0. Instead, it imported fine, and then our validation code flagged the issue to be fixed later. And all the rest of the data could be used in the meantime. Anyway, that was a digression (but explains why we don't use Pydantic). Functionally, we need the "nodes" in the data model to be able to convert themselves to/from JSON compatible Python, and validate themselves, etc. This is done by using the actual type object, as stored in dataclasses.Field.type This has worked nicely for us, and dataclasses saved us a lot of boilerplate code. if we do from future import Annotations then the system breaks, as we have a string instead of the actual type that is needed. Solutions: I *think* PEP 649 would work fine for us, as the annotation would get evaluated when it was added to the field object. But inspect.get_annotations isn't really helpful for this use, as the Field.type attributes aren't annotations anymore. So we would have to eval() the strings ourselves, and I think it would be unclear what namespaces to use for that. If dataclasses were to use inspect.get_annotations(cls, eval_str=rue) in order to extract the types to put into the Field objects, then I think all would be good for us. And as I understand it, dataclasses itself doesn't do anything with that attribute anyway. If that's not decided to be a good thing for dataclasses, then I think we'd have to re-implement a lot ourselves, though it could probably be hacked in - I haven't tried that yet. BTW, I don't think we've heard from the attrs (and especially cattrs) folks in this thread. A search of the repo issues indicates that there has been some discussion of PEP 563's impact, but it's not totally clear to me if it's been resolved. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/de311342220232e618cb27c9936ab9bf.jpg?s=120&d=mm&r=g)
On 11/26/21 1:13 AM, Paul Moore wrote:
I have one such tool. Fortunately for me, I strongly dislike having the annotations inside the function header as it quickly gets difficult for me to read, so I was already using decorator syntax for the annotations. When PEP 484 was accepted I just changed where the annotations were being stored from __annotations__ to my own dunder. -- ~Ethan~
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Thu, 25 Nov 2021 at 15:16, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
This was not my understanding of annotations when they were introduced e.g.: https://www.python.org/dev/peps/pep-3107/#use-cases As I remember it, a decision about the purpose of annotations was *explicitly* not made when they were introduced. It was clear that typing was a major potential use and then at some point (around about the introduction of the typing module) there seemed to be a shift in people's understanding of what annotations were for. Eventually that reached the point that people who were particularly interested in typing had no memory of the fact that the purpose of annotations had not really been specified as being about typing in the first place. It looks to me like Chris has identified in PEP 563 what is potentially the earliest reference (in an accepted PEP) to the idea that non-typing uses of annotations are to be discouraged. -- Oscar
![](https://secure.gravatar.com/avatar/880ac02012dcaf99f0392f69af4f8597.jpg?s=120&d=mm&r=g)
My 2¢ from a position of *extreme ignorance*, not to mention zero interest in annotations (all welcome to shoot me down in flames, or treat me with contemptuous silence. But *occasionall*y the "idiot's" point of view is worth considering, so here goes). ISTM that typing/annotations have been allowed to drift without a clear end in view, rather like a ship that is steered in one direction by one person, then in another by someone else, with nobody knowing what the destination port is. As witness the conflicting views in this thread. At the risk of being rude, the phrase "headless chickens" comes to mind. ISTM that it is time to take a step back and decide on a definite policy. Perhaps a definitive pronouncement from Guido. Or the SC. Or a discussion from all parties that reaches an acceptable conclusion. Then stick to it. Even if it causes some backward incompatibility - ISTM that things have changed so rapidly that this really would be an acceptable evil if it results in clarity for the future. Meanwhile, do not change anything, do not approve any PEPs, until this conclusion is reached. I apologise if I am being presumptuous in commenting on a subject I know almost nothing about. /I'm just a soul whose intentions are good,// //Oh Lord, please don't let me be misunderstood.// / Best wishes Rob Cliffe On 25/11/2021 23:51, Oscar Benjamin wrote:
![](https://secure.gravatar.com/avatar/d5d40a40d75ba32b1bdff02b57201cd1.jpg?s=120&d=mm&r=g)
Am 26.11.21 um 01:48 schrieb Rob Cliffe via Python-Dev:
I believe the problem is less within the typing community itself (whether static or runtime), but a fundamental conflict between the typing community and what Stephen called the "typing-suspicious crowd". typing in Python has always been hampered by sometimes valid, often unfounded fears that typing is "taking over" and somehow becomes a requirement. For example, PEP 484 made no changes to Python or the stdlib, except the introduction of a new module, often to the detriment of readability and usability of type annotations. And while progress is made slowly (for example with generics in standard collections or the new callable syntax proposal), I'm sure the typing community would love to progress faster and with less obstacles. - Sebastian
![](https://secure.gravatar.com/avatar/1fee087d7a1ca17c8ad348271819a8d5.jpg?s=120&d=mm&r=g)
On Thu, 25 Nov 2021 23:51:58 +0000 Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
This is also what I remember from the discussions at the time of PEP 3107. Annotations were purposefully use case-agnostic, and there was no stated desire to push for one use case or another. I don't think gradual typing was even on the radar, not in public comments anyway. Regards Antoine.
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 12:15:10AM +0900, Stephen J. Turnbull wrote:
I don't think that's what PEP 563 says. Annotations are not *restricted* to only be strings, it is merely that when the function or class object is built, the annotations are left as strings. So we can post-process annotations and destringify them:
There may be scoping issues to be sorted out, but I don't think they are insurmountable. -- Steven
![](https://secure.gravatar.com/avatar/8da339f04438d3fcc438e898cfe73c47.jpg?s=120&d=mm&r=g)
Steven D'Aprano writes:
You're right, I was imprecise. I meant as PEP 563 is implemented now, I doubt it can handle non-strings nicely.
So we can post-process annotations and destringify them:
True, but you face the dread "double-decode problem": if the annotation happens to destringify to a string, then the "post- processor" may try to destringify again. I guess we can mark each annotation as already destringified or not. As long as this is done by some sort of API, you're probably OK (but it makes me nervous since __annotations__ is completely exposed to the program, as in your example).
There may be scoping issues to be sorted out, but I don't think they are insurmountable.
But isn't there still the issue of forward reference, which motivates these PEPs in the first place?
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
There may > be scoping issues to be sorted out,
Yes, the scoping issues are the main problem.
but I don't think they are insurmountable.
Probably not — and LLukas Langa has some good ideas that the SC is considering. (Sorry I can’t get your name right in a phone) Maybe inspect.get_annotations() does, or will be able to, solve many of these issues. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Sat, Nov 20, 2021 at 11:46:56PM -0800, Christopher Barker wrote:
I don't think that's an insurmountable problem. I think that all you need is a small class decorator to evaluate the stringified annotations back to the real things. The PEP tells us the right way to evaluate annotations, so all(?) you need is a decorator to do that to each method in your class, and Bob's your uncle. Maybe PEP 563 could include a decorator in the typing module to destringify all the annotations in a class or function? As far as I can see from a brief scan of the PEP, and based on knowing next to nothing about your use-case, the only hypothetical problem might be this line in the PEP: "Consequently, using local state in annotations is no longer possible in general." but it's not entirely clear to me what Łukasz means by that, or whether it will affect your use-case. -- Steve
![](https://secure.gravatar.com/avatar/d91ce240d2445584e295b5406d12df70.jpg?s=120&d=mm&r=g)
Steven D'Aprano wrote:
On Sat, Nov 20, 2021 at 11:46:56PM -0800, Christopher Barker wrote:
Maybe PEP 563 could include a decorator in the typing module to destringify all the annotations in a class or function?
If it were in an annotations module, that would probably be sufficient. If it is in typing, then it is a very heavyweight dependency -- heavy enough that even the people actually using that module for development (and not for production runs) are worried about the costs. If the costs of the typing module are that high, it is not acceptable to impose them on people not otherwise using the module. -jJ
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 5:47 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:
As of Py 3.10 there is: inspect.get_annotations one could write a decorator around that, but I think the real question is when do you want the annotations to be "finalized"? -CHB heavy enough that even the people actually using that module for
![](https://secure.gravatar.com/avatar/351a10f392414345ed67a05e986dc4dd.jpg?s=120&d=mm&r=g)
On Thu, Nov 18, 2021 at 8:00 AM Barry Warsaw <barry@python.org> wrote:
In my opinion, we can agree that we can not make PEP 563 default in the future because it will break too many use cases. Anyone against making a statement that "PEP 563 will never be the default behavior"? Then, we do not need to decide "PEP 563 or 649". We can focus on whether we can replace "stock semantics + opt-in PEP 563" with PEP 649 or not. Regards, -- Inada Naoki <songofacandy@gmail.com>
![](https://secure.gravatar.com/avatar/53c166c5e1f0eef9ff4eb4d0b6ec9371.jpg?s=120&d=mm&r=g)
On 11/29/21 7:10 PM, Inada Naoki wrote:
Anyone against making a statement that "PEP 563 will never be the default behavior"?
I think only the SC is empowered to make such a statement.
I doubt the current status quo--keeping PEP 563 as optional behavior, permanently--is viable long-term. It causes problems for most runtime uses of annotations, which we were collectively basically ignoring until now. As runtime use of annotations /and/ typing both become more pervasive, these problems are only going to grow. //arry/
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
I don't think the issue of how typing is making its way intot he community is particular relevant to this thread, but I was directly asked a question, so I'll answer it: I don't know exactly where the impression is coming from that typing is a best practice, but it's certainly creeping into example code, etc. So folks will see examples of code that have nothing to do with typing, and probably have no need for it, and there will be type annotations. Many (most?) folks learn from examples more than targeted instruction, so as those examples grow, so will the impression that it's an inherent part of Python. David Mertz' experience is similar to mine as an instructor. Another anecdotal example: I've been teaching an intro to Python class for years, based on the same material, that we try to keep up to date. But typing is not yet included. However, about a year ago, one of the other instructors (that had not previously developed materials for the class) added a new example / exercise to the "intro to classes" lesson. And that example had type annotations. I ended up removing them, as I thought it could be pretty confusing to students to see them for the first time with no explanation (would they think it was something specific to classes?). I wish I'd discussed it with the other instructor, I don't know why that was done -- was it standard practice in their workplace? Seemed like a best practice? I have no idea. Another observation is that more and more library authors are being asked to add type annotations -- maybe that IS a best practice for widely used libraries, but that's one more piece of data. Also: I know this isn't intentional, but Guido has an enormous influence on the Python community -- so I suspect that the fact that he's taken an active role in developing static typing has influenced how it's being perceived. I do think this is a topic for the community to grapple with, but not really a Python-dev responsibility. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
Thanks to Barry and the SC for giving us this update.
<snip> I want to draw attention to this point:
all the use cases and requirements across the static and dynamic typing constituents.
TL;DR: Annotations can be, and are, used for other things than "typing". I just noticed that PEP 563 apparently deprecated those other uses (well, sort of: "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated"), but if the SC is reconsidering PEP 563, then it would be nice to be clear about whether non-typing uses of annotations are indeed deprecated. If not, then the challenge is to come up with a way forward that not only supports both static and dynamic typing, but also other potentially arbitrary use cases. And now onto more detail ... One example is a use case of mine -- I have built a hierarchical object system, built on dataclasses, in which the annotation absolutely has to be an actual type(class) object. PEP 563 will very much break this use case. Of course, there are other ways to write that code, but I'm pretty sure it would require a complete change to the API. I'm not entirely sure if PEP 646 would work for my use case -- likely it would, at least with modest modifications, rather than a new API. I have no idea how many other similar use cases are out in the wild, but we know that the Pydantic project has concerns. To be fair -- I wrote all my code after PEP 563 was accepted. And I only just now read it carefully, and found that it says: "With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated." If that's the case, then that's the case, and I'll deal, but it is disappointing. But I suspect I'm not alone in not really noticing that statement, so I think it's worth it for the SC to explicitly reiterate at this point that all non-typing uses of annotations are depreciated, if indeed, that is still the intent. Also, that statement is not entirely clear. In fact, I think my use case IS compatible with the "aforementioned PEPs", it's just not compatible with PEP 653 itself. It all depends on what is meant by "compatible". I would like to provide a bit of perspective on this. I was criticised on this list a while back for, essentially, not paying attention and then complaining later. After all, PEP 563 was accepted over four years ago. Fair enough. But the fact is that I, among others, have been a bit uncomfortable about the focus on typing in Python for years. But when issues are raised, we have been repeatedly told that typing is, and always will remain, optional. In short, it was made clear that anyone not interested in typing could safely ignore the discussions about it. And thus, a number of PEPs came and went, and those among us that did not choose to involve ourselves in the conversation did not pay attention to the details. And thus we didn't notice that buried in what seemed like a typing PEP, was, in fact, a depreciation of any non-typing uses of an existing Python feature. (also to be fair, the title of the PEP is "Postponed Evaluation of Annotations" -- so should have caught the attention of anyone interested in annotations for any use. In my personal case, I didn't notice it at the time, as I wasn't using annotations for anything at all until fairly recently. In fact, it was the introduction of dataclasses, which I think are the first use of annotations in the standard library, that drew my attention. dataclasses, as I'm sure you all know, use the presence of an annotation on a class attribute to define a field. This is a pretty nifty use of annotations as an API for defining the fields. I liked that, and decided to use it. dataclasses create a Field object, which has an attribute called "type", that gets populated with the annotation's value. PEP 563 would very much change what gets put in the "type" attribute of a Field. Eric V. Smith has indicated that he never intended that to actually be a type, but rather was intended to be whatever the annotation was, so that PEP 563 wouldn't change anything. I'm happy to take his word for it (and PEP 567 says that the actual type is not used or modified by dataclasses), but as users, all we have to go on is the documentation and the choice of names, which very much gave the impression that a Field.type would be a type -- at least if a type was set in the annotation. I also notice that PEP 567 (dataclasses) and PEP 563 were both developed and approved at about the same time. So I suspect the impact of PEP 563 was not considered when designing or documenting dataclasses. However, PEP 567 does make a number of references to typing, and was clearly written with the idea in mind that the annotations would, in fact, be used for type analysis. But again, no mention of PEP 563, or what might be in the field.type attribute -- one might wonder why it was there at all if it was not intended to be used. We need help in defining those requirements clearly, in uncovering the use
cases we’re not aware of, and most importantly, being an interface to typing enthusiasts
Again -- is it only "typing enthusiasts" that you want to engage? Or "users of annotations"? -- maybe it is, but it would be nice if that was a clear statement from the SC. - CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Sun, 21 Nov 2021 at 07:50, Christopher Barker <pythonchb@gmail.com> wrote:
TL;DR:
Annotations can be, and are, used for other things than "typing". I just noticed that PEP 563 apparently deprecated those other uses (well, sort of: "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated"), but if the SC is reconsidering PEP 563, then it would be nice to be clear about whether non-typing uses of annotations are indeed deprecated. If not, then the challenge is to come up with a way forward that not only supports both static and dynamic typing, but also other potentially arbitrary use cases.
I agree with the points made in this post. It's becoming harder and harder for people not particularly interested in static typing to simply ignore it, and any use of annotations to affect runtime behaviour is in a weird grey area. And as a library author, I'm now finding that I'm getting requests to add typing to my code "for my users" (i.e., using types is no longer just a choice I make for my project, it's an API design issue). So I too would appreciate clarity on where annotations, and more generally typing, stand as Python language features¹. Paul ¹ In particular, typing features seem to change so rapidly that supporting a wide range of Python versions is a real pain (in practical terms - not getting syntax errors is straightforward, but writing type annotations that work well across versions very definitely isn't, at least for someone who doesn't really care about static typing).
![](https://secure.gravatar.com/avatar/8da339f04438d3fcc438e898cfe73c47.jpg?s=120&d=mm&r=g)
Executive summary: The typing-suspicious crowd has a valid complaint about PEPs 563 and 649, but it's not that they weren't warned. Christopher Barker writes:
Not a PEP proponent (or even a typing user), but I thought this had been made clear long ago. My understanding is that optional, incremental type hints are and have always been considered the primary use case for annotations by the BDFL and AFAICT the SC following the BDFL. If compatibility with typing is an issue, then the burden of implementing that is on the other application. Typing *might* do something to help, but it's not obligated to do so.
If PEP 649 works for you, I expect you could work with PEP 563 by moving Larry's data descriptor concept out of the compiler and into your library code. You could also appeal to the "might do something" caveat above. Yury suggested providing both the immediately evaluated object (if there were no forward references, I guess) and the string representation, which would be backward compatible with your use case. The idea was rejected on the basis that it doesn't help with the primary use in typing, and doesn't remove the need for a __future__ import during the transition period. You could ask that that decision be revisited, since it would ensure that uses of __annotations__ like yours continue to work.
But I suspect I'm not alone in not really noticing that statement,
I'm not surprised people haven't noticed that statement. I am surprised that a lot of folks haven't noticed that this comes up occasionally and the answer every time is "we're not going to go out of our way to break other use cases, but typing *is* the primary use case and will take precedence if the question comes up". I will grant that restricting the type of compiled annotations from typing.Any to "string serializing an object's AST" could reasonably be said to be "going out of your way to break other use cases". You have a valid beef here, although it's not obvious to me how it should be resolved. So I'm not saying that you don't have an interest in the future semantics of this dunder. I am +1 on Python providing relief for your use case.
But the fact is that I, among others, have been a bit uncomfortable about the focus on typing in Python for years.
You mean y'all are uncomfortable with the popularity of typing. You wouldn't care if it wasn't used outside of hugely complex proprietary codebases you'll never see. Thing is, reasoning about programs is What We Do as programmers, and efforts to make it easier to do that for complex programs, and to provide software to help with that task, are here to stay and they will be used in mixed company. Even in Python. (And that's why, as someone who doesn't use typing, I support typing. Anything that helps *other people* to write better code, I can get behind. My code, I'm going to write as incompetently as ever.[1] ;-)
But when issues are raised, we have been repeatedly told that typing is, and always will remain, optional.
This has always been in the context of *your* source code. For example, nobody ever claimed you could keep it out of anybody else's source code (although that's clearly what a lot of those "raising issues" want!) Still, features like stub files were provided so typists could work and play nicely with non-typists if they wanted to (and many typists want that, themselves). But dunders are the property of the language (or sometimes the implementation), and they always have been. If you use them in a way that's not documented to work, you're at risk even if it happens to work now. PEP 3107 didn't document that anything would work. ;-) Footnotes: [1] Barry, do not tell Abhilash I wrote that.
![](https://secure.gravatar.com/avatar/880ac02012dcaf99f0392f69af4f8597.jpg?s=120&d=mm&r=g)
On 25/11/2021 15:15, Stephen J. Turnbull wrote:
“As you will no doubt be aware, the plans for development of the outlying regions of the Galaxy require the building of a hyperspatial express route through your star system, and regrettably your planet is one of those scheduled for demolition. The process will take slightly less than two of your Earth minutes. Thank you.” Well we were warned, so everything's fine. Rob Cliffe
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
First: At this point, I am not advocating anything in particular. I am just asking that the SC makes a clear statement about the intention at this point. From Barry's email: """ This is also a call to you, the folks who both care deeply about typing in Python, and have a solid understanding of the subject matter (or willingness to learn) to help us! There are use cases we don’t know about, and unclear requirements from both the static and dynamic typing users. """ I don't care deeply about typing, and I'm not a static or dynamic typing user. So this implies that a decision has been made that annotations are for typing, and typing only. (not that you're not allowed to use for anything else, of course you are, but that other uses won;t be taken into account when designing the new interface) But I have never seen that clearly stated anywhere. The closest is from PEP 563, where it says: """ With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated. """ Which pretty much makes the point, but it's a bit subtle -- what does "incompatible' mean? On Thu, Nov 25, 2021 at 7:15 AM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
The typing-suspicious crowd has a valid complaint about PEPs 563 and 649, but it's not that they weren't warned.
Well, "typing-suspicious" is part of it, but the real problem is folks that might be using annotations for non-typing purposes. As for warning: yes and no -- what's new here is that as far as I know, this is the first time since the py3 transition that a __future__ import would be made standard behavior. And you'd have to be paying a lot of attention to know that was going to happen. And I suspect that only folks involved in the development of typing tools were paying that much attention. In the case of the py3 transition -- it was very well known and documented that lots of things would be changing, and we had a very long transition period. But the challenge with this is that there is no way to raise a depreciation warning for not using a __future__ import. So while we all should have been testing our code with: from __future__ import annotations For the last few versions, I doubt that very many people not developing typing tools were doing that.
well no -- not "always" -- look at PEP 3107, when annotations were first introduced: "By itself, Python does not attach any particular meaning or significance to annotations." and it goes on to give non-typing examples. Since then, the shift from "nothing more than a way of associating arbitrary Python expressions with various parts of a function at compile-time" to a system for defining types has been gradual, with a number of PEPs, and, as far as i can tell, PEP 563 is the first time that a change was proposed that would affect other uses for annotations. Anyway, that is water under the bridge. But the reason I bring it up is because of all the many thousands of Python users, I pay probably more attention to changes than most -- I have followed Python-ideas and Python-dev for years. So I'm guessing that there are other folks out there that will be surprised if and when the behavior of annotations changes.
If compatibility with typing is an issue, then the burden of implementing that is on the other application.
That does indeed seem to be the current intent. And I'm happy to deal with that -- I just want it to be clear that I (and others) have to wait and see how it shakes out, and then figure out what to do, or if I should contribute to the conversation now. If non-typing use cases of annotations are still considered important, then we should encourage folks like me to be part of the conversation.
That would indeed require maybe not a change to the API, but certainly some complex code. But yeah, probably doable one way or another.
I think the "breaking other use cases" has been subtle to those not paying attention to typing.
well, yes. The issue is that, intended or not, typing is making it's way into Python culture. As an instructor of beginning python users, I am unsure at this point when to introduce type annotations. What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required. But that's getting pretty OT here.
Anything that helps
*other people* to write better code, I can get behind. My code, I'm going to write as incompetently as ever.[1] ;-)
me too -- actually, not quite. I write a lot of "scripts" that I will continue to keep simple, and I also develop complex systems -- and those require more care.
Sure -- my point with that is that if I don't want to use typing in my code, then I didn't have to pay attention to the development of the typing systems. But my point here is that what changed with PEP 563 is that while typing is still optional, this is the first time that the language itself may be changed to accommodate typing -- so it can no longer be completely ignored. But dunders are the property of the language (or sometimes the
no, but it did document that annotations were "arbitrary Python expressions" -- that is changed in PEP 563 -- intentionally, but it is changing something about the language that was documented. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Thu, 25 Nov 2021 at 21:45, Christopher Barker <pythonchb@gmail.com> wrote:
The issue is that, intended or not, typing is making it's way into Python culture. As an instructor of beginning python users, I am unsure at this point when to introduce type annotations.
What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.
Agreed this is somewhat OT, but I also think the messaging around annotations needs to be reviewed. I suspect that students will also not clearly understand the fact that annotations aren't checked at runtime. And IMO this is particularly difficult to get across in the case of non-typechecker uses of annotations. The following, for example, is extremely non-intuitive to me:
Even though the "typing is optional" message is well-understood, my instincts still lead me to expect that declaring the type of n will result in a runtime type-check, in the constructor at least. (If I check the code with mypy, it does raise an error, so that's good. Although the more I think about it, given that I believe dataclasses use eval "under the hood", the less I understand *how* it manages to do that without special-case knowledge of the dataclass decorator...) I'd like to see a clearer statement from "somewhere" about how APIs should use annotations at runtime, such that Python users have a much clearer intuition about APIs like the dataclass one, and library designers can build their APIs based on a clear "common understanding" of what to expect when annotations are used. Paul
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 1:37 AM Paul Moore <p.f.moore@gmail.com> wrote:
Static checkers special-case the @dataclass decorator. Eric Traut has a proposal to generalize this support (sorry, I'm in a rush, otherwise I'd dig up the link, but it's in the typing-sig archives).
Note that @dataclass itself is very careful not to use the annotations, it only looks for their *presence*. With one exception for ClassVar. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Fri, 26 Nov 2021 at 17:13, Guido van Rossum <guido@python.org> wrote:
:-( That's what I suspected, but it does mean that dataclasses has a privilege that other libraries (like attrs, I guess?) don't get.
Understood. What I'm suggesting is that it would be good to have a clear "common understanding" about whether libraries should be careful like this, or whether it's OK to base runtime behaviour on type annotations. And if it is OK, then what are good patterns of design and behaviour? This is where the proposal to store annotations as strings hit issues, because it appears to take the view that libraries *shouldn't* be looking at the actual types specified by annotations (or maybe that they should only do so via something like `typing.get_type_hints`). There are other subtleties here (runtime code needs to deal with the fact that int and "int" should be treated the same) that there's no guidance on, again possibly because no-one is really considering that use case. Paul PS I've never written code myself that does runtime introspection of type annotations - so it's quite possible that there *is* guidance that I've just missed. But it wasn't obvious to me from a quick search - the "introspection helpers" section of the typing module docs is pretty basic...
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 11:58 AM Paul Moore <p.f.moore@gmail.com> wrote:
Actually both are done through plugins (since what they do just doesn't fit in the PEP 484 type system) and both have the same status, at least in mypy. (In fact we have twice as many lines of code dedicated to attrs than to dataclasses. https://github.com/python/mypy/tree/master/mypy/plugins) The proposal I mentioned by Eric Traut (pyright's author) would make it easier to support many similar libraries across all static type checkers. ( https://mail.python.org/archives/list/typing-sig@python.org/thread/TXL5LEHYX... )
You are hitting the nail on the head here. I'd say that so far the recommendation has been "use typing.get_type_hints(x) rather than x.__annotations__" -- this handles the equivalence between int and "int" (and hence forward references as well as 'from __future__ import annotations'). There's now also inspect.get_annotations() which has roughly the same functionality without quite so much bias towards typing, so perhaps we should recommend it over typing.get_type_hints() -- though before 3.10 inspect.get_annotations() didn't exist so you might have to fall back on using the other. (There are some semantic differences between the two that I'm glossing over here, because I'm not sure about what exactly they are. :-) We now also have a document that recommends best practices ( https://docs.python.org/3/howto/annotations.html) although it's very new -- it appears Larry Hastings wrote it while he was pushing for PEP 649 (but it received buy-in from the static typing community as well). So perhaps it isn't as bad as it seems, *if* you know where to look? That said, I don't think the current static typing infrastructure would be prepared for an onslaught of modules that use type introspection at runtime to modify the behavior of classes a la attrs and dataclasses. I don't know enough about pydantic to say whether it also falls in this category. But it's definitely difficult to write code that makes use of type annotations at runtime that *also* passes static type checks by mypy etc.; you certainly shouldn't attempt to do so without having CI jobs to run the static checker and test the runtime-introspecting framework you're using. (It's not quite like trying to write code that's valid Python and Fortran at the same time, but it's not trivial. :-) There are also features of our static type system that tend not to be supported by the runtime-introspecting frameworks -- in particular, I'd expect generics and callable types to be hard to deal with at runtime. It's easy enough to do something at runtime with `def f(a: list[int]) -> int`. It's not so simple to handle `def f(a: Sequence[T]) -> T` or `def f(cb: (T) -> tuple[str, T], Sequence[T]) -> Mapping[str, T]`. Presumably frameworks like pydantic just don't support such things and tell the user not to do that. (Someone should look where pydantic draws the line and report back here.)
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 1:47 PM Guido van Rossum <guido@python.org> wrote frameworks like pydantic just don't support such things and tell the user not to do that. I don’t know about Pydantic, but that’s exactly my use case: the type needs to be an actual intstantiatable type. So list and tuple is fine, and Sequence[t] won’t work. But it’s fine that this use case is restricted. In fact, other than the basic core types, you need to use specialized types with this system anyway. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
So does your library work both with and without ‘from __future__ import annotations’? If it does, you shouldn’t have to worry. If it doesn’t, it would be useful if you could post some detailed examples of what goes wrong. On Fri, Nov 26, 2021 at 14:24 Christopher Barker <pythonchb@gmail.com> wrote:
-- --Guido (mobile)
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
I hope my understanding of where the SC’s debate about this currently sits isn’t a misrepresentation, but since I sent the email on behalf of the SC, let me try to clarify what I was trying to say. Aside: A little insight into how the SC works. For communications like this, after whatever debate we have in our live weekly meetings (and sometimes off-line discussions in Slack and email), one of us volunteers to write a draft of the email. We then all comment and wordsmith until we’re in unanimous agreement that it’s ready to go out. Then usually the person who wrote it will send the email. We all have slightly different styles, but I usually sign it with my name “on behalf of the Steering Council”. That’s my way of saying that the communication comes from all of us and represents the SC’s opinion, but it generally has the written style of the original drafter.
Practically speaking, I think that’s current reality: annotations are for typing, be they static type checking or dynamic runtime uses. I was inelegantly saying that even with that narrower interpretation of annotations, there are use cases, interactions, side-effects, semantics, and implications that we do not fully understand. PEP 563 and 649 have visible effects that even within that domain can have important side effects. For example, PEP 563’s loss of local scope, which even “de-stringify-ing” can’t recover. This is what we need help with.
You make a good point. I agree that while all the signs are there for “annotations are for typing”, this has never been explicitly or sufficiently codified, and I’ve been proposing that Someone write a PEP that makes this an official pronouncement. That someone may very well be a person on the SC, but given the timing of elections, it’s likely to fall to the next SC, if they still agree with that position! :-D My recollection of the history of annotations falls somewhere between Greg’s and Guido’s. Annotations as a feature were inspired by the typing use case (with no decision at the time whether those were to be static or runtime checks), but at the same time allowing for experimentation for other use cases. Over time, annotations-for-typing clearly won the mindset and became the predominant use case. Personally, I was strongly against type annotations because of my experience in other languages, but over time I was also won over. From library documentation, to complex code bases, to the transient nature of contributors, type annotations are pretty compelling, and most (but not all) of my worries really didn’t come to fruition. We can lament the non-typing use of annotations, but I think that horse is out of the barn and I don’t know how you would resolve conflicts of use for typing and non-typing annotations. It’s been a slow boil, with no definitive pronouncement, and that needs to be fixed, but I think it's just acknowledging reality. That’s my personal opinion.
well, yes. The issue is that, intended or not, typing is making it's way into Python culture. As an instructor of beginning python users, I am unsure at this point when to introduce type annotations.
What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.
This is an important observation. As much as I’m in the "type annotations are good” crowd now, I still think they should always be optional. Python’s use is so broad these days, I for one don’t want to have to add annotations to every bit of Python I write. -Barry
![](https://secure.gravatar.com/avatar/53c166c5e1f0eef9ff4eb4d0b6ec9371.jpg?s=120&d=mm&r=g)
On 11/29/21 2:56 PM, Barry Warsaw wrote:
PEP 563 and 649 have visible effects that even within that domain can have important side effects. For example, PEP 563’s loss of local scope, which even “de-stringify-ing” can’t recover. This is what we need help with.
Well, sure. If PEP 563 and 649 didn't have visible effects, there'd be no point in doing them. That said, I suggest 649 does a lovely job of avoiding /undesirable/ side-effects. Sure, 649 has observable side effects. For example, you can detect whether or not 649 is active by rebinding a name after it's used in an annotation but before examining that annotation at runtime. This seems harmless--unlikely to happen in production code, and easily remedied if someone did trip over it. A more credible side effect: if you use an undefined name in an annotation, you won't notice at compile-time. Now, this is actually 649's major feature! But there are also scenarios where this feature could cover up a bug, like if you misspell a name--you won't notice until you examine the annotation at runtime (or, more likely, until you run your static analyzer). 563 has this same behavior--and it wasn't enough to prevent 563 being accepted. So I assume this wouldn't be enough to prevent accepting 649 either. 649 has effects on memory usage and performance, but honestly I'm not worried about these. I don't think the memory usage and performance of the prototype were particularly bad. Anyway, as I've said many times: we should figure out the semantics we want first, and then we can worry about optimization. The Python core dev community has no end of smart people who love optimizing things--I'm sure if 649 was accepted and merged, the optimizations would start rolling in. Then of course there are also things 649 simply doesn't do, e.g. resolve the "if TYPE_CHECKING" situation. But it's not appropriate to call that a "side effect" per se. And that's my list. If anybody knows of other visible side effects from 649, naturally you should contact the SC. And/or me, if you think we could change 649 to mitigate it without losing its major features. Happy holidays, //arry/
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Nov 29, 2021, at 15:57, Larry Hastings <larry@hastings.org> wrote:
PEP 563 also has visible side effects, but the real implication of them wasn’t well understood at the time the PEP was approved. Or maybe it was and we underestimated the impact on certain important users and use cases. I’m particularly wary about getting into that same (um) pickle again.
A more credible side effect: if you use an undefined name in an annotation, you won't notice at compile-time. Now, this is actually 649's major feature! But there are also scenarios where this feature could cover up a bug, like if you misspell a name--you won't notice until you examine the annotation at runtime (or, more likely, until you run your static analyzer). 563 has this same behavior--and it wasn't enough to prevent 563 being accepted. So I assume this wouldn't be enough to prevent accepting 649 either.
How common is it for code to examine annotations at run time, outside of specialized libraries that have to be more defensive anyway (especially given they already have to be defensive in light of 563)?
649 has effects on memory usage and performance, but honestly I'm not worried about these. I don't think the memory usage and performance of the prototype were particularly bad. Anyway, as I've said many times: we should figure out the semantics we want first, and then we can worry about optimization. The Python core dev community has no end of smart people who love optimizing things--I'm sure if 649 was accepted and merged, the optimizations would start rolling in.
I agree about that priority (semantics first, optimization later), as long as those semantics don’t make such future optimizations impossible, or severely degrade important performance constraints such as import times.
Then of course there are also things 649 simply doesn't do, e.g. resolve the "if TYPE_CHECKING" situation. But it's not appropriate to call that a "side effect" per se.
Although, it would be nice to have a comprehensive solution to this and other situations too, even if they happen outside of PEP 563/649 discussion…
And that's my list. If anybody knows of other visible side effects from 649, naturally you should contact the SC. And/or me, if you think we could change 649 to mitigate it without losing its major features.
That list is exactly what I'm (we’re?) looking for. -Barry
![](https://secure.gravatar.com/avatar/d6b9415353e04ffa6de5a8f3aaea0553.jpg?s=120&d=mm&r=g)
On 11/29/2021 5:56 PM, Barry Warsaw wrote:
On Nov 25, 2021, at 13:41, Christopher Barker <pythonchb@gmail.com> wrote:
What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.
This is an important observation. As much as I’m in the "type annotations are good” crowd now, I still think they should always be optional. Python’s use is so broad these days, I for one don’t want to have to add annotations to every bit of Python I write.
Maybe it should be reiterated with whatever decision comes forth that
and other duck-typed code will always be legal, idiomatic, and even expected as good practice for beginner, informal, exploratory, and similar python code. -- Terry Jan Reedy
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Mon, Nov 29, 2021 at 07:56:16PM -0500, Terry Reedy wrote:
[...]
Many of the typing-related PEP comes with such a disclaimer, listed as "Non-goals". For example: https://www.python.org/dev/peps/pep-0484/#non-goals Łukasz's stringified annotations PEP has a non-goals section: https://www.python.org/dev/peps/pep-0563/#non-goals Larry's deferred evaluation PEP does not: https://www.python.org/dev/peps/pep-0649/ but I don't think we should hold it against PEP-649. It's not trying to sneakily sneak mandatory static typechecking in by the back door like some sort of sneaking sneak :-) -- Steve
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Mon, Nov 29, 2021 at 5:01 PM Terry Reedy <tjreedy@udel.edu> wrote:
Why would it need to be reiterated? Are there really people who believe that such code would become invalid? AFAIK *everybody* here agrees that this should stay valid. So who would we be reiterating it for? (Yes, several static type checkers have options that cause the checker to complain about unannotated code. But that's not the default behavior and running a static checker is a choice, like running a linter. Three-space indents or capitalized function names are never going to be disallowed either, even though PEP 8 says that's not how you ought to code.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/d6b9415353e04ffa6de5a8f3aaea0553.jpg?s=120&d=mm&r=g)
On 11/29/2021 8:16 PM, Guido van Rossum wrote:
Chris, can you say anything more about why people would get such a mis-impression?
The people that Chris B. referred to, and their 'sources'. But we need more info to know how to best counter such wrong beliefs. -- Terry Jan Reedy
![](https://secure.gravatar.com/avatar/be200d614c47b5a4dbb6be867080e835.jpg?s=120&d=mm&r=g)
On 11/30/2021 1:16 AM, Guido van Rossum wrote:
I know it sounds like a straw-man argument, but I don't think it is. I just think the non-straw-men making the argument aren't "important" enough to call out specifically - they are "random individual programmers who have gotten the wrong impression from another language and applied it to Python" rather than "influencers trying to change how we code".[1] And we can always say that the *language* does not require running the type checker, but the reality for many developers is that the projects and teams they are part of will require it. And often, they require it because it's some kind of global "Best Practice" rather than because it makes sense for their particular situation. For example, I saw a snippet fly by on Twitter the other day (again, not the fault of the individual who posted it, so I'm not naming-and-shaming) showing how to use Black to format a big directory of Jupyter notebooks. Convenient, perhaps, but the vast majority of people with "big directories of Jupyter notebooks" are going to be much happier if you just leave them alone! And yet, "The Community" is suggesting we should format all of them in bulk. THAT'S the kind of thing that also has been happening with typing, and why some of us feel the need to publicly re-state things that are all agreed upon within this group, but are struggling to be heard over the public discourse on the topics. And this kind of reiteration is easier when our official documents (PEPs, etc.) state it explicitly. Cheers, Steve [1]: Okay, I will call out the Python Bytes podcast - which I've been on a couple of times - as one that regularly opines on topics based on very little information and says things more definitively than they've been said anywhere else. They have a much bigger (and more easily influenced) audience than python-dev. I'll also say that what I see from LWN is generally very good in this respect, their writers handle things well. Though I don't tend to read or listen to either of these that often.
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 02:52, Steve Dower <steve.dower@python.org> wrote:
This is very definitely the case. There's a subtle (maybe not so subtle, actually) and increasing pressure on projects to add typing. Often with little or no justification beyond "you should", as if having typing is a sort of "obvious best practice". Sometimes "because it will make it easier for your users who use typing" is given as a justification, but while that's fair, it's also a disturbing gradual pressure for typing to extend everywhere, manifesting by making it feel like not adding typing is somehow "not caring about your users". Also, related to the question Terry raised, IMO it would be useful to have a clear statement on code that *does* use type annotations, but violates them at runtime. To be specific, is the following considered as an error?
If (as in the current interpreter) typing is optional, this isn't an error. Certainly, running mypy over it fails, but the point here is that mypy is an external tool - the Python interpreter itself (and the language definition) allows it. I think it would be useful to have a clear message that this is intended, and won't be altered lightly. Remember that the "correct" annotation here is either muladd(x: Any, y: Any, z: Any) -> Any, or muladd(x: SomeMultiplyProtocol, y: SomeAddProtocol, z: SomeAddProtocol) -> Any, neither of which is particularly helpful... Paul
![](https://secure.gravatar.com/avatar/d67ab5d94c2fed8ab6b727b62dc1b213.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 8:19 PM Paul Moore <p.f.moore@gmail.com> wrote:
My understanding is that it's precisely as wrong, or not wrong, as this: def add(x, y): """Multiply two numbers""" return x / y To my mind, annotations are machine-readable metadata, with no inherent "correctness" to them (from the language's point of view), other than syntactically. ChrisA
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 09:17:13AM +0000, Paul Moore wrote:
Isn't that just duck-typing? You've got a function that is documented as taking arguments of one type (int), but actually will work with any numeric type (and some non-numeric types) that quacks like an int. muladd(2, 'ab', 'cd') # returns 'abcdabcd' If you replaced the type annotations with a docstring that listed the parameters x, y, z as int, would duck-typing be wrong? Maybe. I don't think we can make any concrete claims here. Sometimes duck-typing is perfectly fine. Sometimes its not. It depends on the function's implementation and its semantics. Sometimes calling a function with a duck-typed value seems to work fine but the result is meaningless junk. (Garbage In, Garbage Out.) I guess all we can say is that the Python language is agnostic and neutral on this matter. -- Steve
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 09:23, Paul Moore <p.f.moore@gmail.com> wrote:
I suspect that a big source of this is editors that use typing for autocompletion etc. Speaking as a vim user this is something that has passed me by except that I have students who I see adding type hints even though I deliberately haven't taught them anything about type hints. I can only presume that some editor is doing this for them or telling them that they need to do this (students often can't tell the difference between editor warnings and actual errors). Others have mentioned the pressure on libraries to adopt typing and I've certainly noticed this with SymPy. I think type hints could be good for SymPy for internal use but it seems that a lot of users want it for external reasons that I don't always understand but that also seems to come partly from editors as well. Some people apparently want to add type hints that look completely useless to me like def f() -> Union[MyClass, Any] As I understand it this does not give any meaningful information to a type checker but it apparently makes vscode work better: https://github.com/sympy/sympy/pull/22180#discussion_r718917577 Here in SymPy you can see a suggestion that type hints are needed because otherwise vscode is slow and uses 4GB of memory just to load documentation for SymPy functions: https://github.com/sympy/sympy/issues/17945#issuecomment-928798977 Not to do with editors but also indicative is this PR whose title implies that SymPy is "incompatible" with PEP 561 (a PEP that I had never heard of before seeing the PR): https://github.com/sympy/sympy/pull/22337 There are other open "issues" like this for SymPy where the presumption is that not having type hints is now to be considered a deficiency of the library regardless of whether the hints have any benefit for internal use. I don't object to adding the hints but it's a huge amount of work that someone would have to do and I don't want to add useless/inaccurate hints temporarily (as some have suggested to do). -- Oscar
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 12:39, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
This is precisely the sort of concern I have. Although it is awfully tempting to passive-aggressively annotate everything as Any, just to shut people up :-( And to be clear, it's often very non-obvious how to annotate something - in https://github.com/pfmoore/editables I basically gave up because I couldn't work out how to write a maintainable annotation for an argument that is "a Path, or something that can be passed to the Path constructor to create a Path" (it's essentially impossible without copy/pasting the argument annotation for the Path constructor). I also spent a lot of time trying to deal with a typeshed bug. And even worse, if I put the types in a .pyi file (which I want to do, because I don't want the library to import typing at runtime because it's a significant overhead for something that's supposed to be lightweight), apparently mypy won't check them so I gain no benefit for the project itself. Anyway, we're *way* off topic now, and I doubt there's much that the SC or python-dev can do to change the community view on typing, anyway. Paul
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 02:30:18PM +0000, Paul Moore wrote:
I thought that type inference was supposed to solve that sort of problem? If the typechecker can see that an argument is passed to the Path constructor, it should be able to infer that it must be the same types as accepted by Path. Aside: I'm a little disappointed in the way the typing ecosystem has developed. What I understood was that we'd get type inference like ML or Haskell use, so we wouldn't need to annotate *everything*, only the bits needed to resolve ambiguity. But what we seem to have got is typing like C, Pascal and Java, except gradual. Am I being unreasonable to be disappointed? I'm not a heavy mypy user, I just dabble with it occasionally, so maybe I've missed something.
Heh. We could update PEP 8 to ban type annotations, then watch as the people who over-zealously apply PEP 8 to everything AND over-zealously insist on adding type annotations to everything have their heads explode. -- Steve
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano <steve@pearwood.info> wrote:
You're after https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike: `str | PathLike[str]` (if you're only accepting string paths).
I would change that "should" to "may". Python's dynamism makes inferencing really hard.
It really depends on the code base. Type checkers can make guesses based on the code they have available to them, but that only works if the usage is really clear and the dynamic nature of the code doesn't make things murky. For instance, look at open() and how whether you opened a file with `b` or not influences whether the object's methods return strings or bytes. What would you expect to be inferred in that case if you didn't annotate open() with overrides to specify how its arguments influence the returned object? It also depends on how lenient your type checker is willing to be. Compiled languages get to know all potential inputs and outputs of a function upfront, while Python it's a guess based on what you happen to have installed since you can always just `importlib.import_module()` anything at any time. But since type checkers typically prefer false-positives over false-negatives for correctness, it means having to clarify more code.
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 19:07, Brett Cannon <brett@python.org> wrote:
Well, it's not really. What I'm after, as I stated, is "anything that can be passed to the Path constructor". Yes, str | PathLike[str] is probably close enough (although why is it OK for me to prohibit bytes paths?) but that's what I mean about copying the Path constructor's annotations. If Path changes, I have to change my code. This is a very common idiom: def f(p: ???): p = Path(p) ... Why isn't it correspondingly straightforward to annotate? If PathLike[str] included str, then it would be a lot easier. It's not at all obvious to me why it doesn't (well, that's not entirely true - it's because PathLike is an ABC, not a protocol, and it's not intended to define "the type of objects that the Path constructor takes"). It would still not be documented anywhere, though.
That's fair. That's why I think it should be straightforward for the user to explicitly say "this argument should accept the same types as pathlib.Path does". If inference can't do it automatically, and the user can't (easily) let the checker know that it's OK to do it, then we're left with no easy way to express a very common pattern.
Personally, I'd be quite happy leaving open() as duck typed. I see this as what Steven was getting at - the "typing ecosystem" has moved into a situation where it's acknowledged that some typing problems are really hard, due to Python's dynamism, and yet there's still a drive to try to express such highly dynamic type constraints statically. And worse still, to insist that doing so is somehow necessary. Whatever happened to "practicality beats purity", and typing being "gradual"? Surely annotating everything except open() is practical and gradual? Paul
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
Another concern I have is the over specification of types. I have seen many examples of, e.g. func(x: int, y: float, stuff: List(int]): but very few of: func(x: SupportsInt, y: SupportsFloat, stuff: Sequence[SupportsInt]): (or even Iterable[int]) Is that even the right thing to do to get generic number types? How do I specify a type that could be any number type (float, int, Fraction, Decimal, numpy.float32) ... I just spent a few minutes perusing the typing module and MyPy docs, and didn't find a discussion of that. But I'd really love to see the community as a whole start with more generic types in examples, and be clear that the concrete types should be used only when necessary -- even the typing module docs use a lot of concrete types in the examples. I think this is an issue for two reasons: 1) Folks learn from examples more than instruction -- people are very likely to follow the first example they see that seems to do the job. 2) as mentioned in this thread, library authors are being asked to type hint their libraries. I think it will be all too common for that type hinting to be more specific than required. For the most part, that won't cause any real issues right away, but as soon as someone uses that lib in a large project that enforces static type checking, it's could get ugly. -CHB On Tue, Nov 30, 2021 at 1:37 PM Paul Moore <p.f.moore@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 2:52 PM Christopher Barker <pythonchb@gmail.com> wrote:
There is some discussion of the numeric tower in PEP 484 but the PEP says you should just use 'int', 'float' and be happy. *If* we were to change our mind on that we would let you spell it using numbers.Integer, numbers.Real etc., not SupportsInt, SupportsFloat. But the usability of any of those generic solutions is just terrible (too much typing for too little benefit) and honestly in many cases the runtime is not very good at these either (e.g. Decimal refuses to play). In practice it is just fiction that you can write your own numeric type. Regarding Iterable[int] vs. Sequence[int], there are subtle semantic differences so it depends, but these have much better support and *are* encouraged. (What helps is that List is invariant but Sequence is covariant, and people often want covariance, so there is a real incentive to use Sequence, Iterable etc.)
Maybe you are speaking with a lot of enthusiasm but not a lot of experience? Generics aren't always better, and in many cases they just aren't worth the effort to get them right (just like not every bit of code you write needs to be published on PyPI). I also have a feeling that most frameworks that use *runtime* introspection of types strongly prefer concrete types, i.e. list[int] rather than Sequence[T].
It would be nice if someone did some work and collected a list of tutorials about type annotations that exist (especially the ones that are discoverable with a simple Bing query) and ranked them by quality.
That's just the tip of the iceberg. In practice, there are quite a few reasons why the first few iterations of type hints for many popular packages fall short of expectations. Often the issue isn't just sufficiently wide types (like you speculate here) but missing types (for example, stubs containing functions with arguments but no annotations), missing methods/attributes, missing classes, and missing submodules. Not to mention signatures that are impossible to type with the current type system (you can't actually type zip() or filter(), for example). We should definitely push back on zealous new converts to typing who insist that everything should be annotated. But we should also recognize that even in their current, far from perfect state, type annotations can provide a lot of value, when used right. (Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Tue, 30 Nov 2021 at 23:37, Guido van Rossum <guido@python.org> wrote:
We should definitely push back on zealous new converts to typing who insist that everything should be annotated. But we should also recognize that even in their current, far from perfect state, type annotations can provide a lot of value, when used right. (Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.)
I might be misunderstanding but if "hover-doc" is the same as "mouse-over" then it apparently needs many gigabytes of memory and a lot of CPU time when you put the mouse over a sympy function that you are using in your code. It has been suggested that SymPy should fix this by adding hints like Any but I don't see the point of that: SymPy has plenty of bugs but this particular bug is in VS Code. I can see the potential value that type hints can bring for an editor like VS Code but is the editor also pushing an agenda that all code should be *explicitly* typed? Does VS Code pressurise people into using type hints in their own code? I presume it does because I see the effect in my students (although I might be misattributing this to VS Code rather than something else). -- Oscar
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 4:48 PM Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
(Yes, I meant mouse-over. I had a senior moment. :-) I cannot help you much here, there's a tracker and generally speaking the team does pay attention to tracker issues, so I recommend starting a discussion there. Feel free to CC me on the issue if you decide to create one.
There are two modes in VS Code, a lenient and a strict mode. The strict mode does insist on typing, but you have to opt in to it first. Once you've opted in, it does indeed encourage you to type everything -- but that's pretty much the meaning of strict type checking (mypy has a --strict flag that also requires this). That said, your students probably copied some configuration that turns on strict mode from someone else who is thereby pushing this agenda unwittingly. Sadly most people forget where they got their initial configuration (I know I forget it as soon as it works :-). And the strict mode also provides useful error messages in certain cases. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/be200d614c47b5a4dbb6be867080e835.jpg?s=120&d=mm&r=g)
On 12/1/2021 12:47 AM, Oscar Benjamin wrote:
For the record, this is the *old* support that has since been replaced (and was based on full program analysis in the Python of ~2010 that had no annotations at all, so always struggled with programs that could not be thoroughly inferred and particularly with those that generated runtime types - Sympy was a special case in it since about 2012 to avoid those issues, but that may have been lost when it was migrated from Visual Studio to VS Code). The latest versions of VS Code have a completely new system, that does rely very heavily on programs and libraries being annotated, in exchange for being much lighter on memory use and preprocessing. But it should stay out of your way on unannotated code if you haven't enabled its more strict modes. Cheers, Steve
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 3:31 PM Guido van Rossum <guido@python.org> wrote:
There is some discussion of the numeric tower in PEP 484 but the PEP says you should just use 'int', 'float' and be happy.
Thanks -- I didn't think to look there. And this: "when an argument is annotated as having type float, an argument of type int is acceptable" makes it pretty workable. I do wonder what happens with oddball numbers, like, say numpy.float32. But maybe that's Ok, as for though most part one will be dealing with arrays of them -- the scalars are fairly rare in real code. however, IIRC, __index__ was introduced at least partially so that numpy integer types could be used as indexes. Which would mean SupportsIndex for a type that's going to be used as an index, yes?
Yes, Decimal and Fraction are pretty special purpose, really.
well, yes, for sure.
Generics aren't always better, and in many cases they just aren't worth the effort to get them right
I suspect that depends a bit on the use case. For something internal to one system, probably not worth the effort at all. But for a general purpose library, I'm (perhaps needlessly) concerned about types getting locked down more than they need to be, just because its easier, and "it works for me"
(just like not every bit of code you write needs to be published on PyPI).
Someone should tell that to all the folks publishing 0.0.0.1 version of packages :-)
This is getting into what it means to use Types at runtime -- if it's runtime type checking, then more abstract types might be fine. But if it's using the type itself (like my use case), then absolutely -- my system will only work with real concrete type objects. Though that may not play that well with type checkers :-(
yes, it would.
We should definitely push back on zealous new converts to typing who insist that everything should be annotated.
well, we got folks wanting to change PEP 8 becuase they don't want their linter to complain -- so it will be a battle.
(Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.)
Some of the folks on my team use it -- I've been meaning to check it out -- one more reason now. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/15c3eef50a943fdefa0cb50a21d11203.jpg?s=120&d=mm&r=g)
It would be nice if someone did some work and collected a list of tutorials about type annotations that exist (especially the ones that are discoverable with a simple Bing query) and ranked them by quality.
I went with Google rather than Bing but here's what I found: https://gist.github.com/stroxler/ade7977ed07e27448222a468796bc467 I did check a few Bing queries, the results are mostly a bit less helpful, I get more discussions of the `type` function and fewer about type annotations. But the Real Python tutorial still shows up on the first page most of the time.
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Tue, Nov 30, 2021 at 1:34 PM Paul Moore <p.f.moore@gmail.com> wrote:
It's your code; do what you want. 😁 If you want to accept bytes, then you can accept bytes. I personally just don't care about the bytes edge case, so I leave it out. Plus I don't think pathlib supports bytes paths to begin with, so if you are after just the pathlib.Path case then you don't want bytes unless you're going to handle the decoding. But if you want bytes then you're after https://github.com/python/typeshed/blob/8542b3518a36313af57b13227996168e592f... which is what `open()` takes (plus ints): `str | bytes | os.PathLike[str] | os.PathLike[bytes]`.
but that's what I mean about copying the Path constructor's annotations. If Path changes, I have to change my code.
Yep. You are writing down your expectations of what is acceptable to pass in and you're choosing to be very flexible, and so you have to write down more. I don't think anyone is claiming that typing your code takes no effort. But practically speaking the constructor to pathlib.Path isn't going to change.
If we added an object to pathlib that represented the type annotation for the constructor of pathlib then it wouldn't be. It's probably not an unreasonable idea since this idiom is common enough, I just don't know if anyone has bothered to suggest it for the module.
You're essentially wrapping the constructor to `pathlib.Path` as broadly as possible, which means you want types as broad as possible while still being accurate. You could also tell users, "give me pathlib.Path objects" and your troubles go away or "an object that implements the `__fspath__` protocol" (which is what `os.PathLike` represents, hence why `str` isn't covered by it); it all depends on what sort of assumptions you want to be able to make about what you're given. But because you're wanting to accept multiple types to support both the "old" way of string paths and the "new" way of `__fspath__` you then need to put the work in to accept those types. Personally, I treat string paths vs pathlib object paths like encoding and decoding strings; get it converted immediately at the boundary of your code and then just assume pathlib everywhere else. Pathlib was added in Python 3.4 and is well-known enough at this point that I don't worry about people not being aware of it, so I let users do the glue from any string paths they get to my APIs. But to be clear, path representations != pathlib.Path type parameters != `__fspath__` protocol; there's subtlety here regardless of the types.
But how would you specify that? And what if the thing you're wrapping isn't typed itself? And what if the object you're wrapping takes multiple arguments? You could talk to the typing-sig and see if they have discussed some sort of `Parameter[pathlib.Path][0]` object to specify the type of the first argument to `pathlib.Path`.
Sure, but how would you have expected old code that isn't about to change to be typed?
Yep, hence why people can still use code you didn't type and get *some* benefit from typing still. But there are limits as flowing types through untyped code is hard, hence why your users are asking for types; t limits what *they* can check for without writing type annotations for your code (at which point they are now the ones worrying about a drifting of types, much like you are with pathlib.Path). Code like pathlib.Path that can take various types or other things that can return different types become something of a blackhole for typing because the type checkers just can't figure out what you want or it figures it out to be so broad as to be useless.
Paul
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
I know this isn't really the place for this conversation, but:
which is what `os.PathLike` represents, hence why `str` isn't covered by it);
wait, what? It seems so clear to me that "PathLike" (as a type specifier) would mean: anything that can be passed into os.fspath to give me a path. (or, of course to the stdlib functions that take paths) Isn't the entire purpose of os.fspath that you can write code like: def fun(some_kind_of_path): some_kind_of_path = os.fspath(some_kind_of_path) (or just pass it to a function you takes PathLIke) and go on your merry way -- e.g. duck typing, baby! Is there really no way to annotate that simply now? -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/3b73b776444fa777acfa37bbdcff23fe.jpg?s=120&d=mm&r=g)
On Wed, Dec 1, 2021 at 10:50 PM Christopher Barker <pythonchb@gmail.com> wrote:
Assuming you want the return value of 'fun' to be covariant with the path input, I believe you would say this: def fun(some_kind_of_path: str) -> str: ... def fun(some_kind_of_path: bytes) -> bytes: ... def fun(some_kind_of_path: os.PathLike[AnyStr]) -> AnyStr: some_kind_of_path = os.fspath(some_kind_of_path) # transform it return some_kind_of_path I would love to be shown how to do this with just a one-line declaration of 'fun', but I've given up trying to figure it out.
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Thu, Dec 2, 2021 at 9:48 AM Eric Fahlgren <ericfahlgren@gmail.com> wrote:
I just read the wikipedia page on "Covariant return type", and I either misunderstood what it means, or it's not relevant to my intended example.
Ahh I see, I wasn't intending, in my example, to return the path -- my intent was a function that would need to use the path, to, e.g. open a file. This Strikes me as a very common use case -- you want your users to be able to pass multiple different types in, as long as they can be used as a path. I have a LOT of code that does this -- for years it accepted only string paths, and now I'm updating it to take PathLike as well. My usual code is one of: def fun_that_opens_a_file(the_path): # if I'm working with an "old" library, or one that, e.g. wraps a C lib: the_path = os.fspath(the_path) # if I'm working with things I know will take a Path object: the_path = Path(the_path)
I would love to be shown how to do this with just a one-line declaration of 'fun', but I've given up trying to figure it out.
Darn. could you at least use a Union type: def fun_that_opens_a_file(the_path: Union[str, Path]): ... which isn't too bad. Personally, I think accepting bytes is the way of madness, but you could add that too. On Thu, Dec 2, 2021 at 12:32 PM Brett Cannon <brett@python.org> wrote:
in my code I just take pathlib.Path or pathlib.PurePath and I'm done as I don't want to be dealing with encoded file paths and instead with objects
Sorry, I wasn't clear. By "PathLike" (as a type specifier)" I meant using it as, well, a type specifier, not as an ABC -- often we can use an ABC as a type specifier, but not always. But you've prompted me to go re-read the PEP and I see this: """ Provide specific type hinting support There was some consideration to providing a generic typing.PathLike class which would allow for e.g. typing.PathLike[str] to specify a type hint for a path object which returned a string representation. While potentially beneficial, the usefulness was deemed too small to bother adding the type hint class. This also removed any desire to have a class in the typing module which represented the union of all acceptable path-representing types as that can be represented with typing.Union[str, bytes, os.PathLike] easily enough and the hope is users will slowly gravitate to path objects only. """ I didn't pay any attention to that at the time, as I wasn't paying attention to typing. But personally, I think I do have that desire :-) But I can see that "the hope is users will slowly gravitate to path objects only". would lead to: that represent file paths. For my part, I'm not interested in encoded paths (e.g. bytes) -- that is absolutely not the right boundary to deal with file system encoding. (that's mentioned in the PEP) But I also don't want my users (or me, with old code, scripts, etc) to have to suddenly go in and wrap a call to Path() everywhere in order to continue to use my library. String paths are ubiquitous, and I think here to stay. I really like PEP 519 -- I think it not only provides a transition, but also a future in which Path objects and string paths can continue to play well together. While Union[PathLike, str] is a pretty light lift, this is one tiny example of what we "typing skeptics" are concerned about: a transition away from full on dynamic typing to a more static style. Maybe that won't come to pass -- we'll see. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Wed, Dec 1, 2021 at 10:40 PM Christopher Barker <pythonchb@gmail.com> wrote:
That is not what the docs say: https://docs.python.org/3/library/os.html#os.PathLike. And as the creator of that ABC it's very much on purpose (see https://www.python.org/dev/peps/pep-0519/ for details).
Depends on what "your merry way" is. As the docs say, os.fspath() is about getting the file system representation. That's not something to directly manipulate in a pathlib world unless you're using os.path to manipulate that string encoding (but which PEP 519 was specifically created to avoid such encoding headaches): https://docs.python.org/3/library/os.html#os.fspath .
Is there really no way to annotate that simply now?
Once again, depends on what "simply" means to you. The examples I gave I don't find complicated, especially when we are talking about APIs that are trying to accept a broad set of types to convert into a single type. If you want to create a type variable that represents anything that is valid to `os.fspath()` or `pathlib.Path` to avoid typing out 2 or 4 types, then that can be considered for inclusion in the stdlib somewhere. But as I said previously, in my code I just take pathlib.Path or pathlib.PurePath and I'm done as I don't want to be dealing with encoded file paths and instead with objects that represent file paths.
![](https://secure.gravatar.com/avatar/392880c03b3481024eca1b762ffe3ff2.jpg?s=120&d=mm&r=g)
On 2021-11-30 09:01, Steven D'Aprano wrote:
Captain Kirk would be proud: https://memory-alpha.fandom.com/wiki/Induced_self-destruction -Mike
![](https://secure.gravatar.com/avatar/d5d40a40d75ba32b1bdff02b57201cd1.jpg?s=120&d=mm&r=g)
Am 30.11.21 um 13:39 schrieb Oscar Benjamin:
Please note that users of you library usually won't care that the library uses type hints. It's more important that there are type hints for the API, which can also be supplied using a stub file.
We have several resources that can help with questions like these: * https://github.com/python/typing/discussions is a forum for questions about typing. * The forum also has a separate section where users can ask for reviews of type annotations, for example in stub files. * https://gitter.im/python/typing is a chat for typing-related questions. We are also working on a documentation hub at https://typing.readthedocs.io/, but there is not much to see at the moment.
Please note that users of you library usually won't care that the library uses type hints. It's more important that there are type hints for the API, which can also be supplied using a stub file. This is usually quite a bit easier than typing an existing code base. Alternatively, just adding type annotations to the public API, but not using type checking yourself, can be useful for your users. - Sebastian
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Wed, 1 Dec 2021 at 12:08, Sebastian Rittau <srittau@rittau.biz> wrote:
Please note that users of you library usually won't care that the library uses type hints. It's more important that there are type hints for the API, which can also be supplied using a stub file.
I tried that route, but I was informed that if I do that, mypy will not check my stubs against the source code. Which means that there's no easy way of testing that the stubs are correct - is that accurate? Paul PS I appreciate the links you posted to various typing forums, but IMO the most critical missing resource is a central set of typing documentation that includes examples, FAQs and best practices as well as reference materials. Typically when I hit an issue with types, I want to research my own solution, not ask a question, which requires me to bother other people, as well as interrupting my flow while I wait for a response. I'd much rather see work being done on documenting what we currently have in typing, rather than yet more changes which while no doubt useful, is effectively just churn that makes maintaining a codebase that has to support multiple Python versions harder. PPS Sorry if this sounds negative. TBH, I'd quite happily not use typing if I didn't want to and stay quiet. A lot of the frustration I see being expressed here (including my own) seems to come from the fact that it's so difficult to actually take that sort of "I can ignore it if I don't use it" attitude, whether that's because of community pressure, tool requirements, or whatever.
![](https://secure.gravatar.com/avatar/d5d40a40d75ba32b1bdff02b57201cd1.jpg?s=120&d=mm&r=g)
Am 01.12.21 um 13:36 schrieb Paul Moore:
mypy includes a "stubtest" tool that compares a stub to introspected runtime data. We use it in typeshed as part of the CI process and it's a very powerful tool to help keeping stubs up-to-date and correct.
Completely agree. This is what typing.readthedocs.io is supposed to become, but time constraints mean that it's very incomplete at the moment.
I completely understand this pressure, especially for library authors. Providing high quality stubs and the best user experience is not easy. But I believe that referring people to typeshed can help. While we of course prefer high quality stubs or type annotations shipped with the package in question, typeshed can provide a fairly low barrier of entry for projects that don't have the resources to maintain type annotations themselves. It can also be used as an "incubator", where stubs are created and improved iteratively, until they are deemed ready for inclusion in an upstream package. - Sebastian
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
for library authors.
Providing high quality stubs and the best user experience is not easy. But I believe that referring people to typeshed can help.
This is actually very helpful. It provides an answer for open source projects for which do users want typing. One can say to the users that want type stubs for the library that they are encouraged to contribute those stubs to type shed themselves and once they have been tested and vetted they can be included in the project. it’s not unlike any other feature request. The developer of an open source project is under no obligation to provide any feature asked for, but a well-managed project will encourage useful contributions from users. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/7b5bbadd9baf9c6b33a053e9687ce97e.jpg?s=120&d=mm&r=g)
@Paul
... missing resource is a central set of typing documentation that includes examples, FAQs and best practices as well as reference materials
Like Sebastian, I agree, and this is something we're making progress on.
... easy way of testing that the stubs are correct
mypy ships with a tool called stubtest. It's what typeshed uses to validate itself against CPython. I recently wrote up documentation for it, see here: https://mypy.readthedocs.io/en/latest/stubtest.html
Thanks for not staying quiet and helping us make typing better. In particular, I think one takeaway from this thread is that a lot of our documentation is focussed on "how to get started with typing my code", and not much addressed at library authors with limited time to deal with typing questions (like "what is a PEP 561" and "if my type hints are incomplete does that cause problems for my users") On Thu, 2 Dec 2021 at 15:34, Rob Cliffe via Python-Dev < python-dev@python.org> wrote:
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Fri, 3 Dec 2021 at 00:10, Shantanu Jain <hauntsaninja@gmail.com> wrote:
That's great to know - it's easy for me to say "we need more docs" but I know it's hard to produce them. I'm glad it's on the radar :-)
... easy way of testing that the stubs are correct
mypy ships with a tool called stubtest. It's what typeshed uses to validate itself against CPython. I recently wrote up documentation for it, see here: https://mypy.readthedocs.io/en/latest/stubtest.html
Thanks, I wasn't aware of that. Although TBH I like the suggestion of using typeshed as a kind of staging point for interested users to develop annotations, which can then be moved back into the library once they've been proven stable. I'm still surprised that mypy doesn't check stub files though. If I write stubs for my external API, and then later decide I want to add types to my internal functions, to do static checking on my library, am I therefore required to move the annotations out of the stub file into the main code? What if I want to avoid the runtime overhead of importing the typing module? Or I want to keep the source code clear, by not making function definitions into multi-line monsters because of type declarations?
A lot of the frustration I see being expressed here (including my own) seems to come from the fact that it's so difficult to actually take that sort of "I can ignore it if I don't use it"
Thanks for not staying quiet and helping us make typing better. In particular, I think one takeaway from this thread is that a lot of our documentation is focussed on "how to get started with typing my code", and not much addressed at library authors with limited time to deal with typing questions (like "what is a PEP 561" and "if my type hints are incomplete does that cause problems for my users")
That's a good point that I hadn't spotted - thanks for picking up on it! Paul
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Thu, Dec 2, 2021 at 3:35 PM Rob Cliffe via Python-Dev < python-dev@python.org> wrote:
I assume you accidentally pressed Send prematurely.
Actually, it was my phone making the text white -- what the heck? why is it so hard to send plain text email? resent now. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Wed, 1 Dec 2021 at 12:12, Sebastian Rittau <srittau@rittau.biz> wrote:
The distinction between internal code and API is not always that clear in SymPy but in any case it is still very hard to type many of SymPy's public API functions. You are right that there is a distinction here because when I imagine that type hints could be useful for SymPy I think of adding hints in places that users definitely don't care about. Adding hints for public API is more immediately useful for users but also much harder because many public APIs are quite liberal/flexible in what they accept and return.
I'm not sure what your point is here. Are you suggesting that I use these forums or that I redirect other people there?
My previous statement still stands: it's a huge amount of work and I do not want to add bogus hints as a stopgap. If the hints are to be added then someone needs to do that hard work to make sure that they are accurate. -- Oscar
![](https://secure.gravatar.com/avatar/cdc87637918eccd37ca88e9079e73705.jpg?s=120&d=mm&r=g)
On Mon, Nov 29, 2021 at 8:17 PM Guido van Rossum <guido@python.org> wrote:
I'm certainly not alone, among people on this list, in regard to the following. But I teach a lot of people who are coders, but not necessarily senior (in Python, or otherwise). I also mentor/lead such junior programmers. I find it quite common when people who haven't known Python for 20 years see type annotations, they have trouble getting the optional and gradual nature of them. Explaining that is certainly not impossible, nor even all that difficult. However, it DOES need to be explained. Especially when less experienced Pythonistas have worked with statically typed languages, they often see code with type annotations and overgeneralize to their requirement. I don't know exactly what that means about *where* this needs to be explained. Maybe just by trainers and authors. But maybe things in official Python documentation as well, which seem more definitive. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
![](https://secure.gravatar.com/avatar/dbdddb64dc47a7853e836edfed6b1f3f.jpg?s=120&d=mm&r=g)
On 29 Nov 2021, at 23:56, Barry Warsaw wrote:
But isn't that the reason why we have `typing.Annotated`, so that annotations used for typing and annotations used for other purposes can coexist? An annotation used for typing only is: ```
An annotation used for something else is:
`typing.Annotated` gives us both:
Granted, for someone who only wants to use annotations for their own
purpose, digging out the original not typing related stuff from
`typing.Annotated` is more work, but doable.
Or am I missing something here?
> [...]
Servus,
Walter
![](https://secure.gravatar.com/avatar/72ee673975357d43d79069ac1cd6abda.jpg?s=120&d=mm&r=g)
On 26/11/21 4:15 am, Stephen J. Turnbull wrote:
I'm not sure that's true. The way I remember it, back when annotations were first introduced, the BDFL didn't say that, or at least didn't say it very clearly. It sounded more like type hints were just one of many possible uses, and he encouraged people to experiment. There were even discussions about coming up with a convention to manage conflicting uses of annotations in the same code. That wouldn't have happened if typing were considered the only supported use. -- Greg
![](https://secure.gravatar.com/avatar/047f2332cde3730f1ed661eebb0c5686.jpg?s=120&d=mm&r=g)
On Thu, Nov 25, 2021 at 3:49 PM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
My memory is also hazy, but I'm quite sure that *in my mind* annotations were intended as a compromise between conflicting proposals for *typing*. We didn't have agreement on the syntax or semantics, but we did know we wanted to do something with types eventually. Some folks wanted to enforce types at runtime. Others wanted to use them to generate faster code. Yet others wanted types to be checked by the compiler. The term "gradual typing" wasn't invented (or hadn't reached our community) yet, and offline static type checking wasn't something we had thought of either (I think). But it was clear that typing would have to be optional. Fortunately the current situation is that there are few people who want to use annotations for non-typing, but there are still conflicting use cases, in particular offline static type checking on the one hand (which started with mypy and PEP 484) and some runtime use of types on the other hand ( pydantic <https://pydantic-docs.helpmanual.io/> being the main example IIUC). Tools like pydantic seem to have adopted the *notation* of PEP 484 fully, but their *interpretation* is different. This was explicitly provided for by PEP 484 (the design there carefully ensures that annotations are fully introspectable) but it wasn't anticipated to be a big use case, hence the mistaken belief at the time PEP 563 was accepted that stringifying annotations would be sufficient for the introspection use case. So I'm in agreement with Stephen, but since he left out any mention of offline static checking, his observation isn't helping much. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
![](https://secure.gravatar.com/avatar/d995b462a98fea412efa79d17ba3787a.jpg?s=120&d=mm&r=g)
On Fri, 26 Nov 2021 at 05:14, Guido van Rossum <guido@python.org> wrote:
My memory is also hazy, but I'm quite sure that *in my mind* annotations were intended as a compromise between conflicting proposals for *typing*. We didn't have agreement on the syntax or semantics, but we did know we wanted to do something with types eventually.
More hazy memories here, but I think the original proposal left open the possibility of annotations not being types at all - for example, being docstrings for the arguments, or option names for a "function call to CLI" tool, etc. I think that some libraries took this approach (in particular the "CLI builder" idea). At some point (probably around PEP 484) it became clear that the expectation was that annotations would be *types*, and in particular would be the expected/intended type of the annotated value. I think there was some discussion at that point about "what do we do about the CLI builders" but the conclusion was that they had relatively low adoption rates, and could adapt (or more accurately, abandon the annotation approach). That was the point, in my recollection, where the "annotations are for types" principle was established. However, I feel that many people got (or were given) the impression that static type checkers were the core use case (probably as a result of the emphasis on the message that "type annotations will never be mandatory"), for better or worse. I think that by now, it's long been understood by most people that annotations are types. This may actually be one reason why people are so uncomfortable with the idea of stringified annotations, because it violates that assumption - personally, I have the same discomfort about using explicit string annotations for forward references, it feels like I'm not declaring a "proper type". If what I say above is right, the debate here isn't about whether annotations "are for types", but rather about whether reading the types in annotations and using them to affect behaviour *at runtime* is a legitimate use of annotations. That is the use case that stringifying annotations makes more difficult, and which doesn't seem to have a strong enough voice in the direction of typing proposals. I lurk on the typing-sig, and from an outsider's perspective, the participants seem to be almost entirely designers or heavy users of static type checkers. That gives a certain emphasis to the proposals coming from that group. I'd therefore interpret Barry's plea as being for *anyone* with a use for annotations to provide their feedback (at least, anyone who accepts that annotations are types), with particular emphasis on people who want to use the types declared in annotations to affect runtime behaviour, as that's the most under-represented group at the moment (and it's not clear whether it's under-represented because there aren't many such uses, or because the users aren't being heard from). Paul
![](https://secure.gravatar.com/avatar/d91ce240d2445584e295b5406d12df70.jpg?s=120&d=mm&r=g)
Paul Moore wrote:
Absolutely. While it was clear that Guido's own use cases were about typing, annotations were explicitly not limited to typing, which is one reason why some of the later changes have felt to some people like bait and switch. Maybe it is already too late to avoid that.
... the expectation was that annotations would be *types*,
Even from the start, it was assumed that they would be objects. (Specifically types was expected to be common, but not universal.) The particular way strings are being substituted for evaluated objects has sometimes reminded me of raising a string instead of an exception class/object. It will work, but it can seem sloppy, and it can be annoying if you were assuming otherwise and suddenly have to add a bunch of evals. (That said, I haven't yet been sufficiently motivated to even tease out exactly what the problems are, let alone to propose an alternative that also satisfies the typing fans -- in part because it feels like the obvious optimization is to just not run typing, and it isn't clear what middle grounds are generally worthwhile.)
I see that as a second dispute, which I had previously missed. I think you're right, though. On the other hand, I'm not sure the solution to both isn't just a helper function that does the 2nd-pass resolution -- preferably without requiring that all the rest of typing be imported, since even the people who want to use the typing package agree that importing it is not lightweight.
At times, it sort of reminds me of OWL and "Semantic Web". There are plenty of people who will want to use annotations as a tool, but won't be willing to wade through what can feel like "How many angels can dance on the head of a pin?" discussions. That said, I'm not sure how to best reach people who just want a rough-and-ready usually-good-enough tool. -jJ
![](https://secure.gravatar.com/avatar/2828041405aa313004b6549acf918228.jpg?s=120&d=mm&r=g)
Here's a use case for runtime type annotations: dataclasses use annotations to define fields. With the exception of ClassVar, the actual type is ignored. There's code in dataclasses.py to deal with stringized annotations, specifically just looking for ClassVar. I'd like to see this special case go away, assuming 649 is accepted. And while dataclasses itself doesn't care about the types, apparently users do, and want to be able to call get_type_hints on the annotations. Many people actually want dataclasses to call this for them, but I've resisted. See https://bugs.python.org/issue36643 There are plenty of bugs around this issue, some of which have been fixed. But there are also at least a couple that haven't been fixed, and I'm not sure they will be. I'm hoping that 649 gets accepted and I don't have to worry about the problem at all. Here's an example of an open bug around get_type_hints with dataclasses: https://bugs.python.org/issue45524 Eric On 11/29/2021 6:00 PM, Barry Warsaw wrote:
![](https://secure.gravatar.com/avatar/034bb4232e7e14e4cbf258aad8ca2a57.jpg?s=120&d=mm&r=g)
So maybe this is my time to chime in. I have used annotations for runtime behavior. My primary use case is an injection library that I wrote. It allows something along the lines of: class IMyService(IService): pass @inject def call_something(arg1: str, arg2: int, svc: IMyService = None): assert svc is not None # do stuff... At runtime the `inject()` decorator will scan the function signature and insert an instance of `IMyService` based on a registry lookup. (We use zope.component utilities, but that's an implementation detail.) When testing, one can simply pass a mock version of the service. Using the mypy-zope plugin, the above also passes all mypy type checking. It is a simple pattern in terms of annotation but a very effective pattern that mimics other injection systems in Java and TypeScript (Angular). We have used it for a few years now and like it a lot. Regards, Stephan On Monday, November 29, 2021 6:00:04 PM EST Barry Warsaw wrote:
-- Stephan Richter Entrepreneur & Geek
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Thu, Dec 2, 2021 at 6:57 PM Stephan Richter <stephan.richter@gmail.com> wrote:
So maybe this is my time to chime in. I have used annotations for runtime behavior. My primary use case is an injection library that I wrote.
Does it work with __future__ annotations? -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/489cdabedd1112f98474038939668778.jpg?s=120&d=mm&r=g)
On Fri, 26 Nov 2021 at 09:14, Paul Moore <p.f.moore@gmail.com> wrote:
I don't know if this helps, but I'm a maintainer/creator of a Python library that uses type annotations at runtime. The library is used to create GraphQL APIs[1]. We took a lot of inspiration from dataclasses, our API looks like this: ```python @strawberry.type class User: name: str ``` The `strawberry.type` decorator reads the annotations and creates an object with a list of fields and types, but it doesn't try to resolve them (that happens later to prevent issues with circular dependencies and so on). I remember the discussion around PEP 563 and how it would break libraries using type annotations, in our case things are fine since the types we define should be defined in the global namespace[2] anyway. I think the only issue we have with type annotations is when you have circular references inside your python code (where you'd use `if TYPE_CHECKING: import X`), we "solved" that with a custom class called `LazyType[type, module]`, that allows to specify a type and where it comes from, so our library can resolve the proper type, even if it was not imported in the namespace of the class using that type. Sorry for the digression into some of the details of how the library works, I hope it helps a bit understanding the use case. I think there's definitely use cases for type hints being used at runtime. I personally love the fact that we are allowed to do this, in other languages (like TypeScript) you don't have access to the types at runtime and it does prevent the creation of libraries like dataclasses or mine, which is a bit of a shame (you'd need to use code generation to get a similar experience). There's also other libraries that use type hints at runtime (not for runtime type validation[3]): - Typer - http://typer.tiangolo.com - SQLModel - http://sqlmodel.tiangolo.com - Beanie - https://github.com/roman-right/beanie/ - Pydantic - http://pydantic-docs.helpmanual.io - odmantic - https://github.com/art049/odmantic/ - singledispatch - https://docs.python.org/3/library/functools.html#functools.singledispatch [1] If you're not familiar with GraphQL there's some info here: https://graphql.org [2] I did ask if there was a way to resolve types when having them encapsulated into functions, but that was a specific use case with tests and I was/am fine with using global there https://mail.python.org/archives/list/typing-sig@python.org/thread/SNKJB2U5S... [3] while I see runtime validation being something useful, I think it is worth taking a look at different use cases for type hints :) -- Patrick Arminio
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
This is great Patrick, thanks. My use case is similar to Patrick's, except it builds on dataclasses directly: It's part of a larger system, but I've just pulled it out into its own poorly documented and poorly tested package: https://github.com/PythonCHB/flexi (I think it's generally useful, so may one day make a proper package out of it) Anyway, the principle is this: The goal is to have a flexible data model that enforces the structure, but not necessarily the validity of values, or even types. This is useful because we need to store / edit / import complex hierarchy of data, and we very much don't want to have all of it to have to be valid in order to even store it. An example is a dataset we recently imported (from spreadsheets--arrgghh), where in a thousand records, they had put "too viscous to measure" instead of a value for interfacial tension. That, of course, would be helpful to a person reading the spreadsheet, but would have been a pain if we had enforced that that field had to be a float > 0. Instead, it imported fine, and then our validation code flagged the issue to be fixed later. And all the rest of the data could be used in the meantime. Anyway, that was a digression (but explains why we don't use Pydantic). Functionally, we need the "nodes" in the data model to be able to convert themselves to/from JSON compatible Python, and validate themselves, etc. This is done by using the actual type object, as stored in dataclasses.Field.type This has worked nicely for us, and dataclasses saved us a lot of boilerplate code. if we do from future import Annotations then the system breaks, as we have a string instead of the actual type that is needed. Solutions: I *think* PEP 649 would work fine for us, as the annotation would get evaluated when it was added to the field object. But inspect.get_annotations isn't really helpful for this use, as the Field.type attributes aren't annotations anymore. So we would have to eval() the strings ourselves, and I think it would be unclear what namespaces to use for that. If dataclasses were to use inspect.get_annotations(cls, eval_str=rue) in order to extract the types to put into the Field objects, then I think all would be good for us. And as I understand it, dataclasses itself doesn't do anything with that attribute anyway. If that's not decided to be a good thing for dataclasses, then I think we'd have to re-implement a lot ourselves, though it could probably be hacked in - I haven't tried that yet. BTW, I don't think we've heard from the attrs (and especially cattrs) folks in this thread. A search of the repo issues indicates that there has been some discussion of PEP 563's impact, but it's not totally clear to me if it's been resolved. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/de311342220232e618cb27c9936ab9bf.jpg?s=120&d=mm&r=g)
On 11/26/21 1:13 AM, Paul Moore wrote:
I have one such tool. Fortunately for me, I strongly dislike having the annotations inside the function header as it quickly gets difficult for me to read, so I was already using decorator syntax for the annotations. When PEP 484 was accepted I just changed where the annotations were being stored from __annotations__ to my own dunder. -- ~Ethan~
![](https://secure.gravatar.com/avatar/664d320baa05c827ff08ed361fe77769.jpg?s=120&d=mm&r=g)
On Thu, 25 Nov 2021 at 15:16, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
This was not my understanding of annotations when they were introduced e.g.: https://www.python.org/dev/peps/pep-3107/#use-cases As I remember it, a decision about the purpose of annotations was *explicitly* not made when they were introduced. It was clear that typing was a major potential use and then at some point (around about the introduction of the typing module) there seemed to be a shift in people's understanding of what annotations were for. Eventually that reached the point that people who were particularly interested in typing had no memory of the fact that the purpose of annotations had not really been specified as being about typing in the first place. It looks to me like Chris has identified in PEP 563 what is potentially the earliest reference (in an accepted PEP) to the idea that non-typing uses of annotations are to be discouraged. -- Oscar
![](https://secure.gravatar.com/avatar/880ac02012dcaf99f0392f69af4f8597.jpg?s=120&d=mm&r=g)
My 2¢ from a position of *extreme ignorance*, not to mention zero interest in annotations (all welcome to shoot me down in flames, or treat me with contemptuous silence. But *occasionall*y the "idiot's" point of view is worth considering, so here goes). ISTM that typing/annotations have been allowed to drift without a clear end in view, rather like a ship that is steered in one direction by one person, then in another by someone else, with nobody knowing what the destination port is. As witness the conflicting views in this thread. At the risk of being rude, the phrase "headless chickens" comes to mind. ISTM that it is time to take a step back and decide on a definite policy. Perhaps a definitive pronouncement from Guido. Or the SC. Or a discussion from all parties that reaches an acceptable conclusion. Then stick to it. Even if it causes some backward incompatibility - ISTM that things have changed so rapidly that this really would be an acceptable evil if it results in clarity for the future. Meanwhile, do not change anything, do not approve any PEPs, until this conclusion is reached. I apologise if I am being presumptuous in commenting on a subject I know almost nothing about. /I'm just a soul whose intentions are good,// //Oh Lord, please don't let me be misunderstood.// / Best wishes Rob Cliffe On 25/11/2021 23:51, Oscar Benjamin wrote:
![](https://secure.gravatar.com/avatar/d5d40a40d75ba32b1bdff02b57201cd1.jpg?s=120&d=mm&r=g)
Am 26.11.21 um 01:48 schrieb Rob Cliffe via Python-Dev:
I believe the problem is less within the typing community itself (whether static or runtime), but a fundamental conflict between the typing community and what Stephen called the "typing-suspicious crowd". typing in Python has always been hampered by sometimes valid, often unfounded fears that typing is "taking over" and somehow becomes a requirement. For example, PEP 484 made no changes to Python or the stdlib, except the introduction of a new module, often to the detriment of readability and usability of type annotations. And while progress is made slowly (for example with generics in standard collections or the new callable syntax proposal), I'm sure the typing community would love to progress faster and with less obstacles. - Sebastian
![](https://secure.gravatar.com/avatar/1fee087d7a1ca17c8ad348271819a8d5.jpg?s=120&d=mm&r=g)
On Thu, 25 Nov 2021 23:51:58 +0000 Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
This is also what I remember from the discussions at the time of PEP 3107. Annotations were purposefully use case-agnostic, and there was no stated desire to push for one use case or another. I don't think gradual typing was even on the radar, not in public comments anyway. Regards Antoine.
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 12:15:10AM +0900, Stephen J. Turnbull wrote:
I don't think that's what PEP 563 says. Annotations are not *restricted* to only be strings, it is merely that when the function or class object is built, the annotations are left as strings. So we can post-process annotations and destringify them:
There may be scoping issues to be sorted out, but I don't think they are insurmountable. -- Steven
![](https://secure.gravatar.com/avatar/8da339f04438d3fcc438e898cfe73c47.jpg?s=120&d=mm&r=g)
Steven D'Aprano writes:
You're right, I was imprecise. I meant as PEP 563 is implemented now, I doubt it can handle non-strings nicely.
So we can post-process annotations and destringify them:
True, but you face the dread "double-decode problem": if the annotation happens to destringify to a string, then the "post- processor" may try to destringify again. I guess we can mark each annotation as already destringified or not. As long as this is done by some sort of API, you're probably OK (but it makes me nervous since __annotations__ is completely exposed to the program, as in your example).
There may be scoping issues to be sorted out, but I don't think they are insurmountable.
But isn't there still the issue of forward reference, which motivates these PEPs in the first place?
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
There may > be scoping issues to be sorted out,
Yes, the scoping issues are the main problem.
but I don't think they are insurmountable.
Probably not — and LLukas Langa has some good ideas that the SC is considering. (Sorry I can’t get your name right in a phone) Maybe inspect.get_annotations() does, or will be able to, solve many of these issues. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Sat, Nov 20, 2021 at 11:46:56PM -0800, Christopher Barker wrote:
I don't think that's an insurmountable problem. I think that all you need is a small class decorator to evaluate the stringified annotations back to the real things. The PEP tells us the right way to evaluate annotations, so all(?) you need is a decorator to do that to each method in your class, and Bob's your uncle. Maybe PEP 563 could include a decorator in the typing module to destringify all the annotations in a class or function? As far as I can see from a brief scan of the PEP, and based on knowing next to nothing about your use-case, the only hypothetical problem might be this line in the PEP: "Consequently, using local state in annotations is no longer possible in general." but it's not entirely clear to me what Łukasz means by that, or whether it will affect your use-case. -- Steve
![](https://secure.gravatar.com/avatar/d91ce240d2445584e295b5406d12df70.jpg?s=120&d=mm&r=g)
Steven D'Aprano wrote:
On Sat, Nov 20, 2021 at 11:46:56PM -0800, Christopher Barker wrote:
Maybe PEP 563 could include a decorator in the typing module to destringify all the annotations in a class or function?
If it were in an annotations module, that would probably be sufficient. If it is in typing, then it is a very heavyweight dependency -- heavy enough that even the people actually using that module for development (and not for production runs) are worried about the costs. If the costs of the typing module are that high, it is not acceptable to impose them on people not otherwise using the module. -jJ
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
On Fri, Nov 26, 2021 at 5:47 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:
As of Py 3.10 there is: inspect.get_annotations one could write a decorator around that, but I think the real question is when do you want the annotations to be "finalized"? -CHB heavy enough that even the people actually using that module for
![](https://secure.gravatar.com/avatar/351a10f392414345ed67a05e986dc4dd.jpg?s=120&d=mm&r=g)
On Thu, Nov 18, 2021 at 8:00 AM Barry Warsaw <barry@python.org> wrote:
In my opinion, we can agree that we can not make PEP 563 default in the future because it will break too many use cases. Anyone against making a statement that "PEP 563 will never be the default behavior"? Then, we do not need to decide "PEP 563 or 649". We can focus on whether we can replace "stock semantics + opt-in PEP 563" with PEP 649 or not. Regards, -- Inada Naoki <songofacandy@gmail.com>
![](https://secure.gravatar.com/avatar/53c166c5e1f0eef9ff4eb4d0b6ec9371.jpg?s=120&d=mm&r=g)
On 11/29/21 7:10 PM, Inada Naoki wrote:
Anyone against making a statement that "PEP 563 will never be the default behavior"?
I think only the SC is empowered to make such a statement.
I doubt the current status quo--keeping PEP 563 as optional behavior, permanently--is viable long-term. It causes problems for most runtime uses of annotations, which we were collectively basically ignoring until now. As runtime use of annotations /and/ typing both become more pervasive, these problems are only going to grow. //arry/
![](https://secure.gravatar.com/avatar/176220408ba450411279916772c36066.jpg?s=120&d=mm&r=g)
I don't think the issue of how typing is making its way intot he community is particular relevant to this thread, but I was directly asked a question, so I'll answer it: I don't know exactly where the impression is coming from that typing is a best practice, but it's certainly creeping into example code, etc. So folks will see examples of code that have nothing to do with typing, and probably have no need for it, and there will be type annotations. Many (most?) folks learn from examples more than targeted instruction, so as those examples grow, so will the impression that it's an inherent part of Python. David Mertz' experience is similar to mine as an instructor. Another anecdotal example: I've been teaching an intro to Python class for years, based on the same material, that we try to keep up to date. But typing is not yet included. However, about a year ago, one of the other instructors (that had not previously developed materials for the class) added a new example / exercise to the "intro to classes" lesson. And that example had type annotations. I ended up removing them, as I thought it could be pretty confusing to students to see them for the first time with no explanation (would they think it was something specific to classes?). I wish I'd discussed it with the other instructor, I don't know why that was done -- was it standard practice in their workplace? Seemed like a best practice? I have no idea. Another observation is that more and more library authors are being asked to add type annotations -- maybe that IS a best practice for widely used libraries, but that's one more piece of data. Also: I know this isn't intentional, but Guido has an enormous influence on the Python community -- so I suspect that the fact that he's taken an active role in developing static typing has influenced how it's being perceived. I do think this is a topic for the community to grapple with, but not really a Python-dev responsibility. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
participants (29)
-
Antoine Pitrou
-
Barry Warsaw
-
Brett Cannon
-
Chris Angelico
-
Christopher Barker
-
David Mertz, Ph.D.
-
Eric Fahlgren
-
Eric V. Smith
-
Ethan Furman
-
Greg Ewing
-
Guido van Rossum
-
Inada Naoki
-
Jim J. Jewett
-
Larry Hastings
-
Mike Miller
-
Oscar Benjamin
-
Patrick Arminio
-
Paul Moore
-
Rob Cliffe
-
Sebastian Rittau
-
Shantanu Jain
-
Stephan Richter
-
Stephen J. Turnbull
-
Steve Dower
-
Steve Holden
-
Steven D'Aprano
-
Steven Troxler
-
Terry Reedy
-
Walter Dörwald