I should point out that "accept PEP 649" and "break pydantic" are not the only options here. The thing that will break pydantic is the end of PEP 563's deprecation period, not a failure to implement PEP 649. Other viable options include: - Leave PEP 563 opt-in until we can agree on a solution to the problem. - Leave PEP 563 opt-in forever. - Deprecate PEP 563 and go back to status quo ante. I haven't followed this closely enough — if PEP 649 were accepted today, would it even be ready for use before the 3.10 code freeze (which is in a few weeks)? Assuming this is a real problem (and based in part on how long it took for attrs to get what support it has for PEP 563 https://github.com/python-attrs/attrs/issues/288, I wouldn't be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649. Best, Paul On 4/15/21 4:20 PM, Bluenix wrote:

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:

```py import discord from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command() # discord.py reads the typehints and converts the arguments accordingly async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed await ctx.send(f'{member.mention}! {text}')

bot.run('token') ```

I must say, this is extremely ergonomic design! Don't break it :) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4Q... Code of Conduct: http://python.org/psf/codeofconduct/

Hello, I'm a bit surprised that this topic is brought up just days before the feature freeze of Python 3.10. How did tools work until now with from future import annotations? This feature has been in the language for years (since 3.7), and I haven't heard until today anyone complaining about this feature gap. Did everyone just told users not to use lazy evaluation of type hints until now? Also there's been advertised for at least a year that next python will make it default 🤔 Thanks, Bernat On Thu, Apr 15, 2021, 23:13 Paul Bryan wrote:

I've been working on a similar framework, Fondat [1], that shares similar objectives as Pydantic and FastAPI, albeit implementing in a different manner.

I believe other frameworks will have to "run a gauntlet" (as I already have) to deal with the affects of PEP 563. If anyone on these teams are interested in how I've dealt with such issues, please feel free to contact me directly.

I also recommend members of these teams monitor and/or participate in the typing-sig (and maybe the python-dev) mailing list, as discussions in these lists are likely to affect their work in the future.

Paul

[1] https://github.com/fondat/fondat-core

On Thu, 2021-04-15 at 17:49 -0400, Paul Ganssle wrote:

I should point out that "accept PEP 649" and "break pydantic" are not the only options here. The thing that will break pydantic is the end of PEP 563's deprecation period, not a failure to implement PEP 649.

Other viable options include:

- Leave PEP 563 opt-in until we can agree on a solution to the problem. - Leave PEP 563 opt-in forever. - Deprecate PEP 563 and go back to status quo ante.

I haven't followed this closely enough — if PEP 649 were accepted today, would it even be ready for use before the 3.10 code freeze (which is in a few weeks)?

Assuming this is a real problem (and based in part on how long it took for attrs to get what support it has for PEP 563 https://github.com/python-attrs/attrs/issues/288, I wouldn't be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649.

Best, Paul On 4/15/21 4:20 PM, Bluenix wrote:

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:

```py

import discord

from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command()

# discord.py reads the typehints and converts the arguments accordingly

async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed

await ctx.send(f'{member.mention}! {text}')

bot.run('token')

```

I must say, this is extremely ergonomic design! Don't break it :)

_______________________________________________

Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-leave@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4Q...

Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PBGFXPC7... Code of Conduct: http://python.org/psf/codeofconduct/

It seems evident that the problems with PEP 563 have been well-known at least to pydantic for several years, as you can see in the issue Samuel Colvin linked: https://github.com/samuelcolvin/pydantic/issues/2678 That said, while I do think that "please contact upstream when you see a problem developing, not just before a major deadline" is a good lesson to take away from this, it is probably not worth litigating the question of the particular manner of the notification. As much as I think it would have been good for this discussion to happen 6 months, 1 year, 2 years or 3 years ago, "before the code freeze" is a heck of a lot better than "after the release" (which is often when the notification comes in, and why we have been encouraging people to test against alphas and betas). Hopefully we can let it rest by saying that the timing of learning about this apparently urgent situation could have been much better, but it could have been worse as well. Best, Paul On 4/15/21 7:12 PM, Christopher Barker wrote:

...

I'm a bit surprised that this topic is brought up just days before

On Thu, Apr 15, 2021 at 3:29 PM Bernat Gabor mailto:gaborjbernat@gmail.com> wrote: the feature freeze of Python 3.10.

I have not idea about the technical details, but I think there is a bit of a discomment in the community:

Annotations have been around for years.

More recently, there was an official specification of how to use them for type hinting (was that PEP 484? a bit confused as that appears to still be provisional -- but anyway)

Since the specification for type hinting, a lot of work has been done on/with static type checkers (e.g. MyPy).

That work has informed the changes / improvements / proposals to the annotation and type system in Python.

However, those of us that are not actively involved (and maybe not that interested) in static typing have been not paying a great deal of attention to those efforts.

But some of us may, in fact, be using annotations for other reasons, but since we haven't been involved in the discussion, some issues may have been missed.

-CHB

-- Christopher Barker, PhD (Chris)

Python Language Consulting   - Teaching   - Scientific Software Development   - Desktop GUI and Web Development   - wxPython, numpy, scipy, Cython

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5TGKKRZX... Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, Apr 16, 2021 at 6:03 AM Bluenix wrote:

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:

```py import discord from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command() # discord.py reads the typehints and converts the arguments accordingly async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed await ctx.send(f'{member.mention}! {text}')

bot.run('token') ```

I must say, this is extremely ergonomic design! Don't break it :)

Are you sure about PEP 563 break it and discord.py can not fix it? As far as my understanding, PEP 563 don't hurt this use case so much: * This use case evaluates runtime type information only once. So eval() overhead is not a problem. * Currently, annotation is very very complex and varies. For example, List[int] will be `List[int]`, `List['int']`, `'List[int]'`, `'List["int"]'`, `List[ForwardRef('int')]` etc... After PEP 563, only `'List[int]'` is practical so we can stop supporting `List["int"]` and others at some point. So playing with runtime type will become easier in the future. Am I wrong? -- Inada Naoki

On Fri, Apr 16, 2021 at 8:27 AM Inada Naoki wrote:

After PEP 563, only `'List[int]'` is practical so we can stop supporting `List["int"]` and others at some point.

There's a lot of code written before PEP 563 was available (and code that needs to support earlier Python versions) that uses string literal quoting in various places. If Python stops supporting List["X"], it will be a significant backward compatibility break. Some of these string literal escapes are made unnecessary by PEP 563, but there are still cases where string literals are needed that neither PEP 563 nor PEP 649 address. Examples include type aliases, type variable definitions and base classes, but there are others [1]. These aren't evaluated in a type annotation context, so manual string literal escaping would still be needed.

So playing with runtime type will become easier in the future.

Am I wrong?

In some cases things will become easier, but other common use cases still seem to be unresolved. I haven't seen any proposal that can completely replace all uses of string literal escapes. I guess one option would be to only allow string literal escaping to be used outside type annotations in the future. I think that this could be feasible with a suitable deprecation period. Also, the use of "if TYPE_CHECKING" breaks the runtime use of annotations. PEP 563 and PEP 649 reduce the need for string literal escaping in this use case, but it mostly helps static type checkers. This is pretty common in codebases that use static type checking. My main issue with PEP 649 is that it only addresses a subset of remaining issues, i.e. it doesn't go far enough. It's also not clear to me if PEP 649 (or PEP 563) can be extended to cover the remaining runtime issues, or if we'd need a *third* approach to solve them. Jukka [1] https://www.python.org/dev/peps/pep-0563/#forward-references

On Thu, Apr 15, 2021 at 4:20 PM Brett Cannon wrote:

Making this an "us versus them" discussion just makes the whole situation feel confrontational when instead everyone is trying to figure out the best thing for everybody when there's no perfect answer.

I agree that that was strong language, but from the perspective of folks not interested in Static Typing, there has been a lot of activity, or at least discussion, that makes it seem like this optional feature is driving the evolution of the language. So folks are going to get concerned, and maybe upset, if it looks like changes might be made that may break other features in order to make static typing work better. And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Fri, 16 Apr 2021 at 01:13, Guido van Rossum wrote:

On Thu, Apr 15, 2021 at 16:48 Christopher Barker wrote:

...

And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them.

It seems a little disingenuous to claim discussions about annotations don’t concern you when you’re actively using them (for typing, no less, in the case of pydantic). And I am sure a project as popular (by their own description) as pydantic will find a way forward if PEP 649 is rejected, despite overdramatized claims.

It looks to me like pydantic were well aware that this affected them and have been working over time to try and support this. They probably also had their eye on PEP 649 or hoped that something else would come along to fix this but they can now see the crunch point approaching. As Chris says though other libraries/developers have been using annotations for some time and might not have realised that typing considerations could undermine their other uses for annotations since annotations predate typing and typing is an optional feature.

That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.

Would it be problematic to postpone making __future__.annotations the default? Oscar

On Thu, Apr 15, 2021 at 18:20 Barry Warsaw wrote:

On Apr 15, 2021, at 17:47, Oscar Benjamin wrote:

...

Would it be problematic to postpone making __future__.annotations the

default?

This is a good question, and I think the SC would really like to know if anybody has objections to postponing this to 3.11. We haven’t discussed it yet (this whole topic is on our agenda for next Monday), but it might be the best thing to do given where we are in the 3.10 release cycle. It would give everyone a chance to breathe and come up with the right long term solution.

As long as you know that’s not zero cost either. Undoing the work will be complex (merge conflicts will abound) and and all testing and implementation people have done assuming PEP 563 semantics will still have to be redone. And note that there are some corner cases where the “PEP 563 semantics” in 3.10 could not be made fully compatible with the behavior under `from __future__ Import annotations` in 3.9 (in truth, because not enough testing was done under those semantics in 3.7-3.9). But it could be done, and that’s the way PEP 649 is currently written (using `from __future__ import co_annotations`). -- --Guido (mobile)

On Fri, Apr 16, 2021 at 9:49 AM Oscar Benjamin wrote:

...

That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.

Would it be problematic to postpone making __future__.annotations the default?

__future__.annotation is the default since 2020-10-04. https://github.com/python/cpython/commit/044a1048ca93d466965afc027b91a5a9eb9... After that, many changes are made on compiler and other places. So reverting the change is not so simple. And personally, I love static typing but I don't use type hint for performance/memory usage reason. I spend much effort to optimize PEP 563 to minimize type hinting overhead. So it's very sad that if I can not use type hinting when I can drop Python 3.9 support. So if PEP 649 is accepted, I want to use it since Python 3.10. Otherwise, I can not use type hinting even after I dropped Python 3.9 support. But it is up to release manager and steering council. -- Inada Naoki

Thank you everyone for your responses. I entirely accept that I should have brought this up earlier, perhaps much earlier. In my defence, when PEP 563 first came on my radar I assumed that get_type_hint() would be improved before it became the default behaviour, AFAIK it hasn't really changed. In particular the subtle changes to scope described in detail in PEP 649 [1] are still there and are still a big headache for pydantic. As Paul Ganssle says, my timing was not great, but it could also have been worse. I also never meant to create an "us versus them" style discussion, I was just worried about the narrow time window I had and the likelihood that my voice would not be heard, hence sounding as impassioned as I did. For a fairly comprehensive description of why PEP 563 represents challenges to pydantic, I defer to Łukasz Langa (the instigator of PEP 563) who has explained it very clearly in another thread "[Python-Dev] PEP 563 in light of PEP 649" [2]. Samuel [1] - https://www.python.org/dev/peps/pep-0649/#id9 [2] - https://mail.python.org/archives/list/python-dev@python.org/message/ZBJ7MD6C...

On 4/17/2021 12:28 AM, Christopher Barker wrote:

I wonder if anyone has considered the impact of PEP 563 on dataclasses ? I have!

I did find this SO post:

https://stackoverflow.com/questions/52171982/new-annotations-break-inspectio... https://stackoverflow.com/questions/52171982/new-annotations-break-inspectio...

which is related, but not quite the same -- THAT issue was fixed.

My issue does show up in this BPO:

https://bugs.python.org/issue39442 https://bugs.python.org/issue39442

Somewhere in the middle of that thread, Eric wrote: "At this point, this seems more like fodder for python-ideas."

Then there's a bit more to the thread, which peters out without resolution.

On to the issue:

dataclasses may be the only part of the stdlib that uses annotations.

dataclasses store the annotations in Field objects type attribute, in the __dataclass_fields__ attribute.

We can see that with this code:

@dataclass class Simple:     x : int     y : float     name : str

s = Simple(3, 4.0, "fred")

print("The field types:") for f in s.__dataclass_fields__.values():     print(f"name: {f.name http://f.name}, type: {f.type}, type of type: {type(f.type)}")

Which results in:

The field types: name: x, type: , type of type: name: y, type: , type of type: name: name, type: , type of type:

with:

from __future__ import annotations

The result is:

The field types: name: x, type: int, type of type: name: y, type: float, type of type: name: name, type: str, type of type:

This of course is completely as expected.

I have no idea how dataclasses uses the Field type attribute -- as far as I can tell, for nothing at all. However, it is there, and it is called "type", rather than say, "annotation". In retrospect, that field probably should have been named "annotation". Regardless, the intent was always "store what's in __annotations__[field_name]", or what the user specified in field(..., type=whatever, ...).

And I have a whole pile of code that fully expects the Fields' type attribute to be an actual type object that I can call to crate an instance of that type (or call a method on it, which is what I am actually doing)

So my code will very much break with this change. True, unfortunately. To be clear to everyone not paying close attention, "this change" is PEP 563.

I fully understand that the __dataclass_fields__ attribute was probably never intended to be part of the public API, so I get what I deserve.

However, the Field object is documented, as such:

""" class dataclasses.Field

Field objects describe each defined field. These objects are created internally, and are returned by the fields() module-level method (see below). Users should never instantiate a Field object directly. Its documented attributes are:

name: The name of the field. type: The type of the field. default, default_factory, init, repr, hash, compare, and metadata have the identical meaning and values as they do in the field() declaration.

Other attributes may exist, but they are private and must not be inspected or relied on. """

That last sentence implies that the type attribute CAN be inspected and relied upon, which is what I am doing. Yes, Field.type is very much part of the public dataclasses API as available through dataclasses.fields(), not through cls.__dataclass_fields__.

And I haven't tried to solve this problem in my use case, but I'm not sure it can be done -- when I get around to inspecting the type of the Field objects, I have no idea what namespace they are in -- so I can't reconstruct them from the string. I really need the type object itself. @dataclass pretty much has the same problem with respect to calling typing.get_type_hints().

So I'll probably need to re-write much of the dataclasses decorator, to call eval() early -- though even then I'm not sure I'll have access to the proper namespace.

Anyway -- one more data point:  PEP 563 changes the (semi-public?) API of dataclasses.

Though *maybe* that could be addressed with a dataclasses update -- again, I've only started to think about it -- there was some discussion of that in the BPO, though Eric didn't seem particularly interested.

I still think it's working as intended: it uses what's in __annotations__ (or passed in to field()). As everyone who has tried to call typing.get_type_hints() knows, it's hard to get right, if it's even possible, because, as you say "when I get around to inspecting the type of the Field objects, I have no idea what namespace they are in". My opinion is that the person who wants a real type object probably has a better idea of that namespace than @dataclass does, but there's a very good chance they don't know, either. @dataclass goes out of its way to not call typing.get_type_hints(). The original reason for this is not wanting to force typing to be imported, if it wasn't already being used. That may have been addressed with PEP 560, but I've never really checked on the impact. Another reason for not calling typing.get_type_hints(): there's really only one thing [*] dataclasses wants to know, with regard to the type/annotation of the field: is the type of this field typing.ClassVar? It doesn't seem that the performance issues and possible failures make it worth calling typing.get_type_hints() just for this case. @dataclass uses other tricks (not described here). In any event, all of this mess is one reason I'd like to see PEP 649 get accepted: there would never be a reason to call typing.get_type_hints(), and the values in the Field object would again be type objects. Back to my original point: If you ignore the test for ClassVar, then dataclasses completely ignores the values in __annotations__ or Field.type. It's no different from typing.NamedTuple in that regard. I do have sympathy for users looking at Field.type and getting a string instead of a type object: but that's really no different from non-dataclass users looking at any occurrence of __annotations__ and now getting a string: that's a result of PEP 563 across the board, not just with dataclasses. As I said in bpo-39442: Isn't that the entire point of "from __future__ import annotations"? Eric [*]: Actually two things: the other being "is the field of type dataclasses.InitVar?". It has some of the same problems as ClassVar, but we know that dataclasses has been imported, so it's less of a big deal.

On Sat, Apr 17, 2021 at 6:12 AM Eric V. Smith mailto:eric@trueblade.com> wrote:

...

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I have!

Thanks Eric.

In retrospect, that field probably should have been named "annotation". Regardless, the intent was always "store what's in __annotations__[field_name]", or what the user specified in field(..., type=whatever, ...).

intent is all well and good, but what was the intent of __annotations__ In the first place. Static typing was certainly one of the intents, but as it DID store a value, not a string from the beginning, we can only assume that there was some intent that the value might be used. Yes, but wouldn't that say that PEP 563 could never be implemented then, since it would break that intent? We already knew it broke all code that was expecting a type object in __annotations__. I'm just saying my expectation is 563 will break any code expecting a type object in __annotations__, or anything that's derived from __annotations__ (specifically in my case, dataclass users looking at Field.type). My

On 4/17/2021 6:44 PM, Christopher Barker wrote: probably unarticulated intent was that Field.type agree with __annotations__[field_name].

As for dataclasses -- why store it at all? It seems that dataclasses use the fact that a class attribute has an annotation to determine what to add to the field -- which is a nifty use of teh syntax. But presumably the intent of storing the annotation in the Field object, and providing a public attribute for it, was so that it could be used.

I didn't want to predict what someone was using dataclasses for. Especially someone using field(..., metadata=something) might want access to any of the field() values, including .type. I supposed I could have said "for defaults, default_factory, repr, etc., look at the Field object, but for the type look at __annotations__[field_name]", but I decided it would be easier if everything about a field would be in the Field object. Not that it makes much difference: I believe that Field.type always agrees with __annotations__[field_name]. This allows someone to say (where some_dataclass_field_object is a Field): process_field(some_dataclass_field_object) instead of this, if the .type field weren't present: process_field(some_dataclass_field_object, SomeDataclass.__annotations__[some_dataclass_field_object.name])

I suppose your point is that the type attribute stored the annotation, and PEP 653 changes what the annotation will be so there's nothing specific at all about dataclasses here. Which is true. But users of dataclasses may have been less aware of all the implications as they didn't write that annotation etrating code themselves. I know I wasn't.

In any event, all of this mess is one reason I'd like to see PEP 649 get accepted:

Indeed -- that is the title of this thread, after all :-)

Well, not everyone is not in support of the subject! Eric

And see others' notes, there seems to be two other places in the stdlib that will be affected.

-CHB

-- Christopher Barker, PhD (Chris)

Python Language Consulting   - Teaching   - Scientific Software Development   - Desktop GUI and Web Development   - wxPython, numpy, scipy, Cython

-- Eric V. Smith

Related, `inspect.Parameter.annotation` is affected too, but at least this attribute is called `annotation` instead of `type`. I noticed this today with `multipledispatch` (though [reported](https://github.com/mrocklin/multipledispatch/issues/104) in 2019) and some other internal code, both using `inspect.signature`. `multipledispatch` could reasonably swap to `typing.get_type_hints`, but the other code I was working with also used `Parameter.default` and `Parameter.kind`, so would require both `inspect.signature` and `get_type_hints`. Not the worst and arguably necessary now/pre PEP 563 with explicit string annotations, but the PEP 649 semantics would be a welcome improvement.

As far as I know, both Pydantic and marshmallow start using annotation for runtime type after PEP 563 is accepted. Am I right? When PEP 563 is accepted, there are some tools using runtime type in annotation, but they are not adopted widely yet. But we didn't have any good way to emit DeprecationWarning for use cases that PEP 563 breaks. Maybe, we should have changed the default behavior soon in Python 3.8. With long preparation period without DeperecationWarning, use cases broken by PEP 563 beccome much. I still love PEP 563. Python is dynamic language so runtime type and static type can not be 100% consistent. Annotation is handy tool. But there is only one annotation per object. When we start using annotation for multiple purposes, it becomes ugly monster soon. Using annotation syntax only for static typing like TypeScript is the ideal. TypeScript is far more succeeded than Python about Gradual Typing. But, current status is went to where I hate already. Annotation is used for multiple purposes already. I'm sad and disappointed about it, but that's that. I'm OK to keep PEP 563 opt-in. And I think we should do it. (off course, I'm not a SC member. I follow the SC decision). And if we keep PEP 563 opt-in, no need to compare PEP 563 with PEP 649. PEP 649 should be compared with "stock semantics + opt-in PEP 563 semantics". Of course, one semantics is better than two semanticses. But we need to have three semanticses until we can remove stock + PEP 563 semantices. We should think about PEP 649 very carefully, spending more time. Regards,