Please accept PEP 649! Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code: ```py import discord from discord.ext import commands bot = commands.Bot(command_prefix='>') @bot.command() # discord.py reads the typehints and converts the arguments accordingly async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed await ctx.send(f'{member.mention}! {text}') bot.run('token') ``` I must say, this is extremely ergonomic design! Don't break it :)

I've been working on a similar framework, Fondat [1], that shares similar objectives as Pydantic and FastAPI, albeit implementing in a different manner.  I believe other frameworks will have to "run a gauntlet" (as I already have)  to deal with the affects of PEP 563. If anyone on these teams are interested in how I've dealt with such issues, please feel free to contact me directly. I also recommend members of these teams monitor and/or participate in the typing-sig (and maybe the python-dev) mailing list, as discussions in these lists are likely to affect their work in the future. Paul  [1] https://github.com/fondat/fondat-core On Thu, 2021-04-15 at 17:49 -0400, Paul Ganssle wrote:

I should point out that "accept PEP 649" and "break pydantic" are not the only options here. The thing that will break pydantic is the end of PEP 563's deprecation period, not a failure to implement PEP 649. Other viable options include: - Leave PEP 563 opt-in until we can agree on a solution to the problem. - Leave PEP 563 opt-in forever. - Deprecate PEP 563 and go back to status quo ante.

I haven't followed this closely enough — if PEP 649 were accepted today, would it even be ready for use before the 3.10 code freeze (which is in a few weeks)?

Assuming this is a real problem (and based in part on how long it took for attrs to get what support it has for PEP 563, I wouldn't be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649.

Best, Paul On 4/15/21 4:20 PM, Bluenix wrote:

...

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:

```py import discord from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command() # discord.py reads the typehints and converts the arguments accordingly async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed await ctx.send(f'{member.mention}! {text}')

bot.run('token') ```

I must say, this is extremely ergonomic design! Don't break it :) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4Q... Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2... Code of Conduct: http://python.org/psf/codeofconduct/

Hello, I'm a bit surprised that this topic is brought up just days before the feature freeze of Python 3.10. How did tools work until now with from future import annotations? This feature has been in the language for years (since 3.7), and I haven't heard until today anyone complaining about this feature gap. Did everyone just told users not to use lazy evaluation of type hints until now? Also there's been advertised for at least a year that next python will make it default 🤔 Thanks, Bernat On Thu, Apr 15, 2021, 23:13 Paul Bryan <pbryan@anode.ca> wrote:

I've been working on a similar framework, Fondat [1], that shares similar objectives as Pydantic and FastAPI, albeit implementing in a different manner.

I believe other frameworks will have to "run a gauntlet" (as I already have) to deal with the affects of PEP 563. If anyone on these teams are interested in how I've dealt with such issues, please feel free to contact me directly.

I also recommend members of these teams monitor and/or participate in the typing-sig (and maybe the python-dev) mailing list, as discussions in these lists are likely to affect their work in the future.

Paul

[1] https://github.com/fondat/fondat-core

On Thu, 2021-04-15 at 17:49 -0400, Paul Ganssle wrote:

I should point out that "accept PEP 649" and "break pydantic" are not the only options here. The thing that will break pydantic is the end of PEP 563's deprecation period, not a failure to implement PEP 649.

Other viable options include:

- Leave PEP 563 opt-in until we can agree on a solution to the problem. - Leave PEP 563 opt-in forever. - Deprecate PEP 563 and go back to status quo ante.

I haven't followed this closely enough — if PEP 649 were accepted today, would it even be ready for use before the 3.10 code freeze (which is in a few weeks)?

Assuming this is a real problem (and based in part on how long it took for attrs to get what support it has for PEP 563 <https://github.com/python-attrs/attrs/issues/288>, I wouldn't be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649.

Best, Paul On 4/15/21 4:20 PM, Bluenix wrote:

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:

```py

import discord

from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command()

# discord.py reads the typehints and converts the arguments accordingly

async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed

await ctx.send(f'{member.mention}! {text}')

bot.run('token')

```

I must say, this is extremely ergonomic design! Don't break it :)

_______________________________________________

Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-leave@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4Q...

Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PBGFXPC7... Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Apr 15, 2021 at 12:01 PM Samuel Colvin <samcolvin@gmail.com> wrote:

I've read the recent discussions <https://mail.python.org/archives/list/python-dev@python.org/thread/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/#UITB2A657TAINAGWGRD6GCKWFC5PEBIZ> regarding PEP 649 and PEP 563 with interest, Larry Hastings recently contacted me when prompted <https://mail.python.org/archives/list/python-dev@python.org/message/YKVYJMLUWUVT4KMLUNEQYVBZWNAPR4GV/> to do so in a related discussion.

I maintain pydantic <https://pydantic-docs.helpmanual.io/> which uses type annotations to provide runtime data validation.

I'm extremely worried that if PEP 649 gets rejected, pydantic will not be able to fully support python 3.10 and may even end up getting abandoned, at best it will be much slower and more brittle.

Please, please put pragmatism over principle and accept PEP 649.

Please don't phrase the decision in these terms. While for Pydantic PEP 649 is more pragmatic, that does not mean PEP 563 isn't pragmatic for other people for other reasons. Making this an "us versus them" discussion just makes the whole situation feel confrontational when instead everyone is trying to figure out the best thing for everybody when there's no perfect answer.

On Thu, Apr 15, 2021 at 4:20 PM Brett Cannon <brett@python.org> wrote:

Making this an "us versus them" discussion just makes the whole situation feel confrontational when instead everyone is trying to figure out the best thing for everybody when there's no perfect answer.

I agree that that was strong language, but from the perspective of folks not interested in Static Typing, there has been a lot of activity, or at least discussion, that makes it seem like this optional feature is driving the evolution of the language. So folks are going to get concerned, and maybe upset, if it looks like changes might be made that may break other features in order to make static typing work better. And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Fri, 16 Apr 2021 at 01:13, Guido van Rossum <guido@python.org> wrote:

On Thu, Apr 15, 2021 at 16:48 Christopher Barker <pythonchb@gmail.com> wrote:

...

And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them.

It seems a little disingenuous to claim discussions about annotations don’t concern you when you’re actively using them (for typing, no less, in the case of pydantic). And I am sure a project as popular (by their own description) as pydantic will find a way forward if PEP 649 is rejected, despite overdramatized claims.

It looks to me like pydantic were well aware that this affected them and have been working over time to try and support this. They probably also had their eye on PEP 649 or hoped that something else would come along to fix this but they can now see the crunch point approaching. As Chris says though other libraries/developers have been using annotations for some time and might not have realised that typing considerations could undermine their other uses for annotations since annotations predate typing and typing is an optional feature.

That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.

Would it be problematic to postpone making __future__.annotations the default? Oscar

On Fri, Apr 16, 2021 at 9:49 AM Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

...

That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.

Would it be problematic to postpone making __future__.annotations the default?

__future__.annotation is the default since 2020-10-04. https://github.com/python/cpython/commit/044a1048ca93d466965afc027b91a5a9eb9... After that, many changes are made on compiler and other places. So reverting the change is not so simple. And personally, I love static typing but I don't use type hint for performance/memory usage reason. I spend much effort to optimize PEP 563 to minimize type hinting overhead. So it's very sad that if I can not use type hinting when I can drop Python 3.9 support. So if PEP 649 is accepted, I want to use it since Python 3.10. Otherwise, I can not use type hinting even after I dropped Python 3.9 support. But it is up to release manager and steering council. -- Inada Naoki <songofacandy@gmail.com>

On 4/15/21 6:09 PM, Barry Warsaw wrote:

On Apr 15, 2021, at 17:47, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

...

Would it be problematic to postpone making __future__.annotations the default? This is a good question, and I think the SC would really like to know if anybody has objections to postponing this to 3.11. We haven’t discussed it yet (this whole topic is on our agenda for next Monday), but it might be the best thing to do given where we are in the 3.10 release cycle. It would give everyone a chance to breathe and come up with the right long term solution.

I don't have any objections, and I certainly see the wisdom in such a decision. Best wishes, //arry/

On Fri, 16 Apr 2021 10:35:19 +0900 Inada Naoki <songofacandy@gmail.com> wrote:

And personally, I love static typing but I don't use type hint for performance/memory usage reason. I spend much effort to optimize PEP 563 to minimize type hinting overhead. So it's very sad that if I can not use type hinting when I can drop Python 3.9 support.

AFAIU, we're not talking about dropping PEP 563 (yet), but about making it opt-in again. Regards Antoine.

On Fri, Apr 16, 2021 at 8:27 AM Inada Naoki <songofacandy@gmail.com> wrote:

After PEP 563, only `'List[int]'` is practical so we can stop supporting `List["int"]` and others at some point.

There's a lot of code written before PEP 563 was available (and code that needs to support earlier Python versions) that uses string literal quoting in various places. If Python stops supporting List["X"], it will be a significant backward compatibility break. Some of these string literal escapes are made unnecessary by PEP 563, but there are still cases where string literals are needed that neither PEP 563 nor PEP 649 address. Examples include type aliases, type variable definitions and base classes, but there are others [1]. These aren't evaluated in a type annotation context, so manual string literal escaping would still be needed.

So playing with runtime type will become easier in the future.

Am I wrong?

In some cases things will become easier, but other common use cases still seem to be unresolved. I haven't seen any proposal that can completely replace all uses of string literal escapes. I guess one option would be to only allow string literal escaping to be used outside type annotations in the future. I think that this could be feasible with a suitable deprecation period. Also, the use of "if TYPE_CHECKING" breaks the runtime use of annotations. PEP 563 and PEP 649 reduce the need for string literal escaping in this use case, but it mostly helps static type checkers. This is pretty common in codebases that use static type checking. My main issue with PEP 649 is that it only addresses a subset of remaining issues, i.e. it doesn't go far enough. It's also not clear to me if PEP 649 (or PEP 563) can be extended to cover the remaining runtime issues, or if we'd need a *third* approach to solve them. Jukka [1] https://www.python.org/dev/peps/pep-0563/#forward-references

I wonder if anyone has considered the impact of PEP 563 on dataclasses ? I did find this SO post: https://stackoverflow.com/questions/52171982/new-annotations-break-inspectio... which is related, but not quite the same -- THAT issue was fixed. My issue does show up in this BPO: https://bugs.python.org/issue39442 Somewhere in the middle of that thread, Eric wrote: "At this point, this seems more like fodder for python-ideas." Then there's a bit more to the thread, which peters out without resolution. On to the issue: dataclasses may be the only part of the stdlib that uses annotations. dataclasses store the annotations in Field objects type attribute, in the __dataclass_fields__ attribute. We can see that with this code: @dataclass class Simple: x : int y : float name : str s = Simple(3, 4.0, "fred") print("The field types:") for f in s.__dataclass_fields__.values(): print(f"name: {f.name}, type: {f.type}, type of type: {type(f.type)}") Which results in: The field types: name: x, type: <class 'int'>, type of type: <class 'type'> name: y, type: <class 'float'>, type of type: <class 'type'> name: name, type: <class 'str'>, type of type: <class 'type'> with: from __future__ import annotations The result is: The field types: name: x, type: int, type of type: <class 'str'> name: y, type: float, type of type: <class 'str'> name: name, type: str, type of type: <class 'str'> This of course is completely as expected. I have no idea how dataclasses uses the Field type attribute -- as far as I can tell, for nothing at all. However, it is there, and it is called "type", rather than say, "annotation". And I have a whole pile of code that fully expects the Fields' type attribute to be an actual type object that I can call to crate an instance of that type (or call a method on it, which is what I am actually doing) So my code will very much break with this change. I fully understand that the __dataclass_fields__ attribute was probably never intended to be part of the public API, so I get what I deserve. However, the Field object is documented, as such: """ class dataclasses.Field Field objects describe each defined field. These objects are created internally, and are returned by the fields() module-level method (see below). Users should never instantiate a Field object directly. Its documented attributes are: name: The name of the field. type: The type of the field. default, default_factory, init, repr, hash, compare, and metadata have the identical meaning and values as they do in the field() declaration. Other attributes may exist, but they are private and must not be inspected or relied on. """ That last sentence implies that the type attribute CAN be inspected and relied upon, which is what I am doing. And I haven't tried to solve this problem in my use case, but I'm not sure it can be done -- when I get around to inspecting the type of the Field objects, I have no idea what namespace they are in -- so I can't reconstruct them from the string. I really need the type object itself. So I'll probably need to re-write much of the dataclasses decorator, to call eval() early -- though even then I'm not sure I'll have access to the proper namespace. Anyway -- one more data point: PEP 563 changes the (semi-public?) API of dataclasses. Though *maybe* that could be addressed with a dataclasses update -- again, I've only started to think about it -- there was some discussion of that in the BPO, though Eric didn't seem particularly interested. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

... dataclasses may be the only part of the stdlib that uses annotations.

There's another one, actually: functools.singledispatch, at https://github.com/python/cpython/blob/3.9/Lib/functools.py#L860. It uses

El vie, 16 abr 2021 a las 21:32, Christopher Barker (<pythonchb@gmail.com>) escribió: the annotations on the first argument to an `@register`ed function to figure out how to dispatch. Currently it uses `typing.get_type_hints` to get the type. Presumably, this would break with PEP 563 semantics if you try to create a singledispatch function in a nested namespace.

On Sat, Apr 17, 2021 at 6:12 AM Eric V. Smith <eric@trueblade.com> wrote:

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I have!

Thanks Eric.

In retrospect, that field probably should have been named "annotation". Regardless, the intent was always "store what's in __annotations__[field_name]", or what the user specified in field(..., type=whatever, ...).

intent is all well and good, but what was the intent of __annotations__ In the first place. Static typing was certainly one of the intents, but as it DID store a value, not a string from the beginning, we can only assume that there was some intent that the value might be used. As for dataclasses -- why store it at all? It seems that dataclasses use the fact that a class attribute has an annotation to determine what to add to the field -- which is a nifty use of teh syntax. But presumably the intent of storing the annotation in the Field object, and providing a public attribute for it, was so that it could be used. I suppose your point is that the type attribute stored the annotation, and PEP 653 changes what the annotation will be so there's nothing specific at all about dataclasses here. Which is true. But users of dataclasses may have been less aware of all the implications as they didn't write that annotation etrating code themselves. I know I wasn't.

In any event, all of this mess is one reason I'd like to see PEP 649 get accepted:

Indeed -- that is the title of this thread, after all :-) And see others' notes, there seems to be two other places in the stdlib that will be affected. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

I'd like to chime in with an example of how PEP 563 breaks code that uses dataclasses. I've written a library instant_api (https://github.com/alexmojaki/instant_api) that is heavily inspired by FastAPI but uses dataclasses for complex types instead of pydantic. The example at the beginning of the README is short and demonstrates it nicely. Basically it lets you write code on both the client and server sides that work seamlessly with standard dataclasses, type hints, and type checkers without any plugins, instead of untyped dicts parsed from the JSON that is communicated behind the scenes. `from __future__ import annotations` breaks that README example, even though there are no locally defined types, because as mentioned the dataclass field now contains a string instead of a type. Going a bit deeper, instant_api is powered by https://github.com/alexmojaki/datafunctions, which is more generic than instant_api so that others can build similar tools. Again, the idea is that you can write code with nice dataclasses and type hints, but call it with basic JSON serializable types like dicts. For example: ``` from dataclasses import dataclass from datafunctions import datafunction @dataclass class Point: x: int y: int @datafunction def translate(p: Point, dx: int, dy: int) -> Point: return Point(p.x + dx, p.y + dy) assert translate({"x": 1, "y": 2}, 3, 4) == {"x": 4, "y": 6} # This is equivalent to the following without @datafunction # assert translate(Point(1, 2), 3, 4) == Point(4, 6) ``` In the same way as before, `from __future__ import annotations` breaks this code. The reason is that datafunctions itself is powered by https://github.com/lovasoa/marshmallow_dataclass. Here's an example: ``` from dataclasses import dataclass from marshmallow_dataclass import class_schema @dataclass class Point: x: int y: int schema = class_schema(Point)() assert schema.load({"x": 1, "y": 2}) == Point(1, 2) ``` Again, in the same way as before, `from __future__ import annotations` breaks this code. Specifically `class_schema(Point)` breaks trying to deal with the string `'int'` instead of a type. This problem was raised in https://github.com/lovasoa/marshmallow_dataclass/issues/13 two years ago. It's by far the oldest open issue in the repo. It was clear from the beginning that it's a difficult problem to solve. Little progress has been made, there's one PR that's not in good shape, and it seems there's been no activity there for a while. A couple of other issues have been closed as duplicates. One of those issues is about being unable to use recursive types at all. marshmallow_dataclass has 266 stars. It builds on https://github.com/marshmallow-code/marshmallow, an extremely popular and important data (de)serialization and validation library. Here's a little timeline: - 2013: marshmallow 0.1.0 first released in 2013 - 2014: marshmallow 1.0.0 released - 2015: attrs (precursor to dataclasses) first released - 2016: Python 3.6.0 final released, allowing the variable annotations which make pydantic and dataclasses possible. - 2017: First version of pydantic released - 2018: Python 3.7.0 final released, introducing dataclasses Nowadays pydantic is the natural successor/alternative to marshmallow - Google autocompletes "pydantic vs " with marshmallow as the first option, and vice versa. But marshmallow is clearly well established and entrenched, and thanks to marshmallow_dataclass it was the better fit for my particular use case just last year when I made instant_api. If someone wants to keep combining dataclasses and marshmallow, but without marshmallow_dataclass (e.g. if PEP 563 goes through before marshmallow_dataclass is ready) then they need to revert to the raw marshmallow API which doesn't use type hints. The previous example becomes much uglier: ``` from dataclasses import dataclass from marshmallow import Schema, fields, post_load @dataclass class Point: x: int y: int class PointSchema(Schema): x = fields.Int() y = fields.Int() @post_load def make_point(self, data, **kwargs): return Point(**data) schema = PointSchema() assert schema.load({"x": 1, "y": 2}) == Point(1, 2) ``` This post turned out longer than I initially planned! In summary, my point is that type hints and dataclasses as they work right now make it possible to write some really nice code - nice for humans to both write and read, nice for type checkers and other static analysis, and providing very nice features using annotations at runtime. And despite clear demand and benefits and ample time, people haven't managed to make this code continue working with stringified type annotations. Clearly doing so is not easy. So there's a good case for the dataclasses module to resolve these annotations to actual types, especially if PEP 563 goes through but even if it doesn't.

Hi Samuel, I have a naive question about pydantic. IIRC there's something in your tracker about not being able to handle recursive definitions correctly with PEP 563. But I must be misremembering, because PEP 563 actually makes that easier, not harder. Without PEP 563, how does it handle classes like this? @<pydantic decorator> class C: def method(self) -> C: ... (My apologies for not doing more research, but I figured this would be quicker for you to answer than for me to research, given that I've never used pydantic.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

As far as I know, both Pydantic and marshmallow start using annotation for runtime type after PEP 563 is accepted. Am I right?

Not quite, pydantic was released in June 2017 (see HN post: https://news.ycombinator.com/item?id=14477222) and always used annotations, PEP 563 was created in September 2017, a few months later. I can't speak about Marshmallow, back then I don't think it used annotations at all, but maybe it does now. When PEP 563 is accepted, there are some tools using runtime type in

annotation, but they are not adopted widely yet.

Yes, that's completely fair. Samuel