I've used code along the lines of: ``` if TYPE_CHECKING: thing_with_dataclass_like_semantics = dataclasses.dataclass else: # define thing_with_dataclass_like_semantics ``` with moderate success. I do sort of agree with Paul: to the extent that libraries have exact dataclass semantics, they shouldn't need to exist (or can use the above trick), and to the extent that they have different semantics, this proposal would fall short (or get very complicated). And I do think there are a variety of semantics here, e.g. https://github.com/python/cpython/pull/17322 On Fri, 23 Apr 2021 at 23:26, Paul Bryan wrote:

If such libraries were able to implement their data objects as dataclasses, would such a PEP be necessary?

Paul

On Sat, 2021-04-24 at 06:12 +0000, Eric Traut wrote:

PEP 557 introduced the dataclass to the Python stdlib. Several popular libraries (including attrs, pydantic, and various libraries that support database ORMs such as django and edgedb) have behaviors that are similar to dataclass, but these behaviors cannot be described using standard type annotations.

To work around this limitation, mypy custom plugins have been developed for many of these libraries, but these plugins don't work with other type checkers, linters or language servers. They are also costly to maintain for library authors, and they require that Python developers know about the existence of these plugins and download and configure them within their environment.

I've been exploring ways to extend support for dataclass semantics to other other libraries in a standardized manner. I've posted a [proposal in the form of a proto-PEP]( https://github.com/microsoft/pyright/blob/master/specs/dataclass_transforms.... ).

The latest versions of pylance and pyright implement this proposal as a proof of concept. It was relatively easy to implement (on the order of a couple hundred lines of code), and I assume that it would be similarly easy for other type checkers to add this support.

I've created a [github discussion and feedback forum]( https://github.com/microsoft/pyright/discussions/1782), and I welcome feedback on the proposal. I've already been in contact with the maintainers of attrs and pydantic. They are generally supportive, and I've incorporated their initial input into the latest draft of the proposal.

-Eric

-- Eric Traut Contributor to Pyright & Pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca

_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com

Other libraries presumably came up with their own data transfer objects prior to the introduction of dataclasses. But dataclasses are here, and are now the standard for data structures for Python. So, a part of my thinking is, yes, somehow existing libraries should adapt in some way. (I don't see much reason to support those who have no interest in understanding dataclasses.) As I am naive about Python static type checking, I would suggest a Protocol that defines methods that dataclass proper and others should implement. While I'm at it, I would then also deprecate `is_dataclass` and rely on subclass checking against the protocol. Adapting other implementations could be just to add any additional "dataclass" semantics over what they've already built.  Paul On Sat, 2021-04-24 at 15:54 +0000, Eric Traut wrote:

Yes, it's possible to use `TYPE_CHECKING` to get close to the desired type checking semantics in some cases (at least those that use decorator functions to transform the original class), but I don't think this solution is adequate. Shantanu, you are a very experienced and clever developer. Most Python developers would not come up with this solution.

The vast majority of Pylance users (I would guess >95%) are not interested in type checking but _are_ interested in high-quality completion suggestions and tooltip-based documentation during editing. These features rely on type information under the covers, but these users generally don't have any understanding of (or interest in understanding) Python type annotations. They simply expect it to work — in their code editor and in they Jupyter notebooks.

Of the Pylance users who _are_ interested in type checking (a number that is growing all the time!), I would estimate that >90% of them are not sophisticated enough in their understanding of types to think of a solution like the one you suggested above. They expect type checking to "just work" with all of the popular libraries. That's not an unreasonable expectation, in my opinion.

Paul, I'm not sure I understand what you mean by "implement their data objects as dataclasses". Are you suggesting that the authors of all of these popular libraries should reimplement their library's functionality by subclassing from dataclass in some manner? I don't think that's a realistic ask. It would be a large undertaking, would be a breaking change in terms of the library's interface (at least for those that are metaclass-based), would introduce other subtle incompatibilities for programs that use these libraries (and other libraries that build on these libraries), and it would take years for the Python community to adopt these new versions. I'm also skeptical that all of these libraries could be built on top of dataclass because some of the functionality they include (such as pydantic's runtime type checking) likely does not compose well with the implementation of dataclass. So if I understand you correctly, I don't think that proposal would achieve the goals I'm trying to achieve here. If I misinterpreted your suggestion, please elaborate.

-Eric

-- Eric Traut Contributor to Pyright & Pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca

I took a crack at updating the spec with additional explanations and examples. Let me know if that makes things clearer. I don't think it makes sense to document the full implementation of these decorator functions or metaclasses in the spec. They're quite complex. For example, here's an implementation of the `@dataclass` decorator function: https://github.com/ericvsmith/dataclasses/blob/95d2a4ca30e9dfdbc6837921cc5a5.... The implementations in `attrs`, `pydantic` and other libraries are similarly complex, but they're all doing roughly the same thing — synthesizing methods including an `__init__` method that includes one parameter for each declared data field. The phrase "dataclass semantics" is meant to refer to the behaviors listed in the "Motivation" section at the beginning of the spec. Additional details can be found in the subsection titled "Dataclass Semantics". -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.

On Sat, 2021-04-24 at 21:33 -0700, Guido van Rossum wrote:

Another thing that surprises me is that apparently the names of the various allowable keyword arguments (eq, final etc.) is fixed, and you can only specify whether they are supported and their default setting? That seems super dependent on the current set of popular "dataclass-like" functions. (And unlike Paul, I don't think that now dataclasses exist in the stdlib, the other libraries should defer to its interface or implementation -- I think there's plenty of room for new class decorators along these lines.)

If so, then why limit to those intended to provide equivalent of dataclass? Perhaps this is a cross-cutting case here. If this is a method to provide templates for types (or perhaps factories of said types), I guess that could make sense. If this is to somehow imbue dataclass-like properties on non-dataclass-dataclasses that don't implement dataclass' interface, I'm still skeptical. Paul

I agree we don't need to over-generalize, I just worry about applicability to lesser-known libraries that we haven't heard of yet. So it would be a good idea to have a plan *if* that need arises. In any case I assume this is just a stopgap measure until we have true portable type system extensions... On Sat, Apr 24, 2021 at 11:10 PM Eric Traut wrote:

Guido, thanks for the feedback. I've updated the spec to incorporate your suggestions.

Yes, I'm recommending that we fix the names of the various allowable keyword parameters. The popular libraries use these parameter names, and I don't see a need to provide additional flexibility here. Of course, we could add more configurability if there's a need, but I'd prefer to keep it simple.

-Eric

-- Eric Traut Contributor to Pyright & Pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

I've been adding pyi dataclass and attr declarations to pytype, and I've spent the last month trying to track down edge cases that pytype should support. In particular, this part of your proposal differs from dataclass behaviour: @dataclass class Employee: # Field with no descriptor name: str # Field that uses field descriptor class instance age: Optional[int] = field(default=None, init=False) # Field with type annotation and simple initializer to # describe default value is_paid_hourly: bool = True # Field with inferred type and simple initializer to # describe default value office_number = "unassigned" office_number will be a classvar rather than a field, since it's not annotated with a type. Another thing to address is whether the entire range of dataclass field constructors, e.g. InitVars, will be supported in pyi files (initvars in particular would need something like field_descriptors to specify the generic class used to mark an initvar, unless we just say that dataclasses are in the stdlib and other libraries should just use dataclasses.InitVar). martin On Sat, Apr 24, 2021 at 11:10 PM Eric Traut wrote:

Guido, thanks for the feedback. I've updated the spec to incorporate your suggestions.

Yes, I'm recommending that we fix the names of the various allowable keyword parameters. The popular libraries use these parameter names, and I don't see a need to provide additional flexibility here. Of course, we could add more configurability if there's a need, but I'd prefer to keep it simple.

-Eric

-- Eric Traut Contributor to Pyright & Pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com

Another corner case I just had to add to pytype - attrs supports `kw_only` annotations on individual fields. It seems worth adding that to the list of standard `field` parameters. martin On Thu, Apr 29, 2021 at 2:09 PM Eric Traut wrote:

Thanks for the feedback, Martin.

That's a good point about `office_number` being interpreted as a class variable. My intent was to match the semantics of dataclass, and this was simply a bug in the spec. I've updated the spec accordingly.

As for `InitVar`, none of the other libraries that I've reviewed have a similar concept (at least that I've noticed), so I didn't see a need to add support for custom `InitVar` classes. If one of these libraries decides to add support in the future, I would encourage them to use `dataclass.InitVar` rather than creating a new class with the same semantics.

-Eric

-- Eric Traut Contributor to Pyright & Pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com

Thanks for identifying those behavioral mismatches between attrs and dataclass. These differences are unfortunate, especially given that dataclass evolved from attrs. I've updated the [Limitations](https://github.com/microsoft/pyright/blob/main/specs/dataclass_transforms.md...) section of the spec to capture these and other pieces of feedback so we can collectively decide which of these limitations need to be addressed (either in the dataclass_transform spec or in attrs) and which can potentially be ignored (because they are esoteric edge cases that impact very few uses in practice). Keep the feedback coming! -Eric