Re: [Python-ideas] Augmented assignment syntax for objects.
I'm gonna take a shot at elaborating this point.
We Python programmers often tout Python as a high-level, high-productivity language where complex and useful things are easy to do. However, vanilla Python classes are anything but high level; they're basically glorified dictionaries that require an awful lot of boilerplate for basic features. __init__ verbosity is just one of the symptoms of this.
I'm going to posit we need declarative classes. (This is what a library like attrs provides, basically.) For a class to be declarative, it needs to be possible to inspect the class for its attributes and more.
I'm putting forward three examples. These examples are based on attrs since that's what I consider to be the best way of having declarative classes in Python today.
First example. Here's a slide from a recent talk I've given to a local Python meetup: http://tinche.github.io/talks/attrs/#/8/5 This is a class decorator that, when applied to any attrs class, will generate a simple shallow copy() method for that class. This copy method will be faster than copy.copy() and IMO less surprising since it will actually invoke __init__ (and all logic inside). This isn't production code, just a proof of concept for the talk.
Second example: I'm working on a library that can turn dictionaries into attrs classes (i.e. turn unstructured data into structured data). Example: http://cattrs.readthedocs.io/en/latest/structuring.html#attrs-classes. When dealing with nested classes the library not only needs to know exactly which fields a class has but also the types of those fields, recursively (for which I'm planning to use attrs metadata). This functionality is useful when preparing data to be stored or sent elsewhere, since serialization libraries and clients in other languages expect json/msgpack etc.
Third example: I work at a mobile games company. The backends are Python, the games are written in other languages. The backends and the games share data structures - requests, responses, the save game, game data. I want to have a single source of truth for these structures, so I want to generate classes/structs in other languages from the backend Python classes.
All of these examples require declarative classes to do properly. Other approaches are, I suppose, possible in some cases - like inspecting __init__ and whatnot. But I claim the underlying issue is that we should be declaring classes differently, both to enable these new, high-level use cases and because declarative classes are much more readable and contain less boilerplate than what we have now, and this is (for me) Python is about.
Message: 1
Date: Fri, 28 Apr 2017 09:28:55 +1000 From: Steven D'Aprano steve@pearwood.info To: python-ideas@python.org Subject: Re: [Python-ideas] Augmented assignment syntax for objects. Message-ID: 20170427232853.GC22525@ando.pearwood.info Content-Type: text/plain; charset=iso-8859-1
On Wed, Apr 26, 2017 at 08:52:46PM -0400, Juancarlo A?ez wrote:
In my experience, what Python is lacking is a way to declare attributes outside of the constructor. Take a look at how it's done in C#, Swisft,
or
Go.
Since you apparently already know how they do it, how about telling us (1) how they do it, and (2) *why* they do it?
Object attributes outside of the constructor would solve things more relevant than the vertical space used when assigning constructor
parameters
to attributes.
Solve which things?
For example, multiple inheritance is well designed in Python, except that it often requires constructors with no parameters, which leads to objects with no default attributes.
Can you elaborate?
-- Steve
On 28 April 2017 at 12:55, Tin Tvrtković tinchester@gmail.com wrote:
I'm putting forward three examples. These examples are based on attrs since that's what I consider to be the best way of having declarative classes in Python today.
Your comments and examples are interesting, but don't they just come down to "attrs is a really good library"? I certainly intend to look at it based on what's been said in this thread. But I don't see anything much that suggests that anything *beyond* attrs is needed (except maybe bringing attrs into the stdlib, if its release cycle would make that reasonable and the author was interested, and honestly "put it in the stdlib" could be said about any good library - in practice though publishing on PyPI and accessing via pip is 99% of the time the more reasonable option).
Am I missing some point? Paul
On 28 April 2017 at 22:26, Paul Moore p.f.moore@gmail.com wrote:
On 28 April 2017 at 12:55, Tin Tvrtković tinchester@gmail.com wrote:
I'm putting forward three examples. These examples are based on attrs since that's what I consider to be the best way of having declarative classes in Python today.
Your comments and examples are interesting, but don't they just come down to "attrs is a really good library"? I certainly intend to look at it based on what's been said in this thread. But I don't see anything much that suggests that anything *beyond* attrs is needed (except maybe bringing attrs into the stdlib, if its release cycle would make that reasonable and the author was interested, and honestly "put it in the stdlib" could be said about any good library - in practice though publishing on PyPI and accessing via pip is 99% of the time the more reasonable option).
Am I missing some point?
Yes, the point I attempted to raise earlier: at the language design level, "How do we make __init__ methods easier to write?" is the *wrong question* to be asking. It's treating the symptom (writing an imperative initialiser is repetitive when it comes to field names) rather than the root cause (writing imperative initialisers is still part of the baseline recommendation for writing classes, and we don't offer any supporting infrastructure for avoiding that directly in the standard library)
Accordingly, a potentially better question to ask is "How do we make it so that writing a custom __init__ method for each class definition seems as strange and esoteric in 2025 as writing a custom metaclass seems now?".
For a *lot* of classes, what we want to be able to define is:
- a set of data fields - a basic initialiser to set those fields by name - a repr based on those fields (or a subset) - equality comparisons based on those fields (or a subset) - ordering comparisons based on those fields (or a subset) - conversion to a JSON-friendly dict - pickling support - copying support
Given the initial set of data fields, reasonable default behaviours for all of the rest can be derived automatically, but we don't provide the tools to do that derivation by default. This leaves teachers of Python in a quandary: they can either teach "native Python classes" (which are boilerplate heavy and error prone), or else they can pick a third party library like attrs, and teach "Python-with-attrs", in the same way that it's recommended to teach "Python-with-requests" rather than the native urllib APIs when it comes to accessing resources over HTTPS.
The difference between this case and requests is that HTTPS and related protocols experience a high level of churn independently of Python language updates, so there are real logistical benefits to maintaining it as a third party module.
By contrast, if we add an "autoclass" module to the standard library that adds more tools like functools.total_ordering to handle injecting boilerplate functionality into class definitions at runtime, then that's a pure win - folks can either delegate the details of their instance behaviour to the standard library maintainers (in the same way that most folks already delegate the behaviour of their metaclasses), or else they can continue to write out all those supported methods by hand if they really want or need the fine-grained control.
Cheers, Nick.
On 2017-04-28 06:07, Nick Coghlan wrote:
…the root cause (writing imperative initialisers is still part of the baseline recommendation for writing classes, and we don't offer any supporting infrastructure for avoiding that directly in the standard library)
For a *lot* of classes, what we want to be able to define is:
- a set of data fields
- a basic initialiser to set those fields by name
- a repr based on those fields (or a subset)
- equality comparisons based on those fields (or a subset)
- ordering comparisons based on those fields (or a subset)
- conversion to a JSON-friendly dict
- pickling support
- copying support
Nice! I've implemented a higher-level base class in a few projects that does just this. Things like pickling and JSON I might want to put into "mixins", but yes the majority of these would be nice to have implemented by default "batteries included" style. I prefer a base class to decorators since they seem to stay out of the way more:
class Vehicle(BaseObject, Picklable): …
That still leaves the problem of how to define instance attributes concisely. Perhaps a "defaults" member or marker, or one of the other examples in the larger thread would do.
On 29 April 2017 at 03:00, Mike Miller python-ideas@mgmiller.net wrote:
On 2017-04-28 06:07, Nick Coghlan wrote:
For a *lot* of classes, what we want to be able to define is:
- a set of data fields
- a basic initialiser to set those fields by name
- a repr based on those fields (or a subset)
- equality comparisons based on those fields (or a subset)
- ordering comparisons based on those fields (or a subset)
- conversion to a JSON-friendly dict
- pickling support
- copying support
Nice! I've implemented a higher-level base class in a few projects that does just this. Things like pickling and JSON I might want to put into "mixins", but yes the majority of these would be nice to have implemented by default "batteries included" style. I prefer a base class to decorators since they seem to stay out of the way more:
class Vehicle(BaseObject, Picklable): …
Putting any questions of aesthetics aside, the main argument in favour of using a base class and __init_subclass__ for autoclass definitions would be the fact that further subclasses would automatically get the magic method construction executed, rather than requiring that the class decorator be repeated on the subclass.
Cheers, Nick.
On 28 April 2017 at 14:07, Nick Coghlan ncoghlan@gmail.com wrote:
Am I missing some point?
Yes, the point I attempted to raise earlier: at the language design level, "How do we make __init__ methods easier to write?" is the *wrong question* to be asking. It's treating the symptom (writing an imperative initialiser is repetitive when it comes to field names) rather than the root cause (writing imperative initialisers is still part of the baseline recommendation for writing classes, and we don't offer any supporting infrastructure for avoiding that directly in the standard library)
[...]
Ah, OK. I see what you're saying here. I agree, that's the direction we should be looking in. I'd sort of lumped all of that side of things in my mind under the header of "must take a look at attrs" and left it at that for now. My mistake.
So basically yes, I agree we should have better means for writing common class definition patterns in a declarative way, retaining the current underlying mechanisms while making it so that people typically don't need to interact with them. I suspect that those means will probably take the form of stdlib/builtin support (decorators, custom base classes, etc) but the design will need some thrashing out. Beyond that, I don't really have much more to add until I've done some research into the current state of the art, in the form of libraries like attrs (and Stephan Hoyer mentioned typing.NamedTuple).
Paul
On Fri, Apr 28, 2017 at 3:07 PM Nick Coghlan ncoghlan@gmail.com wrote:
Yes, the point I attempted to raise earlier: at the language design level, "How do we make __init__ methods easier to write?" is the *wrong question* to be asking. It's treating the symptom (writing an imperative initialiser is repetitive when it comes to field names) rather than the root cause (writing imperative initialisers is still part of the baseline recommendation for writing classes, and we don't offer any supporting infrastructure for avoiding that directly in the standard library) [...]
Very well put. I also can't help but hope these efforts lead to Python also acquiring better tools for dealing with structured data (instances of classes and enumerations) down the road. Like a Pythonic version of Rust's match, for example. That would be really something.
On 28 April 2017 at 21:55, Tin Tvrtković tinchester@gmail.com wrote:
Third example: I work at a mobile games company. The backends are Python, the games are written in other languages. The backends and the games share data structures - requests, responses, the save game, game data. I want to have a single source of truth for these structures, so I want to generate classes/structs in other languages from the backend Python classes.
Some additional metaclass based prior art for declarative class definitions:
- Django ORM models - SQL Alchemy ORM models - JSL template definitions (http://jsl.readthedocs.io/en/latest/tutorial.html)
(I'm sure there's also plenty of prior art in Zope and Plone, but I'm not personally all that familiar with either of those)
All of these examples require declarative classes to do properly. Other approaches are, I suppose, possible in some cases - like inspecting __init__ and whatnot. But I claim the underlying issue is that we should be declaring classes differently, both to enable these new, high-level use cases and because declarative classes are much more readable and contain less boilerplate than what we have now, and this is (for me) Python is about.
Exactly.
I don't think we should change anything about the basic semantics of class or method definitions (since they're part of what makes it possible to reason about Python as an inherently procedural language), but I *do* think we should be considering the kinds of helpers we can provide as class decorators to take the boilerplate and tedium out of defining well-behaved classes.
Cheers, Nick.
On Fri, Apr 28, 2017 at 4:55 AM, Tin Tvrtković tinchester@gmail.com wrote:
I'm going to posit we need declarative classes. (This is what a library like attrs provides, basically.) For a class to be declarative, it needs to be possible to inspect the class for its attributes and more.
I'm putting forward three examples. These examples are based on attrs since that's what I consider to be the best way of having declarative classes in Python today.
My favorite example of declarative classes in Python today is Python 3.6.1's typing.NamedTuple:
class Employee(NamedTuple): """Represents an employee.""" name: str id: int = 3
NamedTuple fixes many of the warts of the original namedtuple: 1. It supports docstrings out of the box, not requiring a subclass (which require setting __slots__ to get right) 2. It supports default values. 3. It has a sane declarative syntax. 4. It's fields can be type checked.
What's not to love? There are two valid complaints [1] about namedtuple that still apply: 1. It's a tuple, which brings along extra baggage (numeric indexing/iteration, comparing equal to tuples, etc.) 2. It's immutable. Sometimes it's convenient to have a mutable class.
It seems like a simple solution would be make variants of NamedTuple that solve each of these issues. We might call these collections.Struct and collections.FrozenStruct.
[1] e.g., see https://glyph.twistedmatrix.com/2016/08/attrs.html
Third example: I work at a mobile games company. The backends are Python, the games are written in other languages. The backends and the games share data structures - requests, responses, the save game, game data. I want to have a single source of truth for these structures, so I want to generate classes/structs in other languages from the backend Python classes.
If you're willing to define these classes outside of Python in a separate file (which is quite reasonable if you need cross-language support), then this is exactly the problem solved by protocol buffers ( https://developers.google.com/protocol-buffers/). So I'm not sure we really need this fix in Python.
participants (5)
-
Mike Miller -
Nick Coghlan -
Paul Moore -
Stephan Hoyer -
Tin Tvrtković