Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.
Sometimes users inherit from builtin types only to find that their overridden methods are not called. Instead of this being a trap for unsuspecting users, I suggest overriding the __new__ method of these types so that it will raise with an informative exception explaining that, e.g., instead of inheriting from dict, you should inherit from UserDict. I suggest this modification to any Python implementation that has special versions of classes that cannot easily be extended, such as CPython. If another Python implementation allows dict (e.g.) to be extended easily, then it doesn't have to raise. Best, Neil
On 04/20/2016 07:51 PM, Neil Girdhar wrote:
How, exactly, does that help? And even if it does help (which I doubt), you're willing to break currently working code that successfully subclasses dicts, strings, tuples, etc., that then pass through dicts, strings, tuples, etc., to build their new class? No thank you. -- ~Ethan~
On Wed, Apr 20, 2016 at 07:51:37PM -0700, Neil Girdhar wrote:
-1 What about those who don't want or need to inherit from UserDict, and are perfectly happy with inheriting from dict? Why should we break their working code for the sake of people whose code already isn't working? Not everyone who inherits from dict tries to override __getitem__ and __setitem__ and are then surprised that other methods don't call their overridden methods. We shouldn't punish them. -- Steve
I think inheriting directly from dict is simply bad code because CPython doesn't promise that any of your overridden methods will be called. The fact that it silently doesn't call them is an inscrutable trap. And really, it's not much of a "punishment" to simply change your base class name. On Wed, Apr 20, 2016 at 11:16 PM Steven D'Aprano <steve@pearwood.info> wrote:
Except from the fact that a lot of code requiring actual dicts (or str, list…) will break because they are no longer receiving a dict instance (or an instance of (one of) its subclasses), but a collections.UserDict instance which is also entirely unrelated to dict except that they have methods with similar names. As I understand it, you prefer to put the burden of supporting collections.UserDict not on the people subclassing dict, but on the library developers expecting actual dicts. I’m sure there are a couple of built-in functions (written in C) accepting dicts that would not work if they didn’t get dicts. And getting built-in functions to accept instances of pure Python code is nontrivial and tricky at best, and requires a lot more work than it would make sense to in this case. I’m -1 on the idea. -Emanuel From: Neil Girdhar Sent: Thursday, April 21, 2016 12:34 AM I think inheriting directly from dict is simply bad code because CPython doesn't promise that any of your overridden methods will be called. The fact that it silently doesn't call them is an inscrutable trap. And really, it's not much of a "punishment" to simply change your base class name.
On Thu, Apr 21, 2016 at 2:33 PM, Neil Girdhar <mistersheik@gmail.com> wrote:
There are way too many cases that work just fine, though.
Why should I inherit from UserDict instead? I have to import that from somewhere (is it in types? collections? though presumably your error message would tell me that), and then I have to contend with the fact that my class is no longer a dictionary.
If you *do* push forward with this proposal, incidentally, I would recommend not doing it in __new__, but changing it so the class is no longer subclassable, as per bool:
But I am firmly -1 on disallowing dict subclasses just because they aren't guaranteed to call all of your overridden methods. ChrisA
On Thu, Apr 21, 2016 at 1:15 AM Chris Angelico <rosuav@gmail.com> wrote:
Of course it's a dictionary. It's an abc.Mapping, which is all a user of your class should care about. After all, it "quacks like a duck", which is all that matters.
That's a fair point, but it seems like a bug in JSON. They should have checked if it's an abc.Mapping imho.
Good point.
On Thu, Apr 21, 2016 at 3:33 PM, Neil Girdhar <mistersheik@gmail.com> wrote:
Your first sentence is in conflict with your other statements. My type is simply *not* a dictionary. You can start raising bug reports all over the place saying "JSON should look for abc.Mapping rather than dict", but I'm not even sure that it should. ChrisA
Lol, well, I think it should. Anyway, the cons are that yes some people's bad code won't work. The pros are that unsuspecting users won't fall into the trap associated with inheriting from dict, list or set. Of course, people on python-ideas know the pitfalls of inheriting from builtin types. You're not users that benefit from this change. It's people who don't know the problems and have no way of finding out until they spend a day debugging why overriding some method on a derived class of a derived class that ultimately inherits from dict doesn't work. Experts always want to make their own lives better. That's the problem with a language made by experts. New users fall into traps that you will never fall into (again). On Thu, Apr 21, 2016 at 1:48 AM Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Apr 21, 2016 at 1:58 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I feel your pain. However, what you see as a failure or defect of the builtins is actually a feature! :-) The downstream programmer should not be expected to know the implementation details of a dict. You don't want to read the source, you just want to use the public interface. You should be able to subclass and override methods as you like without worrying about hidden internal relationships. You should be able to override ``__getitem__`` without accidentally affecting things like ``values``. I guess it's a case of "a little knowledge is dangerous". Someone who knows nothing about dict implementation would not expect to see a change in method A because of an override of method B. The expert is keeping the novice safe. It's the journeyman who suffers.
Wow, I disagree with this. The fact that you can override getitem and it doesn't fix initialization in the constructor to match is definitely confusing. The fact that __add__ called on a dict subclass doesn't return a class of the same type is confusing. This is why every Python book advises people not to inherit from builtins. Let's not get carried away. This is an optimization for CPython — not a feature. On Thu, Apr 21, 2016 at 3:27 AM Michael Selik <mike@selik.org> wrote:
On Thu, Apr 21, 2016 at 04:33:33AM +0000, Neil Girdhar wrote:
I think inheriting directly from dict is simply bad code because CPython doesn't promise that any of your overridden methods will be called.
Python makes the same promise that it makes for *all* classes: if you override a method, and call that method on an instance of a subclass, the subclass' overridden implementation will be called. Dicts are no different from any other class, and have been since version 2.2 when types and classes where unified and inheriting from built-ins was first allowed. In fact, dicts were *explicitly* listed by Guido as one of the built-ins which can be subclassed: "Let's start with the juiciest bit: you can subtype built-in types like dictionaries and lists." https://www.python.org/download/releases/2.2.3/descrintro/#subclassing Your proposal would break at least two standard types, both of which subclass dict: py> from collections import defaultdict, Counter py> defaultdict.__mro__ (<class 'collections.defaultdict'>, <class 'dict'>, <class 'object'>) py> Counter.__mro__ (<class 'collections.Counter'>, <class 'dict'>, <class 'object'>) and it runs counter to a documented feature of dicts, that they can be subclassed and given a __missing__ method: "If a SUBCLASS OF DICT [emphasis added] defines a method __missing__() and key is not present, the d[key] operation calls that method ..." https://docs.python.org/2/library/stdtypes.html#mapping-types-dict
The fact that it silently doesn't call them is an inscrutable trap.
That is not a fact. What you should say is: "The fact that dicts don't obey the undocumented assumptions I made about the implementation of methods is a trap." and I will agree: correct, but its a trap for hubris and foolishness, and the answer to that is, don't make unjustified assumptions about how dicts are implemented. And I can say that because I made exactly that wrong assumption too. Nowhere does the documentation say that dict.update calls __setitem__. Nowhere does it say that dict.clear calls __delitem__. Nowhere does it say that dict.values calls __getitem__. And yet I assumed that they did all that. When somebody makes the unjustified assumption that they do, then they will discover that calling the subclass' clear method fails to call the overridden __delitem__. It also fails to call the overridden __str__. The only difference is that nobody assumes that clear() calls __str__, but many people foolishly assume that it calls the __delitem__ method. Both assumptions have *exactly* the same justification: none at all. I made a bunch of stupid, foolish assumptions, based on absolutely nothing more than the idea that it stands to reason that dict must be implemented in this way, and got bitten. And I deserved it. I was wrong, and anyone making the same assumption is wrong. Well, I say I was "bitten", but that over-dramatises the situation. What actually happened was that I wrote a subclass without doing any tests, then tested it, and discovered that it didn't work how I expected. I spent a few minutes playing with the class, added a few print statements, discovered that my assumptions were wrong, and then overrode the classes I actually wanted to override. Twenty minutes of googling and reading the docs convinced me that, no, Python doesn't promise that dict.clear calls __delitem__. It would have been five minutes except I was especially stubbon and pig-headed that day and didn't want to admit that I was in the wrong. But I was. Getting bitten by this was, in fact, a valuable lesson. I learned what I should have already known, what I had *intellectually* known but had ignored because "it stands to reason". Namely, if an implementation isn't documented, you cannot assume that it works in a particular way. My sympathy level is zero, and my support for this proposal is negative. -- Steve
On Thu, Apr 21, 2016, at 07:17, Steven D'Aprano wrote:
So what method should be overridden to make a dict subclass useful as a class or object dictionary (i.e. for attribute lookup to work with names that have not been stored with dict.__setitem__)? Overriding __getitem__ or __missing__ doesn't work. My only consolation is that defaultdict doesn't work either. I can't even figure out how to get the real class dict, as I would need if I were overriding __getattribute__ explicitly in the metaclass (which also doesn't work) - cls.__dict__ returns a mappingproxy. Alternatively, where, other than object and class dicts, are you actually required to have a subclass of dict rather than a UserDict or other duck-typed mapping? Incidentally, why is __missing__ documented under defaultdict as "in addition to the standard dict operations"?
On Thu, Apr 21, 2016 at 09:43:17AM -0400, Random832 wrote:
I don't understand your question. Or rather, if I have understood it, the question has a trivial answer: you don't have to override anything. class MyDict(dict): pass d = MyDict() d.attr = 1 print(d.attr) will do what you appear to be asking. If that's not what you actually mean, then you need to explain more carefully. I will also point out that your question is based on a point of confusion. You ask: "So what method should be overridden..." but the answer to this in general must be "What makes you think a single method is sufficient?" This doesn't just apply to dicts, it applies in general to any class. "What method do I override to make a subclass of int implement arithmetic with wrap-around (e.g. 999+1 = 0, 501*2 = 2)?"
This doesn't make anything any clearer for me.
Anywhere you have to operate with code that does "if isinstance(x, dict)" checks.
Incidentally, why is __missing__ documented under defaultdict as "in addition to the standard dict operations"?
Because defaultdict provides __missing__ in addition to the standard dict operations. Is there a problem with the current docs? -- Steve
On Fri, Apr 22, 2016 at 12:00 PM, Steven D'Aprano <steve@pearwood.info> wrote:
"As a class or object dictionary". Consider:
Now, what can I replace that with? A regular dict works fine:
But defining __missing__ doesn't create attributes automatically:
And neither does overriding __getitem__:
The question is a fair one. What can you do to make an object's dictionary provide attributes that weren't put there with __setitem__? ChrisA
On Thu, Apr 21, 2016, at 22:00, Steven D'Aprano wrote:
I don't understand your question. Or rather, if I have understood it, the question has a trivial answer: you don't have to override anything.
I thought it went without saying that I want to be able to get items other than by having them already stored in the dict under the exact key. Like if I wanted to make a case-insensitive dict, or one where numbers are equivalent to strings, or have it materialize a default value for missing keys.
Well, I was assuming it doesn't call *more than one* method for the specific task of looking up an attribute by name, even if there are other methods for other tasks that I would have to override to get the whole package of what behavior I want. As near as I can tell, it doesn't call *any method at all*, but instead reaches inside PyDictObject's internal structure.
Right, but I'm asking which method *one particular* expression calls. This is more like "Which method on an object implements the + operator", only it's "Which method on an object's type's namespace dict implements the ability to look up attributes on that object?" In code terms: class mydict(dict): def ????????(self, name): if name == 'foo': return 'bar' mytype = type('C',(),mydict()) => desired result: mytype().foo == mytype.foo == 1
Given a type, how do I get a reference to the dict instance that was passed to the type's constructor? Or, in code terms:
def f(x): ???????? => desired result: f(mytype) is mydict
Fix the other code, because it's wrong. If you can't, monkey-patch its builtins.
When I read it earlier, the wording in defaultdict's documentation seemed to suggest that what it provides is the ability to define a __missing__ method and have it be called - and that, itself, *is* a "standard dict operation" - rather than an implementation of the method. It looks like I misinterpreted it though.
On Thu, Apr 21, 2016 at 10:35:01PM -0400, Random832 wrote:
Obviously not.
"Items"? Like items in a dict? Then you already have at least two ways: you override __getitem__, or add a __missing__ method to the dict. What makes you think these techniques don't work?
Where does attribute lookup come into this?
Are you still talking about *attribute lookup* or are you back to *key lookup*? Honestly Random, you have to be clear as to which you want, because they are different thing. You can't just jump backwards and forwards between talking about dicts and attributes and expect people to understand what you mean. Classes and instances may not have a __dict__ at all, if they define __slots__ instead, or if they are builtins: py> (1).__dict__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'int' object has no attribute '__dict__'
An excellent example, because the answer is *two* methods, __add__ and __radd__. You've made my point for me again.
only it's "Which method on an object's type's namespace dict implements the ability to look up attributes on that object?"
What makes you think that such a method exists? Have you read the rest of this thread? If not, I suggest you do, because the thread is all about making unjustified assumptions about how objects are implemented. You are doing it again. Where is it documented that there is a method on the object __dict__ (if such a __dict__ even exists!) that does this? So you are asking the wrong question. Don't ask about an implementation detail that you *assume* exists. You should ask, not "which method", but "how do I customise attribute lookup?" And you know the standard answer to that: override __getattribute__ or __getattr__. Do they not solve your problem? Another smart question might be, what are the constraints and rules for customizing behaviour by setting the object __dict__ to something other than a regular builtin dict? I don't know the answer to that. But don't make assumptions about the implementation, and having made those assumptions, assume that everyone else shares them and that they "go without saying". *Especially not* in a thread that talks about how silly it is to make those assumptions.
Given a type, how do I get a reference to the dict instance that was passed to the type's constructor?
I believe that the answer to that is, you can't.
You cannot assume that it is "wrong" just because it is inconvenient for you. Do you seriously think that monkey-patching the built-ins in production code is a good idea? -- Steve
On Thu, Apr 21, 2016, at 23:49, Steven D'Aprano wrote:
The fact that I tried them and they didn't work.
Where does attribute lookup come into this?
Because looking up an attribute implies getting an item from the object or class's __dict__ with a string key of the name of the attribute (see below for the documented basis for this assumption). How are you not following this? I don't believe you're not messing with me.
Classes and instances may not have a __dict__ at all, if they define __slots__ instead, or if they are builtins:
If a class defines __slots__, the *instances* don't have a __dict__, but the *class* sure as hell still does. Even, as it turns out, if the metaclass had __slots__. (Not sure what's up with that, actually).
My main concern here is the class dict. So, let's see... ### Class attribute references are translated to lookups in this dictionary, e.g., C.x is translated to C.__dict__["x"] Now, that is technically true (C.__dict__ is, we've established, not the actual dict, but a "mappingproxy" object), but this behavior itself contradicts the documentation: ### [type] With three arguments, [...] and the dict dictionary is the namespace containing definitions for class body and becomes the __dict__ attribute. Except, it *doesn't* become the __dict__ attribute - its contents are *copied* into the __dict__ object, which is a new "mappingproxy" whose contents will not reflect further updates to the dict that was passed in. And regarding the object __dict__, when such a __dict__ *does* exist (since, unlike class dicts, you actually can set object dicts to be arbitrary dict subclasses) ### The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.
I hadn't yet realized, when I asked this question, that the class __dict__ "mappingproxy" is a new object (that isn't even a dict! "A class has a namespace implemented by a dictionary object" was also wrong) and doesn't retain a reference to the passed-in dict nor reflect changes to it.
Random832 <random832@fastmail.com> writes:
No, that's not what it implies. The ‘__dict__’ of an object is an implementation detail, and is not necessarily used for attribute lookup. As the documentation says: A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g., C.x is translated to C.__dict__["x"] (although there are a number of hooks which allow for other means of locating attributes). <URL:https://docs.python.org/3.5/reference/datamodel.html> So no, attribute lookup does not imply getting an item from any particular dictionary. Nothing in the documentation implies that — if you find an exception, please file a bug report for that part of the documentation.
How are you not following this? I don't believe you're not messing with me.
Steven follows quite well; he is trying to get you to explain your meaning so we can find where the mismatch is.
That's right. As documented, attribute lookup does not imply any dictionary lookup for the attribute. Attribute lookup by name is one thing. Dictionary lookup by key is quite a different thing. Please let's keep the two distinct in any discussions. -- \ “Ubi dubium, ibi libertas.” (“Where there is doubt, there is | `\ freedom.”) | _o__) | Ben Finney
On Fri, Apr 22, 2016 at 3:17 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Okay. That definitely implies that the class itself has a real dict, though. And it should be possible, using a metaclass, to manipulate that, right? prepare: (<class '__main__.DemoMeta'>, 'Demo', ()) new: (<class '__main__.DemoMeta'>, 'Demo', (), AutoCreateDict{'__qualname__': 'Demo', '__module__': "<'__name__'>"})
Maybe I'm just misunderstanding how metaclasses should be written, but this seems like it ought to work. And it's not making any use of the AutoCreateDict - despite __prepare__ and __new__ clearly being called, the latter with an AutoCreateDict instance. But by the time it gets actually attached to the dictionary, we have a mappingproxy that ignores __missing__. The docs say "are translated to", implying that this will actually be equivalent. Using a non-dict-subclass results in prompt rejection in the type() constructor: prepare: (<class '__main__.DemoMeta'>, 'Demo', ()) DictLike get __name__ DictLike set __module__ <'__name__'> DictLike set __qualname__ Demo new: (<class '__main__.DemoMeta'>, 'Demo', (), <__main__.DictLike object at 0x7f2b41b96ac8>) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __new__ TypeError: type() argument 3 must be dict, not DictLike Is there any way to make use of this documented transformation? ChrisA
On 04/21/2016 11:06 PM, Chris Angelico wrote:
On Fri, Apr 22, 2016 at 3:17 PM, Ben Finney wrote:
Random832 writes:
A class does have a real dict() -- not a subclass, not a look-a-like, but a real, honest-to-goodness, freshly minted {}.
And it should be possible, using a metaclass, to manipulate that, right?
Only for the purposes of class creation. One of the final steps of creating a class (after __prepare__ and __new__ have been called) is to copy everything from whatever dict (sub)class you used into a brand-new dict.
Maybe I'm just misunderstanding how metaclasses should be written, but this seems like it ought to work.
You have the metaclass portion right.
Well, it would if you tried creating any attributes without values during class creation.
No, it's a dict -- we just don't get to directly fiddle with it (hence the return of a mappingproxy).
The docs say "are translated to", implying that this will actually be equivalent.
Is there any way to make use of this documented transformation?
There are already many hooks in place to supplement or replace attribute lookup -- use them instead. :) -- ~Ethan~
On 22 April 2016 at 16:06, Chris Angelico <rosuav@gmail.com> wrote:
The mapping used during class body execution can be customised via __prepare__, but the resulting contents of that mapping are still copied to a regular dictionary when constructing the class object itself. We deliberately don't provide a mechanism to customise the runtime dictionary used by object instances, regardless of whether they're normal instances or type definitions. In combination with the __dict__ descriptor only exposing a mapping proxy, this ensures that all Python level modifications to the contents go through the descriptor machinery - you can't get your hands on a mutable pointer to the post-creation namespace. It looks like there *is* a missing detail in the data model docs in relation to this, though: https://docs.python.org/3/reference/datamodel.html#creating-the-class-object should state explicitly that the namespace contents are copied to a plain dict (which is then never exposed directly to Python code), but it doesn't. Cheers, Nick. P.S. I actually played around with an experimental interpreter build that dropped the copy-to-a-new-namespace step back when I was working on https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes. It's astonishingly broken in the number of ways it offers to corrupt the interpreter state (since you can entirely bypass the descriptor machinery, which the rest of the interpreter expects to be impossible if you're not messing about with ctypes or C extensions), but kinda fun in the quirky action at a distance it makes possible :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Apr 22, 2016 at 11:24 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Okay, that explains my confusion, at least :) I can't think of any situations where I actually *want* a dict subclass (__getattr[ibute]__ or descriptors can do everything I can come up with), but it'd be good to document that that's not possible. ChrisA
On Fri, Apr 22, 2016 at 03:17:34PM +1000, Ben Finney wrote:
I think it's a bit more than an implementation detail. For example, I don't think it would be legitimate for (let's say) FooPython to decide to use instance.__tree__ (a binary tree) for instance attribute look-ups. But I think that the details are far more complex than "attribute lookups are done by key access to a dict __dict__" (for example, if attribute lookups are done by inspecting obj.__dict__, how do you look up obj.__dict__?). And I don't think that the docs suggest that they are replacable by dict subclasses.
Not quite: my first email was genuinely confused. When I said I didn't understand what Random was trying to say, I meant it. By my second email, or at least the end of it, I could guess what he was trying to do, which (I think) is to replace __dict__ with a subclass instance in order to customise attribute lookup. -- Steve
On Fri, Apr 22, 2016 at 12:45:34AM -0400, Random832 wrote: [...]
That is a very good point. I think that's a documentation bug.
True, but the documentation doesn't say that attribute lookup goes through the *full* dict key lookup, including support of __missing__ and __getitem__. I'll grant you that neither does the documentation say that it doesn't, so I'd call this a documentation bug. -- Steve
On 04/22/2016 04:16 AM, Steven D'Aprano wrote:
On Fri, Apr 22, 2016 at 12:45:34AM -0400, Random832 wrote:
No, it's a real dict() -- we just don't get direct access to it anymore.
See above about the 'mappingproxy'. As for updates, IIRC setattr() is the way to make changes these days.
That is a very good point. I think that's a documentation bug.
Patches welcome. ;) -- ~Ethan~
On Fri, Apr 22, 2016 at 5:16 AM, Steven D'Aprano <steve@pearwood.info> wrote:
mappingproxy (a.k.a. types.MappingProxyType) is exactly what it says: a proxy for a mapping. Basically, it is a wrapper around a collections.abc.Mapping. The namespace (a dict or dict sub-class, e.g. OrderedDict) passed to type() is copied into a new dict and the new type's __dict__ is set to a mappingproxy that wraps that copy. So if there is a documentation bug then it is the ambiguity of the word "becomes". Perhaps it would be more correct as "is copied into".
What would you say is the specific documentation bug? That the default attribute lookup (object.__getattribute()) does not use the obj.__dict__ attribute but rather the dict it points to (if it knows about it)? Or just that anything set to __dict__ is not guaranteed to be honored by the default __getattribute__()? -eric
On Thu, Apr 21, 2016 at 10:45 PM, Random832 <random832@fastmail.com> wrote:
Attribute access and item access communicate different things about the namespaces on which they operate. The fact that objects use mappings under the hood for the default attribute access behavior is an implementation detail. Python does not document any mechanism to hook arbitrary mappings into the default attribute access behavior (i.e. object.__getattribute__()).
Not exactly. You still have to factor in descriptors (including slots). [1] In the absence of those then you are correct that the current implementation of type.__getattribute__() (not the same as object.__getattribute__(), BTW) is a lookup on the type's __dict__. However, that is done directly on tp_dict and not on <TYPE>.__dict__, if you want to talk about implementation details. Of course, the point is moot, as you've pointed out, since a type's __dict__ is both unsettable and a read-only view.
Again, you also have to factor in descriptors. [2] Regardless of that, it's important to realize that object.__getattribute__() doesn't use any object's __dict__ attribute for any of the lookups you've identified. This is because you can't really look up the object's __dict__ attribute *while* doing attribute lookup. It's the same way that the following doesn't work: class Spam: def __getattribute__(self, name): if self.__class__ == Spam: # infinite recursion!!! ... return object.__getattribute__(self, name) Instead you have to do this: class Spam: def __getattribute__(self, name): if object.__getattribute__(self, "__class__") == Spam: ... return object.__getattribute__(self, name) Hence, object.__getattribute__() only does lookup on the dict identified through the type's tp_dictoffset field. However, as you've noted, objects of custom classes have a settable __dict__ attribute. This is because by default the mapping is tied to the object at the tp_dictoffset of the object's type. [3] Notably, the mapping must be of type dict or of a dict subclass. What this implies to me is that someone went to the trouble of allowing folks to use some other dict (or OrderedDict, etc.) than the one you get by default with a new object. However, either they felt that using a non-dict mapping type was asking for too much trouble, there were performance concerns, or they did not want to go to the effort to fix all the places that expect __dict__ to be an actual dict. It's probably all three. Keep in mind that even with a dict subclass the implementation of object.__getattribute__() can't sensibly use normal lookup on when it does the lookup on the underlying namespace dict. Instead it uses PyDict_GetItem(), which is basically equivalent to calling dict.__getitem__(ns, attr). Hence that underlying mapping must be a dict or dict subclass, and any overridden __getitem__() method is ignored. -eric [1] https://hg.python.org/cpython/file/default/Objects/typeobject.c#l2924 [2] https://hg.python.org/cpython/file/default/Objects/object.c#l1028 [3] https://hg.python.org/cpython/file/default/Objects/object.c#l1195
On 21 April 2016 at 12:51, Neil Girdhar <mistersheik@gmail.com> wrote:
Builtins can be extended, you just have to override all the methods where you want to change the return type:
This isn't hard as such, it's just tedious, so it's often simpler to use the more subclass friendly variants that dynamically look up the type to return and hence let you get away with overriding a smaller subset of the methods. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Apr 21, 2016 at 04:36:54PM +1000, Nick Coghlan wrote:
Builtins can be extended, you just have to override all the methods where you want to change the return type:
I wonder whether we should have a class decorator which automatically adds the appropriate methods? It would need some sort of introspection to look at the superclass and adds overridden methods? E.g. if the superclass is float, it would add a bunch of methods like: def __add__(self, other): x = super().__add__(other) if x is NotImplemented: return x return type(self)(x) but only if they aren't already overridden. Then you could do this: @decorator # I have no idea what to call it. class MyInt(int): pass and now MyInt() + MyInt() will return a MyInt, rather than a regular int. Getting the list of dunder methods is easy, but telling whether or not they should return an instance of the subclass may not be. Thoughts? -- Steve
On Thu, Apr 21, 2016 at 7:30 AM Steven D'Aprano <steve@pearwood.info> wrote:
If I'm not mistaken, some of the dunders need to be overridden as part of the class definition and can't be added dynamically without some exec a la namedtuple. If so, are you still interested in creating that decorator?
On Thu, Apr 21, 2016 at 9:44 PM, Michael Selik <mike@selik.org> wrote:
Which ones? __new__ is already covered, and AFAIK all operator dunders can be injected just fine. def operators_return_subclass(cls): if len(cls.__bases__) != 1: raise ValueError("Need to have exactly one base class") base, = cls.__bases__ def dunder(name): def flip_to_subclass(self, other): x = getattr(super(cls, self), name)(other) if type(x) is base: return cls(x) return x setattr(cls, name, flip_to_subclass) for meth in "add", "sub", "mul", "div": dunder("__"+meth+"__") return cls @operators_return_subclass class hexint(int): def __repr__(self): return hex(self) I'm not sure what set of dunders should be included though. I've gone extra strict in this version; for instance, hexint/2 --> float, not hexint. You could go either way. ChrisA
On 04/20/2016 07:51 PM, Neil Girdhar wrote:
How, exactly, does that help? And even if it does help (which I doubt), you're willing to break currently working code that successfully subclasses dicts, strings, tuples, etc., that then pass through dicts, strings, tuples, etc., to build their new class? No thank you. -- ~Ethan~
On Wed, Apr 20, 2016 at 07:51:37PM -0700, Neil Girdhar wrote:
-1 What about those who don't want or need to inherit from UserDict, and are perfectly happy with inheriting from dict? Why should we break their working code for the sake of people whose code already isn't working? Not everyone who inherits from dict tries to override __getitem__ and __setitem__ and are then surprised that other methods don't call their overridden methods. We shouldn't punish them. -- Steve
I think inheriting directly from dict is simply bad code because CPython doesn't promise that any of your overridden methods will be called. The fact that it silently doesn't call them is an inscrutable trap. And really, it's not much of a "punishment" to simply change your base class name. On Wed, Apr 20, 2016 at 11:16 PM Steven D'Aprano <steve@pearwood.info> wrote:
Except from the fact that a lot of code requiring actual dicts (or str, list…) will break because they are no longer receiving a dict instance (or an instance of (one of) its subclasses), but a collections.UserDict instance which is also entirely unrelated to dict except that they have methods with similar names. As I understand it, you prefer to put the burden of supporting collections.UserDict not on the people subclassing dict, but on the library developers expecting actual dicts. I’m sure there are a couple of built-in functions (written in C) accepting dicts that would not work if they didn’t get dicts. And getting built-in functions to accept instances of pure Python code is nontrivial and tricky at best, and requires a lot more work than it would make sense to in this case. I’m -1 on the idea. -Emanuel From: Neil Girdhar Sent: Thursday, April 21, 2016 12:34 AM I think inheriting directly from dict is simply bad code because CPython doesn't promise that any of your overridden methods will be called. The fact that it silently doesn't call them is an inscrutable trap. And really, it's not much of a "punishment" to simply change your base class name.
On Thu, Apr 21, 2016 at 2:33 PM, Neil Girdhar <mistersheik@gmail.com> wrote:
There are way too many cases that work just fine, though.
Why should I inherit from UserDict instead? I have to import that from somewhere (is it in types? collections? though presumably your error message would tell me that), and then I have to contend with the fact that my class is no longer a dictionary.
If you *do* push forward with this proposal, incidentally, I would recommend not doing it in __new__, but changing it so the class is no longer subclassable, as per bool:
But I am firmly -1 on disallowing dict subclasses just because they aren't guaranteed to call all of your overridden methods. ChrisA
On Thu, Apr 21, 2016 at 1:15 AM Chris Angelico <rosuav@gmail.com> wrote:
Of course it's a dictionary. It's an abc.Mapping, which is all a user of your class should care about. After all, it "quacks like a duck", which is all that matters.
That's a fair point, but it seems like a bug in JSON. They should have checked if it's an abc.Mapping imho.
Good point.
On Thu, Apr 21, 2016 at 3:33 PM, Neil Girdhar <mistersheik@gmail.com> wrote:
Your first sentence is in conflict with your other statements. My type is simply *not* a dictionary. You can start raising bug reports all over the place saying "JSON should look for abc.Mapping rather than dict", but I'm not even sure that it should. ChrisA
Lol, well, I think it should. Anyway, the cons are that yes some people's bad code won't work. The pros are that unsuspecting users won't fall into the trap associated with inheriting from dict, list or set. Of course, people on python-ideas know the pitfalls of inheriting from builtin types. You're not users that benefit from this change. It's people who don't know the problems and have no way of finding out until they spend a day debugging why overriding some method on a derived class of a derived class that ultimately inherits from dict doesn't work. Experts always want to make their own lives better. That's the problem with a language made by experts. New users fall into traps that you will never fall into (again). On Thu, Apr 21, 2016 at 1:48 AM Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Apr 21, 2016 at 1:58 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I feel your pain. However, what you see as a failure or defect of the builtins is actually a feature! :-) The downstream programmer should not be expected to know the implementation details of a dict. You don't want to read the source, you just want to use the public interface. You should be able to subclass and override methods as you like without worrying about hidden internal relationships. You should be able to override ``__getitem__`` without accidentally affecting things like ``values``. I guess it's a case of "a little knowledge is dangerous". Someone who knows nothing about dict implementation would not expect to see a change in method A because of an override of method B. The expert is keeping the novice safe. It's the journeyman who suffers.
Wow, I disagree with this. The fact that you can override getitem and it doesn't fix initialization in the constructor to match is definitely confusing. The fact that __add__ called on a dict subclass doesn't return a class of the same type is confusing. This is why every Python book advises people not to inherit from builtins. Let's not get carried away. This is an optimization for CPython — not a feature. On Thu, Apr 21, 2016 at 3:27 AM Michael Selik <mike@selik.org> wrote:
On Thu, Apr 21, 2016 at 04:33:33AM +0000, Neil Girdhar wrote:
I think inheriting directly from dict is simply bad code because CPython doesn't promise that any of your overridden methods will be called.
Python makes the same promise that it makes for *all* classes: if you override a method, and call that method on an instance of a subclass, the subclass' overridden implementation will be called. Dicts are no different from any other class, and have been since version 2.2 when types and classes where unified and inheriting from built-ins was first allowed. In fact, dicts were *explicitly* listed by Guido as one of the built-ins which can be subclassed: "Let's start with the juiciest bit: you can subtype built-in types like dictionaries and lists." https://www.python.org/download/releases/2.2.3/descrintro/#subclassing Your proposal would break at least two standard types, both of which subclass dict: py> from collections import defaultdict, Counter py> defaultdict.__mro__ (<class 'collections.defaultdict'>, <class 'dict'>, <class 'object'>) py> Counter.__mro__ (<class 'collections.Counter'>, <class 'dict'>, <class 'object'>) and it runs counter to a documented feature of dicts, that they can be subclassed and given a __missing__ method: "If a SUBCLASS OF DICT [emphasis added] defines a method __missing__() and key is not present, the d[key] operation calls that method ..." https://docs.python.org/2/library/stdtypes.html#mapping-types-dict
The fact that it silently doesn't call them is an inscrutable trap.
That is not a fact. What you should say is: "The fact that dicts don't obey the undocumented assumptions I made about the implementation of methods is a trap." and I will agree: correct, but its a trap for hubris and foolishness, and the answer to that is, don't make unjustified assumptions about how dicts are implemented. And I can say that because I made exactly that wrong assumption too. Nowhere does the documentation say that dict.update calls __setitem__. Nowhere does it say that dict.clear calls __delitem__. Nowhere does it say that dict.values calls __getitem__. And yet I assumed that they did all that. When somebody makes the unjustified assumption that they do, then they will discover that calling the subclass' clear method fails to call the overridden __delitem__. It also fails to call the overridden __str__. The only difference is that nobody assumes that clear() calls __str__, but many people foolishly assume that it calls the __delitem__ method. Both assumptions have *exactly* the same justification: none at all. I made a bunch of stupid, foolish assumptions, based on absolutely nothing more than the idea that it stands to reason that dict must be implemented in this way, and got bitten. And I deserved it. I was wrong, and anyone making the same assumption is wrong. Well, I say I was "bitten", but that over-dramatises the situation. What actually happened was that I wrote a subclass without doing any tests, then tested it, and discovered that it didn't work how I expected. I spent a few minutes playing with the class, added a few print statements, discovered that my assumptions were wrong, and then overrode the classes I actually wanted to override. Twenty minutes of googling and reading the docs convinced me that, no, Python doesn't promise that dict.clear calls __delitem__. It would have been five minutes except I was especially stubbon and pig-headed that day and didn't want to admit that I was in the wrong. But I was. Getting bitten by this was, in fact, a valuable lesson. I learned what I should have already known, what I had *intellectually* known but had ignored because "it stands to reason". Namely, if an implementation isn't documented, you cannot assume that it works in a particular way. My sympathy level is zero, and my support for this proposal is negative. -- Steve
On Thu, Apr 21, 2016, at 07:17, Steven D'Aprano wrote:
So what method should be overridden to make a dict subclass useful as a class or object dictionary (i.e. for attribute lookup to work with names that have not been stored with dict.__setitem__)? Overriding __getitem__ or __missing__ doesn't work. My only consolation is that defaultdict doesn't work either. I can't even figure out how to get the real class dict, as I would need if I were overriding __getattribute__ explicitly in the metaclass (which also doesn't work) - cls.__dict__ returns a mappingproxy. Alternatively, where, other than object and class dicts, are you actually required to have a subclass of dict rather than a UserDict or other duck-typed mapping? Incidentally, why is __missing__ documented under defaultdict as "in addition to the standard dict operations"?
On Thu, Apr 21, 2016 at 09:43:17AM -0400, Random832 wrote:
I don't understand your question. Or rather, if I have understood it, the question has a trivial answer: you don't have to override anything. class MyDict(dict): pass d = MyDict() d.attr = 1 print(d.attr) will do what you appear to be asking. If that's not what you actually mean, then you need to explain more carefully. I will also point out that your question is based on a point of confusion. You ask: "So what method should be overridden..." but the answer to this in general must be "What makes you think a single method is sufficient?" This doesn't just apply to dicts, it applies in general to any class. "What method do I override to make a subclass of int implement arithmetic with wrap-around (e.g. 999+1 = 0, 501*2 = 2)?"
This doesn't make anything any clearer for me.
Anywhere you have to operate with code that does "if isinstance(x, dict)" checks.
Incidentally, why is __missing__ documented under defaultdict as "in addition to the standard dict operations"?
Because defaultdict provides __missing__ in addition to the standard dict operations. Is there a problem with the current docs? -- Steve
On Fri, Apr 22, 2016 at 12:00 PM, Steven D'Aprano <steve@pearwood.info> wrote:
"As a class or object dictionary". Consider:
Now, what can I replace that with? A regular dict works fine:
But defining __missing__ doesn't create attributes automatically:
And neither does overriding __getitem__:
The question is a fair one. What can you do to make an object's dictionary provide attributes that weren't put there with __setitem__? ChrisA
On Thu, Apr 21, 2016, at 22:00, Steven D'Aprano wrote:
I don't understand your question. Or rather, if I have understood it, the question has a trivial answer: you don't have to override anything.
I thought it went without saying that I want to be able to get items other than by having them already stored in the dict under the exact key. Like if I wanted to make a case-insensitive dict, or one where numbers are equivalent to strings, or have it materialize a default value for missing keys.
Well, I was assuming it doesn't call *more than one* method for the specific task of looking up an attribute by name, even if there are other methods for other tasks that I would have to override to get the whole package of what behavior I want. As near as I can tell, it doesn't call *any method at all*, but instead reaches inside PyDictObject's internal structure.
Right, but I'm asking which method *one particular* expression calls. This is more like "Which method on an object implements the + operator", only it's "Which method on an object's type's namespace dict implements the ability to look up attributes on that object?" In code terms: class mydict(dict): def ????????(self, name): if name == 'foo': return 'bar' mytype = type('C',(),mydict()) => desired result: mytype().foo == mytype.foo == 1
Given a type, how do I get a reference to the dict instance that was passed to the type's constructor? Or, in code terms:
def f(x): ???????? => desired result: f(mytype) is mydict
Fix the other code, because it's wrong. If you can't, monkey-patch its builtins.
When I read it earlier, the wording in defaultdict's documentation seemed to suggest that what it provides is the ability to define a __missing__ method and have it be called - and that, itself, *is* a "standard dict operation" - rather than an implementation of the method. It looks like I misinterpreted it though.
On Thu, Apr 21, 2016 at 10:35:01PM -0400, Random832 wrote:
Obviously not.
"Items"? Like items in a dict? Then you already have at least two ways: you override __getitem__, or add a __missing__ method to the dict. What makes you think these techniques don't work?
Where does attribute lookup come into this?
Are you still talking about *attribute lookup* or are you back to *key lookup*? Honestly Random, you have to be clear as to which you want, because they are different thing. You can't just jump backwards and forwards between talking about dicts and attributes and expect people to understand what you mean. Classes and instances may not have a __dict__ at all, if they define __slots__ instead, or if they are builtins: py> (1).__dict__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'int' object has no attribute '__dict__'
An excellent example, because the answer is *two* methods, __add__ and __radd__. You've made my point for me again.
only it's "Which method on an object's type's namespace dict implements the ability to look up attributes on that object?"
What makes you think that such a method exists? Have you read the rest of this thread? If not, I suggest you do, because the thread is all about making unjustified assumptions about how objects are implemented. You are doing it again. Where is it documented that there is a method on the object __dict__ (if such a __dict__ even exists!) that does this? So you are asking the wrong question. Don't ask about an implementation detail that you *assume* exists. You should ask, not "which method", but "how do I customise attribute lookup?" And you know the standard answer to that: override __getattribute__ or __getattr__. Do they not solve your problem? Another smart question might be, what are the constraints and rules for customizing behaviour by setting the object __dict__ to something other than a regular builtin dict? I don't know the answer to that. But don't make assumptions about the implementation, and having made those assumptions, assume that everyone else shares them and that they "go without saying". *Especially not* in a thread that talks about how silly it is to make those assumptions.
Given a type, how do I get a reference to the dict instance that was passed to the type's constructor?
I believe that the answer to that is, you can't.
You cannot assume that it is "wrong" just because it is inconvenient for you. Do you seriously think that monkey-patching the built-ins in production code is a good idea? -- Steve
On Thu, Apr 21, 2016, at 23:49, Steven D'Aprano wrote:
The fact that I tried them and they didn't work.
Where does attribute lookup come into this?
Because looking up an attribute implies getting an item from the object or class's __dict__ with a string key of the name of the attribute (see below for the documented basis for this assumption). How are you not following this? I don't believe you're not messing with me.
Classes and instances may not have a __dict__ at all, if they define __slots__ instead, or if they are builtins:
If a class defines __slots__, the *instances* don't have a __dict__, but the *class* sure as hell still does. Even, as it turns out, if the metaclass had __slots__. (Not sure what's up with that, actually).
My main concern here is the class dict. So, let's see... ### Class attribute references are translated to lookups in this dictionary, e.g., C.x is translated to C.__dict__["x"] Now, that is technically true (C.__dict__ is, we've established, not the actual dict, but a "mappingproxy" object), but this behavior itself contradicts the documentation: ### [type] With three arguments, [...] and the dict dictionary is the namespace containing definitions for class body and becomes the __dict__ attribute. Except, it *doesn't* become the __dict__ attribute - its contents are *copied* into the __dict__ object, which is a new "mappingproxy" whose contents will not reflect further updates to the dict that was passed in. And regarding the object __dict__, when such a __dict__ *does* exist (since, unlike class dicts, you actually can set object dicts to be arbitrary dict subclasses) ### The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.
I hadn't yet realized, when I asked this question, that the class __dict__ "mappingproxy" is a new object (that isn't even a dict! "A class has a namespace implemented by a dictionary object" was also wrong) and doesn't retain a reference to the passed-in dict nor reflect changes to it.
Random832 <random832@fastmail.com> writes:
No, that's not what it implies. The ‘__dict__’ of an object is an implementation detail, and is not necessarily used for attribute lookup. As the documentation says: A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g., C.x is translated to C.__dict__["x"] (although there are a number of hooks which allow for other means of locating attributes). <URL:https://docs.python.org/3.5/reference/datamodel.html> So no, attribute lookup does not imply getting an item from any particular dictionary. Nothing in the documentation implies that — if you find an exception, please file a bug report for that part of the documentation.
How are you not following this? I don't believe you're not messing with me.
Steven follows quite well; he is trying to get you to explain your meaning so we can find where the mismatch is.
That's right. As documented, attribute lookup does not imply any dictionary lookup for the attribute. Attribute lookup by name is one thing. Dictionary lookup by key is quite a different thing. Please let's keep the two distinct in any discussions. -- \ “Ubi dubium, ibi libertas.” (“Where there is doubt, there is | `\ freedom.”) | _o__) | Ben Finney
On Fri, Apr 22, 2016 at 3:17 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Okay. That definitely implies that the class itself has a real dict, though. And it should be possible, using a metaclass, to manipulate that, right? prepare: (<class '__main__.DemoMeta'>, 'Demo', ()) new: (<class '__main__.DemoMeta'>, 'Demo', (), AutoCreateDict{'__qualname__': 'Demo', '__module__': "<'__name__'>"})
Maybe I'm just misunderstanding how metaclasses should be written, but this seems like it ought to work. And it's not making any use of the AutoCreateDict - despite __prepare__ and __new__ clearly being called, the latter with an AutoCreateDict instance. But by the time it gets actually attached to the dictionary, we have a mappingproxy that ignores __missing__. The docs say "are translated to", implying that this will actually be equivalent. Using a non-dict-subclass results in prompt rejection in the type() constructor: prepare: (<class '__main__.DemoMeta'>, 'Demo', ()) DictLike get __name__ DictLike set __module__ <'__name__'> DictLike set __qualname__ Demo new: (<class '__main__.DemoMeta'>, 'Demo', (), <__main__.DictLike object at 0x7f2b41b96ac8>) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __new__ TypeError: type() argument 3 must be dict, not DictLike Is there any way to make use of this documented transformation? ChrisA
On 04/21/2016 11:06 PM, Chris Angelico wrote:
On Fri, Apr 22, 2016 at 3:17 PM, Ben Finney wrote:
Random832 writes:
A class does have a real dict() -- not a subclass, not a look-a-like, but a real, honest-to-goodness, freshly minted {}.
And it should be possible, using a metaclass, to manipulate that, right?
Only for the purposes of class creation. One of the final steps of creating a class (after __prepare__ and __new__ have been called) is to copy everything from whatever dict (sub)class you used into a brand-new dict.
Maybe I'm just misunderstanding how metaclasses should be written, but this seems like it ought to work.
You have the metaclass portion right.
Well, it would if you tried creating any attributes without values during class creation.
No, it's a dict -- we just don't get to directly fiddle with it (hence the return of a mappingproxy).
The docs say "are translated to", implying that this will actually be equivalent.
Is there any way to make use of this documented transformation?
There are already many hooks in place to supplement or replace attribute lookup -- use them instead. :) -- ~Ethan~
On 22 April 2016 at 16:06, Chris Angelico <rosuav@gmail.com> wrote:
The mapping used during class body execution can be customised via __prepare__, but the resulting contents of that mapping are still copied to a regular dictionary when constructing the class object itself. We deliberately don't provide a mechanism to customise the runtime dictionary used by object instances, regardless of whether they're normal instances or type definitions. In combination with the __dict__ descriptor only exposing a mapping proxy, this ensures that all Python level modifications to the contents go through the descriptor machinery - you can't get your hands on a mutable pointer to the post-creation namespace. It looks like there *is* a missing detail in the data model docs in relation to this, though: https://docs.python.org/3/reference/datamodel.html#creating-the-class-object should state explicitly that the namespace contents are copied to a plain dict (which is then never exposed directly to Python code), but it doesn't. Cheers, Nick. P.S. I actually played around with an experimental interpreter build that dropped the copy-to-a-new-namespace step back when I was working on https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes. It's astonishingly broken in the number of ways it offers to corrupt the interpreter state (since you can entirely bypass the descriptor machinery, which the rest of the interpreter expects to be impossible if you're not messing about with ctypes or C extensions), but kinda fun in the quirky action at a distance it makes possible :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Apr 22, 2016 at 11:24 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Okay, that explains my confusion, at least :) I can't think of any situations where I actually *want* a dict subclass (__getattr[ibute]__ or descriptors can do everything I can come up with), but it'd be good to document that that's not possible. ChrisA
On Fri, Apr 22, 2016 at 03:17:34PM +1000, Ben Finney wrote:
I think it's a bit more than an implementation detail. For example, I don't think it would be legitimate for (let's say) FooPython to decide to use instance.__tree__ (a binary tree) for instance attribute look-ups. But I think that the details are far more complex than "attribute lookups are done by key access to a dict __dict__" (for example, if attribute lookups are done by inspecting obj.__dict__, how do you look up obj.__dict__?). And I don't think that the docs suggest that they are replacable by dict subclasses.
Not quite: my first email was genuinely confused. When I said I didn't understand what Random was trying to say, I meant it. By my second email, or at least the end of it, I could guess what he was trying to do, which (I think) is to replace __dict__ with a subclass instance in order to customise attribute lookup. -- Steve
On Fri, Apr 22, 2016 at 12:45:34AM -0400, Random832 wrote: [...]
That is a very good point. I think that's a documentation bug.
True, but the documentation doesn't say that attribute lookup goes through the *full* dict key lookup, including support of __missing__ and __getitem__. I'll grant you that neither does the documentation say that it doesn't, so I'd call this a documentation bug. -- Steve
On 04/22/2016 04:16 AM, Steven D'Aprano wrote:
On Fri, Apr 22, 2016 at 12:45:34AM -0400, Random832 wrote:
No, it's a real dict() -- we just don't get direct access to it anymore.
See above about the 'mappingproxy'. As for updates, IIRC setattr() is the way to make changes these days.
That is a very good point. I think that's a documentation bug.
Patches welcome. ;) -- ~Ethan~
On Fri, Apr 22, 2016 at 5:16 AM, Steven D'Aprano <steve@pearwood.info> wrote:
mappingproxy (a.k.a. types.MappingProxyType) is exactly what it says: a proxy for a mapping. Basically, it is a wrapper around a collections.abc.Mapping. The namespace (a dict or dict sub-class, e.g. OrderedDict) passed to type() is copied into a new dict and the new type's __dict__ is set to a mappingproxy that wraps that copy. So if there is a documentation bug then it is the ambiguity of the word "becomes". Perhaps it would be more correct as "is copied into".
What would you say is the specific documentation bug? That the default attribute lookup (object.__getattribute()) does not use the obj.__dict__ attribute but rather the dict it points to (if it knows about it)? Or just that anything set to __dict__ is not guaranteed to be honored by the default __getattribute__()? -eric
On Thu, Apr 21, 2016 at 10:45 PM, Random832 <random832@fastmail.com> wrote:
Attribute access and item access communicate different things about the namespaces on which they operate. The fact that objects use mappings under the hood for the default attribute access behavior is an implementation detail. Python does not document any mechanism to hook arbitrary mappings into the default attribute access behavior (i.e. object.__getattribute__()).
Not exactly. You still have to factor in descriptors (including slots). [1] In the absence of those then you are correct that the current implementation of type.__getattribute__() (not the same as object.__getattribute__(), BTW) is a lookup on the type's __dict__. However, that is done directly on tp_dict and not on <TYPE>.__dict__, if you want to talk about implementation details. Of course, the point is moot, as you've pointed out, since a type's __dict__ is both unsettable and a read-only view.
Again, you also have to factor in descriptors. [2] Regardless of that, it's important to realize that object.__getattribute__() doesn't use any object's __dict__ attribute for any of the lookups you've identified. This is because you can't really look up the object's __dict__ attribute *while* doing attribute lookup. It's the same way that the following doesn't work: class Spam: def __getattribute__(self, name): if self.__class__ == Spam: # infinite recursion!!! ... return object.__getattribute__(self, name) Instead you have to do this: class Spam: def __getattribute__(self, name): if object.__getattribute__(self, "__class__") == Spam: ... return object.__getattribute__(self, name) Hence, object.__getattribute__() only does lookup on the dict identified through the type's tp_dictoffset field. However, as you've noted, objects of custom classes have a settable __dict__ attribute. This is because by default the mapping is tied to the object at the tp_dictoffset of the object's type. [3] Notably, the mapping must be of type dict or of a dict subclass. What this implies to me is that someone went to the trouble of allowing folks to use some other dict (or OrderedDict, etc.) than the one you get by default with a new object. However, either they felt that using a non-dict mapping type was asking for too much trouble, there were performance concerns, or they did not want to go to the effort to fix all the places that expect __dict__ to be an actual dict. It's probably all three. Keep in mind that even with a dict subclass the implementation of object.__getattribute__() can't sensibly use normal lookup on when it does the lookup on the underlying namespace dict. Instead it uses PyDict_GetItem(), which is basically equivalent to calling dict.__getitem__(ns, attr). Hence that underlying mapping must be a dict or dict subclass, and any overridden __getitem__() method is ignored. -eric [1] https://hg.python.org/cpython/file/default/Objects/typeobject.c#l2924 [2] https://hg.python.org/cpython/file/default/Objects/object.c#l1028 [3] https://hg.python.org/cpython/file/default/Objects/object.c#l1195
On 21 April 2016 at 12:51, Neil Girdhar <mistersheik@gmail.com> wrote:
Builtins can be extended, you just have to override all the methods where you want to change the return type:
This isn't hard as such, it's just tedious, so it's often simpler to use the more subclass friendly variants that dynamically look up the type to return and hence let you get away with overriding a smaller subset of the methods. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Apr 21, 2016 at 04:36:54PM +1000, Nick Coghlan wrote:
Builtins can be extended, you just have to override all the methods where you want to change the return type:
I wonder whether we should have a class decorator which automatically adds the appropriate methods? It would need some sort of introspection to look at the superclass and adds overridden methods? E.g. if the superclass is float, it would add a bunch of methods like: def __add__(self, other): x = super().__add__(other) if x is NotImplemented: return x return type(self)(x) but only if they aren't already overridden. Then you could do this: @decorator # I have no idea what to call it. class MyInt(int): pass and now MyInt() + MyInt() will return a MyInt, rather than a regular int. Getting the list of dunder methods is easy, but telling whether or not they should return an instance of the subclass may not be. Thoughts? -- Steve
On Thu, Apr 21, 2016 at 7:30 AM Steven D'Aprano <steve@pearwood.info> wrote:
If I'm not mistaken, some of the dunders need to be overridden as part of the class definition and can't be added dynamically without some exec a la namedtuple. If so, are you still interested in creating that decorator?
On Thu, Apr 21, 2016 at 9:44 PM, Michael Selik <mike@selik.org> wrote:
Which ones? __new__ is already covered, and AFAIK all operator dunders can be injected just fine. def operators_return_subclass(cls): if len(cls.__bases__) != 1: raise ValueError("Need to have exactly one base class") base, = cls.__bases__ def dunder(name): def flip_to_subclass(self, other): x = getattr(super(cls, self), name)(other) if type(x) is base: return cls(x) return x setattr(cls, name, flip_to_subclass) for meth in "add", "sub", "mul", "div": dunder("__"+meth+"__") return cls @operators_return_subclass class hexint(int): def __repr__(self): return hex(self) I'm not sure what set of dunders should be included though. I've gone extra strict in this version; for instance, hexint/2 --> float, not hexint. You could go either way. ChrisA
participants (12)
-
Ben Finney -
Chris Angelico -
Eric Snow -
Ethan Furman -
Greg Ewing -
Michael Selik -
Neil Girdhar -
Nick Coghlan -
Random832 -
Steven D'Aprano -
Terry Reedy -
Émanuel Barry