[Python-ideas] Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.

[Steven D'Aprano]

On Thu, Apr 21, 2016 at 04:33:33AM +0000, Neil Girdhar wrote:
> I think inheriting directly from dict is simply bad code because CPython
> doesn't promise that any of your overridden methods will be called.

Python makes the same promise that it makes for *all* classes: if you 
override a method, and call that method on an instance of a subclass, 
the subclass' overridden implementation will be called. Dicts are no 
different from any other class, and have been since version 2.2 when 
types and classes where unified and inheriting from built-ins was first 
allowed.

In fact, dicts were *explicitly* listed by Guido as one of the built-ins 
which can be subclassed:

  "Let's start with the juiciest bit: you can subtype built-in 
   types like dictionaries and lists."

https://www.python.org/download/releases/2.2.3/descrintro/#subclassing

Your proposal would break at least two standard types, both of which 
subclass dict:

py> from collections import defaultdict, Counter
py> defaultdict.__mro__
(<class 'collections.defaultdict'>, <class 'dict'>, <class 'object'>)
py> Counter.__mro__
(<class 'collections.Counter'>, <class 'dict'>, <class 'object'>)


and it runs counter to a documented feature of dicts, that they can be 
subclassed and given a __missing__ method:

"If a SUBCLASS OF DICT [emphasis added] defines a method __missing__() 
and key is not present, the d[key] operation calls that method ..."

https://docs.python.org/2/library/stdtypes.html#mapping-types-dict


> The fact that it silently doesn't call them is an inscrutable trap.

That is not a fact.

What you should say is:

"The fact that dicts don't obey the undocumented assumptions I made 
about the implementation of methods is a trap."

and I will agree: correct, but its a trap for hubris and foolishness, 
and the answer to that is, don't make unjustified assumptions about how 
dicts are implemented. And I can say that because I made exactly that 
wrong assumption too.

Nowhere does the documentation say that dict.update calls __setitem__. 
Nowhere does it say that dict.clear calls __delitem__. Nowhere does it 
say that dict.values calls __getitem__. And yet I assumed that they did 
all that.

When somebody makes the unjustified assumption that they do, then they 
will discover that calling the subclass' clear method fails to call the 
overridden __delitem__. It also fails to call the overridden __str__. 
The only difference is that nobody assumes that clear() calls __str__, 
but many people foolishly assume that it calls the __delitem__ method. 
Both assumptions have *exactly* the same justification: none at all.

I made a bunch of stupid, foolish assumptions, based on absolutely 
nothing more than the idea that it stands to reason that dict must be 
implemented in this way, and got bitten. And I deserved it. I was wrong, 
and anyone making the same assumption is wrong.

Well, I say I was "bitten", but that over-dramatises the situation. What 
actually happened was that I wrote a subclass without doing any tests, 
then tested it, and discovered that it didn't work how I expected. I 
spent a few minutes playing with the class, added a few print 
statements, discovered that my assumptions were wrong, and then overrode 
the classes I actually wanted to override.

Twenty minutes of googling and reading the docs convinced me that, no, 
Python doesn't promise that dict.clear calls __delitem__. It would have 
been five minutes except I was especially stubbon and pig-headed that 
day and didn't want to admit that I was in the wrong. But I was.

Getting bitten by this was, in fact, a valuable lesson. I learned what I 
should have already known, what I had *intellectually* known but had 
ignored because "it stands to reason". Namely, if an implementation 
isn't documented, you cannot assume that it works in a particular way.

My sympathy level is zero, and my support for this proposal is negative.



-- 
Steve