[Python-ideas] Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.
steve at pearwood.info
Thu Apr 21 07:17:09 EDT 2016
On Thu, Apr 21, 2016 at 04:33:33AM +0000, Neil Girdhar wrote:
> I think inheriting directly from dict is simply bad code because CPython
> doesn't promise that any of your overridden methods will be called.
Python makes the same promise that it makes for *all* classes: if you
override a method, and call that method on an instance of a subclass,
the subclass' overridden implementation will be called. Dicts are no
different from any other class, and have been since version 2.2 when
types and classes where unified and inheriting from built-ins was first
In fact, dicts were *explicitly* listed by Guido as one of the built-ins
which can be subclassed:
"Let's start with the juiciest bit: you can subtype built-in
types like dictionaries and lists."
Your proposal would break at least two standard types, both of which
py> from collections import defaultdict, Counter
(<class 'collections.defaultdict'>, <class 'dict'>, <class 'object'>)
(<class 'collections.Counter'>, <class 'dict'>, <class 'object'>)
and it runs counter to a documented feature of dicts, that they can be
subclassed and given a __missing__ method:
"If a SUBCLASS OF DICT [emphasis added] defines a method __missing__()
and key is not present, the d[key] operation calls that method ..."
> The fact that it silently doesn't call them is an inscrutable trap.
That is not a fact.
What you should say is:
"The fact that dicts don't obey the undocumented assumptions I made
about the implementation of methods is a trap."
and I will agree: correct, but its a trap for hubris and foolishness,
and the answer to that is, don't make unjustified assumptions about how
dicts are implemented. And I can say that because I made exactly that
wrong assumption too.
Nowhere does the documentation say that dict.update calls __setitem__.
Nowhere does it say that dict.clear calls __delitem__. Nowhere does it
say that dict.values calls __getitem__. And yet I assumed that they did
When somebody makes the unjustified assumption that they do, then they
will discover that calling the subclass' clear method fails to call the
overridden __delitem__. It also fails to call the overridden __str__.
The only difference is that nobody assumes that clear() calls __str__,
but many people foolishly assume that it calls the __delitem__ method.
Both assumptions have *exactly* the same justification: none at all.
I made a bunch of stupid, foolish assumptions, based on absolutely
nothing more than the idea that it stands to reason that dict must be
implemented in this way, and got bitten. And I deserved it. I was wrong,
and anyone making the same assumption is wrong.
Well, I say I was "bitten", but that over-dramatises the situation. What
actually happened was that I wrote a subclass without doing any tests,
then tested it, and discovered that it didn't work how I expected. I
spent a few minutes playing with the class, added a few print
statements, discovered that my assumptions were wrong, and then overrode
the classes I actually wanted to override.
Twenty minutes of googling and reading the docs convinced me that, no,
Python doesn't promise that dict.clear calls __delitem__. It would have
been five minutes except I was especially stubbon and pig-headed that
day and didn't want to admit that I was in the wrong. But I was.
Getting bitten by this was, in fact, a valuable lesson. I learned what I
should have already known, what I had *intellectually* known but had
ignored because "it stands to reason". Namely, if an implementation
isn't documented, you cannot assume that it works in a particular way.
My sympathy level is zero, and my support for this proposal is negative.
More information about the Python-ideas