[Python-ideas] Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.

[Eric Snow]

On Thu, Apr 21, 2016 at 10:45 PM, Random832 <random832 at fastmail.com> wrote:
> On Thu, Apr 21, 2016, at 23:49, Steven D'Aprano wrote:
>> Where does attribute lookup come into this?
>
> Because looking up an attribute implies getting an item from the object
> or class's __dict__ with a string key of the name of the attribute (see
> below for the documented basis for this assumption)

Attribute access and item access communicate different things about
the namespaces on which they operate.  The fact that objects use
mappings under the hood for the default attribute access behavior is
an implementation detail.  Python does not document any mechanism to
hook arbitrary mappings into the default attribute access behavior
(i.e. object.__getattribute__()).

>> What makes you think that such a method exists?
>>
>> Have you read the rest of this thread? If not, I suggest you do, because
>> the thread is all about making unjustified assumptions about how objects
>> are implemented. You are doing it again. Where is it documented that
>> there is a method on the object __dict__ (if such a __dict__ even
>> exists!) that does this?
>
> My main concern here is the class dict. So, let's see...
>
> ### Class attribute references are translated to lookups in this
> dictionary, e.g., C.x is translated to C.__dict__["x"]
>

Not exactly.  You still have to factor in descriptors (including
slots). [1]  In the absence of those then you are correct that the
current implementation of type.__getattribute__() (not the same as
object.__getattribute__(), BTW) is a lookup on the type's __dict__.
However, that is done directly on tp_dict and not on <TYPE>.__dict__,
if you want to talk about implementation details.  Of course, the
point is moot, as you've pointed out, since a type's __dict__ is both
unsettable and a read-only view.

> And regarding the object __dict__, when such a __dict__ *does* exist
> (since, unlike class dicts, you actually can set object dicts to be
> arbitrary dict subclasses)
>
> ### The default behavior for attribute access is to get, set, or delete
> the attribute from an object’s dictionary. For instance, a.x has a
> lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'],
> and continuing through the base classes of type(a) excluding
> metaclasses.

Again, you also have to factor in descriptors. [2]

Regardless of that, it's important to realize that
object.__getattribute__() doesn't use any object's __dict__ attribute
for any of the lookups you've identified.  This is because you can't
really look up the object's __dict__ attribute *while* doing attribute
lookup.  It's the same way that the following doesn't work:

    class Spam:
        def __getattribute__(self, name):
            if self.__class__ == Spam:  # infinite recursion!!!
                ...
            return object.__getattribute__(self, name)

Instead you have to do this:

    class Spam:
        def __getattribute__(self, name):
            if object.__getattribute__(self, "__class__") == Spam:
                ...
            return object.__getattribute__(self, name)

Hence, object.__getattribute__() only does lookup on the dict
identified through the type's tp_dictoffset field.  However, as you've
noted, objects of custom classes have a settable __dict__ attribute.
This is because by default the mapping is tied to the object at the
tp_dictoffset of the object's type. [3]

Notably, the mapping must be of type dict or of a dict subclass.  What
this implies to me is that someone went to the trouble of allowing
folks to use some other dict (or OrderedDict, etc.) than the one you
get by default with a new object.  However, either they felt that
using a non-dict mapping type was asking for too much trouble, there
were performance concerns, or they did not want to go to the effort to
fix all the places that expect __dict__ to be an actual dict.  It's
probably all three.

Keep in mind that even with a dict subclass the implementation of
object.__getattribute__() can't sensibly use normal lookup on when it
does the lookup on the underlying namespace dict.  Instead it uses
PyDict_GetItem(), which is basically equivalent to calling
dict.__getitem__(ns, attr).  Hence that underlying mapping must be a
dict or dict subclass, and any overridden __getitem__() method is
ignored.

-eric

[1] https://hg.python.org/cpython/file/default/Objects/typeobject.c#l2924
[2] https://hg.python.org/cpython/file/default/Objects/object.c#l1028
[3] https://hg.python.org/cpython/file/default/Objects/object.c#l1195